Site icon CormacHogan.com

Data Services Manager version 2.2 now available

It gives me great pleasure to announced that the latest version of Data Services Manger (DSM) is now available. Version 2.2 has a wealth of new features related to the deployment, management and monitoring of data services on VMware Cloud Foundation (VCF). I will share these features in greater detail in this post, but please also refer to the official DSM 2.2 documentation for further information. Let’s get started.

Multi-Cluster Deployments for increased availability

In previous versions of DSM, databases and data services could be provisioned in two modes. The first is standalone mode which deploys the data service as a single node on a single VM in your vSphere infrastructure. The second mode is clustered mode whereby a 3 node topology is deployed onto a single vSphere Cluster.  This second mode places VMs / nodes of the database on different ESXi hypervisors within the same vSphere cluster for improved availability. With DSM version 2.2, even more availability is provided by allowing databases to be provisioned in a cross-cluster topology. This means that databases can have its VMs / nodes placed on different vSphere clusters, providing even higher availability in the event of a maintenance activity or unplanned outage on part of your vSphere infrastructure.

Now customers have the option of deploying databases standalone to a single server, or choosing to make the databases highly available by deploying clustered databases onto either a single vSphere cluster or deploying the clustered databases deployed across multiple vSphere clusters.

Database Delete Protection now available

This was a feature request that we heard from many customers over the past year, so I am delighted to see it included in the DSM 2.2 release. DSM users have now the ability  to recover a database after it has been deleted.  The retention period of a deleted database is 30 days, but this can be reduced or extended if necessary. From the databases view, simply select the Archives tab, and click on the ellipsis button to the left of the database name, and click on the Restore database option. This is also where you can change the retention period of the database.

Disaster Recovery via Replication across multiple sites

DSM 2.2 enables our customers to meet even higher levels of availability across different vSphere infrastructures via a new replication feature. To leverage this functionality of multi-site replication, customers can deploy a second DSM instances which is attached to a different vCenter server. This could be a different VMware Cloud Foundation (VCF) workload domain, for example. A PostgreSQL database deployed and managed by one DSM deployment can now be replicated to a target (secondary) database deployed and managed by another DSM deployment. Configuring replication is quite straight forward as you might expect. On the primary database, under Data Availability and Remote Replication, slide the button across to enable replication. You will then be prompted for two pieces of information. The first is the Replication Slot Name which is used to retain the WAL files that are needed by the secondary. The second field is the Replication User. As the name implies, this is the user used by the secondary instance to connect to the primary instance for replication purposes. Once added, and saved, this will result in some Remote Replication details being created. This information is used to create the secondary database.

To create the Secondary Database, select a new option in the database instances views called “Create Secondary Database”.

The replication connection string and slot will need to be added during the setup of the secondary database. Once the secondary instance has been created and replication initiated, the status can be observed once again via the DSM UI.

During normal operations, the secondary instance only contains a Read Replica node. If a disaster event occurs on the primary and failover is necessary, the secondary database instance at the target site can be promoted to primary via the database dropdown menu.  If necessary, the secondary may be scaled out from a single node database to a clustered database for availability purposes once it has been promoted to primary.

Major Version Upgrade support

Lifecycle Management is always front of mind in Data Services Manager. Historically, minor version updates of the database fleet have been available since the initial release of DSM, both manually and automated via the maintenance window option on each database. In DSM version 2.2 we are extending lifecycle management to support major version upgrades of PostgreSQL databases.  Simply navigate to the Available Upgrades section of the Database Summary page to see which major version upgrades are available. In this example, I am running PostgreSQL major version 13, so versions 14, 15 and 16 are all available for upgrade. I can even jump to version 16 directly from 13.

There are few things to keep in mind with major version upgrades. This operation will cause some downtime, and the length of the upgrade operation is completely dependent on the size of your database. You will also need to pay due diligence to the installed extensions as, for example, there is no guarantee that a PostgreSQL extension that worked in version 13 is supported or even works in version 16. A way to validate this is to first clone the database, and upgrade the cloned database before upgrading your actual database. If the clone works well after upgrade, then this will give you some peace of mind that the actual upgrade will also be successful. additional peace of mind comes from the fact that if there is an issue with the upgrade, you can always restore the original database from backup. The upgrade will take a pre-upgrade backup automatically.

Scale down of VM Classes

When it comes to resource management, DSM supported the scale up of a VM class to provide more compute and memory resources to VMs / nodes running databases. With the release of version 2.2, a new enhancement now enables DSM admins to claim back unused resources by allowing them to apply a lower specification VM class to the database nodes. Thus, this release allows DSM users to both scale up and scale down their PostgreSQL database resources. In this example, I initially deployed a database with a medium VM Class, as shown here:

With both small and large VM classes also defined as part of this infrastructure policy, the VMs which back this database is using can now be scaled up as well as down from a compute and memory resources perspective by simply selecting a different VM Class, as shown here.

Custom Certificate Management expanded and simplified

Certificates provide a way to secure communication between DSM and external services (Object Storage, LDAP, Aria Operations for Logs, etc.). It also ensure that there is secure communication between end-users and the databases provisioned by DSM. DSM has supported RSA certificates for some time. In version 2.2, DSM also introduces support for ECDSA and Ed25519 certificates.

Another enhancement in DSM 2.2 is the customers who wish to use their own custom certificate can now do this via the DSM UI. While DSM was already able to add custom certificates to databases, it could only be done via the API which could prove challenging. Being able to add the certificate chain and key via the DSM UI makes this much easier to implement secure communication channels. While certificates can be added at database creation time, it can also be added later by editing a running database configuration.

Configuration of pg_hba.conf simplified

Along similar lines, we have made a major enhancement to how the pg_hba.conf can be customized for PostgreSQL databases. While this was also possible previously, similar to custom certificate management, it could only be done through the API. In DSM version 2.2, we are including a mechanism to add configuration entries via the DSM UI. The pg_hba.conf basically controls access to databases. It can define which users from which network can access which databases. Thus it is an extremely important configuration to have control over. for example, if I wanted to add all users who had @local usernames and grant them access to all databases from any network, I could add the following entry to the pg_hba.conf file using this configuration: host all “/^(.*)@local$” 0.0.0.0/0 scram-sha-256 . This is how it might appear when added via the DSM UI at database creation time:

This is another feature that many of our customer have been requesting so it is super to see it included in DSM 2.2.

Improved Webhook Notifications

Webhooks are a very important feature for sending alerts and updates to third party notification systems such as ServiceNow or even Slack. We received feedback from our customers that they would like to have better controls over who receives an alert via webhook. To that end, DSM 2.2 now allows admins to select primary receivers of a database alert. If a webhook is set to DSM Admin, then notifications will be automatically received. Admins do not need to go to every database and associate it with the webhook. If the recipient is set to DSM User, then the behaviour remains the same as it is in earlier versions in so far as each database needs to have the webhook manually configured.

Global View of Database Events

This is another customer request. Rather than having to examine the events of each database individually, the request was to have a centralised view for all database events. In DSM version 2.2, this view is now available.

Onboarding of existing PostgreSQL databases into DSM

In this release we have also validated the process to import existing PostgreSQL databases into Data Services Manager. This is currently a manual process but the steps have been included in the official documentation. Good news for customers who already have PostgreSQL deployments in their environments and would like to import those databases into DSM for fleet management as well as automated lifecycle management and backups, along with the many of features and capabilities offered by DSM. See this section of the official DSM 2.2 documentation for further details.

Object Storage Service (Technical Preview)

The final feature that I wish to bring to your attention is that we now have a tech preview for an additional data service in Data Services Manager version 2.2. By once again partnering with MinIO, we are able to offer a technical preview of an Object Storage service through DSM. This will allow DSM customers to provision Object Storage on their VMware Cloud Foundation (VCF) environments while enjoying all of the fleet management capabilities offered by DSM. There are a few steps to get this configured which I will cover in a future post. These steps include the creation of an image Registry to store the MinIO container images. Once the registry is setup and configured in DSM, users can then proceed with the deployment of Object Storage. The operation works much the same as a database provisioning operation, as shown in the screenshot below.

The operation will create a new PostgreSQL database for storing Object Storage Metadata, as well as creating one or more storage pools to provide capacity for the S3 compatible buckets. Multiple storage pools with different numbers of nodes and disk capacity can be provisioned. Once provisioned, the object store view gives details of the Portal URL as well as the Service URLs of the storage pool nodes. Also provided is a button, located at the top of the window, to connect directly to the Object Store Portal.

Clicking on the “Open Object Store Portal” button at the top of the view will take you to the Portal UI. At this point, you can login as the DSM administrator that created the Object Storage, or login as a user that has the new DSM 2.2 role of Object Storage Admin. Note that this user can also be used to create Object Storage. Once logged in, the admin will presented with the following UI which allows them to create and manage S3 compatible buckets.

More details will be forthcoming in an upcoming blog post. Be aware that a MinIO license will be required to enable this service. You will need to contact MinIO directly to get such a license. And finally, this is a tech preview, so please do not put it into production. Support of this feature will be best efforts.

Summary

As you can see, a lot of effort has gone into delivering an abundance of new features into Data Services Manager version 2.2. There are some pretty awesome features. And while the product is specifically for VMware Cloud Foundation customers, VCF is not needed to implement a proof of concept. A simple vSphere environment is enough to evaluate many of the features of DSM outlined above. However, you might find that DSM is an excellent reason for your organization to considering moving your existing vSphere infrastructure to VCF. If you would like to learn more, please don’t hesitate in reaching out and I can provide you with additional information.

Exit mobile version