We have just released version 10.0.0.0 of the Ping Identity Directory Server. See the release notes for a complete overview of changes, but here’s my summary:

Important Notices

• As of the 10.0 release, the Directory Server only supports Java versions 11 and 17. Support for Java 8 has been removed, as a critical component (the embedded web container we use to support HTTP requests, including the Directory REST API, SCIM, and the Administration Console) no longer supports Java 8.
• As of the 10.0 release, we are no longer offering the Metrics Engine product as part of the Directory Server suite (the Directory Proxy Server and Synchronization Server are still included, as is the Server SDK for developing custom extensions). You should instead rely on the server’s ability to integrate with other monitoring software, through mechanisms like our support for OpenMetrics (used by Prometheus and other software), StatsD, and the Java Management Extensions (JMX).

Summary of New Features and Enhancements

• Added support for inverted static groups [more information]
• Added support for post-LDIF-export task processors, which can be used to perform custom processing after successfully exporting an LDIF file, including the option to upload the resulting file to an Amazon S3 bucket [more information]
• Added a new log file rotation listener that can be used to upload newly rotated log files to a specified Amazon S3 bucket [more information]
• Added a new amazon-s3-client command-line tool [more information]
• Added authentication support to the Directory REST API [more information]
• Added support for a generate access token request control [more information]
• Added support for configuring the server with a single database cache that may be shared across all local DB backends [more information]
• Added an option to automatically re-encode passwords after changing the configuration of the associated password storage scheme [more information]
• Exposed a request-handler-per-connection configuration property in the LDAP connection handler configuration [more information]
• Updated the encrypt-file tool to add a --re-encrypt argument [more information]
• Updated the encrypt-file tool to add a --find-encrypted-files argument [more information]
• Updated the replication server and replication domain configuration to add a new missing-changes-policy property that can be used to customize the way the server behaves in the event that missing changes are detected in the environment, and the server will now remain available by default under a wider range of circumstances that may not represent actual problems
• Significantly improved performance for creating a backup, restoring a backup, or performing online replica initialization
• Significantly improved static group update performance
• Improved performance for the process of validating the server state immediately after completing an update
• Added a split-ldif tool that can be used to split a single LDIF file into multiple sets for use in setting up an entry-balanced deployment with the Directory Proxy Server
• Updated the bcrypt password storage scheme to include support for the 2b variant (in addition to the existing 2y, 2a, and 2x variants)
• Updated the HTTP connection handler to add an option for performing SNI hostname validation during TLS negotiation
• Updated the backup tool to display a warning when creating a compressed backup of an encrypted backend, since encrypted backends cannot be effectively compressed, but attempting to do so will make the backup process take longer
• Updated the dsreplication command so that it uses a separate log file per subcommand, and so that log files representing failed runs of the tool are archived rather than overwritten by subsequent runs
• Removed the dsreplication removed-defunct-server subcommand, which is better provided through the dedicated remove-defunct-server tool
• Removed the dsreplication cleanup-local-server subcommand, which is better provided through the remove-defunct-server --performLocalCleanup command
• Updated dsreplication initialize-with-static-topology to add an --allowServerInstanceDelete argument that can be used to remove servers from the topology if they are not included in the provided JSON file
• Updated dsreplication initialize-with-static-topology to add an --allowDomainIDReuse argument that can be used to allow domain IDs to be used with different base DNs
• Updated the check-replication-domains tool so that it no longer requires the --serverRoot argument
• Updated the replication server configuration to add an option that can be used to include information about all remote servers in monitor messages, which can be useful in large topologies where that can constitute a large amount of data
• Added support for an access log field request control that can be used to include arbitrary fields in the log message for the associated operation
• Updated the configuration API to treat patch operations with empty arrays as a means of resetting the associated configuration property
• Added the ability to configure connect and response timeouts when connecting to certain external services over HTTP, including CyberArk Conjur instances, HashiCorp Vault instances, the Pwned Passwords service, and YubiKey OTP validation servers
• Updated the Synchronization Server to improve performance when setting the startpoint to the end of the changelog for an Active Directory server
• Reduced the default amount of memory allocated for the export-ldif and backup tools

Summary of Bug Fixes

• Fixed an issue in which the Directory REST API could fail to strip out certain kinds of encoded passwords in responses to clients (although only to clients that were authorized to access those attributes)
• Improved the way that the replication generation ID is computed, which can help ensure the same ID is generated across replicas when they are populated by LDIF import instead of online replica initialization
• Fixed an issue that could cause an error while trying to initialize aggregate pass-through authentication handlers
• Fixed an issue that could cause “invalid block type” errors when interacting with compressed files
• Fixed an issue that could prevent the server from properly including an encrypted representation of the new password in the changelog entry for a password modify extended operation when the server was configured with the changelog password encryption plugin
• Fixed an issue in which the server could fail to update a user’s password history on a password change that included a password update behavior request control indicating that the server should ignore password history violations
• Fixed an issue that could cause the server to add two copies of the current password in the password history when changing a password with the password modify extended operation
• Fixed an issue in which the server could incorrectly allow a user to set an empty password. Even though that password could not be used to authenticate, the server should not have allowed it to be set
• Fixed an issue that could cause the dictionary password validator to incorrect accept certain passwords that contained a dictionary word as a substring that was larger than the maximum allowed percentage of the password
• Fixed an issue in which the server could be unable to properly interpret the value of the allow-pre-encoded-passwords configuration property in password policies defined in user data that were created prior to the 9.3 release of the server
• Fixed an issue in which the server may not have properly applied replace modifications for attributes with options
• Fixed an issue in which the first unsuccessful bind attempt after a temporary failure lockout had expired may not be counted as a failed attempt toward a new failure lockout
• Fixed an issue in which running manage-profile generate-profile against an updated server instance could result in a profile that may not be usable for setting up new instances
• Fixed an issue in which dsreplication initialize could suggest using the --force argument in cases where that wouldn’t help, like when attempting to authenticate with invalid credentials
• Fixed an issue with dsreplication enable-with-static-topology in which the server could report an error when trying to connect to a remote instance
• Fixed an issue with dsreplication enable-with-static-topology in which case sensitivity in base DNs was not handled properly
• Fixed an issue in which the remove-defunct-server command could fail in servers configured with the AES256 password storage scheme
• Fixed an issue that could cause a replication error if missing changes were found for an obsolete replica that is not configured in all servers
• Fixed an issue in which the server did not check the search time limit often enough during very expensive index processing, which could allow the server to process a search request for substantially longer than the maximum time limit for that operation
• Fixed an issue that caused the server to incorrectly include client certificate messages in the expensive operations access log
• Fixed an internal error that could be encountered if an administrative alert or alarm is raised at a specific point in the shutdown process
• Fixed an issue with synchronizing Boolean attributes (e.g., “enabled”) to PingOne
• Fixed an issue in which the Synchronization Server could fail to properly synchronize changes involving the unicodePwd attribute to Active Directory if the sync class was not configured with a DN map
• Fixed an issue that could cause the create-sync-pipe-config command to improperly generate correlated attribute definitions for generic JDBC sync destinations
• Fixed an error that could prevent manage-topology add-server from adding a Synchronization Server instance to a topology that already had at least two Synchronization Server instances
• Fixed an issue in which the server did not properly log an alternative authorization DN for multi-update operations that used proxied authorization
• Fixed an issue in which dsjavaproperties --initialize could result in duplicate arguments in the java.properties file
• Fixed an issue that could cause a spurious message to be logged to the server’s error log when accessing the status page in the Administration Console

In the 10.0 release, we’re introducing support for inverted static groups, which try to combine the primary benefits of traditional static groups and dynamic groups without their most significant disadvantages.

Traditional static groups contain an attribute (either member or uniqueMember, depending on the group’s object class) explicitly listing the DNs of the members of that group. They are pretty straightforward to use and are widely supported by LDAP-enabled applications, but as the number of members in the group increases, so does the size of the entry and the cost of reading and writing that entry and updating group membership.

Traditional static groups also support nesting, but it’s not necessarily easy to distinguish between members that are users and those that are nested groups. The server has to maintain an internal cache so that it can handle nested memberships efficiently, and this requires both extra memory consumption and a processing overhead when the group is updated.

Dynamic groups, on the other hand, don’t have an explicit list of members, but instead are defined with one or more LDAP URLs whose criteria will be used for membership determinations. Because there is no member list to maintain, dynamic groups don’t have the same scalability issues as traditional static groups, and the number of members in the group isn’t a factor when attempting to determine whether a specific user is a member. However, dynamic groups aren’t as widely supported as traditional static groups among LDAP-enabled applications, there’s no way to directly add or remove members in a dynamic group (at least, not without altering the entries in a way that causes them to match or stop matching the membership criteria, which varies on a group-by-group basis), and they don’t support nesting.

Inverted static groups provide a way to explicitly manage group membership like with traditional static groups, but with the scalability of dynamic groups. Rather than storing the membership as a list of DNs in the group entry itself, each user entry has a list of the DNs of the inverted static groups in which they’re a member (in the ds-member-of-inverted-static-group-dn operational attribute). This means that the number of members doesn’t affect the performance of many group-related operations, like adding a new member to the group, removing an existing member from the group, or determining whether a user is a member of the group.

The only way in which the size of the group does impact performance is if you want to retrieve the entire list of members for the group (which you can do by performing a subtree search to find entries whose isMemberOf attribute has a value that matches the DN of the target group). While this is slower than simply retrieving a traditional static group entry and retrieving the list of member DNs, this is actually not an analogous comparison for a couple of key reasons:

• Retrieving the list of member DNs from a traditional static group only gives you the DNs of the member entries. That isn’t enough if you need the values of any other attributes from the member entries.

• Retrieving the list of member DNs from a traditional static group doesn’t work well if that group includes one or more nested groups. There’s no good way to tell which of the member DNs reference users and which represent nested groups, and the member DN list won’t include members of the nested groups.

As such, the best way to retrieve a list of all members of a traditional static group is also to perform a subtree search that targets the isMemberOf attribute, and it should be at least as fast to do that for an inverted static group as it is for a traditional static group.

The other key difference that inverted static groups have over traditional static groups lies in the way that we handle nested membership. As previously noted, traditional static groups can include both user DNs and group DNs in their membership attribute, and there’s not a good way to distinguish between them. Inverted static groups distinguish between user members and nested groups. Rather than adding a nested group to the inverted static group as a regular member, you need to add the DN of the nested group to the ds-nested-group-dn attribute of the inverted static group entry. This does make it possible to distinguish between user entries and nested groups, and it allows the server to handle nesting for these types of groups without a separate cache or expensive processing.

The main disadvantage that inverted static groups have in comparison to traditional static groups is that because they are a new feature, existing applications don’t directly support them. If an application only cares about making group membership determinations, makes those determinations using the isMemberOf attribute, and doesn’t need to alter group membership, then it should work just as well with inverted static groups as it does with traditional static groups. However, if it does need to alter group membership, or if it doesn’t support using the isMemberOf attribute, then that’s a bigger hurdle to overcome. To help with that, we’re including a “Traditional Static Group Support for Inverted Static Groups” plugin that can be used to allow clients to interact with inverted static groups in some of the same ways they might try to interact with traditional static groups. This includes:

• The plugin will intercept attempts to modify the group entry to add or remove member or uniqueMember values, and instead make the corresponding updates to the ds-member-of-inverted-static-group-dn attribute in the target user entries.

• The plugin can generate a virtual member or uniqueMember attribute for the group entry. It can do this in a few different ways, which may have different performance characteristics:
  • It can do it in a way that works for compare operations or equality search operations that target the membership attribute but don’t actually attempt to retrieve the membership list. This is the most efficient way to determine if a traditional static group has a specific DN in its list of members, and it should be about as fast to make this determination for an inverted static group as it is for a traditional static group.
  • It can do it in a way that attempts to populate the attribute with a list of all of the direct members of the group (excluding nested members). The performance of this does depend on the number of direct members in the group.
  • It can do it in a way that attempts to populate the attribute with a list of all of the direct and nested members of the group. The performance of this depends both on the number of direct members in the group as well as the types and sizes of the nested groups.

The data stored in the Directory Server is often absolutely critical to the organizations that use it, so it’s vital to have a good backup strategy, and that must include some kind of off-site mechanism. Amazon’s S3 (Simple Storage Service) is a popular cloud-based mechanism that is often used for this kind of purpose, and in the 10.0 release, we’re introducing a couple of ways to have the Directory Server take advantage of it. In particular, you can now easily use it as off-site storage for LDIF exports and log files. We also include a new amazon-s3-client tool that can be used to interact with the S3 service from the command line.

Post-LDIF-Export Task Processors

We’ve introduced a new API in the server that can be used to have the server perform additional processing after successfully exporting data to an LDIF file as part of an administrative task (including those created by a recurring task). You can use the Server SDK to create custom post-LDIF-export task processors that do whatever you want, but we’re including an “Upload to S3” implementation that can copy the resulting export file (which will ideally have already been compressed and encrypted during the export process) to a specified S3 bucket. This processor includes retention support, so you can have it automatically remove previous export files created more than a specified length of time ago, or you can have it keep a specified number of the newest files in the bucket.

The export-ldif command-line tool now has a new --postExportProcessor argument that you can use to indicate which processors should be invoked for after the export file has been successfully written, and you can also specify which processors to use when creating the tasks programmatically (for example, using the task support in the UnboundID LDAP SDK For Java) or by simply adding an appropriate formatted entry to the server’s tasks backend. We’ve also updated the configuration for the LDIF export recurring task to include a new post-ldif-export-task-processor property to specify which processor(s) should be invoked for LDIF exports created by that recurring task.

Note that the post-LDIF-export task processor functionality is only available for LDIF exports invoked as recurring tasks, and not those created using the export-ldif tool in the offline, standalone mode. This is because post-LDIF-export task processors may need access to a variety of server components, and it’s difficult to ensure that all necessary components would be available outside of the running server process.

The Upload to S3 Log File Rotation Listener

The Directory Server already had an API for performing custom processing whenever a log file is rotated out of service, including copying the file to an alternative location on the server filesystem or invoking the summarize-access-log tool on it. In the 10.0 release, we’re including a new “Upload to S3” log file rotation listener that can be used to upload the rotated log file to a specified S3 bucket. This is available for all of the following types of log files:

• Access
• Error
• Audit
• Data recovery
• HTTP operations
• Sync
• Debug
• Periodic stats

Although there are obvious benefits to copying all of these types of log files to an external service, I want to specifically call out the importance of having off-site backups for the data recovery log. The data recovery log is a specialized type of audit log that keeps track of all write operations processed by the server in a form that allows them to be easily replayed or reverted should the need arise. The data recovery log can be used as a kind of incremental backup mechanism for keeping track of changes made since the most recent backup or LDIF export, and in a worst-case scenario in which all server instances are lost and you need to start over from scratch, you can restore the most recent backup or import the most recent LDIF export, and the replay any additional changes from the data recovery log that were made after the backup or export was created.

As with the Upload to S3 post-LDIF-export task processor, the new log file rotation listener also includes retention support so that you can choose to keep either a specified number of previous log files, or those uploaded less than a specified length of time in the past.

The amazon-s3-client Command-Line Tool

The new amazon-s3-client tool allows you to interact with the S3 service from the command line, including in shell scripts or batch files. It supports the following types of operations:

• List the existing buckets in the S3 environment
• Create a new bucket
• Remove an existing bucket (optionally removing the files that it contains)
• List the files in a specified bucket
• Upload a file to a specified bucket
• Download a specified file from a bucket
• Download one or more of the newest files from a specified bucket, based on the number of files to download, the age of files to download, or files created after a specified time
• Remove a file from a bucket

This allows you to perform a number of functions, including manually upload additional files that the server doesn’t support uploading automatically, or to download files for use in bootstrapping new instances. It can generate output as either human-readable text or machine-parsable JSON.

The Directory REST API allows you to submit requests and retrieve data from the server using a JSON-formatted HTTP-based API. It’s always had support for all of the typical operations needed for interacting with the data, like:

• Creating new entries
• Updating existing entries
• Removing existing entries
• Retrieving individual entries
• Searching for all entries matching a given set of criteria

Within the last few releases, we’ve also introduced support for a wide variety of request controls, and also certain extended operations. But one of the big gaps between what the server offered over the Directory REST API versus what you could get via LDAP was in its support for authentication. The Directory REST API has always supported authorizing individual requests using either HTTP basic authorization or OAuth 2 bearer tokens, but it didn’t really provide any good way to authenticate clients and verify client credentials. And if you wanted to authorize requests with stronger authentication than just a DN and password, you had to have an external service configured for issuing OAuth tokens.

This is being addressed in the 10.0 release with a new /authenticate endpoint, which currently supports the following authentication methods:

• password — Username or bind DN and a static password

• passwordPlusTOTP — Username or bind DN, a static password, and a time-based one-time password (TOTP)

• passwordPlusDeliveredOTP — Username or bind DN, a static password, and a one-time password delivered through some out-of-band mechanism like email or SMS

• passwordPlusYubiKeyOTP — Username or bind DN, a static password, and a one-time password generated by a YubiKey device

We’re also adding other new endpoints in support of these mechanisms, including:

• One for generating a TOTP secret, storing it in the user’s entry, and returning it to the client so that it can be imported into an app (like Authy or Google Authenticator) for generating time-based one-time passwords for use with the passwordPlusTOTP authentication method.

• One for revoking a TOTP secret so that it can no longer be used to generate time-based one-time passwords that will be accepted by the passwordPlusTOTP authentication method.

• One for generating a one-time password and delivering it to the user through some out-of-band mechanism so that it can be used to authenticate with the passwordPlusDeliveredOTP authentication method.

• One for registering a YubiKey device with a user’s entry so that it can be used to authenticate with the passwordPlusYubiKeyOTP method.

• One for deregistering a YubiKey device with a user’s entry so that it can no longer be used to authenticate with the passwordPlusYubiKeyOTP method.

If the authentication is successful, the response may include the following content:

• An access token that can be used to authorize subsequent requests as the user via the Bearer authorization method.

• An optional set of attributes from the authenticated user’s entry.

• If applicable, the length of time until the user’s password expires.

• A flag that indicates whether the user is required to choose a new password before they will be allowed to do anything else.

• An optional array of JSON-formatted response controls.

We’ve added support for a new “generate access token” request control that can be included in a bind request to indicate that if the bind succeeds, the server should return a corresponding response control with an access token that can be used to authenticate subsequent connections via the OAUTHBEARER SASL mechanism. While this control is used behind the scenes in the course of implementing the new authentication support in the Directory REST API, it can also be very useful in certain LDAP-only contexts.

For example, this ability may be especially useful in cases where you want to authenticate a client with a mechanism that relies on single-use credentials, like the UNBOUNDID-TOTP, UNBOUNDID-DELIVERED-OTP, or UNBOUNDID-YUBIKEY-OTP SASL mechanism. In such cases, the credentials can only be used once, which means you can’t use them to authenticate multiple connections (for example, as part of a connection pool), or to re-establish a connection if the initial one becomes invalidated.

You can configure the Directory Server with multiple local DB backends. You should do this if you want to have multiple naming contexts for user data, and you can also do it for different portions of the same hierarchy if you want to maintain them separately for some reason (e.g., to have them replicated differently, as in an entry-balanced configuration where some of the DIT needs to be replicated everywhere while the entry-balanced portion needs to be replicated only to a subset of servers).

Previously, each local DB backend had its own separate database cache, which had to be sized independently. This gives you the greatest degree of control over caching for each of the backends, which may be particularly important if you don’t have enough memory to hold everything based on the current caching configuration, but it can also be a hassle in some deployments. And if you don’t keep track of how much you’ve allocated to each backend, you could potentially oversubscribe the available memory.

In the 10.0 release, we’re adding the ability to share the same cache across all local DB backends. To do this, set the use-shared-database-cache-across-all-local-db-backends global configuration property to true, and set the shared-local-db-backend-database-cache-percent property to the percentage of JVM memory to allocate to the cache. Note that this doesn’t apply to either the LDAP changelog or the replication database, both of which intentionally use very small caches because their sequential access patterns don’t really require caching for good performance.

The Directory Server has always had support for declaring certain password storage schemes to be deprecated as a way of transparently migrating passwords from one scheme to another. If a user successfully authenticates in a manner that provides the server with access to the clear-text password (which includes a number of SASL mechanisms in addition to regular LDAP simple binds), and their password is currently encoded with a scheme that is configured as deprecated, then the server will automatically re-encode the password with the currently configured default scheme.

Deprecated password storage schemes can only be used to migrate users from one scheme to another, but there may be legitimate reasons to want to re-encode a user’s password without changing the scheme. For example, several schemes use multiple rounds of encoding to make it more expensive for attackers to attempt to crack passwords, and you may want to have passwords re-encoded if you change the number of rounds that the server uses.

In the 10.0 release, we’re adding a new re-encode-passwords-on-scheme-config-change property to the password policy configuration. If this is set to true, if a client authenticates in a manner that provides the server with access to their clear-text password, and if their password is currently encoded with different settings than are currently configured for the associated scheme, then the server will automatically re-encode that password with the default scheme using current settings. This functionality is available for the following schemes:

• AES256 — If there is a change in the definition used to encrypt passwords.

• ARGON2, ARGON2D, ARGON2I, ARGON2ID — If there is a change in the iteration count, parallelism factor, memory usage, salt length, or derived key length.

• BCRYPT — If there is a change in the cost factor.

• PBKDF2 — If there is a change in the digest algorithm, iteration count, salt length, or derived key length.

• SCRYPT — If there is a change in the CPU/memory cost factor exponent, the block size, or the parallelization parameter.

• SSHA, SSHA256, SSHA384, SSHA512 — If there is a change in the salt length.

It is also possible to enable this functionality for custom password storage schemes created using the Server SDK by overriding some new methods added to the API.

When the Directory Server accepts a new connection from an LDAP client, it hands that connection off to a request handler, which will be responsible for reading requests from that client and adding them to the work queue so that they can be picked up for processing by worker threads. By default, the server automatically determines the number of request handlers to use (although you can explicitly configure the number if you want), and a single request handler may be responsible for reading requests from several connections.

The vast majority of the time, having request handlers responsible for multiple connections isn’t an issue. Just about the only thing that a request handler has to do is wait for requests to arrive, read them, decode them, and add them to the work queue so that they will be processed. While it’s doing this for a request from one client, any other clients that are sending requests at the same time will need to wait, but because the entire process of reading, decoding, and enqueueing a request is very fast, it’s rare that processing for one client will have any noticeable impact on the request handler’s ability to process other clients. However, there are a couple of instances in which this might not be the case:

• If a client is submitting a very large number of asynchronous requests at the same time.

• If the server needs to perform TLS negotiation on the connection to set up a secure communication channel, and some part of that negotiation is taking a long time to complete.

In practice, neither of these is typically an issue. Even if there are a ton of asynchronous requests submitted all at once, it’s still pretty unlikely that it will cause any noticeable starvation in the server’s ability to read requests from other clients. And the individual steps of performing TLS negotiation also tend to be processed very quickly. However, there have been exceptional cases in which these kinds of processing may have had a noticeable impact. In such instances, the best way to deal with that possibility is to have the server use a separate request handler for each connection that is established so that the process of reading, decoding, and enqueueing requests from one client cannot impact the server’s ability to do the same for other clients that may be sending requests at exactly the same time. In the unlikely event that the need arises in your environment, you can now use the request-handler-per-connection property in the connection handler configuration to cause the server to allocate a new request handler for every client connection that is established.

As its name implies, the encrypt-file tool can be used to encrypt (or decrypt) the contents of files, either using a definition from the server’s encryption settings database or with an explicitly-provided passphrase. In many cases, if a file used by the server (or a command-line tool) is encrypted with an encryption settings definition, the server can detect that and automatically decrypt it as it’s reading the contents of that file.

If an administrator wishes to retire an encryption settings definition for some reason, and especially if they want to remove it from the encryption settings database, they need to ensure that it is no longer needed to decrypt any existing encrypted data. In the past, some customers have overlooked encrypted files when ensuring that a definition is no longer needed. To help avoid that, we’ve added two new arguments to the encrypt-file tool:

• --find-encrypted-files {path} — This argument can be used to search for encrypted files below the specified path on the server filesystem. By default, it will find files that have been encrypted with any encryption settings definition or with a passphrase, but you can also provide the --encryption-settings-id argument to indicate that you only want it to find files encrypted with the specified definition.

• --re-encrypt {path} — This argument can be used to re-encrypt an existing encrypted file, using either a different encryption settings definition or a new passphrase.

If you do plan to retire an existing encryption settings definition, then you should use the encrypt-file --find-encrypted-files command to identify any files that have been encrypted with that definition, and then use encrypt-file --re-encrypt to re-encrypt them with a different definition so that the server can still access them even if you remove the retired definition from the encryption settings database.

We have just released several new versions of the Ping Identity Directory Server to address a security issue that we discovered. The issue is in a component of the server that is only enabled when setting up the Delegated Admin product, and customers who are using that product are strongly advised to upgrade. Customers who are not using the Delegated Admin product should not be affected by the issue.

The following new versions are now available and contain the fix for this issue:

• 9.3.0.1
• 9.2.0.2
• 9.1.0.3
• 8.3.0.9

The security issue was discovered internally, and we have no reason to believe that it has been independently discovered or exploited. Ping is not currently prepared to provide additional information about the vulnerability at this time, but is expected to release a security advisory with additional details in the future.

We have just released version 9.3.0.0 of the Ping Identity Directory Server. See the release notes for a complete overview of changes, but here’s my summary:

Summary of New Features and Enhancements

• Added support for data encryption restrictions [more information]
• Added the ability to freeze the encryption settings database [more information]
• Added the ability to set up the server with a pre-existing encryption settings database [more information]
• Added support for monitoring the availability of the encryption settings database [more information]
• Added other data encryption improvements [more information]
• Added an aggregate pass-through authentication handler [more information]
• Added a PingOne pass-through authentication handler [more information]
• Improved dsreplication performance in topologies with a large number of servers and/or high network latency between some of the servers
• Added more options for allowing pre-encoded passwords [more information]
• Added the ability to use the proxied authorization v1 or v2 request control in password modify extended requests
• Updated the Directory REST API to provide support for the password modify, get password quality requirements, and suggest password extended operation types [more information]
• Added a disallowed characters password validator [more information]
• Added a UTF-8 password validator [more information]
• Added the ability to include ds-pwp-modifiable-state-json in add operations [more information]
• Added the ability to automatically apply changes to TLS protocol and cipher suite configuration [more information]
• Added new account-authenticated and account-deleted account status notification types [more information]
• Added configuration properties for managing the configuration archive [more information]
• Added a new replication-missing-changes-risk alert type [more information]
• Added a new replication-not-purging-obsolete-replicas alert type [more information]
• Added a new check-replication-domains tool that can list known replication domains identify any that may be obsolete
• Added a --showPartialBacklog argument to dsreplication status
• Added the ability to synchronize Boolean-valued attributes to the PingOne sync destination
• Updated replace-certificate to support obtaining new certificate information from PEM files
• Added support for encrypted PKCS #8 private keys [more information]
• Added caching support to the PKCS #11 key manager provider [more information]
• Added the ability to specify the start and end times for the range of log messages to include in collect-support-data archives when invoking the tool as an administrative task

Summary of Bug Fixes

• Fixed an issue when modifying ds-pwp-modifiable-state-json with other attributes [more information]
• Fixed an issue that could prevent the server from properly building indexes with very long names
• Fixed an issue that could cause the server to omit matching entries when configuring compact-common-parent-dn values [more information]
• Fixed an issue in which failover may not work properly after updating a Synchronization Server instance with manage-profile replace-profile
• Fixed an issue with replace modifications for attributes containing variants with options [more information]
• Improved support for passwords containing characters with multiple encodings [more information]
• Fixed an issue that could prevent obsolete replicas from being automatically purged in certain circumstances
• Fixed an issue that could prevent the servers in a replication topology from being able to select the authoritative server for maintaining information in the topology registry
• Increased timeouts used by the dsreplication tool to reduce the chance that they would be incorrectly encountered when interacting with a large replication topology
• Fixed an issue that caused the Directory REST API to always include the permissive modify request control when updating entries
• Improved access control behavior for the password policy state extended operation [more information]
• Fixed an issue in which subtree searches based at the server’s root DSE could omit entries from backends with base DNs subordinate to those of other backends
• Fixed an issue that could prevent a user from using grace logins to change their own password in a modify request that contained the proxied authorization request control
• Fixed an issue with substring filters containing logically empty substrings [more information]
• Improved error handling when using automatic authentication with client certificates [more information]
• Improved Directory Proxy Server error handling when using the rebind authorization method [more information]
• Fixed an issue that prevented including permit-export-reversible-passwords privilege in the default set of root privileges
• Fixed an issue that could cause manage-profile setup to complain about being unable to find certain utilities used by the collect-support-data tool
• Fixed an error that could occur if an archived configuration file was removed in the middle of an attempt to back up the config backend
• Fixed an issue that prevented the Directory Proxy Server from logging search result entry messages for entries passed through from a backend server
• Fixed an issue when synchronizing account state from Active Directory when using modifies-as-creates
• Suppressed servlet information in HTTP error messages by default
• Restricted the RSA key size for inter-server certificates to a maximum of 3072 bits
• Fixed an issue with base DN case sensitivity when enabling replication with a static topology
• Changed the result code used when rejecting an attempt to change a password that is within the minimum age from 49 (invalidCredentials) to 53 (unwillingToPerform)
• Fixed an issue that could cause the server to return multiple password validation details response controls in the response to a password modify extended request
• Fixed an issue that could prevent the server from returning a generated password for a password modify extended operation processed with the no-operation request control

We have made a set of changes to the way that the server manages and interacts with the encryption settings database. When used in combination, this can allow for a separation of duties between those responsible for managing the Directory Server itself and those responsible for managing data encryption, which could be used to limit the access that server administrators have to encrypted data. However, even in environments where this strict separation of duties is not required, these changes can still substantially improve the overall security of the directory environment.

These changes come in the form of four key enhancements:

• We have introduced the ability to impose restrictions on the ways that administrators can interact with encrypted data. These restrictions are defined in the encryption settings database itself and can prevent administrators from doing any or all of the following:
  • Disabling data encryption
  • Changing the cipher stream provider used to protect the encryption settings database
  • Exporting the encryption settings definitions
  • Exporting or backing up backend data in unencrypted form
  • Exporting or backing up backend data in a form that is encrypted with a supplied passphrase rather than an encryption settings definition
  • Using the encrypt-file tool to decrypt files

• We have added the ability to freeze the encryption settings database with a passphrase. If the encryption settings database has been frozen, then no changes may be made to it, including creating, importing, or removing definitions, changing which is the preferred definition, and altering the set of data encryption restrictions. If any changes are needed, then the encryption settings database may be un-frozen using the same passphrase that was initially used to freeze it.

• We have added the ability to set up the server with a pre-existing encryption settings database. This database should already have the desired set of definitions, and it may be configured with data encryption restrictions and frozen so that no changes will be allowed. This database will also be tied to a specific cipher stream provider used to protect its contents. To set up the server with a pre-existing encryption settings database, you should use manage-profile setup with a server profile that has the following characteristics:
  • The setup-arguments.txt file must contain the new --encryptDataWithPreExistingEncryptionSettingsDatabase argument.
  • The configuration changes needed to set up the associated cipher stream provider must be included in dsconfig batch files contained in the pre-setup-dsconfig directory.
  • The encryption settings database itself, along with any metadata files that the cipher stream provider might need, should be included in the appropriate locations below the server-root/pre-setup directory.

• We have added support for a new monitor provider that can periodically ensure that the server can read the contents of the encryption settings database without relying on any caching that the cipher stream provider may normally use to improve performance and reliability. Not only does this offer better overall monitoring for the health of the server, but it can also be used to take action if the information that it needs to interact with the encryption settings database has been disabled. For example, if the cipher stream provider relies on an external key or secret (e.g., from Amazon Key Management Service, Amazon Secrets Manager, Azure Key Vault, CyberArk Conjur, or HashiCorp Vault) to be able to unlock the encryption settings database, and that key or secret has been intentionally removed or revoked, then it may be desirable to take action to limit the server’s ability to access encrypted data, like entering lockdown mode or shutting down entirely.

When all of these are combined, and the encryption settings database is protected by a cipher stream provider that relies on an external service, the server administrators could be able to maintain the server with data encryption enabled, but with substantially restricted access to the data that it contains. Although it’s not yet available as an option, this could be useful in environments like PingOne Advanced Services, where Ping personnel can manage a Directory Server deployment on behalf of an organization with limited access to that organization’s data.

We have made a number of other additional improvements in the server’s support for data encryption. These include:

• We added a new --key-factory-iteration-count argument to the encryption-settings create command to make it possible to specify the PBKDF2 iteration count to use for the new definition. When creating a new encryption settings definition that is backed by a randomly generated secret, the server will now default to using an iteration count of 600,000 in accordance with the latest OWASP guidelines.

• We updated most cipher stream providers to make it possible to explicitly specify the PBKDF2 iteration count that they use when deriving the key used to protect the encryption settings database. They also now use a higher default iteration count of 600,000 when creating a new database.

• We updated the file-based cipher stream provider to support using a separate metadata file with additional details about the encryption that it uses to protect the encryption settings database. When setting up a new instance of the server in a manner that uses this cipher stream provider, a metadata file will be automatically generated to allow it to use stronger encryption to protect the database than has been used in the past.

• We have improved the strength of the encryption used to protect encryption settings exports, backups, LDIF exports, encrypted log files, and other types of file encryption. We now prefer 256-bit AES over 128-bit AES when it’s available, and we use a higher PBKDF2 iteration count to protect the key.

• We have improved the performance of file encryption and decryption operations performed by the server in the common case in which the encryption uses an encryption settings definition rather than a separate passphrase. Although the server (or standalone tools that need to access encryption settings definitions) may take a little longer to start up with the stronger encryption settings that are in place, the use of caching should dramatically reduce the cost of subsequent encryption and decryption operations.

• We updated the encryption settings backend to provide additional information about the definitions contained in the encryption settings database. The base entry for the backend will also indicate whether the encryption settings database is frozen and/or configured with any data encryption restrictions.

• We updated setup so that if it is configured to generate a tools.pin file with the default password that command-line tools may use to authenticate to the server, that password will now be automatically encrypted if data encryption is enabled.

The Directory Server has long supported the ability to use pass-through authentication, in which a bind attempt against a local entry may ultimately be forwarded to an external service to verify the credentials, optionally setting those credentials if they are confirmed to be correct.

Initially, pass-through authentication was only supported for other LDAP servers. Later, we introduced support for passing through authentication attempts to PingOne. In the 9.0 release, we introduced a new pluggable pass-through authentication plugin that made it possible to use the Server SDK to develop custom authentication handlers that could be used to target other types of services.

We have only ever supported a single instance of the pass-through authentication plugin (of any type) active in the server at once. In the 9.3 release, we are introducing a new aggregate pass-through authentication handler for use with the pluggable pass-through authentication plugin. This aggregate handler allows you to configure pass-through authentication so that it can support multiple external services, of the same or different types. You can now configure each of the pass-through authentication handlers to indicate which types of bind requests they support, and you can optionally attempt to use multiple handlers for the same bind operation under certain circumstances.

We have also added a new PingOne pass-through authentication handler that works in conjunction with the pluggable pass-through authentication plugin to allow passing through bind requests to PingOne. This handler offers essentially the same functionality as the PingOne pass-through authentication plugin, but the new PingOne handler allows it to be used in conjunction with the aggregate pass-through authentication handler so that pass-through authentication attempts can use PingOne in addition to other services.

By default, the server does not allow clients to set pre-encoded passwords. This is something that we made it possible to disable via the allow-pre-encoded-passwords configuration property in the password policy, but we strongly discourage that because the server can’t perform any validation for pre-encoded passwords. A client could use this to set a password that doesn’t meet the server’s password strength requirements. And because most password storage schemes have many different ways of encoding the same password (which is important to protect against attacks using precomputed password dictionaries), this could also be exploited to allow a user to continue using the same password indefinitely, in spite of password expiration or other related settings.

The biggest risk to allowing pre-encoded passwords lies in self password changes rather than administrative password reset. Previously, administrators could override the server’s prohibition against pre-encoded passwords in one of two ways:

• If they are authorized to use the password update behavior request control, then they can use it to allow or reject pre-encoded passwords on a per-operation basis.

• If they have the bypass-pw-policy privilege, then they will be allowed to set pre-encoded passwords for other users (and do other things that the password policy configuration may prevent by default).

Previously, the allow-pre-encoded-passwords configuration property only offered two values: false (the default setting, in which the server would not allow clients to set pre-encoded passwords) or true (in which the server would allow any client to set pre-encoded passwords). In the 9.3 release, we are making it possible to use three additional values for this property:

• add-only — Indicates that the server will allow administrators to include pre-encoded passwords in add requests, but will continue to reject pre-encoded passwords for both self password changes and administrative password resets.

• admin-reset-only — Indicates that the server will allow administrators to perform an administrative password reset with a pre-encoded password, but will continue to reject pre-encoded passwords in add requests and for self password changes.

• add-and-admin-reset-only — Indicates that the server will allow administrators to include pre-encoded passwords in add requests and when performing administrative password resets, but will continue to reject pre-encoded passwords in self password changes.

In cases where an application may have a legitimate need to be able to set pre-encoded passwords for users, but it’s not feasible to update the application to use the password update behavior request control or to give the account is use the bypass-pw-reset privilege, these new options may make it possible for that application to set pre-encoded passwords for users while still preventing them for self password changes.

Updated the Directory REST API to add support for equivalents to the following LDAP extended operations:

• The standard password modify extended operation, which makes it possible to perform a self password change or an administrative password reset. Although you could previously change a user’s password by updating the password attribute, this operation offers a number of advantages, including:
  • You don’t need to know which attribute is used to store passwords in the target user’s entry.
  • There’s a dedicated field for supplying the user’s current password, which the password policy may require for self password changes.
  • You can optionally omit a new password and have the server automatically generate a new one and return it in the response.
  • It can be used in conjunction with a password reset token to allow a user to perform a self password change in cases where their account may be otherwise unusable.

• The proprietary get password quality requirements extended operation, which can be used to obtain a list of the requirements that new passwords will be required to satisfy, in both machine-parsable and human-readable forms.

• The proprietary generate password extended operation (which is called “suggest passwords” in the Directory REST API), which can be used to cause the server to generate suggested new passwords for a user.

We have added a new disallowed characters password validator that makes it possible to reject passwords that contain any of a specified set of characters. You can define characters that are not allowed to appear anywhere in a password, as well as characters that are not allowed to appear at the beginning and/or the end of a password.

We have also added a new UTF-8 password validator that can be used to ensure that only passwords that are provided as valid UTF-8 strings will be allowed. You can optionally choose to limit passwords to only containing ASCII character or to allow non-ASCII characters, and you can also specify which classes of characters (e.g., letters, numbers, punctuation, symbols, spaces, etc.) should be allowed.

If you enable the Modifiable Password Policy State Plugin in the server, then you can use the ds-pwp-modifiable-state-json operational attribute to set certain aspects of a user’s password policy state, including:

• Whether the account is administratively disabled
• Whether the account is locked as a result of too many failed authentication attempts
• Whether the account is in a “must change password” state
• The password changed time
• The account activation time
• The account expiration time

The password expiration warned time

Previously, the ds-pwp-modifiable-state-json attribute could only be used in a modify operation to alter the password policy state for an existing user. As of the 9.3 release, we now also allow it to be used in an add operation to specify state information for the account being created.

The compact-common-parent-dn configuration property can be used to reduce the amount of disk space consumed by the database by tokenizing portions of entry DNs that lots of entries have in common. For example, in a database in which most of the entries are below “ou=People,dc=example,dc=com”, defining that as a compact-common-parent-dn value could reduce the size needed to store DNs of entries below that by up to 26 bytes. And this compaction can happen not only in the encoded representation of the entry, but also in an internal index that we use to map the DNs of entries to the identifier of the record that contains the data for that entry, so that can double the space savings.

Unfortunately, we discovered a bug in the way that the maintained that DN-to-ID database when custom compact-common-parent-dn values were specified. This bug could only appear under very specific circumstances, including:

• When one or more compact-common-parent-dn values were specified that are two or more levels below the base DN for the backend.

• When processing an unindexed search in which the base DN for the search is below the base DN for the backend, but above one or more of the compact-common-parent-dn values.

In such cases, the search could have incorrectly omitted entries from the search results that were below those compact-common-parent-dn values. This was due to an issue with the way that we ordered records in that DN-to-ID database.

We have fixed the problem in the 9.3 release. However, because the issue relates to the order in which records were stored in the database, if you are affected by this problem, then you will need to export the contents of the database to LDIF and re-import it. This isn’t something that you need to do unless you have configured compact-common-parent-dn values that are at least two levels below the base DN for the backend.

When we initially introduced support for the ds-pwp-modifiable-state-json operational attribute, we did not allow altering it in conjunction with any other attribute in the user’s entry. We also included an optimization in the plugin that handles that attribute so that if the requested change did not actually alter the user’s password policy state (e.g., because the new value only attempted to set state properties to values that they already had), the plugin would tell the server to skip much of the normal processing for that modify operation.

We have since updated the server to allow you to alter other attributes (except the password attribute) in the same request as one that modifies ds-pwp-modifiable-state-json. However, in doing so, we neglected to update the plugin so that it no longer skipped the remainder of the core modify operation processing if the ds-pwp-modifiable-state-json update did not result in any password policy state changes but there were still other, unrelated changes that should be applied. In such cases, the server could have failed to apply changes to those other attributes. This has been fixed, and other modifications in the request will still be processed even if a change to ds-pwp-modifiable-state-json does not alter the user’s password policy state.

By default, the server automatically selects an appropriate set of TLS protocols and cipher suites that it will use for secure communication. This default configuration should provide a good level of security that avoids options with known weaknesses and that don’t support forward secrecy while still remaining compatible with virtually any client from the last fifteen years.

However, the server does allow you to explicitly configure the set of TLS protocols and/or cipher suites if you have a need to do so. Previously, making any such changes required you to either restart the affected connection handler or the entire server for them to actually take effect. As of the 9.3 release, these changes will now automatically take effect for the LDAP connection handler so that any new secure sessions that are negotiated after the change is made will use the updated settings.

Note that this is currently only supported for the LDAP connection handler. It is still necessary to perform a restart to make the change take effect for other types of connection handlers (like the HTTP or JMX handlers), or to make the change take effect for replication.

We have added a new account-authenticated account status notification type that can be used to generate an account status notification any time a user authenticates with a bind operation that matches the criteria contained in an account status notification handler’s account-authentication-notification-result-criteria configuration property.

We have added a new account-deleted account status notification type that can be used to generate an account status notification any time a user’s account is removed with a delete operation that matches the criteria contained in an account status notification handler’s account-deletion-notification-request-criteria configuration property.

We have fixed an issue that could cause the server to behave incorrectly when processing a replace modification for an attribute that has some variants with attribute options in the target entry. If the replace modification does not include any values for the target attribute, the server would have previously removed all variants of the attribute from the entry, including those with and without attribute options. It will now correctly only remove the variant without any attribute options.

Unicode is an international standard that defines the set of characters that computers are intended to support. This includes a wide range of characters encompassing most written languages on earth, including not only the core set of ASCII characters used in English, but also characters with diacritical marks used in many Latin-based languages, non-Latin symbols like those used in Chinese hanzi and Japanese kanji, and even emojis.

In some cases, Unicode supports multiple ways of encoding the same character. For example, the “ñ” (Latin small letter N with tilde) character can be represented in two different ways:

• As the single Unicode character U+00F1
• As a lowercase ASCII letter n followed by Unicode character U+0303, which is a special combining mark that basically means “put a tilde over the previous character”

Previously, when encoding passwords, the Directory Server would always encode them using the exact set of UTF-8 bytes provided by the client in the request used to set that password, and the server would always use the exact set of UTF-8 bytes included in a bind request as a way of attempting to verify whether the provided password was correct. This approach works just fine in the vast majority of cases, but it has the potential to fail in circumstances where the password contains characters that have multiple Unicode representations, and the encoding used in the bind request is different from the encoding used when the password was originally set.

As of the 9.3 release, we have improved our support for authenticating with passwords that contain characters with multiple Unicode representations. If a password storage scheme indicates that a given plaintext password could not have been used to create the encoded stored password, but if the provided plaintext password contains one or more non-ASCII characters, then we will check to see if that password may have alternative encodings that could have been used to generate the stored password.

The password policy state extended operation provides the ability to retrieve and update a wide range of properties that are part of a user’s password policy state. In the 9.3 release, we have improved the way that it operates to avoid a common pitfall that has caused issues in the past for clients that are subject to access control restrictions.

Previously, whenever the extended operation handler retrieved the entry for the target user, it did so only under the authority of the authenticated user. If that user was subject to access control restrictions and didn’t have the permission to retrieve some or all of the operational attributes that are used to maintain password policy state information in a user’s entry, then the version of the entry that was retrieved would not contain those attributes, even if they were present. When using that entry to construct a view of the user’s password policy state, this might result in a state that is different from what the server would actually use when interacting with that account.

For example, if the target account has been locked as a result of too many failed authentication attempts, but the requester doesn’t have permission to see the attribute used to maintain information about those failed attempts, then the password policy state extended operation could report that the account was not locked even though it was. Note that this did not affect the server’s behavior when actually enforcing the account lockout, but it could still provide misleading or unexpected behavior for the client that issued the password policy state extended request.

As of the 9.3 release, we have changed the extended operation handler’s behavior to avoid this kind of problem. The server will still verify that the requester has permission to retrieve the target user’s account, but it will now re-retrieve that account using an internal root user that is not subject to password policy restrictions. This ensures that the extended operation handler will have access to all of the operational attributes that are needed to construct an accurate representation of the user’s password policy state.

Note that this new behavior really only affects attempts to retrieve information about a user’s password policy state. It does not have any effect on attempts to update that state. If the client attempts to use the password policy state extended operation to make a change to a user’s password policy state, but the requester does not have permission to write to the necessary operational attribute(s) in the target user’s entry, then the update attempt will continue to fail.

We have updated the config backend to add support for a few new configuration properties that can help better manage the configuration archive and reduce unnecessary bloat that it may cause. The new properties include:

• maintain-config-archive — Indicates whether the server should maintain a configuration archive at all. The configuration archive is maintained by default, and disabling it will not remove existing archived configurations.

• max-config-archive-size — Specifies the maximum number of archived configurations that the server should maintain. By default, this is unlimited. If this is enabled and there are more than the maximum allowed archived configurations, then the oldest configurations will be removed to make room for newer configurations.

• insignificant-config-archive-base-dn — Specifies base DNs for configuration changes that should not be preserved in the configuration archive. If a configuration change affects only affects entries below one of these base DNs, then it will not result in that change being maintained separately in the configuration archive. By default, we have included a value of “cn=topology,cn=config” so that changes to entries in the topology registry are excluded from the configuration archive. Certain updates to the topology registry, like adding a new replica into the topology, may result in a large number of changes that previously included a lot of bloat in the configuration archive.

Substring search filters are not allowed to contain empty (zero-length) substrings. If a client attempted to process a search with a substring filter that contains an empty substring, the server would have properly rejected it. However, there were cases in which the server did not properly handle substring search filters that contained non-empty substrings that normalized to empty strings (for example, a substring filter that targeted the telephoneNumber attribute in which one of the substrings contained only characters that are considered insignificant when matching telephone number values). The server would have incorrectly considered that normalized-to-empty substring as matching anything rather than nothing, and in some cases, that could cause the search to return unexpected matches. This has been corrected, and substring filters with substrings that normalize to empty values will now properly never match anything.

By default, whenever a client presents their own certificate chain to the server during TLS negotiation and wants to use that certificate chain to authenticate, it needs to send a SASL EXTERNAL bind request to the server to cause it to perform the appropriate authentication processing. However, LDAP connection handlers offer an auto-authenticate-using-client-certificate configuration property that can cause them to attempt to automatically authenticate a client that presented its own certificate chain as soon as the TLS negotiation completes. Because this automatic authentication attempt happens without any explicit request from the client, there’s no way for the server to indicate whether it completed successfully.

Previously, if an automatic authentication attempt failed, the server would keep the connection alive, but in an unauthenticated state. This could yield unexpected behavior in applications that issued request with the expectation that they had authenticated, only to find those requests rejected due to access control restrictions. As of the 9.3 release, the server will now immediately terminate any client connection that presented its own certificate chain when auto-authenticate-using-client-certificate was set to true but the server was unable to successfully authenticate that client for some reason.

The Directory Proxy Server allows you to set an authorization-method value of rebind, which may be useful in cases where the backend server doesn’t support the intermediate client or proxied authorization request controls (e.g., Active Directory). In such cases, the Directory Proxy Server will remember the credentials that the client used to authenticate, and it will re-bind with those credentials before sending a request that should be authorized as that user.

Previously, if the rebind attempt failed for any reason, the Directory Proxy Server would have immediately reported a failure for the attempt to process a request that should have been authorized as that user. We have updated this behavior so that if the failure suggests that the underlying connection used for that attempt may no longer be valid, the server may re-try the attempt in a different server or on a newly created connection to the same server.

We have defined a couple of new administrative alert types that the server can use to notify administrators of replication-related concerns:

• replication-missing-changes-risk — This alert type will be used if the server has developed a replication backlog that is large enough that the server is at risk of missing changes if it can’t catch up.

• replication-not-purging-obsolete-replicas — This alert type will be used when bringing replication online (most likely during server startup) if the replication-purge-obsolete-replicas configuration property is not set to true. This property is set to true by default as of the 9.2 release, so this primarily applies to older servers that have been updated. We strongly recommend enabling automatic purging of obsolete replicas to reduce unnecessary overhead in replication storage and network traffic, and this alert can be used to ensure that administrators are aware of this recommendation.

We have introduced support for encrypted PKCS #8 private key PEM files. This allows you to make use of private key files that don’t expose the key in the clear. Encrypted private keys require a password to access their contents.

It should be possible to use encrypted private key files anywhere that you can use an unencrypted private key file, including:

• When importing a certificate chain with manage-certificates import
• When exporting a private key with manage-certificates export-private-key
• When setting up the server with a certificate chain and private key obtained from PEM files
• When using replace-certificate to replace a listener or inter-server certificate with a certificate chain and private key obtained from PEM files

The PKCS #11 key manager provider can be used to allow the server to obtain a listener certificate chain from a PKCS #11 token, like a hardware security module (HSM). Whenever the server needs to negotiate a new TLS session with a client, it can access the PKCS #11 token to identify the listener certificate chain that should be used for that processing. This processing may require multiple accesses to the PKCS #11 token. In cases where the PKCS #11 token is remotely accessed over a network, and especially when there is a notable network latency involved in that access, this can have a notable impact on the performance of the TLS negotiation process.

In the 9.3 release, we have introduced a new pkcs11-max-cache-duration property in the PKCS #11 key manager provider configuration. By setting this to a nonzero value, the server can use a degree of caching to eliminate the need for some of the interaction with the HSM, which can dramatically reduce the number of requests that need to be made to the PKCS #11 token.

Note that the use of caching does have a risk incorrect or unexpected behavior if the contents of the PKCS #11 token are altered so that the cached results are no longer accurate. As such, if you decide to enable caching, we recommend temporarily disabling that caching when making changes to the contents of the PKCS #11 token, and then re-enabling the caching once the changes are complete.

We have just released version 9.2.0.0 of the Ping Identity Directory Server. See the release notes for a complete overview of changes, but here’s my summary:

Potential Backward Compatibility Issues

• Removed support for incremental backups [more information]
• Updated the Groovy language version from 2.x to 3.x [more information]

Summary of New Features and Enhancements for All Products

• Added support for Java 17 [more information]
• Added support for accessing external services through an HTTP proxy server [more information]
• Added a Prometheus monitoring servlet extension [more information]
• Added support for authenticating to Amazon AWS using an IRSA role [more information]
• Added support for generating digital signatures with encryption settings definitions [more information]
• Updated replace-certificate when running in interactive mode so that it can re-prompt for a certificate file if the initial file existed but did not contain valid certificate data

Summary of New Features and Enhancements for the Directory Server

• Improved support for data security auditors [more information]
• Added new secure, connectioncriteria, and requestcriteria access control keywords [more information]
• Added support for defining resource limits for unauthenticated clients [more information]
• Added Argon2i, Argon2d, and Argon2id password storage schemes to supplement the existing Argon2 scheme [more information]
• Changed the default value of the replication-purge-obsolete-replicas global configuration property from false to true
• Updated migrate-ldap-schema to support migrating attribute type definitions from Active Directory in spite of their non-standards-compliant format
• Improved the usage text for the dsreplication enable command

Summary of New Features and Enhancements for the Directory Proxy Server

• Exposed the maximum-attributes-per-add-request and maximum-modifications-per-modify-request properties in the global configuration

Summary of New Features and Enhancements for the Synchronization Server

• Added support for synchronizing to SCIMv2 destinations [more information]
• Added a sync-pipe-view tool that can display information about the set of sync pipes configured in the server
• Added sync pipe monitor attributes related to account password policy state when synchronizing to a Ping Identity Directory Server

Summary of Bug Fixes

• Fixed an issue that could cause replication protocol messages to be dropped, potentially resulting in paused replication
• Fixed an issue in which a timeout could prevent adding servers to a large topology
• Fixed an issue in which an unexpected error could cause a replication server to stop accepting new connections
• Fixed an issue that prevented resource limits from being set properly for the topology administrator
• Fixed an issue in which the dsreplication tool incorrectly handled DNs in a case-sensitive manner
• Fixed an issue that could cause dsreplication enable to fail if there were any topology administrators without passwords
• Fixed an issue that could cause a configured idle timeout to interfere with replica initialization
• Fixed an issue that could prevent the server from generating an administrative alert when clearing an alarm that triggered an alert when it was originally raised
• Fixed an issue that could cause degraded performance to a PingOne sync destination
• Fixed an issue that could prevent users from changing their own passwords with the password modify extended operation if their account was in a “must change password” state and the request passed through the Directory Proxy Server
• Fixed an issue in which dsconfig would always attempt to use simple authentication when applying changes to servers in a group, regardless of the type of authentication used when launching dsconfig
• Fixed an issue that could cause certain kinds of Directory REST API requests to fail if they included the uniqueness request control
• Fixed an issue in which an unclean shutdown could cause the server to create exploded index databases
• Disabled the index cursor entry limit by default, which could cause certain types of indexed searches to be considered unindexed
• Fixed an issue that could adversely affect performance in servers with a large number of virtual static groups

We have removed support for incremental backups. This feature was deprecated in the 8.3.0.0 release after repeated issues that could interfere with the ability to properly restore those backups. These issues do not affect full backups, which continue to be supported.

As an alternative to full or incremental backups, we recommend using LDIF exports, which are more useful and more portable than backups. They are also typically very compressible and can be taken more frequently than backups without consuming as much disk space. Further, the extract-data-recovery-log-changes tool can be used in conjunction with either LDIF exports or backups to replay changes recorded in the data recovery log since the time the LDIF export or backup was created.

In order to facilitate support for Java 17, we have updated the library providing support for the Groovy scripting language from version 2.x to 3.x. While this should largely preserve backward compatibility, there may be some issues that could prevent existing Groovy scripted extensions from continuing to work without any problems.

The only compatibility issue that we have noticed is that the 3.x version of the Groovy support library cannot parse Java import statements that are broken up across multiple lines, like:

import java.util.concurrent.atomic.
            AtomicLong;

This was properly handled in Groovy 2.x, but the Groovy 3.x library does not appear to support this. To address the problem, you will need to update the script to put the entire import statement on a single line, like:

import java.util.concurrent.atomic.AtomicLong;

If you have any Groovy scripted extensions, we strongly recommend verifying them in a test environment before attempting to update production servers.

We have updated the server to support running on JVMs running Java version 17, which is the latest LTS release of the Java language. Java versions 8 and 11 also continue to be supported.

Note that Java 17 support is limited to the Directory Server, Directory Proxy Server, and Synchronization Server. Java 17 is not supported for the Metrics Engine, although it continues to be supported on Java 8 and 11.

The best way to enable Java 17 support is to have the JAVA_HOME environment variable set to the path of the Java 17 installation when installing the server using either the setup or manage-profile setup commands. It’s more complicated to switch to Java 17 for an existing instance that was originally set up on Java 8 or 11 because there are changes in the set of JVM arguments that should be used with Java 17. As such, if you want to switch to Java 17, then we recommend installing new instances and migrating the data to them.

By default, installations using Java 17 will use the garbage first garbage collection algorithm (G1GC), which is the same default as Java 11. We also support using the Z garbage collector (ZGC) on Java 17, although we have observed that it tends to consume a significantly greater amount of memory than the garbage first algorithm. While ZGC can exhibit better garbage collection performance than G1GC, if you wish to use it, we recommend configuring a smaller JVM heap size and thoroughly testing the server under load and at scale before enabling it in production environments.

We have updated several server components to provide support for issuing outbound HTTP and HTTPS requests through a proxy server. Updated components include:

• The Amazon Key Manager cipher stream provider
• The Amazon Secrets Manager cipher stream provider, passphrase provider, and password storage scheme
• The Azure Key Vault cipher stream provider, passphrase provider, and password storage scheme
• The PingOne pass-through authentication plugin
• The PingOne sync source and destination
• The Pwned Passwords password validator
• The SCIMv1 sync destination
• The SCIMv2 sync destination
• The Twilio alert handler and OTP delivery mechanism
• The UNBOUNDID-YUBIKEY-OTP SASL mechanism handler

To enable HTTP forward proxy support for any of these components, first, create an HTTP proxy external server configuration object with a command like:

dsconfig create-external-server \
     --server-name "Example HTTP Proxy Server" \
     --type http-proxy \
     --set server-host-name:proxy.example.com \
     --set server-port:3128

You can also optionally use the basic-authentication-username and basic-authentication-passphrase-provider properties if the HTTP proxy server requires authentication.

Once the HTTP proxy external server has been created, update the target component to reference that server. For example:

dsconfig set-password-validator-prop \
     --validator-name "Pwned Passwords" \
     --set "http-proxy-external-server:Example HTTP Proxy Server"

We have added support for a new HTTP servlet extension that can be used to expose certain server metrics in a format that can be consumed by Prometheus or other monitoring systems that support the OpenMetrics data format. To enable it, add the servlet extension to the desired HTTP connection handlers and either restart the server or disable and re-enable those connection handlers. For example:

dsconfig set-connection-handler-prop \
     --handler-name "HTTPS Connection Handler" \
     --add "http-servlet-extension:Prometheus Monitoring" \
     --set enabled:false

dsconfig set-connection-handler-prop \
     --handler-name "HTTPS Connection Handler" \
     --set enabled:true

By default, the server is preconfigured to expose a variety of metrics. You can customize this to remove metrics that you don’t care about, or to add additional metrics that we didn’t include by default. Any single-valued numeric monitor attribute can be exposed as a metric. You can also customize the set of labels included in metric definitions, on both a server-wide and per-metric basis.

The server offers a number of components that can interact with Amazon Web Services components,
including:

• A cipher stream provider that can use the Key Management Service
• A cipher stream provider, passphrase provider, and password storage scheme that can use the Secrets Manager

In the past, you could authenticate to AWS using either a secret access key or using an IAM role that is associated with the EC2 instance or EKS container in which the server is running. In the 9.2.0.0 release, we’re introducing support for authenticating with an IRSA (IAM role for service accounts) role. We are also adding support for a default credentials provider chain that can attempt to automatically identify an appropriate authentication method for cases in which the server is running in an AWS environment, or in cases where information about a secret access key is available through either environment variables or Java system properties.

To use the new authentication methods, first create an AWS external server that specifies the desired value for the authentication-method property. Then, reference that external server when creating the desired component. For example:

dsconfig create-external-server \
     --server-name AWS \
     --type amazon-aws \
     --set authentication-method:irsa-role \
     --set aws-region-name:us-east-2

dsconfig create-cipher-stream-provider \
     --provider-name KMS \
     --type amazon-key-management-service \
     --set enabled:true \
     --set aws-external-server:AWS \
     --set kms-encryption-key-arn:this-is-the-key-arn

The server offers a data security auditor framework that can be used to iterate across entries in a number of backends and examine them for potential security-related issues or items of note. In the past, we’ve offered auditors that can do the following:

• Identify entries that define access control rules
• Identify accounts that have been administratively disabled
• Identify accounts that have passwords that are expired, are about to expire, or that have not been changed in longer than a given length of time
• Identify accounts that are locked as a result of too many authentication failures, because it’s been too long since the user last authenticated, or because they did not choose a new password in a timely manner after an administrative reset.
• Identify accounts with multiple passwords
• Identify accounts with privileges assigned by real or virtual attributes
• Identify accounts encoded with a variety of weak password storage schemes, including 3DES, AES, BASE64, BLOWFISH, CLEAR, MD5, RC4, or the default variant of the CRYPT scheme

In the 9.2 release, we’ve introduced support for several new types of data security auditors, including those that can do the following:

• Identify accounts with account usability errors, warnings, and/or notices
• Identify accounts that have an activation time in the future, an expiration time in the past, or an expiration time in the near future
• Identify accounts that have passwords encoded with a deprecated password storage scheme
• Identify accounts that have not authenticated in longer than a specified period of time, or that have not ever authenticated
• Identify accounts that reference a nonexistent password policy
• Identify entries that match a given search filter

We have also updated the Server SDK so that you can create your own data security auditors to use whatever logic you want.

In addition, we have updated the locked account data security auditor so that it can identify accounts that are locked as a result of attempting to authenticate with a password that fails password validator criteria, and we have updated the weakly encoded password data security auditor so that the following schemes are also considered weak: SMD5, SHA, SSHA, and the MD5 variant of the CRYPT scheme.

Finally, we’ve introduced support for a new audit data security recurring task that you can use to have the server automatically perform an audit on a regular basis.

We have introduced three new access control keywords.

The secure bind rule can be used to make access control decisions based on whether the client is using a secure connection (e.g., LDAPS or LDAP with StartTLS) to communicate with the server. Using a bind rule of secure="true" indicates that the ACI only applies to clients communicating with the server over a secure connection, while secure="false" indicates that the ACI only applies to clients communicating with the server over an insecure connection.

The connectioncriteria bind rule can be used to make access control decisions based on whether the client connection matches a specified set of connection criteria. The value of the bind rule can be either the name or the DN of the desired connection criteria.

The requestcriteria target can be used to make access control decisions based on whether the operation matches a specified set of request criteria. The value of the target can be either the name or the DN of the desired request criteria.

Note that because the Server SDK provides support for creating custom types of connection and request criteria, the introduction of these last two bind rules adds support for being able to define custom access control logic if the server’s existing access control framework doesn’t support what you want.

The server’s global configuration includes the following configuration properties that can be used to set default resource limits that will apply to all users that don’t have specific limits set for them:

• size-limit — Specifies the maximum number of entries that can be returned for a search operation
• time-limit — Specifies the maximum length of time the server should spend processing a search operation
• idle-time-limit — Specifies the maximum length of time that a client connection may remain established without any operations in progress
• lookthrough-limit — Specifies the maximum number of entries that the server can examine in the course of processing a search operation

These properties set global defaults for all clients, including those that aren’t authenticated. However, you may want to set lower limits for unauthenticated connections than for users that are authenticated. To make that easier to accomplish, we have added the following new additional properties that specifically apply to unauthenticated clients:

• unauthenticated-size-limit
• unauthenticated-time-limit
• unauthenticated-idle-time-limit
• unauthenticated-lookthrough-limit

By default, these properties don’t have any values, which will cause the server to inherit the value from the property that doesn’t specifically apply to unauthenticated clients (for example, if unauthenticated-size-limit is not set, then the server will use the size-limit value as the default for both authenticated and unauthenticated clients).

The server supports cryptographically signing log messages, backups, and LDIF exports. Previously, those signatures were always generated with MAC keys shared among other servers in the same topology. These keys are difficult to back up and restore, and the resulting signatures cannot be verified outside of the topology.
In the 9.2.0.0 release, we have updated the server so that it now generates digital signatures with encryption settings definitions. The server’s preferred definition will be used by default, but you can specify an alternative definition with the signing-encryption-settings-id property in the crypto manager configuration.

If digital signing is enabled but no encryption settings definitions are available, then a legacy topology key will continue to be used as a fallback.

The Argon2 key derivation function is a popular mechanism for encoding passwords, especially after it was selected as the winner of a password hashing competition in 2015. We introduced support for an ARGON2 password storage scheme in the 8.0.0.0 release.

There are actually three variants of the Argon2 algorithm:

• Argon2i — Provides better protection against side-channel attacks. The existing ARGON2 scheme uses this variant.
• Argon2d — Provides better protection against GPU-accelerated attacks.
• Argon2id — Mixes the strategies used in the Argon2i and Argon2d variants to provide a degree of protection against both types of attacks.

In the 9.2.0.0 release, we are introducing three new password storage schemes, ARGON2I, ARGON2D, and ARGON2ID, which provide explicit support for each of these variants.

Note that if you want to use the Argon2 algorithm to encode passwords, and you need to run in an environment that contains pre-9.2.0.0 servers, then you should use the existing ARGON2 scheme. The newer schemes should only be used in environments containing only servers running version 9.2.0.0 or later.

The Synchronization Server has included support for SCIMv1 servers as a sync destination since the 3.2.2.0 release. This support relies on an XML-based configuration to map LDAP source attributes to SCIM destination attributes.

In the 9.2.0.0 release, we’re introducing support for SCIMv2 servers as a sync destination. For this destination, all of the necessary configuration is held in the server’s configuration framework, so there is no need for a separate file with mapping information. This implementation introduces several new types of configurable components, including:

• HTTP authorization methods, which provide support for a variety of mechanisms for authenticating to HTTP-based services, including basic authentication and OAuth 2 bearer tokens (and in the latter case, you may configure either a static bearer token or have the server obtain one from an OAuth authorization server using the client_credentials grant type).
• A SCIM2 external server, which provides the SCIM service URL, authorization method, and other settings to use when interacting with the SCIMv2 service.
• SCIM2 attribute mappings, which describe how to generate SCIM attributes from the LDAP representation of a source entry.
• SCIM2 endpoint mappings, which associate a set of attribute mappings with an endpoint in the SCIMv2 server.
• The SCIM2 sync destination, which associates the SCIM2 external server and the SCIM2 endpoint mappings.

The documentation describes the process for configuring the Synchronization Server to synchronize changes to a SCIMv2 server. In addition, the config/sample-dsconfig-batch-files/configure-synchronization-to-scim2.dsconfig file provides an example that illustrates a set of changes that can be used to synchronize inetOrgPerson LDAP entries to urn:ietf:params:scim:schemas:core:2.0:UserM. SCIMv2 entries.