> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/release-notes/data-quality-observability-self-hosted-release-archive/release-2024-10.md). # Release 2024.10 * [Enhancements](#enhancements) * [Fixes](#fixes) * [Features in preview](#features-in-preview) * [DQ security](#dq-security) ## Release Information * Release date of Data Quality & Observability (self-hosted) 2024.10: October 29, 2024 * Release notes publication date: September 23, 2024 {% hint style="danger" %} Some customers have encountered issues while upgrading to Data Quality & Observability (self-hosted) 2024.10 due to a change in our flyway library that is not backwards compatible. The fix for this issue is included in the Data Quality & Observability (self-hosted) 2024.11.1 patch. As always, we recommend [backing up and restoring](/data-quality-self-hosted/installation/upgrade-data-quality-observability-self-hosted/prepare-your-environment-for-an-upgrade/create-and-restore-a-backup.md) your Metastore before upgrading Data Quality & Observability (self-hosted) versions. {% endhint %} {% hint style="info" %} The above issue only impacts upgrades to Data Quality & Observability (self-hosted) 2024.10. New installations will not encounter this issue. {% endhint %} ## Enhancements ### Pushdown * [SAP HANA Pushdown](/data-quality-self-hosted/core-concepts/explorer/processing-methods/pushdown-processing.md) is now generally available. * When creating and running DQ Jobs on [SQL Server Pushdown](/data-quality-self-hosted/core-concepts/explorer/processing-methods/pushdown-processing.md) connections, you can now perform schema, profile, and rules checks. * You can now scan for fuzzy match duplicates in DQ Jobs created on [BigQuery Pushdown](/data-quality-self-hosted/core-concepts/explorer/processing-methods/pushdown-processing.md) connections. * You can now scan for numerical outliers in DQ Jobs created on [Trino Pushdown](/data-quality-self-hosted/core-concepts/explorer/processing-methods/pushdown-processing.md) connections. * DQ Jobs created on [Snowflake Pushdown](/data-quality-self-hosted/core-concepts/explorer/processing-methods/pushdown-processing.md) connections now support union lookback for advanced outlier configurations. * DQ Jobs created on [Snowflake Pushdown](/data-quality-self-hosted/core-concepts/explorer/processing-methods/pushdown-processing.md) connections now support source to target validation to ensure data moves consistently through your data pipeline and identify changes when they occur. ### Integration * You can now define custom integration data quality rules, which are also known as aggregation paths, in the Collibra Platform operating model setting to allow you to view data quality scores for assets other than databases, schemas, tables, and columns.

You need a minimum of Collibra Platform 2024.10 and Data Quality & Observability (self-hosted) 2024.07.

* To allow you to manage the scope of the DGC resources to which OAuth can grant access during the integration, a new OAuth parameter in the Web ConfigMap is now set to `DQ_DGC_OAUTH_DEFAULT_REQUEST_SCOPE: "dgc.global-manage-all-resources"` by default. This configuration grants Data Quality & Observability (self-hosted) access via OAuth to all DGC resources during the integration. For more granular control over the DGC resources to which Data Quality & Observability (self-hosted) is granted access via OAuth, we plan to introduce additional allowable values in a future release. * Users of the Quality tab in Collibra Platform who do not have a Data Quality & Observability (self-hosted) account can now view the history table to track the evolution of the quality score of a given asset. * When using Replay to run DQ Jobs over a defined historical period, for example 5 days in the past, the metrics from each backrun DQ Job is included in the DQ History table and the quality calculation. * After integrating a dataset from Data Quality & Observability (self-hosted) and Collibra Platform, you can now see the number of passing and failing rows for a given rule on the Data Quality Rule asset page. * The JSON of a dataset integration between Data Quality & Observability (self-hosted) and Collibra Platform now shows the number of passing and breaking records. ### Jobs * You can now [edit the schedule details](/data-quality-self-hosted/core-concepts/scheduler/jobs-schedule.md) of DQ Jobs from the Jobs Schedule page. * A banner now appears when a data type is not supported. ### Rules * Data Class Rules now have a maximum character length of 64 characters for the Name option and 256 for the Description. * Email Data Classes where the email contains a single character domain name now pass validation. For example, * The Rule Definitions page now has a Break Record Preview Available column to make it easier to see when a rule is eligible for previewing break records. * You can now use the search field to search for Rule Descriptions on the Rule Definitions page. ### Alerts * You can now [toggle individual alerts](/data-quality-self-hosted/core-concepts/alerts/working-with-the-alert-builder.md) from the Active column on the Alert Builder page to improve control over when you want alerts to send. This can prevent unnecessary alerts from being sent during certain occasions, such as setup and debugging. ### Dataset Manager * The [Dataset Manager](/data-quality-self-hosted/core-concepts/dataset-directory/dataset-manager.md) table now contains a searchable and sortable column called Connection Name to help identify your datasets more easily. * We aligned the roles and permissions requirements for the Dataset Manager API. * PUT /v2/updatecatalogobj requires its users to have ROLE\_ADMIN, ROLE\_DATA\_GOVERNANCE\_MANAGER, or ROLE\_DATASET\_ACTIONS. * PUT /v2/updatecatalog requires its users to be the dataset owner or have ROLE\_ADMIN or ROLE\_DATASET\_ACTIONS. * DELETE /v2/deletedataset requires its users to be the dataset owner or have ROLE\_ADMIN or ROLE\_DATASET\_MANAGER. * PATCH /v2/renameDataset requires its users to be the owner of the source dataset or have ROLE\_ADMIN, ROLE\_DATASET\_MANAGER, or ROLE\_DATASET\_ACTIONS. * POST /v2/update-run-mode requires its users to have ROLE\_DATASET\_TRAIN, ROLE\_DATASET\_ACTIONS, or dataset access. * POST /v2/update-catalog-data-category requires its users to have ROLE\_DATASET\_TRAIN, ROLE\_DATASET\_ACTIONS, or dataset access. * PUT /v2/business-unit-to-dataset requires its users to have ROLE\_DATASET\_ACTIONS or dataset access. * POST /v2/business-unit-to-dataset requires its users to have ROLE\_DATASET\_ACTIONS or dataset access. * POST /dgc/integrations/trigger/integration requires its users to have ROLE\_ADMIN, ROLE\_DATASET\_MANAGER, or ROLE\_DATASET\_ACTIONS. * POST /v2/postjobschedule requires its users to have ROLE\_OWL\_CHECK and either ROLE\_DATASET\_ACTIONS or dataset access. ### Connections * The authentication dropdown menu for any given connection now displays only its supported authentication types. * We've upgraded the following drivers to the versions listed: * [Db2](/data-quality-self-hosted/connections/supported-connections/supported-jdbc-data-sources/connecting-to-db2.md) 4.27.25 * [Snowflake](/data-quality-self-hosted/connections/supported-connections/supported-jdbc-data-sources/connecting-to-snowflake.md) 3.19.0

If you use additional encryption algorithms for JWT authentication, you must set one of the following parameters during your deployment of Data Quality & Observability (self-hosted), depending on your deployment type:Helm-based deploymentsSet the following parameter in the Helm Chart: --set global.web.extraJvmOptions="-Dnet.snowflake.jdbc.enableBouncyCastle=true"Standalone deploymentsSet the following environment variable in the owl-env.sh: -export EXTRA_JVM_OPTIONS=”-Dnet.snowflake.jdbc.enableBouncyCastle=true"

* [SQL Server](/data-quality-self-hosted/connections/supported-connections/supported-jdbc-data-sources/connecting-to-sql-server.md) 12.6.4 (Java 11 only)

While the Java 8 version is not officially supported, you can replace the SQL Server driver in the /opt/owl/drivers/mssql folder with the Java 8 version of the supported driver. You can find Java 8 versions of the supported SQL Server on the Maven Repository.

### Admin Console * A new access control layer, **Require DATASET\_ACTIONS role for dataset management actions**, is available from the [Security Settings](/data-quality-self-hosted/admin-configuration/admin-console/security-settings.md) page. When enabled, a new out of the box role, [**ROLE\_DATASET\_ACTIONS**](/data-quality-self-hosted/admin-configuration/admin-console/role-management/role-management.md), is required to allow its users to edit, rename, publish, assign data categories and business units, and enable integrations from the [Dataset Manager](/data-quality-self-hosted/core-concepts/dataset-directory/dataset-manager.md). * A new out of the box role, [**ROLE\_ADMIN\_VIEWER**](/data-quality-self-hosted/admin-configuration/admin-console/role-management/role-management.md), allows users who are assigned to it to access the following Admin Console pages, but restricts access to all others: * Actions * All Audit Trail subpages * Dashboard

Users with ROLE_ADMIN_VIEWER cannot access the pages to which the quick access buttons on the Dashboards page are linked.

* Inventory * Schedule Restrictions

Users with ROLE_ADMIN_VIEWER cannot add or delete schedule restrictions.

* Usage * You can now set both size- and time-based data purges from the [Data Retention Policy](/data-quality-self-hosted/admin-configuration/admin-console/data-retention-policy.md) page of the Admin Console. Previously, you could only set size-based data retention policies. * The Data Retention Policy now supports auto-clean in both single- and multi-tenant environments. ### APIs * We’ve made several changes to the API documentation. First, we aligned the role checks between the Product APIs (V3 endpoints) and the Data Quality & Observability (self-hosted) UI. We’ve also enhanced the documentation in Swagger to include more detailed descriptions of endpoints. Lastly, we reproduced the Swagger documentation of the Product API in the [Collibra Developer Portal](https://developer.collibra.com/api/rest/dq-product#/) to ensure a more unified user experience with the broader Collibra platform and allow for easier scalability of API documentation in the future. ## Fixes ### Integration * The Overview - History section of Data Quality Job assets now displays the correct month of historical data when an integration job run occurs near the end of a given month. ### Jobs * DQ Jobs on SAP HANA connections with SQL referencing table names containing semicolons **;** now run successfully when you escape the table name with quotation marks **" "**. For example, the SQL query `select * from TEST."SPECIAL_CHAR_TABLE::$/-;@#%^&*?!{}~\+="` now runs successfully. * You can now run DQ Jobs that use Amazon S3 as the secondary dataset with Instance Profile authentication. ### Rules * Native rules on DQ Jobs created on connections authenticated by password manager now run successfully and return all related break records when their conditions are met. ### Alerts * The Assignee column is no longer included in the alert email for Rule Status and Condition alerts with rule details enabled. ### APIs * When using the POST /v2/controller-db-export call on a dataset with an alert condition, then using the POST /v2/controller-db-import call, now returns a successful 200 response instead of an unexpected JSON parsing error. ### Latest UI * When running DQ Jobs on NFS connections, data files with the date format **${yyyy}${MM}${dd}** within their file name are now supported. * Native Rules now display the variable name of parameters such as @runId and @dataset with the actual value in the Condition column of the Rules tab on the Findings page. * The Jobs Schedule page now shows the time zone offset (+ or - a number of hours) in the Last Updated column. Additionally, the TimeZone column is now directly to the right of the Scheduled Time column to improve its visibility. * You can now sort columns on the Job Schedule page. * The Agent Configuration, Role Management - Connections, Business Units, Inventory pages of the Admin Console now have fixed column headers and the Actions button and horizontal scrollbar are now visible at all times. * After adding or deleting rules, the rule count on the metadata bar now reflect any updates. ## Features in preview * The Rule Workbench now contains an additional query input field called “Filter,” which allows you to [narrow the scope of your rule query](/data-quality-self-hosted/core-concepts/dq-rules/rule-builder/working-with-the-rule-workbench/filter-a-data-quality-rule.md) so that only the rows you specify are considered when calculating the rule score. A filter query not only helps to provide a better representation of your quality score but improves the relevance of your rule results, saving both time and operational costs by reducing the need to create multiple datasets for each filter.

This feature is currently available as a public preview option. For more information about features in preview, see Previews at Collibra.

## Known limitations * In the this release, table names with spaces are not supported because of dataset name validation during the creation of a dataset. This will be addressed in an upcoming release. ## DQ Security
Show security metrics The following image shows a chart of Collibra DQ security vulnerabilities arranged by release version. ![a table showing the number of critical security vulnerabilities over a period of 5 releases](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-5faf70b553b8bdda8811b8d807751f0d498e4ad2%2Fcritical-table-2024-10.png?alt=media) The following image shows a table of Collibra DQ security metrics arranged by release version. ![a chart showing the number of critical security vulnerabilities over a period of 5 releases](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-920a0b1ed7528ed22a3f99b9fe70ca3cf6cea7b9%2Fvulns-over-time-2024-10.png?alt=media)