> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/release-notes/data-quality-observability-self-hosted-release-archive/release-2021-11.md). # Release 2021.11 #### Enhancements * Rules * \*Tech Preview\* \[TP] Semantics and data concepts management * The application now supports dynamic semantics checks. This allows you to create custom semantics that can be checked for when running a DQ check on a data set. Previously the application checked against a predefined set of semantics. You also have access to controls to organize and apply these semantics checks. The following is a list of changes: * There is a new data concepts management page. You can access it from Catalog or Admin Console. You can assign multiple semantics to a data concept. * When running a DQ check, you can select a data concept. The semantics assigned to this data concept will be checked against each column of dataset. * You have a list of predefined semantics that are not editable. You also have the ability to create/edit/delete custom semantics. * Repo on rules page has been added to Rules Library where semantics will be viewable. * Resource Limits * You can edit the Performance Settings to supply limits to executors, cores, memory and cells so that a user can be warned if submitting a job that requires a lot of resources and admins can control maximum resources submitted. #### Enhancements * Explorer * \*Tech Preview\* \[TP] Dynamic query reload allows you to view JOIN query columns in other activities. * User can update and reload the schema table with the custom query in the scope section by clicking the \[Update Scope] button. It will enable using the new columns from the custom query in all activities (Profile, Outlier, Dupes, Patterns, Source) * Since the first tab is for compositing the query, updating fields will change the user's custom query. Therefore, all areas are locked except the "query" field in the first tab to keep the query unchanged after updating the scope table * Support for some special characters in table name. * Fixed the ability to add additional libs that were previously not being properly saved on subsequent runs. Under DQ Job tag, please utilize -fllb boolean (union lookback) and libsrc input box for lib directory path (will materialize as -addlib). * Connections * \*Tech Preview\* \[TP] BigQuery Views and Joins * Please add the following to the BigQuery connection property ``` viewsEnabled=true ``` * API * You can perform multiple imports without conflicts. * You can have an incremental import such as updating matching records / insert new / leave existing. There is no requirement to delete tables first before running import. * Profile * Fixed backrun timebin to work with weeks and quarters instead of days. * Outliers * Split historical load to avoid historical query rounding up. * \*Tech Preview\* \[TP] Dynamic history options. * Source * Fixed an issue where settings were not sticky for subsequent runs. * Security * SAML Enhancements * New configuration settings are available when the Load Balancer is set for SSL Termination. * You can now set theRelayState property to route SSO to the proper tenant. #### Patches * 2021.11.1 Explorer * Allow ampersand in metastore host name for additional parameters * In below example, support for ampersand needed for required SSL flags ``` metastore01.us-east1-b.c.customer-dq-prod.internal:5432/dev?sslmode=required¤tSchema=public ``` #### Known Limitations * Rules * Semantics and data concepts: * Not supported in pushdown mode * Exporting RegEx semantics not currently supported * While it is possible to create joins and cross-dataset rules using Freeform SQL, it is best practice to create a view and handle the join prior to running the DQ Job. * Behavior * Schema is not eligible for invalidate * Files * Local files using UPLOAD\_PATH, UPLOAD\_FILE\_PATH, and temp files are only eligible to be deployed using the default NO\_AGENT option. These are only intended for quick tests and not intended for production-scale use. Best practice is to use a remote file system connection (S3, Google storage or ADLS). * Delimiter support for special characters is limited. Supported file delimiters are comma, pipe, tab, semicolon, double quote and single quote. Custom delimiters will work for many characters, but not all combinations. * Temp files and NO\_AGENT should have -master local\[\*] or -master spark://:7077 defined in freeform append of the agent options * DQ Job * When submitting jobs via API from a different machine with a different timezone, timezone discrepancies are not accounted for automatically. Best practice is to align each component to use UTC. * Jobs submitted via API with a run date that include HH:MM in the -rd (run date) will submit to the job queue and leave a remnant ‘STAGED’ job * Connections * Postgres limits max connections per spark job. The default is 100. Please refer to Postgres official documentation how to increase max\_connection and shared\_buffers. * * BigQuery * Updating scope to include joins in BigQuery can only be materialized when tables are part of the same dataset collection * Should you receive an error for pre-existing BigQuery jobs, please add -dssafeoff to the cmd line or select ‘Allow Overwrite’ to enable this from Edit mode in the Explorer * Alerts * After an upgrade to 2021.11, you may need to set the environment variable `ALERT_SCHEDULE_ENABLED=true` in **owl-env.sh** and restart **owl-web** to enable email alerts to work again.