> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/connections/supported-connections/supported-remote-files/connecting-to-azure-data-lake-storage.md).

# Connecting to Azure Data Lake Storage

This section contains an overview of Azure Data Lake Storage Gen2 (ADLS).

## General information

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td>Data source</td><td>Azure Data Lake Storage Gen2 (ADLS)</td></tr><tr><td>Supported versions</td><td>N/A</td></tr><tr><td>Connection string</td><td><code>abfss://</code></td></tr><tr><td>Packaged?</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Certified?</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td colspan="2">Supported features</td></tr><tr><td>Analyze data</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Archive breaking records</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Estimate job</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Pushdown</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td></tr><tr><td colspan="2">Processing capabilities</td></tr><tr><td>Spark agent</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Yarn agent</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr></tbody></table>

## Minimum user permissions

In order for Collibra DQ to access your Azure Storage containers, you need the following user permissions:

* ROLE\_ADMIN in Collibra DQ.
* Read access on your Azure Storage containers to create a basic ADLS connection.
* Read and write access to the Azure Storage containers where breaking records are archived from Collibra DQ. This is only necessary if you use the archive breaking records feature.
* For the Download CSV and Copy Signed URL features to work, the Azure Service Principal used by the Azure Data Lake connection must be assigned the **Storage Blob Delegator** role on the Azure Storage Account.

## Recommended and required connection properties

<table><thead><tr><th>Required</th><th>Connection Property</th><th>Type</th><th>Value</th></tr></thead><tbody><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td>Name</td><td>Text</td><td>The unique name used for your connection.</td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td>Connection URL</td><td>String</td><td><p>The connection string value of your ADLS connection.</p><pre><code>abfss://&#x3C;file system>@&#x3C;account name>.dfs.core.windows.net/
</code></pre><div data-gb-custom-block data-tag="hint" data-style="success" class="hint hint-success"><p>See the <a href="#connection-url-elements">Connection URL elements</a> section for more information on the syntax.</p></div></td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td><td>Target Agent</td><td>Text</td><td>The <a href="/data-quality-self-hosted/installation/agent.md">Agent</a> used to submit your DQ Job.</td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td>Auth Type</td><td>Option</td><td><p>The method to authenticate your connection.</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>The configuration requirements are different depending on the Auth Type you select. See <a href="#authentication">Authentication</a> for more details on available authentication types.</p></div></td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td><td>Driver Properties</td><td>String</td><td>The configurable driver properties for your connection. Multiple properties must be semicolon-delimited, and each property=value pair, including the last one, must end with a semicolon (;). <code>abc=123;test=true;</code></td></tr></tbody></table>

## Connection URL elements

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td>File scheme</td><td>The <code>abfss</code> protocol is used as the scheme identifier.</td></tr><tr><td>File system</td><td>The parent location that holds the files and folders. This is the same as containers in the Azure Storage Blob service.</td></tr><tr><td>Account name</td><td>The name given to your storage account during creation.</td></tr></tbody></table>

## Authentication

Select an authentication type from the dropdown menu. The options available in the dropdown menu are the currently supported authentication types for this data source.

{% tabs %}
{% tab title="Key" %}

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td>Storage Account</td><td>The Azure Storage account name.</td></tr><tr><td>Key</td><td>The authentication key for the storage account.</td></tr></tbody></table>
{% endtab %}

{% tab title="Service Principal" %}

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td>TenantId</td><td>The Microsoft Online tenant being used to access data. If not specified, your default tenant will be used.</td></tr><tr><td>ClientId</td><td>The client Id assigned when you register your application with an OAuth authorization server.</td></tr><tr><td>Secret</td><td>The client secret assigned when you register your application with an OAuth authorization server.</td></tr></tbody></table>
{% endtab %}
{% endtabs %}

## Configure private endpoints for key-based authentication

{% stepper %}
{% step %}
Fill out the required fields on the Details tab.
{% endstep %}

{% step %}
Select **Key** from the Authentication type dropdown menu, then fill out the **Storage Account** and **Key** fields.
{% endstep %}

{% step %}
In the Driver Properties field on the Properties tab, add **cloud.endpoint=\<endpoint>** to enable the Azure Data Lake Storage (Gen2) private endpoint for key-based authentication.

{% hint style="info" %}
The **\<endpoint>** value can be any Microsoft cloud endpoint. Depending on the location, Azure uses **\*.windows.net** and **\*.chinacloudapi.cn**, and Microsoft Entra ID uses **<https://login.microsoftonline.com>** and **<https://login.chinacloudapi.cn>**. For more information, go to the official [Microsoft documentation](https://learn.microsoft.com/en-us/azure/china/concepts-service-availability#azure-in-china-rest-endpoints).
{% endhint %}

{% tabs %}
{% tab title="Example" icon="glasses-round" %}
To use the **usgovcloudapi.net** endpoint, add **cloud.endpoint=usgovcloudapi.net**.
{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}
Click **Submit**.
{% endstep %}
{% endstepper %}

## Configure private endpoints for service principal-based authentication

{% stepper %}
{% step %}
Fill out the required fields on the Details tab.
{% endstep %}

{% step %}
Select **Service Principal** from the Authentication type dropdown menu, then fill out the **TenantId**, **ClientId**, and **Secret** fields.
{% endstep %}

{% step %}
In the Driver Properties field on the Properties tab, add **cloud.endpoint=usgovcloudapi.net;authority.host=gov** to enable the Azure Data Lake Storage (Gen2) private endpoint for key-based authentication.
{% endstep %}

{% step %}
Click **Submit**.
{% endstep %}
{% endstepper %}

## Limitations

Creating a job from an ADLS folder is not supported.