> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/installation/agent/set-up-a-dq-agent.md).

# Set up a DQ agent

This topic shows you how to configure a DQ agent.

## Setting up a DQ Agent with `setup.sh` as part of the DQ package

Use the `setup.sh` script located in `/opt/owl/` (or other **Base Path** that your installation used). See the example in the code block below for setting up a DQ Agent with a Postgres server running `localhost` on port `5432` with database postgres and Postgres username/password combo `postgres/password`.

```
# PATH TO DIR THAT CONTAINS THE INSTALL DIR
                export BASE_PATH=/opt
 
                # PATH TO AGENT INSTALL DIR
                export INSTALL_PATH=/opt/owl
 
                # DQ Metadata Postgres Storage settings 
                export METASTORE_HOST=localhost
                export METASTORE_PORT=5432
                export METASTORE_DB=postgres
                export METASTORE_USER=postgres
                export METASTORE_PASSWORD=password 
 
                cd $INSTALL_PATH
 
                # Install DQ Agent only
                ./setup.sh \
                -owlbase=$BASE_PATH \
                -options=owlagent \
                -pguser=$METASTORE_USER \
                -pgpassword=$METASTORE_PASSWORD \
            -pgserver=${METASTORE_HOST}:${METASTORE_PORT}/${METASTORE_DB}
```

The setup script automatically generates the `/opt/owl/config/owl.properties` file and encrypts the provided password.

## Setting up a DQ agent manually

{% stepper %}
{% step %}
Open a terminal session and go to the directory with the installer.
{% endstep %}

{% step %}
Run the following command to encrypt your DQ Metastore password before it is stored in the `/opt/owl/config/owl.properties` file:

```
# PATH TO AGENT INSTALL DIR
                        export INSTALL_PATH=/opt/owl
 
                        cd $INSTALL_PATH
 
                        #Encrypt DQ Metadata Postgres Storage password
            ./owlmanage.sh encrypt=password
```

{% hint style="info" %}
`owlmanage.sh` generates an encrypted string for the plain text password input. You can use the encrypted string in the `/opt/owl/config/owl.properties` configuration file to avoid exposing the DQ Metadata Postgres Storage password.
{% endhint %}
{% endstep %}

{% step %}
Run the following command to open the `/opt/owl/config/owl.properties` configuration file:

```
vi $INSTALL_PATH/config/owl.properties
```

{% endstep %}

{% step %}
Add the following properties to the configuration file:

```
spring.datasource.url=jdbc:postgresql://{DB_HOST}:{DB_PORT}/{METASTORE_DB}
                        spring.datasource.username={METASTORE_USER}
                        spring.datasource.password={METASTORE_PASSWORD}
                        spring.datasource.driver-class-name=com.owl.org.postgresql.Driver
 
                        spring.agent.datasource.url=jdbc:postgresql://{DB_HOST}:{DB_PORT}/{METASTORE_DB}
                        spring.agent.datasource.username={METASTORE_USER}
                        spring.agent.datasource.password={METASTORE_PASSWORD}
            spring.agent.datasource.driver-class-name=org.postgresql.Driver
```

{% endstep %}

{% step %}
Restart the DQ Web App.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
In `SPRING_AGENT_DATASOURCE_URL`, the connection parameter `currentSchema` must be set at the end of the URL.
{% endhint %}

## Setting up the DQ Agent from the Admin Console

{% stepper %}
{% step %}
On the Collibra DQ home page, hover your pointer over <img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-75245abf5c50ddf56f1d2d727ccaacf757d11145%2Fcogwheel-large%402x.png?alt=media" alt="" data-size="line"> **Settings** and select **Admin Console**.\ <i class="fa-chevrons-right">:chevrons-right:</i> The Admin Console opens.
{% endstep %}

{% step %}
Click **Agent Configuration**.\ <i class="fa-chevrons-right">:chevrons-right:</i> The Agent Configuration page opens.
{% endstep %}

{% step %}
In the Actions column, click <img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-5face7f577747d86367bf6698315172b6a3536f5%2FEditIcon.svg?alt=media" alt="Edit icon" data-size="line"> to edit your agent.\ <i class="fa-chevrons-right">:chevrons-right:</i> The **Edit Agent** modal appears.
{% endstep %}

{% step %}
Enter the required information.

{% tabs %}
{% tab title="Standalone" %}

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td><strong>Agent Id</strong></td><td><p>The numerical identifier of your agent. For example, <em>6</em>.</p><p>This field auto-generates and cannot be edited.</p></td></tr><tr><td><strong>Agent Name</strong></td><td><p>The unique name of your agent.</p><p>This field auto-generates and cannot be edited.</p></td></tr><tr><td><strong>Agent Display Name</strong></td><td><p>The descriptive name of your agent that displays anywhere agent information is present in the DQ Web App. You can customize the Agent Display Name to make it easier to identify your agent.</p><div data-gb-custom-block data-tag="hint" data-style="success" class="hint hint-success"><p>There are no character restrictions for the Agent Display Name field, but it is best practice to use only alphanumeric characters, hyphens, and underscores.</p></div></td></tr><tr><td><strong>Is Local</strong></td><td>Select this option for Hadoop deployments only.</td></tr><tr><td><strong>Use Livy</strong></td><td>Not applicable.</td></tr><tr><td><strong>Livy Host</strong></td><td>Not applicable.</td></tr><tr><td><strong>Base Path</strong></td><td><p>The installation folder path for DQ. All other paths in the DQ Agent are relative to this installation path.</p><p>This is the location that is set as <code>OWL_BASE</code> in Full Standalone Setup and other installation setups followed by <code>owl/</code> folder. For example, if the setup command is <code>export OWL_BASE=/home/centos</code> then the <strong>Base Path</strong> in the Agent configuration should be set to <code>/home/centos/owl/</code>.</p><p>Default: <code>/opt/owl/</code>.</p></td></tr><tr><td><strong>Collibra DQ Core JAR</strong></td><td><p>The file path to the DQ Core jar file.</p><p>Default <code>&#x3C;Base Path>/owl/bin/</code></p></td></tr><tr><td><strong>Collibra DQ Core Logs</strong></td><td><p>The folder path where DQ Core logs are stored. Logs from DQ Jobs are stored in this folder.</p><p>Default: <code>&#x3C;Base Path>/owl/log</code></p></td></tr><tr><td><strong>Collibra DQ Script</strong></td><td><p>The file path to DQ execution script <code>owlcheck.sh</code>. This script is used to run DQ Job via command line without using the agent. Using <code>owlcheck.sh</code> for running DQ Jobs is superseded by DQ Agent execution model.</p><p>Default: <code>&#x3C;Base Path>/owl/bin/owlcheck</code></p></td></tr><tr><td><strong>Collibra DQ Web Logs</strong></td><td><p>The folder path where DQ Web logs are stored. Logs from the DQ Web App are stored in this folder.</p><p>Default: <code>&#x3C;Base Path>/owl/log</code></p></td></tr><tr><td><strong>Default Queue</strong></td><td>Only used for Yarn.</td></tr><tr><td><strong>Deploy Deployment Mode</strong></td><td><p>The Spark deployment mode can be either <strong>Client</strong> or <strong>Cluster</strong>. While we recommend <strong>Cluster</strong>, there are best practices to follow:</p><ul><li>If you only have one Spark Worker node, it is best practice to select <strong>Client</strong>.</li><li>If you have more than one Spark Worker node, it is best practice to select <strong>Cluster</strong>.</li></ul></td></tr><tr><td><strong>Default Master</strong></td><td>The Spark Master URL copied from the Spark cluster verification screen. For example, <code>spark://...</code></td></tr><tr><td><strong>Dynamic Spark Allocation</strong></td><td>Not applicable.</td></tr><tr><td><strong>Spark Configuration Key</strong></td><td>Not applicable.</td></tr><tr><td><strong>Spark Configuration Value</strong></td><td>Not applicable.</td></tr><tr><td><strong>Number of Executor(s)</strong></td><td>The default number of executors allocated per DQ Job when using this Agent to run DQ Scans. The default is 1.</td></tr><tr><td><strong>Executor Memory</strong></td><td><p>The default RAM per executors allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1 gigabyte.</p></td></tr><tr><td><strong>Number of Core(s)</strong></td><td><p>The default number of cores per executors allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1.</p></td></tr><tr><td><strong>Driver Memory</strong></td><td><p>The default driver RAM allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1 gigabyte.</p></td></tr><tr><td><strong>Free Form (Appended)</strong></td><td>Other <code>spark-submit</code> parameters to append to each DQ Job when using this Agent to run DQ Scans.</td></tr></tbody></table>
{% endtab %}

{% tab title="Yarn" %}

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td><strong>Agent Id</strong></td><td><p>The numerical identifier of your agent. For example, <em>6</em>.</p><p>This field auto-generates and cannot be edited.</p></td></tr><tr><td><strong>Agent Name</strong></td><td><p>The unique name of your agent.</p><p>This field auto-generates and cannot be edited.</p></td></tr><tr><td><strong>Agent Display Name</strong></td><td><p>The descriptive name of your agent that displays anywhere agent information is present in the DQ Web App. You can customize the Agent Display Name to make it easier to identify your agent.</p><div data-gb-custom-block data-tag="hint" data-style="success" class="hint hint-success"><p>There are no character restrictions for the Agent Display Name field, but it is best practice to use only alphanumeric characters, hyphens, and underscores.</p></div></td></tr><tr><td><strong>Is Local</strong></td><td>You can select this option to form the driver location path, which is normally applicable only when you run your agent in the master or edge node.</td></tr><tr><td><strong>Use Livy</strong></td><td>Not applicable.</td></tr><tr><td><strong>Livy Host</strong></td><td>Not applicable.</td></tr><tr><td><strong>Base Path</strong></td><td><p>The installation folder path for DQ. All other paths in the DQ Agent are relative to this installation path.</p><p>This is the location that is set as <code>OWL_BASE</code> in Full Standalone Setup and other installation setups followed by <code>owl/</code> folder. For example, if the setup command is <code>export OWL_BASE=/home/centos</code> then the <strong>Base Path</strong> in the Agent configuration should be set to <code>/home/centos/owl/</code>.</p><p>Default: <code>/opt/owl/</code></p></td></tr><tr><td><strong>Collibra DQ Core JAR</strong></td><td><p>The file path to the DQ Core jar file.</p><p>Default: <code>&#x3C;Base Path>/owl/bin/</code></p></td></tr><tr><td><strong>Collibra DQ Core Logs</strong></td><td><p>The folder path where DQ Core logs are stored. Logs from DQ Jobs are stored in this folder.</p><p>Default: <code>&#x3C;Base Path>/owl/log</code></p></td></tr><tr><td><strong>Collibra DQ Script</strong></td><td><p>The file path to the DQ execution script <code>owlcheck.sh</code>. This script is used to run DQ Jobs via the command line without using an agent. Using <code>owlcheck.sh</code> for running DQ Jobs is superseded by the DQ Agent execution model.</p><p>Default: <code>&#x3C;Base Path>/owl/bin/owlcheck</code></p></td></tr><tr><td><strong>Collibra DQ Web Logs</strong></td><td><p>The folder path where DQ Web logs are stored. Logs from the DQ Web App are stored in this folder.</p><p>Default: <code>&#x3C;Base Path>/owl/log</code>.</p></td></tr><tr><td><strong>Default Queue</strong></td><td>The default resource queue to submit jobs.</td></tr><tr><td><strong>Default Deployment Mode</strong></td><td>The Spark deployment mode for Yarn is <strong>Cluster</strong>.</td></tr><tr><td><strong>Default Master</strong></td><td><p>Set to <code>Yarn</code>.</p><p>Click <strong>Edit Yarn Config</strong> to ensure you have the necessary Hadoop xml files. Edit the file templates as necessary:</p><ul><li><p><strong>core-site.xml</strong>: Contains information about where authentication protocol, HDFS_RPC_PROTECTION, and the NAME_NODE run in the Hadoop cluster.<strong>Example template</strong></p><pre><code>&#x3C;configuration>
  &#x3C;property>
    &#x3C;name>hadoop.security.authentication&#x3C;/name>
    &#x3C;value>kerberos&#x3C;/value>
  &#x3C;/property>
  &#x3C;property>
    &#x3C;name>hadoop.rpc.protection&#x3C;/name>
    &#x3C;value>$HDFS_RPC_PROTECTION&#x3C;/value>
  &#x3C;/property>
  &#x3C;property>
    &#x3C;name>fs.defaultFS&#x3C;/name>
    &#x3C;value>hdfs://$NAME_NODE:8020&#x3C;/value>
  &#x3C;/property>
&#x3C;/configuration>
</code></pre></li><li><p><strong>hdfs-site.xml</strong>: Contains the configuration settings for authentication protocol, the NAME_NODE, and DATA_NODE.<strong>Example template</strong></p><pre><code>&#x3C;configuration>
  &#x3C;property>
    &#x3C;name>hadoop.security.authentication&#x3C;/name>
    &#x3C;value>HDFS/_HOST@$KERBEROS_DOMAIN&#x3C;/value>
  &#x3C;/property>
&#x3C;/configuration>
</code></pre></li><li><p><strong>yarn-site.xml</strong>: Contains the Yarn resource manager settings.<strong>Example template</strong></p><pre><code>&#x3C;configuration>
  &#x3C;property>
    &#x3C;name>yarn.resourcemanager.scheduler.address&#x3C;/name>
    &#x3C;value>$RESOURCE_MANAGER:8030&#x3C;/value>
  &#x3C;/property>
  &#x3C;property>
    &#x3C;name>yarn.resourcemanager.address&#x3C;/name>
    &#x3C;value>$RESOURCE_MANAGER:8032&#x3C;/value>
  &#x3C;/property>
  &#x3C;property>
    &#x3C;name>yarn.resourcemanager.webapp.https.address&#x3C;/name>
    &#x3C;value>$RESOURCE_MANAGER:8090&#x3C;/value>
  &#x3C;/property>
&#x3C;/configuration>
</code></pre></li></ul></td></tr><tr><td><strong>Dynamic Spark Allocation</strong></td><td>Not applicable.</td></tr><tr><td><strong>Spark Configuration Key</strong></td><td>Not applicable.</td></tr><tr><td><strong>Spark Configuration Value</strong></td><td>Not applicable.</td></tr><tr><td><strong>Number of Executor(s)</strong></td><td><p>The default number of executors allocated per DQ Job when using this Agent to run DQ Scans.</p><p>The default is 1.</p></td></tr><tr><td><strong>Executor Memory</strong></td><td><p>The default RAM per executors allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1 gigabyte.</p></td></tr><tr><td><strong>Number of Core(s)</strong></td><td><p>The default number of cores per executors allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1.</p></td></tr><tr><td><strong>Driver Memory</strong></td><td><p>The default driver RAM allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1 gigabyte.</p></td></tr><tr><td><strong>Free Form (Appended)</strong></td><td>Other <code>spark-submit</code> parameters to append to each DQ Job when using this Agent to run DQ Scans.</td></tr></tbody></table>

{% hint style="info" %}
Ensure that your service account has the permission to launch Spark executor pods. Refer to the executor launch template and permissions.
{% endhint %}
{% endtab %}

{% tab title="K8s" %}

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td><strong>Agent Id</strong></td><td><p>The numerical identifier of your agent. For example, <em>6</em>.</p><p>This field auto-generates and cannot be edited.</p></td></tr><tr><td><strong>Agent Name</strong></td><td><p>The unique name of your agent.</p><p>This field auto-generates and cannot be edited.</p></td></tr><tr><td><strong>Agent Display Name</strong></td><td><p>The descriptive name of your agent that displays anywhere agent information is present in the DQ Web App. You can customize the Agent Display Name to make it easier to identify your agent.</p><div data-gb-custom-block data-tag="hint" data-style="success" class="hint hint-success"><p>There are no character restrictions for the Agent Display Name field, but it is best practice to use only alphanumeric characters, hyphens, and underscores.</p></div></td></tr><tr><td><strong>Is Local</strong></td><td>Select this option for Hadoop deployments only.</td></tr><tr><td><strong>Use Livy</strong></td><td>Not applicable.</td></tr><tr><td><strong>Livy Host</strong></td><td>Not applicable.</td></tr><tr><td><strong>Base Path</strong></td><td><p>The installation folder path for DQ. All other paths in the DQ Agent are relative to this installation path.</p><p>This is the location that is set as <code>OWL_BASE</code> in Full Standalone Setup and other installation setups followed by <code>owl/</code> folder. For example, if the setup command is <code>export OWL_BASE=/home/centos</code> then the <strong>Base Path</strong> in the Agent configuration should be set to <code>/home/centos/owl/</code>.</p><p>Default: <code>/opt/owl/</code></p></td></tr><tr><td><strong>Collibra DQ Core JAR</strong></td><td><p>The file path to the DQ Core jar file.</p><p>Default: <code>&#x3C;Base Path>/owl/bin/</code></p></td></tr><tr><td><strong>Collibra DQ Core Logs</strong></td><td><p>The folder path where DQ Core logs are stored. Logs from DQ Jobs are stored in this folder.</p><p>Default: <code>&#x3C;Base Path>/owl/log</code></p></td></tr><tr><td><strong>Collibra DQ Script</strong></td><td>The file path to DQ execution script <code>owlcheck.sh</code>. This script is used to run DQ Job via command line without using agent. Using <code>owlcheck.sh</code> for running DQ Jobs is superseded by DQ Agent execution model. Default: <code>&#x3C;Base Path>/owl/bin/owlcheck</code>.</td></tr><tr><td><strong>Collibra DQ Web Logs</strong></td><td><p>The folder path where DQ Web logs are stored. Logs from the DQ Web App are stored in this folder.</p><p>Default: <code>&#x3C;Base Path>/owl/log</code></p></td></tr><tr><td><strong>Default Queue</strong></td><td>Only used for Yarn.</td></tr><tr><td><strong>Default Deployment Mode</strong></td><td>The Spark deployment mode for Kubernetes is <strong>Cluster</strong>.</td></tr><tr><td><strong>Default Master</strong></td><td><p>The Kubernetes Master URL copied from the Kubernetes cluster verification screen.</p><p>Set this value to <code>k8s://</code> instead of a specific URL. When you leave this value set to <code>k8s://</code>, this helps Collibra DQ auto-discover the High Availability endpoint of the Kubernetes control plane at runtime.</p><div data-gb-custom-block data-tag="hint" data-style="danger" class="hint hint-danger"><p>Only set this to a specific URL, such as <code>k8s://{hostname}:443</code>, if you are an advanced user or if your specific use case requires it.</p></div></td></tr><tr><td><strong>Dynamic Spark Allocation</strong></td><td>Not applicable.</td></tr><tr><td><strong>Spark Configuration Key</strong></td><td>Not applicable.</td></tr><tr><td><strong>Spark Configuration Value</strong></td><td>Not applicable.</td></tr><tr><td><strong>Number of Executor(s)</strong></td><td>The default number of executors allocated per DQ Job when using this Agent to run DQ Scans. The default is 1.</td></tr><tr><td><strong>Executor Memory</strong></td><td><p>The default RAM per executors allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1 gigabyte.</p></td></tr><tr><td><strong>Number of Core(s)</strong></td><td><p>The default number of cores per executors allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1.</p></td></tr><tr><td><strong>Driver Memory</strong></td><td><p>The default driver RAM allocated per DQ Job when using this Agent to run DQ Scans. Go to <a href="/data-quality-self-hosted/installation/system-requirements.md">System requirements</a> for more information.</p><p>The default is 1 gigabyte.</p></td></tr><tr><td><strong>Free Form (Appended)</strong></td><td>Other <code>spark-submit</code> parameters to append to each DQ Job when using this Agent to run DQ Scans.</td></tr></tbody></table>

{% hint style="info" %}
If you bring in your own Spark executor pod launch template, ensure that the service account used to launch Spark executor pods has the permission to do so. Refer to the [executor launch template](https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template) and for more information.
{% endhint %}
{% endtab %}
{% endtabs %}
{% endstep %}

{% step %}
Click **Save**.
{% endstep %}
{% endstepper %}

## Setting up a DQ Agent on GCP Dataproc

This section provides additional information for installing the DQ Agent on a Google Cloud Dataproc master node.

{% hint style="info" %}
We no longer support Dataproc starting from Data Quality & Observability (self-hosted) version 2026.02 because it does not support Spark 4.
{% endhint %}

DQ Agent requires Java 21 as the default Java runtime for Spark applications. The Spark 3.5 image for GCP Dataproc includes both Java 11 and Java 17, with Java 11 as the default. To set Java 21 as the default, set the JAVA\_HOME environment variable at the session level.

Before installing the DQ Agent, execute the following script:

```
#!/bin/bash
java_home=$(update-alternatives --list java | grep 21-jdk)
update-alternatives --set java $java_home
export JAVA_HOME=$java_home
java --version
```

After installing DQ Agent, edit the settings to set JAVA\_HOME for the DQ jobs. In the DQ Web application, open the Edit Agent modal and add the following parameter to the **Free form (Appended)** field:

```
-conf spark.yarn.appMasterEnv.JAVA_HOME=/usr/lib/jvm/temurin-21-jdk-amd64,spark.executorEnv.JAVA_HOME=/usr/lib/jvm/temurin-21-jdk-amd64
```

## What's next

[Map data source connections to a DQ agent](/data-quality-self-hosted/installation/agent/map-data-source-connections-to-a-dq-agent.md)