> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/installation/configuring-apache-hadoop-to-execute-dq-jobs/configure-aws-emr.md). # Configure AWS EMR Each cloud provider has unique steps and configuration options for running Apache Hadoop clusters. This section describes how to configure Amazon Web Services (AWS) Elastic MapReduce (EMR). For additional information on Hadoop integration, go to [Configuring Apache Hadoop to execute DQ Jobs](/data-quality-self-hosted/installation/configuring-apache-hadoop-to-execute-dq-jobs.md). ## Data Quality & Observability (self-hosted) on EMR Architecture Collibra DQ is able to use EMR as the compute space for DQ jobs. While it is possible to simply operate a long running EMR cluster, EMR's intended operating model is ephemeral infrastructure. Using EMR as an ephemeral compute space is the most cost effective approach both in terms of operational effort and infrastructure costs. Collibra DQ makes it possible to seamlessly leverage EMR in this operating model. When there is not an EMR cluster available, Collibra DQ users are still able to browse datasets and DQ results in DQ Web. However, if a user attempts to deploy a DQ Job, they will see a red light icon next to the target agent. If the user still wants to request a DQ Job, it will simply wait in queue until the target agent comes back online the next time an EMR cluster is available. To ensure a job waits for the same agent, the agent on the replacement cluster must use the same agent ID. When the agent first registers, save the agent ID it records. Then, pass this saved ID as `OWL_AGENT_ID` when deploying the replacement cluster. If you use an ID that the agent did not record, the replacement cluster either creates a new agent or rejects the ID with an `Invalid UUID string` error. Without the saved ID, the replacement cluster registers a new agent, causing connections and scheduled jobs to remain with the old agent. ### Prepare for deployment {% hint style="info" %} Before allowing Collibra DQ to use EMR as the compute space, make sure that DQ Web and the DQ Metastore are already deployed. For more installation details, see [Standalone Install](/data-quality-self-hosted/installation/installing-data-quality-observability-self-hosted-on-self-hosted-spark-standalone.md). {% endhint %} {% stepper %} {% step %} Create a "bootstrap bucket" location in S3 where Collibra DQ binaries and bootstrap script (`install-agent-emr.sh`) will be staged. The EMR cluster instances will need to include an attached Role that has access to this location in order to bootstrap the cluster. This location should not contain any data or any kind of sensitive materials and thus should not require any special permissions. It just needs to be accessible by EMR clusters for bootstrap purposes. {% endstep %} {% step %} Create or modify an instance Profile Role that will be attached to EMR clusters so that it enables read access to the bootstrap bucket location. This Role is separate from the EMR service role that EMR will use to deploy the infrastructure. {% endstep %} {% step %} Stage the bootstrap script and the Collibra DQ binary package in the bootstrap location created above. For EMR 8.0.x, stage the EMR 8 package (see step 6). {% endstep %} {% step %} Make sure that the VPC where the Collibra DQ Metastore is deployed is accessible from the VPC where EMR clusters will be deployed. {% endstep %} {% step %} Make sure that Security Groups applied to the Collibra DQ Metastore are configured to allow access from EMR master and worker Security Groups. If a data source restricts inbound traffic by IP address, you must also allow the IP addresses of the EMR master and core nodes. Keep in mind that a new cluster generates new IP addresses. {% endstep %} {% step %} Decide whether to use EMR 5.x, EMR 6.x, or EMR 8.0.x. This is important because EMR 6 introduces Spark 3 and Scala 2.12. * If you select EMR 6, ensure Collibra DQ binaries were compiled for Spark 3 and Scala 2.12. * If you select EMR 8.0.x, download the AWS EMR 8 package for your DQ version from the Collibra Product Resource Center downloads page. The file name has the form `dq--SPARK410-JDK21-EMR8-package-full.tar.gz`. EMR 8.0.x requires Java 21. Do not use the standard SPARK410 package. {% endstep %} {% step %} (OPTIONAL) Create and store a private key to access EMR instances. {% endstep %} {% endstepper %} ### Deploy an EMR cluster There are several ways to deploy EMR, but for dev-ops purposes, the best path is usually to use the AWS CLI utility. The example below will deploy an EMR cluster bootstrapped with Collibra DQ binaries and a functioning agent to deploy DQ Jobs. {% hint style="info" %} When defining the Bootstrap Location argument, do not include "s3://". For example: If Bootstrap Location is s3://bucket/prefix then BOOTSTRAP\_LOCATION="bucket/prefix". {% endhint %} {% hint style="danger" %} Before you deploy EMR, edit `install-agent-emr.sh` and add the following export variables to it for the setup script to run correctly:`export DQ_ADMIN_USER_PASSWORD``export DQ_ADMIN_USER_EMAIL` {% endhint %} {% hint style="warning" %} As of Collibra DQ version 2023.09, we changed our package names from `owl-*` to `dq-*`. Therefore, you need to replace the following lines in your EMR file referencing **owl** with **dq**:`aws s3 cp s3://${OWL_PACKAGE_LOCATION}/dq-$OWL_VERSION-package-full.tar.gz ./tar -xvf dq-$OWL_VERSION-package-full.tar.gz` {% endhint %} {% hint style="warning" %} For EMR 8.0.x, the `install-agent-emr.sh` script, as provided, does not install the EMR 8 package. Before deploying an EMR 8.0.x cluster, edit the `install-agent-emr.sh` script to do the following: * Use the EMR 8 package. Replace `dq-$OWL_VERSION-package-full.tar.gz` with `dq-$OWL_VERSION-SPARK410-JDK21-EMR8-package-full.tar.gz` in the relevant lines. * Prioritize Java 21 on the PATH before the setup script runs. The default Java version on EMR 8 is Java 8, but Java 21 is preinstalled in `/usr/lib/jvm`. * Set the agent's Java version to Java 21 after the setup script runs and before the agent starts. The setup script leaves the agent's Java version unset. Set Java 21 for the agent only, not system-wide, because Hadoop and YARN require EMR's default Java version. * Use the `--release-label emr-spark-8.0.0` option instead of `emr-6.2.0`. EMR 8.0.x clusters with multiple primary nodes are not supported. In such cases, the script installs an agent on each primary node. {% endhint %} ``` aws emr create-cluster \ --auto-scaling-role EMR_AutoScaling_DefaultRole \ --applications Name=Hadoop Name=Spark Name=Hive Name=Tez \ --name owl-emr \ --release-label emr-6.2.0 \ --region ${EMR_REGION} \ --ebs-root-volume-size 10 \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --enable-debugging \ --bootstrap-actions \ "[{\"Path\":\"s3://${BOOTSTRAP_LOCATION}/install-agent-emr.sh\", \ \"Args\":[ \ \"${OWL_VERSION}\", \ \"${OWL_AGENT_ID}\", \ \"${METASTORE_HOST}:${METASTORE_PORT}/${METASTORE_DB}?currentSchema=owlhub\", \ \"${METASTORE_USER}\", \ \"${METASTORE_PASSWORD}\", \ \"${BOOTSTRAP_LOCATION}\", \ \"${LICENSE_KEY}\", \ \"native\"], \"Name\":\"install-owl-agent\"}]" \ --ec2-attributes "{ \ \"KeyName\":\"${EMR_INSTANCE_PRIVATE_KEY_NAME}\", \ \"InstanceProfile\":\"${BOOTSTRAP_ACCESS_ROLE}\", \ \"SubnetId\":\"${EMR_SUBNET}\", \ \"EmrManagedSlaveSecurityGroup\":\"${EMR_WORKER_SECURITY_GROUP}\", \ \"EmrManagedMasterSecurityGroup\":\"${EMR_WORKER_SECURITY_GROUP}\" \ }" \ --service-role ${EMR_SERVICE_ROLE} \ --log-uri s3n://${EMR_LOG_LOCATION} \ --instance-groups "[ \ {\"InstanceCount\":1,\"InstanceGroupType\":\"MASTER\",\"InstanceType\":\"${EMR_MASTER_INSTANCE_TYPE}\",\"Name\":\"Master - 1\"}, \ {\"InstanceCount\":3,\"InstanceGroupType\":\"CORE\",\"InstanceType\":\"${EMR_CORE_INSTANCE_TYPE}\",\"Name\":\"Core - 2\"} \ ]" ``` Bootstrap arguments are visible to anyone who can describe the cluster and are included in cluster logs. To protect sensitive information, such as METASTORE\_PASSWORD and LICENSE\_KEY, avoid passing them as arguments. Instead, retrieve them from AWS Secrets Manager within the script. Ensure that the cluster's instance profile has the `secretsmanager:GetSecretValue` permission. A DQ Job may fail with a `SocketTimeoutException: Connect timed out` error, even though YARN shows the application as SUCCEEDED. This occurs when a cluster node cannot connect to the data source. To resolve this, ensure the data source allows the cluster nodes' IP addresses. When troubleshooting, check the **Job Exception** column in Job Monitoring in addition to YARN. ### Configure DQ Agent Once the EMR cluster and the DQ Agent is deployed, it needs to be configured in DQ Web. {% stepper %} {% step %} Sign in to your web instance of Collibra DQ. {% endstep %} {% step %} Hover your cursor over the icon and click **Admin Console**.\ :chevrons-right: The Admin Console opens. {% endstep %} {% step %} Click the **Remote Agent** tile.\ :chevrons-right: The **Agent Management** page opens.\ :chevrons-right: In the following example, the newly created agent has a green indicator to show that it is healthy and active. ![Healthy DQ Agent example](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-cf094edac6deac3d515b714bb4024c74603fdf22%2Fdq-agent-management.png?alt=media) {% endstep %} {% step %} Click the pencil icon to the right of the Connections column.\ :chevrons-right: The **Edit Agent** modal appears. {% endstep %} {% step %} Set the **Master Default** field to **Yarn**. {% endstep %} {% step %} Set the **Default Deployment Mode** to **Cluster**. ![Edit Agent modal with Hadoop settings](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-85024f12705ff4d90715eb82dc40813c502347f3%2Fdq-edit-agent-hadoop.png?alt=media) {% endstep %} {% step %} Click **Submit**. {% endstep %} {% step %} On the Agent Management page, click the chain link icon next to the Connections column to map the DQ Agent to Connections.\ :chevrons-right: The **Edit Agent** modal appears. ![Agent to Connection Management example](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-96e2b2e731e217e1dfb615a80e8fd4f658893d64%2Fdq-agent-to-connection-management.png?alt=media) {% hint style="info" %} Any data sources that are not listed in the right hand pane are not visible to the agent. Refer to the [Agent](/data-quality-self-hosted/installation/agent.md) section for more details. {% endhint %} {% endstep %} {% step %} Click **Update**. {% endstep %} {% endstepper %} ### Run DQ Jobs You can now use EMR to run DQ Jobs on your data. {% stepper %} {% step %} In the DQ Web, open the Explorer page. {% endstep %} {% step %} Select a connection, schema, and table. {% endstep %} {% step %} Click **Create DQ Job**. {% endstep %} {% endstepper %} For more information on DQ Jobs, go to[Create a DQ job](/data-quality-self-hosted/core-concepts/explorer/create-a-dq-job.md).