> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/installation/configuring-apache-hadoop-to-execute-dq-jobs.md).

# Configuring Apache Hadoop to execute DQ Jobs

For large-scale processing and concurrency, a single vertically scaled Spark server may not be sufficient. To address large-scale processing, Data Quality & Observability (self-hosted) can push compute jobs to an external Hadoop cluster. This section describes how to configure the DQ Agent to push DQ jobs to Hadoop.

{% hint style="warning" %}
**Dataproc (Managed Service for Apache Spark) Version 3.0 — Advisory** Dataproc 3.0 (Apache Spark 4.1) is not yet available or certified for production use. Customers may use it in non-production environments at their own risk, but we cannot provide full support until it is available.

* **Roadmap**: Official support for Amazon EMR 8 (Spark 4.x) is planned for the 2026.08 release.
* **Compatibility** Customers who want to test Dataproc 3.0 before it is available can use the 2026.08 DQ Spark cluster package, subject to the support limitations above.
* **Recommendation**: We recommend migrating to Cloud Native (Kubernetes) to avoid version compatibility issues in future releases, as we will continue upgrading Spark versions going forward.
  {% endhint %}

The following diagram shows the Data Quality & Observability (self-hosted) architecture with a Hadoop cluster:

![](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-1ab2a0fca8fe89ec32bc466cb58301fd966a3f24%2FScreenshot%202021-06-21%20at%209.05.39%20AM.png?alt=media)