> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/installation/installing-data-quality-observability-self-hosted-on-self-hosted-kubernetes/deploy-on-self-hosted-kubernetes/configure-spark-scratch-disk-space-to-handle-large-datasets.md).

# Configure Spark scratch disk space to handle large datasets

When you run large datasets, your Spark cluster's resources need to be sized appropriately to handle the computation of DQ rules. This section shows you how to configure on-demand Spark scratch disk space to handle huge Spark jobs.

## Prerequisites

Before you configure Spark scratch disk space, ensure you have:

* A Collibra DQ deployment on a cloud native platform, such as EKS, GKS, and so on.
* A Kubernetes service account for Collibra DQ that meets the permission requirements to provision dynamic Persistent Volume Claims (PVCs) in ReadWriteOnce (RWO).
* Execute permissions on the **/tmp** directory or the directory used for temporary storage, to ensure Spark jobs launch successfully.

{% hint style="info" %}
This capability is only available for Collibra DQ versions 2023.02 or newer.
{% endhint %}

## Steps

{% stepper %}
{% step %}
Open a terminal session.
{% endstep %}

{% step %}
Set the following values to deploy or update the Collibra DQ app with the provided Helm charts:

```
--set global.configMap.data.spark_scratch_type="persistentVolumeClaim" --set global.configMap.data.spark_scratch_storage_class="standard" --set global.configMap.data.spark_scratch_storage_size="20Gi" --set global.configMap.data.spark_scratch_local_path="/tmp/scratch"
```

{% hint style="info" %}
The values in the previous example are settings for GKE-based deployments.
{% endhint %}

{% hint style="warning" %}
If you use Snowflake key pair authentication for Pullup jobs, the Spark scratch local path and the private key staging directory must not be identical. The staging directory is `OWL_SCRATCH_DIR` if set, otherwise the JVM temporary directory, which defaults to `/tmp/scratch`. If both resolve to `/tmp/scratch`, Kubernetes rejects the Spark driver pod with the following error: `spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp/scratch": must be unique`To avoid this, set `spark_scratch_local_path` to a different value, for example `/tmp/spark-scratch`, by running a Helm upgrade against your existing Collibra DQ deployment: `helm upgrade ... --set global.configMap.data.spark_scratch_local_path="/tmp/spark-scratch"`After the upgrade completes, rerun the affected Pullup job and confirm that the Spark driver pod is created. For more information, go to [Connecting to Snowflake](/data-quality-self-hosted/connections/supported-connections/supported-jdbc-data-sources/connecting-to-snowflake.md).
{% endhint %}

1. If your Collibra DQ app is managed by an alternative Kubernetes package manager, ensure you update the owl-agent-configmap with the following variables:

   ```
   KUBERNETES_SCRATCH_TYPE: persistentVolumeClaim
   KUBERNETES_SCRATCH_STORAGE_CLASS: standard
   KUBERNETES_SCRATCH_STORAGE_SIZE: 20Gi
   KUBERNETES_SCRATCH_LOCAL_PATH: /tmp/scratch
   ```

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Pick the value of Storage Class, Storage Size and mouthPath specific to your requirements.</p></div>
2. Restart the DQ agent pod to complete the updates.
   {% endstep %}

{% step %}
Sign in to Collibra DQ.
{% endstep %}

{% step %}
Hover your cursor over the <img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-75245abf5c50ddf56f1d2d727ccaacf757d11145%2Fcogwheel-large%402x.png?alt=media" alt="" data-size="line"> icon and click **Admin Console**.\ <i class="fa-chevrons-right">:chevrons-right:</i> The Admin Console opens.
{% endstep %}

{% step %}
Click **Agent Configuration**.\ <i class="fa-chevrons-right">:chevrons-right:</i> The Agent Configuration page opens.
{% endstep %}

{% step %}
Click the **Actions**, then click <img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-5face7f577747d86367bf6698315172b6a3536f5%2FEditIcon.svg?alt=media" alt="Edit icon" data-size="line"> **Edit**.\ <i class="fa-chevrons-right">:chevrons-right:</i> The Edit Agent modal opens.
{% endstep %}

{% step %}
In the **Free form (Appended)** text box option, enter the following string value:

```
-conf spark.kubernetes.driver.podTemplateFile=local:///opt/owl/config/k8s-driver-template.yml,spark.kubernetes.executor.podTemplateFile=local:///opt/owl/config/k8s-executor-template.yml,spark.kubernetes.driver.reusePersistentVolumeClaim=false
```

{% hint style="success" %}
If you add any additional configuration properties based on your specific requirements, separate them with commas.
{% endhint %}
{% endstep %}

{% step %}
Optionally, increase driver pod memory. This may be needed for large datasets, if the driver pod is crashing or jobs end up in Unknown State. Use the following Spark command:

```
-conf spark.kubernetes.memoryOverheadFactor=0.4
```

{% endstep %}

{% step %}
Click **Submit**.
{% endstep %}

{% step %}
Run a sample DQ job to process DQ rules against a dataset of your choice. The corresponding Spark job dynamically provisions PVCs for the driver and each executor to handle large datasets by juggling between Memory and Disk at the mouth path of the PVC.
{% endstep %}
{% endstepper %}

### What's next

For more information, go to the official Spark documentation on:

* [Spark on Kubernetes using Pod Templates](https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template).
* [Spark on Kubernetes using Persistent Volume Claims](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes).