> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/connections/supported-connections/supported-remote-files/connecting-to-network-file-storage-nfs.md).

# Connecting to Network File Storage (NFS)

This section contains an overview of Network File Storage (NFS).

## General information

<table><thead><tr><th width="152">Field</th><th>Description</th></tr></thead><tbody><tr><td>Data source</td><td>Network File Storage (NFS)</td></tr><tr><td>Supported versions</td><td>N/A</td></tr><tr><td>Connection string</td><td><code>nfs:///</code></td></tr><tr><td>Packaged?</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Certified?</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td colspan="2">Supported features</td></tr><tr><td>Analyze data</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Archive breaking records</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td></tr><tr><td>Estimate job</td><td><p><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Estimate job is only available for NFS connections on Standalone deployments of Collibra DQ.</p></div></td></tr><tr><td>Pushdown</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td></tr><tr><td colspan="2">Processing capabilities</td></tr><tr><td>Spark agent</td><td><p><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><ul><li>Spark agent is available for NFS connections on Standalone deployments of Collibra DQ. Additional configurations are required for Kubernetes and Hadoop deployments.</li><li><p>A Spark configuration must be added to the command line to mount the NFS directory path and run the NFS remote connections in Kubernetes. Refer to the <a href="/data-quality-self-hosted/core-concepts/explorer/create-a-dq-job/review.md">Review</a> page for more information on appending using the command line.</p><p>The following Spark configurations are related to mounting NFS volume in Kubernetes:</p><ul><li>spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs-pv-dq-dev.mount.path=/opt/owl/nfs-storage</li><li>spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs-pv-dq-dev.options.claimName=nfs-pvc-dq-dev</li><li>spark.kubernetes.executor.volumes.persistentVolumeClaim.nfs-pv-dq-dev.mount.path=/opt/owl/nfs-storage</li><li>spark.kubernetes.executor.volumes.persistentVolumeClaim.nfs-pv-dq-dev.options.claimName=nfs-pvc-dq-dev</li></ul></li></ul></div></td></tr><tr><td>Yarn agent</td><td><p><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Yarn agent is available for NFS connections on Standalone deployments of Collibra DQ. Additional configurations are required for Kubernetes and Hadoop deployments.</p></div></td></tr></tbody></table>

## Minimum user permissions

In order for Collibra DQ to access your local file system, you need the following permissions.

* Ensure that you can mount the NFS in your Collibra DQ service locally.
* Ensure that the Linux user has read permissions on the path that runs Collibra DQ services.

## Recommended and required connection properties

<table><thead><tr><th>Required</th><th>Connection Property</th><th>Type</th><th>Value</th></tr></thead><tbody><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td>Name</td><td>Text</td><td>The unique name used for your connection.</td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td>Connection URL</td><td>String</td><td><p>The connection string value of your NFS connection.</p><pre><code>nfs:///your/directory/path/
</code></pre></td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td><td>Target Agent</td><td>Text</td><td>The <a href="/data-quality-self-hosted/installation/agent.md">Agent</a> used to submit your DQ Job.</td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td>Auth Type</td><td>Option</td><td><p>The method to authenticate your connection.</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Auth Type is always <code>NFS</code>.</p></div></td></tr><tr><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-de45f34193ae06a2b093c2dfb8fb80cfda945166%2Fclose-large%402x.png?alt=media" alt="" data-size="line"> No</td><td>Driver Properties</td><td>String</td><td>The configurable driver properties for your connection. Multiple properties must be semicolon-delimited, and each property=value pair, including the last one, must end with a semicolon (;). <code>abc=123;test=true;</code></td></tr></tbody></table>

## Authentication

Auth Type must be set to **NFS**.

## Set up shared storage on self-hosted Kubernetes deployments

On self-hosted Kubernetes deployments, in order to successfully run DQ Jobs from an NFS connection type, volumes must be shared across all DQ pods (DQ Web, DQ jobs, and Livy). The following steps describe how to set up shared storage.

{% hint style="warning" %}
Make sure that Hadoop is not configured in the DQ deployment. Otherwise, the DQ job will refer to the HDFS location instead of the local filesystem location of the shared storage.
{% endhint %}

{% stepper %}
{% step %}
In AWS, provision EFS and set up the EFS CSI driver at the AWS EKS cluster where DQ is deployed. For more information, refer to the Amazon EKS document, [Store an elastic file system with Amazon EFS](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html).
{% endstep %}

{% step %}
Using static provisioning, create a persistent volume claim against the EFS storage class at the DQ deployed namespace. For more information, refer to the Amazon EKS document, [Store an elastic file system with Amazon EFS](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html).
{% endstep %}

{% step %}
Update the DQ Web and DQ Agent stateful sets to include the extra mount point based on the previously created PVC. For example:

```
---
volumes:
  - name: owldq-efs-storage
    persistentVolumeClaim:
    claimName: efs-claim
...
volumeMounts:
  - name: owldq-efs-storage
    mountPath: /opt/efs-storage
```

{% endstep %}

{% step %}
In DQ, create an NFS connection. In the Connection URL field, point to /opt/efs-storage location
{% endstep %}

{% step %}
Create the DQ dataset from a file from the above NFS connection.

Optionally, to place the dataset or files at the shared storage, have a temporary pod spin up to copy the files to that location. Refer to the following pod template:

```
---
apiVersion: v1
kind: Pod
metadata:
  name: temporary-copy
spec:
  containers:
    - name: temporary-copy
      image: nginx
      volumeMounts:
       - name: external
        mountPath: /opt/efs-storage
  volumes:
    - name: external
      persistentVolumeClaim:
        claimName: efs-claim
```

Once the above pod is deployed at the DQ namespace, copy the contents from local to the shared storage location using the following command:

```
k cp customer_transactions.csv temporary-copy:/opt/efs-storage/
```

{% endstep %}

{% step %}
Add the following extra configuration options to the owl command line. Use the following example for guidance:

{% hint style="info" %}
These options can be added against the Remote Agent freeform append configurations to set these values by default for the DQ datasets.
{% endhint %}

```
-numexecutors 1 -executormemory 1g -f "/opt/efs-storage/customer_transactions.csv" -h owlpostgres.chzid9w0hpyi.us-east-1.rds.amazonaws.com:5432/dev?currentSchema=validation -ds customer_transactions_csv_2 -master k8s:// -drivermemory 1g -deploymode cluster -rd "2023-10-11" -bhlb 10 -fullfile -loglevel INFO -cxn nfs-dataset 
-conf spark.kubernetes.driver.podTemplateFile=local:///opt/owl/config/k8s-driver-template.yml,spark.kubernetes.executor.podTemplateFile=local:///opt/owl/config/k8s-executor-template.yml,spark.kubernetes.executor.volumes.persistentVolumeClaim.nfs-efs-pv.options.claimName=nfs-efs-pv-claim,spark.kubernetes.executor.volumes.persistentVolumeClaim.nfs-efs-pv.options.sizeLimit=5Gi,spark.kubernetes.executor.volumes.persistentVolumeClaim.nfs-efs-pv.mount.path=/opt/efs-storage,spark.kubernetes.executor.volumes.persistentVolumeClaim.nfs-efs-pv.mount.readOnly=false,spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs-efs-pv.options.claimName=nfs-efs-pv-claim,spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs-efs-pv.options.sizeLimit=5Gi,spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs-efs-pv.mount.path=/opt/efs-storage,spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs-efs-pv.mount.readOnly=false
```

{% endstep %}
{% endstepper %}