> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/admin-configuration/more-admin-settings/livy/working-with-livy.md).

# Working with Livy

## What is Livy?

Apache Livy is a RESTful web service that serves as a session manager for Spark jobs on Kubernetes and Hadoop deployments of Data Quality & Observability (self-hosted). Livy creates temporary Spark pod sessions (Spark Contexts) to efficiently run jobs in the Spark cluster. Data Quality & Observability (self-hosted) uses Livy to help estimate the number of resources it needs to process and preview jobs that run against datasets created from remote file connections, such as S3 and ADLS. Without Livy, you cannot preview the data that you bring in from remote files.

Additionally, when you create rules on the [Rule Workbench](/data-quality-self-hosted/core-concepts/dq-rules/rule-builder/working-with-the-rule-workbench/create-a-data-quality-rule.md), Livy's ability to read and transform large datasets allows you to use the Result Preview component for a convenient sample of available columns in your dataset. With a preview of the columns in your dataset next to the rule builder, Livy enables you to more efficiently write SQL business rules without having to leave the page to reference your dataset.

{% hint style="info" %}
Livy is included as an optional component in most Collibra DQ release packages. Contact your Collibra Account Team to help you choose a release package with Livy.
{% endhint %}

## Livy architecture within Collibra DQ

![livy architecture in dq](https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-68caf960b9d7e3b6cb228415eae540b3dcaf940f%2Fdq-livy-architecture.png?alt=media)

{% stepper %}
{% step %}
Data Quality & Observability (self-hosted) submits a Spark job to the Livy server.
{% endstep %}

{% step %}
Livy:

* Examines the data to determine the optimal number of resources it needs to efficiently process the job.
* Creates a Spark context to communicate with the Spark runtime and coordinate the execution of the job.

  <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Kubernetes provides the compute space for the ephemeral Spark containers that comprise the Spark session needed to run, but there is not a Spark Cluster when running on Kubernetes.</p></div>

{% endstep %}

{% step %}
Spark workers execute the job inside the cluster.
{% endstep %}

{% step %}
The results return to Data Quality & Observability (self-hosted).
{% endstep %}
{% endstepper %}

## Supported deployment types

Livy is currently supported on Kubernetes deployments of Data Quality & Observability (self-hosted).

{% hint style="warning" %}
Livy is not currently supported on Standalone or Hadoop deployments.
{% endhint %}

## Supported remote file data sources

The following table shows the remote file connections that support Livy.

<table><thead><tr><th>Connection</th><th>Supported</th><th>Estimate Job</th></tr></thead><tbody><tr><td>Amazon S3</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>HDFS</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Google Cloud Storage</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Azure Data Lake Storage</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Azure Blob Storage</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td colspan="3">Network File Storage</td></tr><tr><td>CSV</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>ORC</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Parquet</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Avro</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>JSON</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>XML</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Delta</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr><tr><td>Hudi</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td><td><img src="https://4121921244-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWCrP7Albs8DfJjIQxUxw%2Fuploads%2Fgit-blob-d06dd40622fc7d030e21f7c0303a90eb28d229fc%2Fok-large%402x.png?alt=media" alt="" data-size="line"> Yes</td></tr></tbody></table>