> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/installation/agent/upgrade-spark.md). # Upgrade Spark {% hint style="warning" %} Data Quality & Observability (self-hosted) 2026.07 and newer require Java 21 and Spark 4.1.0. Data Quality & Observability (self-hosted) 2026.02 through 2026.06 require Java 17 and Spark 4.1.0. For Data Quality & Observability (self-hosted) versions 2025.08 through 2026.01, you must upgrade to Java 17 and Spark 3.5.6. For more information about version compatibility, click "Java and Spark compatibility matrix" below.

Java and Spark compatibility matrix

| Data Quality & Observability (self-hosted) version | Java 11 | Java 17 | Java 21 | Spark versions | Additional notes | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------- | | 2025.08 |

No |

Yes |

No |

Yes |

No |

Yes |

No |

Yes | 4.1.0 only | |

{% endhint %} This section details how to upgrade on-premises Apache Spark versions. ## Steps {% hint style="info" %} The below steps are intended to reflect examples of what works in a simple Standalone environment. You may need to modify them to accommodate your specific deployment. {% endhint %} {% stepper %} {% step %} Run the following commands to stop Spark and Collibra DQ services. 1. Set the **OWL\_HOME** variable without a trailing slash after the value. ``` export OWL_HOME=\ ``` 2. Stop Spark Master. ``` cd $OWL_HOME/spark/sbin ./stop-master.sh ``` 3. Stop Spark Worker. ``` cd $OWL_HOME/spark/sbin ./stop-worker.sh ``` 4. Stop DQ Web. ``` cd $OWL_HOME/bin ./owlmanage.sh stop=owlweb ``` 5. Stop DQ Agent. ``` cd $OWL_HOME/bin ./owlmanage.sh stop=owlagent ``` 6. Verify that all processes are stopped. ``` ps -ef | grep owl ## No DQ processes should return as running. ## ```

If any DQ processes are still running, you can use the command kill -9 <pid from the ps command in Step 1f> to stop them manually.

{% endstep %} {% step %} Create a package installation folder one level up from **OWL\_HOME** and set the environment variables accordingly. 1. Move up one level from **OWL\_HOME**. ``` cd $OWL_HOME/.. ``` 2. Export the **INSTALLER\_DIRECTORY** variable. ``` export INSTALLER_DIRECTORY=$OWL_HOME/dq_install ``` 3. Create the **INSTALLER\_DIRECTORY** folder. ``` mkdir $INSTALLER_DIRECTORY ``` {% endstep %} {% step %} Create a backup folder. 1. Open the **INSTALLER\_DIRECTORY** folder. ``` cd $INSTALLER_DIRECTORY ``` 2. Create a backup folder. ``` mkdir backup ``` {% endstep %} {% step %} Download the Collibra DQ installer packaged with the necessary Spark version from the [Data Quality & Observability (self-hosted) Downloads page](https://productresources.collibra.com/downloads/#data-quality) in the Product Resource Center. {% endstep %} {% step %} Run the following command to extract the installer. ``` tar -xvf ``` {% endstep %} {% step %} Create backups of the following directories and files by moving or copying them to the backup folder. * spark * spark-extras (if it exists) * drivers * dq-core-\*-dependencies.jar * dq-webapp\*.jar * dq-agent\*.jar * owlcheck * owlmanage.sh {% endstep %} {% step %} Remove the folders and files using the following or similar steps. 1. Open the **OWL\_HOME/bin** folder. ``` cd $OWL_HOME/bin ``` 2. Move the dq-core-\*-dependencies.jar, dq-webapp\*.jar, dq-agent\*.jar, owlcheck, and owlmanage.sh files to the **INSTALLER\_DIRECTORY/backup/** folder. ``` mv dq*.jar owl* $INSTALLER_DIRECTORY/backup/ ``` 3. Move up one folder level. ``` cd .. ``` 4. Move drivers, spark, and spark-extras to the **INSTALLER\_DIRECTORY/backup/** folder. ``` mv drivers spark spark-extras $INSTALLER_DIRECTORY/backup/ ``` {% endstep %} {% step %} Confirm that the files and folders are in the backup folder. ``` ls -ltr $INSTALLER_DIRECTORY/backup/ ``` {% endstep %} {% step %} Run the following command to go to the **packages/install-packages** folder. ``` cd $INSTALLER_DIRECTORY/packages/install-packages ``` {% tabs %} {% tab title="Example" icon="glasses-round" %} You can use the command `tar -xvf spark-4.1.0-bin-hadoop3.tgz` to extract Spark 4.1.0. {% endtab %} {% endtabs %} {% endstep %} {% step %} Extract the Spark installation files. ``` tar -xvf spark-4.1.0-bin-hadoop3.tgz ``` {% endstep %} {% step %} Extract the **spark-extras.tar.gz** file. ``` tar -xvf spark-extras.tar.gz ``` {% endstep %} {% step %} Rename and move the **spark-4.1.0-bin-hadoop3** file from the installer extract folder to the **OWL\_HOME/spark** folder. 1. Rename the **spark-4.1.0-bin-hadoop3** file **spark**. ``` mv spark-4.1.0-bin-hadoop3 spark ``` 2. Move **spark** to the home directory. ``` mv spark $OWL_HOME ``` {% endstep %} {% step %} Run the following command to copy the **spark-extras** folder into **spark/jars**. ``` cp spark-extras/* $OWL_HOME/spark/jars ``` {% endstep %} {% step %} Restore your previous **spark-env.sh** from the backup and verify the cleanup configuration. 1. Copy your previous **spark-env.sh** from the backup to the new Spark installation. ``` cp $INSTALLER_DIRECTORY/backup/spark/conf/spark-env.sh $OWL_HOME/spark/conf/spark-env.sh ``` 2. Verify that the following line is present in **spark-env.sh**. If it is not present, add it to automate the cleanup of Spark work folders and avoid filling up disk space. ``` export SPARK_WORKER_OPTS="${SPARK_WORKER_OPTS} -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=1800 -Dspark.worker.cleanup.appDataTtl=3600" ```

If you do not have a previous spark-env.sh (for example, this is your first Spark upgrade), copy it from the template file instead:

cp $OWL_HOME/spark/conf/spark-env.sh.template $OWL_HOME/spark/conf/spark-env.sh

Then add the cleanup line above.

{% endstep %} {% step %} Copy in the Data Quality & Observability (self-hosted) application files. 1. Open the **INSTALLER\_DIRECTORY**. ``` cd $INSTALLER_DIRECTORY ``` 2. Copy in the dq-core-\*-dependencies.jar, dq-webapp\*.jar, and dq-agent\*.jar files. ``` cp dq-*.jar $OWL_HOME/bin ``` 3. Copy in the owlcheck and owlmanage.sh files. ``` cp owl* $OWL_HOME/bin ``` 4. Make the owlcheck and owlmanage.sh files executable. ``` chmod +x $OWL_HOME/bin/owl* ``` {% endstep %} {% step %} Extract and copy in the updated drivers. 1. Create a **drivers** folder. ``` mkdir drivers ``` 2. Open the **drivers** folder. ``` cd drivers ``` 3. Extract **drivers.tar.gz** into the **drivers** folder. ``` tar -xvf ../drivers.tar.gz ``` 4. Move up one folder level. ``` cd .. ``` 5. Move the **drivers/** folder to the application home directory. ``` mv drivers/ $OWL_HOME ``` {% endstep %} {% step %} Run the following commands to start Spark and Collibra DQ services. 1. Start Spark Master. ``` cd $OWL_HOME/spark/sbin ./start-master.sh ``` 2. Start Spark Worker. ``` cd $OWL_HOME/spark/sbin ./start-worker.sh spark://$(hostname -f):7077 ``` 3. Start DQ Web. ``` cd $OWL_HOME/bin ./owlmanage.sh start=owlweb ``` 4. Tail the log file until you see a "Bootstrap process complete" message. ``` tail -f $OWL_HOME/log/owl-web.log ``` 5. Start DQ Agent. ``` cd $OWL_HOME/bin ./owlmanage.sh start=owlagent ``` {% endstep %} {% step %} Visit the Spark page on port 8080 to confirm that the Spark version is the one you installed and the worker is alive. {% endstep %} {% step %} [Create a test DQ job](/data-quality-self-hosted/core-concepts/explorer.md) without any optional DQ layers or rules to verify that the Spark driver, executors, and containers are able to launch successfully. {% endstep %} {% endstepper %}