> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/apis/using-notebooks-with-data-quality-observability-classic/options-api.md). # Options API ## Field mappings | Field name | CLI prop | Description | | -------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **dataset** | *ds* | dataset name, example: userTable or users or user\_file | | **rundId** | *rd* | run date, must be in format **yyyy-MM-dd** or for incremental use Hours or Minutes **yyyy-MM-dd HH:mm** | | **rundIdEnd** | *rdEnd* | end date for query ranges **t\_date >= $ and t\_date < $**, must be in format **yyyy-MM-dd** or for incremental use Hours or Minutes **yyyy-MM-dd HH:mm** | | **passFail** | *passfaillimit* | Limit for passing or failing runs | | **jobId** | | | | **onReadOnly** | *readonly* | Do not persist results to the DQ metastore - useful during testing. | ## Load Options | Field name | CLI prop | Description | | ------------------- | ---------- | ------------------------------------------------------- | | **fullFile** | *fullfile* | use entire file for lookbacks instead of just filequery | | **fileQuery** | | | | **header** | | | | **headerSrc** | | | | **hasHeader** | | | | **isParallel** | | | | **isJson** | | | | **isMixedJson** | | | | **isMapsJson** | | | | **flatten** | | | | **isMultiLine** | | | | **sparkprinc** | | | | **sparkkeytab** | | | | **jdbcprinc** | | | | **jdbckeytab** | | | | **srcpwdmgr** | | | | **pwdmgr** | | | | **pguser** | pguser | | | **pgpassword** | pgpassword | | | **pghost** | host | | | **pgport** | port | | | **executorcores** | | | | **isParquet** | | | | **isAvro** | | | | **avroSchema** | | | | **isXml** | | | | **xmlRowTag** | | | | **isOrc** | | | | **dateFmt** | | | | **timeFmt** | | | | **datasetSafety** | | | | **filePath** | | | | **delimiter** | | | | **fileLookBack** | | | | **dbLookBack** | | | | **connectionURL** | | | | **userName** | | | | **password** | | | | **sqlQuery** | | | | **connectionProps** | | | | zkHost | \~\~\~\~ | ***Deprecated*** | | zkPort | \~\~\~\~ | ***Deprecated*** | | zkPath | \~\~\~\~ | ***Deprecated*** | ## Outlier Options | Field name | CLI prop | Description | | ---------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **on** | dl | Deep learning. | | **lookback** | dllb |

A deep learning lookback. For example, a value 5 for a lookback of 5 days.

Lookback periods directly correlate to -dllb.

| | **key** | dlkey | A comma-delimited deep learning key. comma delim key ex: symbol,date | | **dateField** | | | | **bin** | | | | **includes** | dlinc | A deep learning col limit. For example, open,close,high,volume. | | **excludes** | dlexc | A deep learning col exclusion. For example, open,close,high,volume. | | **categorical** | | | | **by** | | | | **limit** | | | | **historyLimit** | | | | **minhistory** | dlminhist |

An automatically generated flag based on the outlier lookback setting -dllb that defines the minimum number of days before DQ flags data as potential outliers. -dlminhistensures that the number of days in the algorithm is relative to the total scope of the lookback period.

It is not recommended that you override this flag from the command line.

Lookback periods do not directly correlate to -dlminhist.

| | **score** | | | ## FPG Options | Field name | CLI prop | Description | | -------------- | ------------- | --------------------------------------------------------------------- | | **on** | fpgon | pattern mining | | **lookback** | fpglb | lookback interval for pattern mining. Ex: **-fpglb 5** | | **key** | fpgkey | natural key for pattern mining activity | | **dateField** | fpgdc | date column for pattern mining. Ex: **-fpgdc date\_col** | | **lowFreq** | | ***Deprecated*** | | **includes** | fpginc | pattern mining is expensive use this input to limit the observed cols | | **excludes** | fpgexc | pattern mining is expensive use this input to limit the observed cols | | **timeBin** | fpgtbin | time bin for pattern mining. Ex: **-fpgtbin DAY** | | **score** | fpgscore | score for pattern mining records | | **minSupport** | fpgsupport | | | **confidence** | fpgconfidence | | ## Dupe Options | Field name | CLI prop | Description | | --------------- | ---------- | ----------------------------------------------------- | | **on** | dupe | duplicate record detection | | **includes** | dupeinc | duplicate record detection, column inclusion list | | **excludes** | dupeexc | duplicate record detection, column exclusion list | | **depth** | | | | **lowerBound** | dupelb | duplicate lower bounds on percent match default \[85] | | **upperBound** | | | | **blocksize** | | | | **useCache** | | | | **checkHeader** | | | | **exactMatch** | | | | **ignoreCase** | dupenocase | duplicate record detection, column exclusion list | | **score** | dupescore | | | **limit** | dupelimit | Limit for dupe rows stored | ## Profile Options | Field name | CLI prop | Description | | -------------------- | -------- | ----------- | | **on** | | | | **includes** | | | | **excludes** | | | | **dataShapeOn** | | | | **statsOn** | | | | **correlationOn** | | | | **histogramOn** | | | | **cardinalityOn** | | | | **dataShapeColsInc** | | | | **dataShapeColsExc** | | | ## Source Options | Field name | CLI prop | Description | | ---------------------- | -------- | ----------- | | **on** | | | | **includes** | | | | **excludes** | | | | **key** | | | | **fileQuery** | | | | **map** | | | | **score** | | | | **datasetSrc** | | | | **driverSrc** | | | | **userNameSrc** | | | | **passwordSrc** | | | | **connectionURLSrc** | | | | **sqlQuerySrc** | | | | **connectionPropsSrc** | | | ## Rule Options | Field name | CLI prop | Description | | ------------- | -------- | ----------- | | **on** | | | | **rulesOnly** | | | | **semantic** | | | ## ColMatch Options | Field name | CLI prop | Description | | ----------------------------- | -------- | ----------- | | **colMatchParallelProcesses** | | | | **colMatchDurationMins** | | | | **colMatchBatchSize** | | | | **connectionList** | | | ## Spark Options | Field name | CLI prop | Description | | ------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **numExecutors** | | The number of worker processes responsible for running individual tasks, storing cached data, and handling shuffle operations. The attribute is not cleared even if the API request explicitly tries to update it to be empty. | | **executorMemory** | | | | **driverMemory** | | | | **executorCores** | | The number of CPU cores assigned to each executor process, determining the number of tasks it can run concurrently. The attribute is not cleared even if the API request explicitly tries to update it to be empty. | | **master** | | | | **jars** | | | | **libs** | | | | **driver** | | | ## Misc Options | Field name | CLI prop | Description | | ------------- | -------- | ----------- | | **obslimit** | | | | **nullValue** | | |