> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/core-concepts/explorer/create-a-dq-job/sizing/job-size-estimator-configuration.md).

# Job size estimator configuration

This page illustrates for advanced client-side DQ admin users how to manually calculate Spark configurations to estimate job size.

<table><thead><tr><th>Spark Configuration</th><th>Command Line Option</th><th>Example</th></tr></thead><tbody><tr><td>Memory</td><td><code>-executormemory</code></td><td><p>To determine executor memory, you first need to determine your dataset size. To find the size of your dataset, calculate the following using inputs from your environment:</p><pre><code>(number of rows * number of columns * 10) / 10243
</code></pre><p>Next, determine the upper bound of maxworkermemory parameter on the Admin Limits page in the Admin Console:</p><pre><code>maxworkermemory / 2
</code></pre><p>Lastly, set up a recursive function by dividing the dataset size by 2 until it reaches the new upper bound limit (maxworkermemory / 2).</p><p>If the final value exceeds the upper bound, set Memory to the upper bound value. If the value is less than the upper bound, set Memory to the minworkermemory value.</p></td></tr><tr><td>No. of Cores</td><td><code>-executorcores</code></td><td><pre><code>(maxworkercores / maxworkermemory) * executormemory = executorcores (rounded up)
</code></pre></td></tr><tr><td>Partition Number</td><td><code>-partitionnum</code></td><td><pre><code>dataset size / 128MB = partitionnum
</code></pre><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>128MB refers to the Spark SQL default data frame partition size.</p></div><div data-gb-custom-block data-tag="hint" data-style="success" class="hint hint-success"><p>If the value exceeds 200 based on the results of the calculation, set the Partition Number to 200.</p></div></td></tr><tr><td>No. of Executors</td><td><code>-numexecutors</code></td><td><pre><code>partitionnum / (pergbpartitions*executormemory) = numexecutors
</code></pre><div data-gb-custom-block data-tag="hint" data-style="danger" class="hint hint-danger"><p>The number of executors should be considered a maximum limit and should never exceed 80% of the agent capacity.</p></div></td></tr></tbody></table>