> For the complete documentation index, see [llms.txt](https://docs.collibra.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.collibra.com/data-quality-self-hosted/apis/using-notebooks-with-data-quality-observability-classic/rules-in-notebooks.md).

# Rules in notebooks

This section provides an example of adding rules in a notebook as well as information on creating rules in Scala.

## Add rules

The following example demonstrates how to add a rule in a notebook. In this example, we were provided a file named `atm_cust_file` and want to load it into a database table as well as scan it for all possible errors. We want to provide the following levels of protection:

* Set a business rule that checks if a customer joined before the company was founded.
* Check if the file 100% matches to the DataFrame or db table we've created.
* Check for all possible outliers or anomalies in the dataset.

Each of these issues impacts the business differently and causes a different flow to trigger in our pipeline.

Let's create a simple rule and assign points to the overall scoring system for later delegation.

```
    val rule = new Rule
    rule.setRuleNm("customer_before_company")
    rule.setRuleValue("customer_since_date < '1956-11-01'")
    rule.setPerc(1.0)
    rule.setPoints(1)
    rule.setIsActive(1)
    rule.setUserNm("User")
    rule.setDataset("ATM_CUSTOMER3")
    OwlUtils.addRule (rule)
```

Now let's chain together the remaining two items that were part of our original requirement.

```
val df_source = (spark.read
.format ("csv").option("header", true).option("delimiter", ",")
.load("atmCustFile.csv")
)
val dataset = "atm_customer"
var date = "2018-01-16"
 
val optSource = new SourceOpt
optSource.on = true
optSource.dataset = dataset
optSource.validateValues = true
optSource.key = Array("name")
opt.setSource(optSource)
//Dataframe for the table created from the file
 
val connProps = Map (
"driver"    -> "org.postgresql.Driver",
"user"        -> "postgres",
"password"    -> "xxx",
"url"        -> "jdbc:postgresql://xxx:xxx/postgres",
"dbtable"    -> "public.atmCustomer" )
val df_target = spark.read.format("jdbc").options(connProps).load
val cdq = OwlUtils.OwlContext(df_source, df_target, opt)
.register (opt)
cdq.owlCheck
// first register with catalog if not registered
cdq.register (props)
// Check if dataframe matches the source file 'atm_cust_file'
val breakCountDF = cdq.getSourceBreaksCount()
breakCountDf.show()
if (breakCountDf.count() > 1) {
  // create service now ticket and exit with fail based on not matching to original file
}
val ruleBreaks = cdq1.getRuleBreakRows()
if (ruleBreaks.count() > 1) {
  if (ruleBreaks.where($"score" > 5).count > 1) {
    //create service now ticket and exit with fail based on rules
  }
}
val outliers = cdq.getOutliers()
if (outliers.where($"confidence" < 10).count > 3) {
  // CDQ email Alert to business group for attention
  // where 3 outliers have a confidence below 10
}
```

## Work with rules in Scala

In this section, you can learn how to work with rules in notebooks written in Scala.

### Instantiation

<table><thead><tr><th width="152">Code</th><th>Description</th></tr></thead><tbody><tr><td>val = new Rule()</td><td>Instantiating a Rule object</td></tr><tr><td>rule.setDataset(&#x3C;DATASET>)</td><td>Adding the name of the dataset</td></tr><tr><td>rule.setRuleNm(&#x3C;RULE_NAME>)</td><td>Adding the name of the given rule</td></tr><tr><td>rule.setRuleValue(&#x3C;RULE_EXPRESSION>)</td><td>Setting the simple RULE_EXPRESSION</td></tr><tr><td>rule.setRuleType(&#x3C;RULE_TYPE>)</td><td>Setting the rule type</td></tr><tr><td>rule.setPerc(&#x3C;RULE_PERCENTAGE>)</td><td>Setting the percentage</td></tr><tr><td>rule.setPoints(&#x3C;RULE_POINT>)</td><td>Setting how many points will be deducted from total for each percentage</td></tr><tr><td>rule.setIsActive(&#x3C;RULE_IS_ACTIVE>)</td><td><p>Making rule active/inactive</p><p>Possible values:</p><p>ACTIVE: 1 / true</p><p>INACTive: 0 / false</p></td></tr><tr><td>rule.setUserNm(&#x3C;RULE_OWNER_USER_NAME>)</td><td>Adding the owner</td></tr></tbody></table>

### Rule Types

### Requirements

#### Required imports

```
import com.owl.core.Owl
import com.owl.core.util.OwlUtils
import com.owl.common.options.OwlOptions
import com.owl.common.{Props, Utils}
 
import com.owl.common.domain2.Rule
 
import org.apache.spark.sql.funtions._
```

#### SparkSession initialization

Databricks and other notebook execution frameworks are working with managed/shared Spark sessions, therefore, Collibra DQ recommend using this code snippet in your notebook to initialize the current Spark session properly.

```
// ----- Init Spark ----- //
def sparkInit(): Unit = {
  sparkSession = SparkSession.builder
    .master("local")
    .appName("test")
    .getOrCreate()
}
```

{% hint style="danger" %}
Do not call **spark.stop** at any of your notebooks. Otherwise, the execution engine will exit immediately.
{% endhint %}

{% hint style="warning" %}
Make sure OwlContext is already created before using any method from OwlUtils.
{% endhint %}