---
# SPDX-FileCopyrightText: the secureCodeBox authors
#
# SPDX-License-Identifier: Apache-2.0

title: "ADR-0010: Custom Inheritance Behavior for Affinity and Tolerations"
sidebar_label: "ADR-0010"
---
# ADR-0010: Custom Inheritance Behavior for Affinity and Tolerations

| <!-- -->       | <!-- --> |
|----------------|----------|
| **Status**:    | ACCEPTED |
| **Date**:      | 2021-11-22 |
| **Author(s)**: | Max Maass [max.maass@iteratec.com](mailto:max.maass@iteratec.com) |

## Context

Kubernetes-based cloud environments allow controlling which nodes are used for which workloads through two mechanisms: [Affinity][affinity] and [Tolerations][tolerations].
These can be selected by setting them in the Kubernetes job.

### Problem 1: Sources of Configuration

The secureCodeBox operator is managing Kubernetes jobs for us, and templating their values from the provided `Scan` (for a concrete scan), `ScanType` (for a type of scan, e.g. nmap), `ParseDefinition` (for the parser related to a scan), or `ScanCompletionHook` (for a hook) into the job itself.
Since they each pull information from different places, supporting affinity and tolerations for all of them requires adding them in multiple locations.
Generally, such settings are pulled into the job from one of two (or in some cases three) places:
- The helm values (configured via the values.yaml during install of the `ScanType`, `ParseDefinition` or `ScanCompletionHook`)
- The `Scan` specification for the running scan
- The `CascadingScan` specification when creating a cascaded scan (or, more precisely: both the `Scan` spec of the parent scan and whatever information is given in the `ScanSpec` of the `CascadingScan`)

Specified as a table, this is where values for the different jobs scheduled by the operator normally come from:

| Job type | Helm values | `Scan.spec` | `CascadingScan.spec.scanSpec` |
|----------|-------------|-------------|-------------------------------|
| Scan     | ✅          | ✅           | ✅                            |
| Parse    | ✅          |              |                              |
| Hook     | ✅          |              |                              |

This presents us with a problem: All three job types should be configurable with an affinity and tolerations, but two of them only read their relevant configuration from the helm values (provided during install).
This makes it impossible to ensure that all jobs triggered by a single `Scan` use a specific affinity or toleration (which may be different from the default).
We can address this issue in two ways:

#### Option 1: Accept And Move On
One option is to accept that this is the case and leave it unchanged.
This is unsatifactory, as affinity and tolerations are a powerful tool, not only for controlling the cost of cloud deployments (by using cheaper node types, like preemptible nodes), but also for other aspects like controlling the geographic location of nodes, the presence of special node features, and other aspects.
In some cases, there may be no one valid default value for a single `ScanType` that is correct for all `Scan`s using it.
Additionally, for other features that can have defaults set by the helm values, it is possible to add to these defaults using fields in the `Scan` definition.

#### Option 2: Use Affinity and Tolerations from Scan in all Jobs
The other option is to deviate from the usual method of setting values for the jobs by making all three types of jobs (scans, parsers and hooks) use the affinity and tolerations defined in the `Scan` (_if_ it defines them), and fall back to the defaults from the Helm values otherwise.
This allows the user to specify affinity and tolerations in one place (the `Scan`), and be confident that any jobs started by the `Scan` will use the same affinity and toleration settings.
The downside is that now, values set in the `Scan` will influence the execution of parsers and even hooks, which is different from the behavior of the system in other places, where settings on a `Scan` will not impact the hooks and parsers.

To summarize, this would make the table look as follows:

| Job type | Helm values | `Scan.spec` | `CascadingScan.spec.scanSpec` |
|----------|-------------|-------------|-------------------------------|
| Scan     | ✅          | ✅           | ✅                             |
| Parse    | ✅          | ✅            |                              |
| Hook     | ✅          | ✅            |                              |

(There are no checkmarks on the `CascadingScan.spec.scanSpec` column for parser and hook because it is merged into the `Scan.spec` when creating the cascaded scan. From there, it will influence the parser and hook the same way it would if it had been directly added to the parent `Scan`.)

### Problem 2: Merging vs. Replacing Defaults
Normally, defaults set in the helm values are merged with any additional values provided in the `Scan`, and the same merging behavior governs combining the `Scan.spec` of the triggering scan with the `CascadingScan.spec.scanSpec` of a cascading scan (assuming inheritance is enabled).
However, since the [affinity is defined as a deeply nested dictionary][affinity], merging is both technically challenging and may lead to unexpected results.
In the worst case, it can lead to an invalid configuration, or one that is impossible to schedule because of conflicting requirements.
It is thus advisable to replace any default affinity with one that is specified in the `Scan.spec` (for Helm values) or the `CascadingScan.spec.scanSpec` (for cascaded scans), instead of attempting to merge them.
However, this raises the question of how to handle tolerations.

#### Option 1: Consistency With Other Values
One option is to have it behave in the same way it is done for labels, environmental variables, etc.: merging the default with the values provided in the `Scan`.
The downside of this approach is that it is purely "additive": It is never possible to create a set of tolerations that does not include the default.
Additionally, it is inconsistent with the behavior of the affinity, which is the most closely related feature.

### Option 2: Consistency with Affinity
The alternative is to replace instead of merge for the tolerations as well.
This is inconsistent with the other values, but it ensures that affinity and tolerations behave the same.

## Decision
For the first problem, we choose to use Option 2: Affinity and tolerations defined in the `Scan` will be used by all jobs related to this scan.
They will also be inherited by default, although a special cascading scan flag can be used to disable this (`inheritAffinity` / `inheritTolerations`, as per the standard naming scheme).
If no values are defined in the Scan, the default values from the Helm install are used.

For the second problem, we choose Option 2 as well: Both affinity and tolerations will have the more specific value (`Scan.spec` for scans, `CascadingScan.spec.scanSpec` for cascaded scans) replace the more general value (Helm values for scans, `Scan.spec` of the parent scan for cascaded scans), assuming the more specific value is set.
If the value is not set (i.e., the key is omitted in the configuration), the more general value is used.
If the more specific key is set to an empty value (`[]` for tolerations, `{}` for affinity), it will still replace the more general value and thus remove any affinity or toleration that was previously configured.

## Consequences
This decision leads to a system that is less concerned with consistency in behavior between unrelated features (e.g., tolerations and environmental variables), and more interested in consistency between related features (affinity and tolerations) and convenience for the operator (changing the scheduling behavior of parsers and hooks from the scan definition).

[affinity]:    https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/
[tolerations]: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
