Atomic Testing

Atomic Testing is a way of testing detections in a more repeatable way, similar to how software benefits from unit testing to ensure it still behaves as expected. Detection rulesets can be vulnerable to changes in their underlying technologies, so they benefit from a repeatable testing framework to catch changes in detection efficacy and ensure they still perform as expected against malicious TTPs (tactics, techniques and procedures).

GitLab uses forks of the TTPForge and ForgeArmory frameworks and have customised the collection of provided TTPs specific to our unique tech stack.

For more information, see this epic and the Atomic Testing trial report.

Atomic Testing Process

Get ready

1. Red Team: select a TTP from GitLab’s ForgeArmory fork or develop a new one. SIRT can be involved with the selection since they are familiar with the detection suite, or the Threat Intelligence team who can provide TTPs that are relevant to current threats.
2. Red Team: Do a TTP test run to ensure it works in correct environment. Spin up a standalone VM running the relevant OS and without the detection agent. This allows testing without creating unnecessary alerts. Clone and set up TTPForge and ForgeArmory from our forks and install any dependencies.
3. Red/Blue team: Create a new issue for testing the TTP under this epic
4. Red Team: Ask a SIRT member who can analyse the logs (for example, who is not on call that week)
5. Red/Blue team: Agree on a time (synchronously works best initially if possible, but not required)

Run the TTP

1. Red Team: Notify SIRT in the #security-operations channel of the selected TTP (with a link to the yml file) and the machine it’s being run on, so that an incident isn’t raised.
2. Red Team: Run the TTP live. On a machine or VM with the detection agent installed, clone TTPForge and ForgeArmory from our forks, install dependencies. Run the TTP and immediately notify the Blue Team member to find and analyse logs.
3. Red Team: Ensure the issue created in point 3 above is updated with any commands run and output generated. Use <details> and <summary> to collapse long output sections.

Analysis and recommendation

1. Blue Team: Analyse logs and identify improvements to detections
2. Red Team: If there is a potential improvement (even if not eventually adopted), create a recommendation issue with ~RTRec::Detection and under this epic. Here’s an example.
3. Blue Team: Create detection improvement MR (if applicable) and link in recommendation issue, and apply appropriate SIRT labels.
4. Red/Blue Team: Close issue with appropriate RecOutcome:: label

Repeat as needed!