Sleep Soundly: Reliable Alerting with Unit Testing in Prometheus

You are in on-call rotation, and you do not want to be woken up at 3am in the morning due to a false alarm. The day before you just deployed a new alert rule in production, and you are now sleeping with an eye open praying for the rule to work as expected and to not to fire for something that was not intended to. Well, there is a solution for this, and it is unit testing. Prometheus has a built-in testing mechanism for alerts. If you learn to use it, you will always be sure in which cases the alert would fire and in which cases it should not. Furthermore, you can integrate this unit testing in your CI to make sure that all alerts meet the proper criteria, and if something changes in the alert rule, the test should fail. This is the fastest and painless feedback cycle that does not involve waking you up in the middle of the night. Unit Testing in Prometheus can be tricky and a bit confusing at first. This talk is intended to give an overview and tips and tricks on how to code reliable unit tests for your alerts.