Konubinix' opinionated web of thoughts

Manual Progressive Chaos Monkey


In case you want to run some chaos monkey but don’t feel ready yet.

  1. plan to disturb the system manually once,
  2. then make the system robust against that disturbance,
  3. then plan it another time,
  4. and again
  5. and again
  6. at some point you will want to automatize this test => automatize it
  7. then think about another disturbance and plan it,
  8. and again
  9. and again
  10. automatize it
  11. rinse and repeat

When you think you bullied the system enough, may be you are ready for actually chaos monkey.

In the spirit of the voluntary discomfort, by willingly adjusting to voluntary disturbances, you will get more ready when the discomfort actually happen.

[2023-01-27 Fri] some thoughts based on experience

  • I sometimes feel some cognitive resistance doing it, but I still manage to do it anyway,
  • Doing the disruption at a schedule time even when I don’t want to do it put me in the mental state of fixing the issue. I think that makes it a good drill,
  • It is reassuring thought to have the hardware unplugged but not actually broken, so that I feel like I will be able to put it back if things go FUBAR,
    • yet, I did not need that,
  • some stuff that appeared to be easy and did not need automation at the previous iteration where annoying to do at this iteration (wysiati)
    • this led me to add automation progressively in a dogfooding way and make the PRA more and more seamless, with monitoring, error handling etc.

Notes linking here