Manual Progressive Chaos Monkey
FleetingIn case you want to run some chaos monkey but don’t feel ready yet.
- plan to disturb the system manually once,
- then make the system robust against that disturbance,
- then plan it another time,
- and again
- and again
- …
- at some point you will want to automatize this test => automatize it
- then think about another disturbance and plan it,
- and again
- and again
- …
- automatize it
- rinse and repeat
When you think you bullied the system enough, may be you are ready for actually chaos monkey.
In the spirit of the voluntary discomfort, by willingly adjusting to voluntary disturbances, you will get more ready when the discomfort actually happen.
some thoughts based on experience
- I sometimes feel some cognitive resistance doing it, but I still manage to do it anyway,
- Doing the disruption at a schedule time even when I don’t want to do it put me in the mental state of fixing the issue. I think that makes it a good drill,
- It is reassuring thought to have the hardware unplugged but not actually
broken, so that I feel like I will be able to put it back if things go
FUBAR,
- yet, I did not need that,
- some stuff that appeared to be easy and did not need automation at the
previous iteration where annoying to do at this iteration (wysiati)
- this led me to add automation progressively in a dogfooding way and make the PRA more and more seamless, with monitoring, error handling etc.