I feel your pain. I have been maintaining a probably larger cluster with a similar stack for a few years. Right now I am working on a simulation system that will enable testing arbtrary failure scenarios for arbitrary services across corosync/pacemaker in a virtualized environment. Things like pulling cables, machine death, random upticks in latency, that sort of thing. This will also help to test new designs for new data centers, and corosync/pacemaker's performance over (potentially) high latency links such as the internet when physical site / legal jurisdiction redundancy is also required. What I've done, since it's unrealistic to expect average service authors to grok n-layers of complexity in the networking/storage/service setup fabric and related security processes, is to reduce the problem to one of rigid developer workflow neatly segregating 'platforms', 'services' and 'cloud providers'. The latter, which are either external commercial black boxes or your own infrastructure, includes one that implements corosync/pacemaker, writes your cluster rules for you, segments things neatly in to VLANs, dynamically reconfigures routers, disklessly provisions physical nodes, maintains a detailed hardware inventory including all serial numbers, etc. While it's impossible to replicate real hardware precisely in simulation, I came to the conclusion that I basically needed this degree of automation to have any confidence that things are working and will continue to work.