I make a similar product offering fast Github actions runners[1] and we've been down this rabbit hole of boot time optimization.
Eventually, we realized that the best solution is to actually build scale. There are two factors in your favor then:
1) Spikes are less pronounced and the workloads are a lot more predictable.
2) The predictability means that you have a decent estimate of the workload to expect at any given time, within reason for maintaining an efficient warm pool.
This enables us to simplify the stack and not have high-maintenance optimizations while delivering great user experience.
We have some pretty heavy use customers that enable us to do this.
I make a similar product offering fast Github actions runners[1] and we've been down this rabbit hole of boot time optimization.
Eventually, we realized that the best solution is to actually build scale. There are two factors in your favor then: 1) Spikes are less pronounced and the workloads are a lot more predictable. 2) The predictability means that you have a decent estimate of the workload to expect at any given time, within reason for maintaining an efficient warm pool.
This enables us to simplify the stack and not have high-maintenance optimizations while delivering great user experience.
We have some pretty heavy use customers that enable us to do this.
[1] https://www.warpbuild.com