I've been thinking a lot lately about how much of tech, and life more broadly, is ruined by the pursuit of scale.
Taking on problems you don't (and will never) have because some vanishingly small minority has experienced them is nuts. Over-engineering is as incorrect as under-engineering. The correctly sized, correctly complicated answer is what we're after.
> Over-engineering is as incorrect as under-engineering.
I know what you mean. The catch is figuring out 'the correctly sized' answer. Uncertainty is a huge factor here.
I think a lot of projects that are called 'over-engineered' have the benefit of hindsight. At least some of such projects were rationally designed based on what people knew at the time. There is often significant uncertainty. When engineering resources are hard to acquire, the powers-that-be often want to 'get it right' the first time. Translated: over-engineer it, because we don't know when we'll get resources for the next iteration.
I suppose my point is this: the next time somebody finds some 'over-engineered' code, it can be insightful to learn about the project history. (Example: some projects have overly optimistic prediction of usage. This perhaps suggests that pessimism is one antidote to over-engineering.)
Taking on problems you don't (and will never) have because some vanishingly small minority has experienced them is nuts. Over-engineering is as incorrect as under-engineering. The correctly sized, correctly complicated answer is what we're after.