Even here on HN plenty of people seem to misunderstood the fundamentals of "hardware-rich development" and assume that it must be fantastically expensive and/or reckless.

It is not exactly cheap to do it like SpaceX does, but the savings in time and increases in robustness may very well offset the cost of all the prototypes that undergo a RUD.

The focus on making lots of prototypes reduces cost. They streamline the largest and most expensive manufacturing bottlenecks.

Each Starship has 30+ engines required per flight and tons of precision welding. All of it can now be made in a month. That means massive cost savings going forward, the faster each can be made

Hmmm, from a manufacturing prespective this is probably the safest way too, because you have already scaled production, you arent suddenly trying out all these new manufacturing processes when starship enters production. All your processes have been continuously validated already.

Failing fast also reduces cost. It cuts down on the time wasted on things that aren't going to work. With traditional design it can take years to find out if something will work or not. And when they don't, it leads to lengthy and expensive redesigns. You avoid all of that by iterating rapidly and building on and perfecting what works and scrapping/replacing what doesn't work before you sink a lot of time and resources in these things

We could have had dozens of James Webb Space Telescopes for the cost of, like, five.

Even better, since larger payloads can fit, the design can be far less complex due to weight and folding limitations.

Rather than spend 10 years designing and building one telescope, they could spend 2-3 years and keep refining the hardware, with multiple types on the same platform

Even SpaceX's methods of fail fast (and often explosively) is relatively cheap. SLS is doing it the hard and expensive way.

The downside is that rapid iteration doesn't always give the same insight in terms of scientific understanding.

E.g.,

Design_A fails. We quickly pivot to Design_B, which works. However, we don't spend the time ultimately understanding why Design_A failed. This can be operationally great, but it can also risk conflating being lucky with being good. If you don't fundamentally understand why the thing failed the first time, it's much harder to understand if those failure modes are fully mitigated.

Personally, I think the ideal approach is to have SpaceX continue the rapid enginneering iterations, but give NASA (or some other entity) the resources to research the rest.

That's just your assumption. SpaceX is actually doing all that. They invested early in good simulation tools and they are developing their own. They have great material science teams and engineering teams. They do invest a lot in actually understanding the fundamentals and they make good use of the data from things that failed.

Take for example the work on the failure of Amos 6. They went really deep into that one and its a pretty interesting result from a scientific perspective.

On the other hand, there is no point in doing endless science on why the pad blew up during the first orbital flight test. They solution was already in development anyway and its pretty clear that the solution would migrate the problem. It might introduce new problems but the old one wasn't happening again.

So there is a limited need to the exact detail of underground steam explosion or whatever exactly happened there.

You can understand why design A failed. Just don’t need design B to be blocked on that with agile because the last iteration was working.

Sure, you can if you dedicate the resources to understanding the failure. What I saw personally is that is not always the case with SpaceX. In my experience, they changed a design and seemingly used that as an excuse to not further investigate the original design. (This was related to COPV failures. The general consensus seemed to be “we don’t need to understand why it failed because we’ve moved on to a different design.”)

Some counter-points:

1) The failures themselves aren't all too complex (relatively speaking), and by the time the design made it to a flight vehicle it has been vetted for all immediately knowable risks. The rest you discover during flight. And even then the actual nature of the problems rarely (if at all) are esoteric or present breakthrough scientific discoveries.

2) Failure in rocket launches are catastrophic leaving behind very little evidence. There could be a limit to how much you can learn. But...

3) ...SpaceX is a business serving customers, and if a pivot means going away from a complex cloud of possible failures (known and discovered) to more robust, more deterministic, more reliable, you should do it.

4) At what level do you stop and go "this is fundamental enough"?

I think I disagree on many points here in this specific case.

>The failures themselves aren't all too complex

The COPV failure mechanisms may not be terribly complex (e.g., voids or delamination), but managing the processes to mitigate them isn't.

>by the time the design made it to a flight vehicle it has been vetted for all immediately knowable risks.

SpaceX has already shown some bad processes that disprove this statement. E.g., the strut failure was due to poor supplier quality control, which is a relatively mature process in the industry. I'd give them the benefit of the doubt and say they've gotten better as they've matured as a company.

Failure in rocket launches are catastrophic leaving behind very little evidence.

The difference, I think, is that you are constraining this to launch failures. I'm talking about knowledge and data about individual component failures, from which you can derive the overall reliability of the design.

if a pivot means going away from a complex cloud of possible failures (known and discovered) to more robust, more deterministic, more reliable, you should do it.

I don't disagree that more simple tends to be more reliable. However, the point was that iteration doesn't necessarily advance knowledge of failure. I think these two points are not mutually exclusive. A good business case doesn't mean it makes for good engineering or good science.

At what level do you stop and go "this is fundamental enough"?

It's a tough question. But if you can't characterize why a failure mode occurred reasonably well, that's probably not good enough. So if they say, "We know voids in the COPV manufacturing process contributed to the failure" they should probably make sure they have a good understanding of why they occurred. (And maybe they have at this point)

You're dealing with incredibly smart people here. Now of course perfect knowledge is almost impossible. But each of the designs is probably specked out with at least pluses and minuses and theory is as to whether or not the design is viable, which ways it will perform well or fail.

This is just basic science: theory experiment result.

And with all the documentation and instrumentation on the flights, The theories can be experimentally validated or invalidated.

This isn't like some agile programmer shoving in a couple hundred lines of code over the course of a one week sprint.

Would you have said the same about NASA pre-Challenger/pre-Columbia, or Boeing pre-CST-100? I don't think SpaceX is inherently unique in either the ability of their people or their innate human biases. They may have a different culture, and related to that, I'm pointing out what I've experienced as a potential downside of rapid iteration philosophy. I'm not even saying it's bad, just that we need to be cognizant of it's downsides.

What's stopping them from choosing to revise design A over chosing design B?

Nothing, other than it may just be easier to change the design to avoid the effort of a deeper investigation.

Luck tends to average out over many trials.

Not a particularly good strategy here for a couple reasons:

1) Risk on safety critical operations is not something people tend to want to roll the dice with.

2) The nature of the industry makes each trial pretty expensive. Over decades, the shuttle only had something like 135 launches and the managers still didn't have a good handle on the actual risk.

> the shuttle

Sort of proving the point here, no? There is a reason SpaceX has gone to orbit more times than the Shuttle has. Soon they will have launched more Starship prototypes than the shuttle ever did. And they will be safer, because SpaceX understands why their rockets fail.

Obviously this strategy does not work with humans in the loop. It might have been impossible to do what SpaceX is doing now in the 80s given the advancements in computer control and simulation that have happened since then.

>Sort of proving the point here, no?

I don't think so, for the same reason you identified: it only works when the risk isn't safety critical (ie when humans are in the loop). In other words, it's acceptable when risks are low. It's the same reason NASA is more risk-tolerant with non-human-rated missions. But keep in mind CCP is also meant to transport humans. The risk we'd want to bolster against is the human biases that lead us to get complacent. There are some instances that make me wonder about that with SpaceX, but I'm giving them the benefit to the doubt that they've learned from those.

> However, we don't spend the time ultimately understanding why Design_A failed

Did you read TFA? It's about the release of an extensive report documenting why Design_A failed and what corrections are to be made regarding each specific failure.

Falcon 9 reached orbit on it's first attempt, so it didn't have failed experimental flights unless you count Falcon 1 as being it's experiments.

You are right. It was essentially a bigger Falcon 1 and not a complete redesign. So, the risk was not as high. It was an incremental upgrade. Not a small one but still.