Well, that's kinda the point, isn't it? Have the tests run after each PR, and it flags all the changes for you. For the intentional ones, that means the tests have to be updated (often just changing a line or two). But it also catches all your unintentional changes that you didn't mean to break.
This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch.
Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?
Realistically the build will fail, but since QAs used some action capturing thing, devs will have to either recapture the same flow or delegate it back to QA. Now you have the pipeline blocked, QAs most likely working on some other stuff so they need to reprioritize, devs are also annoyed as their build is not going through.
I completely understand there are pros and cons to every set up, but if higher velocity is more important than complete correctness, such set up might be detrimental.
In our setup, we had the ability to either fix the test ourselves, comment out those tests as useless, or just force a build and bypass any hooks if we needed to force a new deploy and we were absolutely certain the tests aren't important. But at least they would be caught, even if they served as a soft warning rather than a forcible barrier.
> but if higher velocity is more important than complete correctness, such set up might be detrimental.
I guess it's a cultural value thing: move fast and break things vs polish before shipping. As a user I always prefer the latter, but hell, I don't run these multibillion dollar companies that are always shipping buggy things, lol
Totally agreed, it really depends on the case. I’ve been user of both types of products where I value extreme correctness (think of anything financial, or core software that I want to never crash like a browser), but also really don’t care if something like Spotify or Twitter doesn’t work 5% of the time I use it. Obviously that barrier is different for everyone, but that’s where the product development planning and prioritization comes into play.
This is especially important in complex UIs where everything from localization to state management to A/B testing can accidentally break things you didn't even mean to touch.
Isn't it better for the devs to spend a few minutes looking at a broken test (which they can easily fix themselves) than for a user to discover it and have to go through triage months later, when it's out of everyone's memories already?