My problem with Math education was always that speed was an enormous factor in testing. You can methodically go through each question aiming for 100% accuracy and not finish the test paper, while other students can comfortably breeze through all the questions and get 80% accuracy but ultimately score higher on the test. This kind of penalizing for a lack of speed can lead to younger kids who are maximizing for grades to move away from Math for the wrong reasons.
Source: I'm slow but good at Math and ended up dropping it as soon as I could because it would not get me the grades I needed to enter a top tier university.
In my view, speed plays a role, though I can't say that it should play such a central role.
I think that in a lot of disciplines, you have to become fluent at manipulations, and at seeing and thinking in higher level patterns. Being able to think your way through a more complex problem would seem to benefit from, if not require, seeing multiple steps ahead in a progression. At least this is my perception.
My experience in school math was that it wasn't enough to satisfy myself that I knew how to solve a problem. I then had to work my way through a whole bunch of similar problems until I could perform the manipulations quickly. This is also how I managed to commit the definitions, axioms, and theorems to memory. If I didn't do that stuff, then I got my arse handed to me on the exam. I gave my kids pretty much the same advice.
I've never been evaluated but I don't think test anxiety fits in my case - the speed issue was only ever in math / physics.
My accuracy on the questions I got to was very high, I just couldn't go fast enough to complete enough questions. Same deal on SAT type math papers too.
Different tests challenge people differently. It may be that you only needed an accommodation in computationally heavy exams.*
Truthfully, my mention of accommodations isn't for you--I'm assuming you're out of high school already and you've found a professional niche that works for you. I mention test anxiety and the professional workarounds for high schoolers or their parents.
*) This is actually moving well beyond my expertise. I know about the existence of test anxiety, and I've accommodated students with it.
A brief consult with Dr. Google surfaced some clinically recommended accommodations that probably would have helped you. Sorry I don't have a time machine!
Source: I'm slow but good at Math and ended up dropping it as soon as I could because it would not get me the grades I needed to enter a top tier university.