Because of the cost. Performance (if you optimizer is any good) will be at most a few thousand CPU cycles and so only rarely worth worrying about, though even this can add up if everything is an interface.
The larger cost is maintenance. Every time you want to change/add/remove something you now need to touch not only every place that uses the thing in question, but also all the tests. That is by mocking your are tightly coupling your tests to how your code is implemented instead of the code should do. If there is more than one implementation this is okay as the code can do different things in different situations anyway, but if there is only one then you are adding coupling that isn't useful.
If you have never worked with non-trivial code that is more than 10 years old the above won't make sense. However as code ages you will discover that the original beautiful architecture ideas where always wrong for some need that happened since and so you have had to change things. Mocks make all those changes much harder.
You apparently have never been in the situation of making a trivial change to production code and then have thousands of tests fail because of overuse of mocks.
The larger cost is maintenance. Every time you want to change/add/remove something you now need to touch not only every place that uses the thing in question, but also all the tests. That is by mocking your are tightly coupling your tests to how your code is implemented instead of the code should do. If there is more than one implementation this is okay as the code can do different things in different situations anyway, but if there is only one then you are adding coupling that isn't useful.
If you have never worked with non-trivial code that is more than 10 years old the above won't make sense. However as code ages you will discover that the original beautiful architecture ideas where always wrong for some need that happened since and so you have had to change things. Mocks make all those changes much harder.