I spent my undergraduate thesis on automating the production of mock objects and unit tests using program tracing via the JDK's debug APIs [0]. I'd thought it might be useful to have a tool which, requiring no user input, would output when expected contracts of behaviour (in the context of two objects interactions with each other at production) had changed. I wrote my tool to generate JUnit/Mockito tests. It was almost bulletproof in producing working tests for arbitrary Java bytecode (see the link for examples).
At the end of it I was utterly disillusioned that the vast majority of tests with mocks were actually useful to an end developer. It felt too fragile and made refactoring work much more of a chore. Since then I've tested via constructing independent slices of implementation and pushing sample input/output in. In my experience, it has being vastly easier to maintain such a test suite.
I've meant to go back and open source my tool for a while, but found other parts of life getting in the way. If anyone's interested, would love to chat about it (might help me find motivation to continue the work).
I have recently ventured into auto-mocking, see http://zealake.com/2015/01/05/unit-test-your-service-integra..., which allows me to run super fast unit-tests, that are really doing end-to-end testing, giving me 100's of end-to-end tests per second. Is that the same line of thought as your "independent slices of implementation and pushing sample input/output in"?
That's a really interesting idea. It looks like you're capturing and proxying the latest backend request/response patterns used in its test suite into a frontend test requirement so that you can run them both independently, and thus much faster, whilst still keeping most of the assurances of an end-to-end test. I'm going to have to think about applying that approach to my own projects.
What I actually meant is slightly more literal. The backend of my latest web project is 100% web client invocable functions in AWS Lambda. The frontend is a static site built in redux with sagas. The view is a dumb layer mapping state to html and dispatching redux actions which are picked up by the sagas. Each saga only invokes one lambda function, though does send/receive signals with other sagas. Responses break down into state changes which map to a single react view.
By taking this path all implementation can be visualised as discrete slices of logic. Moving signals `up` and `down` the slice requires request/response patterns between the components. To test, I run a script against the developer staging area which sends API calls to each function and diffs the result with expected response back. It is fragile to any change which affects API responses, but is dumb enough that it gives me confidence to flip the switch turning staging environment into production. Its super easy to maintain and update, though I suspect it'd be difficult to apply the approach to more complex application domains (this is a virtual telephone number service).
Do you have any examples of where mocking has let you down? I found creating unit tests with mocks and DI extremely easy and satisfying. At least for me, they also provide a pretty convincing argument that the tested code is correct.
Also, they are basically required if you want to do any testing with external services. I'm not sure how else you could create unit tests for DBs etc.
Yes. Every project I have ever used with a web of mocks, especially around final things made refactoring near impossible. At some point the codebase got so brittle, it fell apart.
I find it funny that in most discussions around mocking, a database is the given example. 100% of the time when I need a database for "unit tests", I use a database. Not a mock. I don't want to know that when I do select blar from flar I get back the value 7, and hardcode it. I want to actually load the data and see it happen. Because if I don't, how do I know that my code will actually work in production?
Sometimes that involves sqllite (easy but not preferred because I don't use it in production). Sometimes it just involves running a local database that gets built up for a unit test run. Sometimes something like docker, sometimes not. Sometimes a shared "dev" database. Lots of options. But literally every project I have been involved with that used a REAL database instead of mocking it out, has worked better and been more stable and easier to change.
Now for actual external services.. say you call some Netflix API. Yes don't call that for real in a unit test. But you can do that with either:
a) Basic mocking, because you created a very simple API. You don't just call some random driver directly with your code, right? You have an interface in front of it. Then you can mock it with no final or static override crap going on.
b) If your API is clean enough, you can just code up a very simple dummy implementation. Lets you set the return value for methods, then the next call it gives them back.
I don't feel that strongly between A and B above. But I feel very very very strongly about people mocking databases or, even worse, every random call between classes in their own codebase, or someone elses codebase.
> I find it funny that in most discussions around mocking, a database is the given example. 100% of the time when I need a database for "unit tests", I use a database. Not a mock. I don't want to know that when I do select blar from flar I get back the value 7, and hardcode it. I want to actually load the data and see it happen. Because if I don't, how do I know that my code will actually work in production?
That is not a unit test. What if your database is corrupted? What if your mock Netflix API is buggy? The second you do anything like that you're no longer unit testing. The whole point is you're testing a single method/class in isolation -- Mockito gives you the power to assume external structures (the database) work as expected.
If you don't like unit testing, sure. But unit testing is what Mockito is all about.
What if my DB is corrupted - Why did it get corrupted? Sounds like you have a good failure to chase down. It would be even worse if your prod DB got corrupt!
For most use cases, a given piece needs 1 test around it. "The test". If you can test your logic + view + db all in 1 test, you get a pretty huge efficiency gain.. over sitting there writing "unit tests" where you mock what a select * from a database returns. Life is too short to do that.
Having just a single set of tests has a lot of draw backs compared to splitting up your tests.
Consider this, we have a simple CRUD app that uses a datastore.
Since you want everyone to be able to run the test suite locally, setting up a local datastore manually isn't an option. You need a script/function that will setup the database.
Because you want your tests to be as deterministic as possible, and you don't want the order that tests are executed in to change the results, you need to setup the database before every test, and then clean it up again after the test.
However, setting up the datastore and cleaning it up again can take significant amount of time. Even if it only takes 5 seconds, when you multiply it out for each test your test suite can now take 100x longer to run.
This means you have now limited yourself to being able to run tests after you have completed large changes, and not incrementally.
Also when a test breaks, you now need to determine if it is broken because your DB isn't configured correctly, or because something is wrong with your code.
You can help mitigate these problems by creating 2 sets of tests. Unit tests and integration test. Unit tests use mocks, and run quickly. Integration tests will create a environment that mimics what you have in prod, and will take a while to complete.
I think you also overestimate how long it takes to write mock tests.
Hopefully this provides some insight into why I think unit + integration tests are the way to go.
> Since you want everyone to be able to run the test suite locally, setting up a local datastore manually isn't an option. You need a script/function that will setup the database.
Since we want our environment to be reproducible and stored in git, this is a requirement anyway. If your schema is not versioned in Git, and cannot automatically be applied in all environments, including prod, you are doing it wrong.
> Because you want your tests to be as deterministic as possible, and you don't want the order that tests are executed in to change the results, you need to setup the database before every test, and then clean it up again after the test.
No not really. Lots of ways to solve this. For example DJango will run a test in a transaction, ensure it passes, then roll back the transaction. Order thus does not matter, the DB is cleaned up for free.
> However, setting up the datastore and cleaning it up again can take significant amount of time. Even if it only takes 5 seconds, when you multiply it out for each test your test suite can now take 100x longer to run.
This means you have now limited yourself to being able to run tests after you have completed large changes, and not incrementally.
So the old wisdom on this is that unit tests should be able to run in 5 minutes or less. Just a silly Django crud app with a parallel test runner, you can do something like 50k-100k tests in 5 minutes (Most tests take about 1/10th of a second, with several going on in parallel). Past the 50-100k test mark it gets harder to do locally, but is trivial to do in CI (for example, there are several CI as a service services that split your tests between containers, so you can run any number of parallel containers).
> Also when a test breaks, you now need to determine if it is broken because your DB isn't configured correctly, or because something is wrong with your code.
Because database state and schema is stored in git, and applied programmatically, we already ruled out the DB. So if a test breaks, it is the code.
> You can help mitigate these problems by creating 2 sets of tests. Unit tests and integration test. Unit tests use mocks, and run quickly. Integration tests will create a environment that mimics what you have in prod, and will take a while to complete.
To me, you had a problem (Database not consistent), and rather solve it in a sane way (store schema in source control, apply automatically) - You solved it in a very roundabout way (worry about which class of a tests a given test is in, write two types of tests, still fight bad schema in prod issues because you didn't solve the root problem (schema not in source control) )
I'm just pointing out that your problem is with unit tests, not with Mockito. I think unit tests are invaluable on large projects, so I find Mockito to be an incredibly valuable tool. I'm also not arguing against integration testing, just that unit testing is not the time to worry about your prod environment.
Why should your tools having to spin up a database in 3 seconds conflate local dev testing and your prod environment? Not advocating a 8 database cluster with hot failover on your laptop.. just a simple database.
Would all those scenarios really be a part of a "unit" test? I generally approach unit tests to ensure that given pre-conditions of the "external structures" you have a defined set of post-conditions that the test can exercise.
The cases you've mentioned - I'd most likely leave up at integration & failure injection testing. Still handled by a CI/CD infrastructure but done at a level where the availability of an entire environment (DBs, multiple nodes, h/w, etc.) can be expected.
I don't really want to split hairs about what test is what test.
I do not write a test that would involve mocking a database. If I have 1 test, it is going to be 1 test that uses the database. Its "the test".. not a "unit test" or an "acceptance test" or a "integration test".. just, "the test."
That's fair if you don't want to make that distinction but it is common vernacular and each testing layer has a set of properties which lead to separate pros and cons. The discussion then becomes, "what are the advantages/disadvantages of testing in isolation versus testing integrated components" which is a well trodden topic.
In a previous comment you mentioned that you get huge efficiency gains testing everything together. Well, during writing you could make that argument (though I would consider that contentious) but when it comes to maintenance and refactoring, I would argue having "the test" is going to result in a much larger surface area for you to debug when you change something and "the test" breaks.
Are you working with other developers on these projects? "The test" failing is much less useful information than a "unit test" failing. If the unit test fails I know it's something pretty fundamental -- I can isolate it to a single method. If the integration test fails, but the unit test passed, I can be reasonably assured the problem is with the database.
Tests are just as much about making software maintainable as they are about making sure it's correct.
Yes, these were for fortune 2000 companies or late state startups... projects with piles of "unit tests" where you try to test one class at a time and hand type what a database does were all inevitably crap, and either failed and got rewritten, or all the crap got written out.
My goal is to be pragmatic with testing. We have "tests" that actually spin up a real database too, but not all of our tests do that.
Where I'm with you is fighting the "architects". The guys that will constantly say you're not doing "real unit testing" and that you should mock every function (and then have to change every mock with every change and blah blah).
To me, being pragmatic with testing is a key component of GTD in modern software development.
A web of mocks is a sign that your design is not clean enough to be easily unit testable. The frustration it causes is a whisper that you need to refactor things, and that refactoring has always ended up making my software far more robust and understandable.
In my experience, most people just throw up their hands and integration test instead of aggressively working the design into shape.
What do you propose instead of mocks? Custom test-versions of every class you need to control the behaviour of?
I've had lots of challenges with some frameworks (EasyMock makes it easy to make a mess) but Mockito is so nice to use, I can't imagine writing Java code without it for my basic unit tests.
You don't need a "Custom test-version" of every class. If you're using dependency injection you can create a "Custom test-version" only for the things that sit at the border of your domain and use real code for everything but those.
The classic example is again storage, suppose you use Redis for storing key-value strings.
Create an interface that has the store and get method and have two implementations of that, one that actually injects a Redis client and one that injects nothing and saves everything in a map.
Now you can test the real implementation with mocks and a real Redis if you want, but all the other tests can run on the in-memory implementation just fine and super fast. No need to mock anything.
I've found mocks useful to exercise interactions (and Mockito is awesome), but once you have a fake for the types at the boundary of your dependency graph you're pretty much sorted.
Mockito makes it a lot worse by not having strict mocks be first-class. EasyMock is much more usable in this regard - you're usually going to have to mock every called method anyway, so it's much better to have the framework tell you when an unmocked method is called than have it silently return null and leave you guessing.
I believe Mockito can verify for you that unmocked stubs haven't been called. [0]
For people looking for an alternative take on why it is good practice to let mocked objects have default stubs for unmocked methods, see this article [1]. This article is referenced in the mocktio documentation.
In my experience, Mockito never actually gets to that point, because my program has called an unmocked stub, tried to use the return result, and NPEd in an obscure and hard-to-debug manner before verification call.
In my experience with Mockito and unit tests I haven't ever ran into this. However it seems like a very valid concern, and I could definitely see it causing problems.
I wonder if it is possible to use reflection and have mockito throw an exception if a stub method is called.
Not really. You can set a custom default answer and you can write a method that creates mocks with that answer, but there are enough rough edges that it ends up amounting to writing your own test framework. Much easier to just use EasyMock.
> I believe Mockito can verify for you that unmocked stubs haven't been called. [0]
As the other reply says, that doesn't help because it never gets that far.
To my mind the article you link has the wrong premise, because I do make my data tests that way - I assert that the data is exactly what I expected. Surprise changes to data would be bad, as would surprise changes to state.
After journeying through many languages since my long time with Java that ended in 2011 I have to say I have yet found a mock object framework in other languages that match the expressiveness that Mockito did. Python's mock has the least memorable API I have ever worked with. :-(
This is because using Mocks in a language like Python is a bit awkward and unnatural in itself... if you look at the way tests are usually written, there is just little need for it. That's also the reason why only recently unittest.mock was added in the first place.
I think maybe because there is no type checking in Python. This makes it more trivial to have objects that quack like a duck, without the need to explicitly create a class that conforms to an interface.
- Another commonly seen way is that there is a backend of sorts passed around and can be injected with little hassle from the tests. Often you'll see what boils down to a Mock, just written by hand, because it tends to be little hassle, and often results in quite clear testing code.
Generating types on the fly using mocking and then doing somewhat awkward indirect assertions is by comparison a bit convoluted and less clear.
There's been a lot of discussions on Mockists vs Classicists. See Martin Fowler's article: http://martinfowler.com/articles/mocksArentStubs.html. In my experience, projects that use mocks extensively usually have the most bugs. If you also have integration tests then mocks are ok. But then it's more code to maintain.
Given a choice I would always do "integration test". Fact is you can test a single class/method, reality is that no class/method exists alone.
We recently implemented an extensive test suite that can be run with either mocks for service dependencies or in a live configuration, depending on config settings. Before, we simply had integration tests that relied on very specific data across disparate services and running it in CI was a nightmare (to the point that we would typically only run the full suite in a QA environment shortly before a production deploy).
Whereas our integration testing has been responsible for catching hundreds of obscure bugs over the past year, most of the bugs found by our mocks have been bugs in the mocks themselves and I expect that issue to compound due to the brittle nature of mocks in general. Mocks make every change some multiple more expensive in both initial cost and maintenance.
At the end of it I was utterly disillusioned that the vast majority of tests with mocks were actually useful to an end developer. It felt too fragile and made refactoring work much more of a chore. Since then I've tested via constructing independent slices of implementation and pushing sample input/output in. In my experience, it has being vastly easier to maintain such a test suite.
I've meant to go back and open source my tool for a while, but found other parts of life getting in the way. If anyone's interested, would love to chat about it (might help me find motivation to continue the work).
[0] http://www.doc.ic.ac.uk/teaching/distinguished-projects/2015...