> Since you want everyone to be able to run the test suite locally, setting up a local datastore manually isn't an option. You need a script/function that will setup the database.
Since we want our environment to be reproducible and stored in git, this is a requirement anyway. If your schema is not versioned in Git, and cannot automatically be applied in all environments, including prod, you are doing it wrong.
> Because you want your tests to be as deterministic as possible, and you don't want the order that tests are executed in to change the results, you need to setup the database before every test, and then clean it up again after the test.
No not really. Lots of ways to solve this. For example DJango will run a test in a transaction, ensure it passes, then roll back the transaction. Order thus does not matter, the DB is cleaned up for free.
> However, setting up the datastore and cleaning it up again can take significant amount of time. Even if it only takes 5 seconds, when you multiply it out for each test your test suite can now take 100x longer to run.
This means you have now limited yourself to being able to run tests after you have completed large changes, and not incrementally.
So the old wisdom on this is that unit tests should be able to run in 5 minutes or less. Just a silly Django crud app with a parallel test runner, you can do something like 50k-100k tests in 5 minutes (Most tests take about 1/10th of a second, with several going on in parallel). Past the 50-100k test mark it gets harder to do locally, but is trivial to do in CI (for example, there are several CI as a service services that split your tests between containers, so you can run any number of parallel containers).
> Also when a test breaks, you now need to determine if it is broken because your DB isn't configured correctly, or because something is wrong with your code.
Because database state and schema is stored in git, and applied programmatically, we already ruled out the DB. So if a test breaks, it is the code.
> You can help mitigate these problems by creating 2 sets of tests. Unit tests and integration test. Unit tests use mocks, and run quickly. Integration tests will create a environment that mimics what you have in prod, and will take a while to complete.
To me, you had a problem (Database not consistent), and rather solve it in a sane way (store schema in source control, apply automatically) - You solved it in a very roundabout way (worry about which class of a tests a given test is in, write two types of tests, still fight bad schema in prod issues because you didn't solve the root problem (schema not in source control) )
Since we want our environment to be reproducible and stored in git, this is a requirement anyway. If your schema is not versioned in Git, and cannot automatically be applied in all environments, including prod, you are doing it wrong.
> Because you want your tests to be as deterministic as possible, and you don't want the order that tests are executed in to change the results, you need to setup the database before every test, and then clean it up again after the test.
No not really. Lots of ways to solve this. For example DJango will run a test in a transaction, ensure it passes, then roll back the transaction. Order thus does not matter, the DB is cleaned up for free.
> However, setting up the datastore and cleaning it up again can take significant amount of time. Even if it only takes 5 seconds, when you multiply it out for each test your test suite can now take 100x longer to run. This means you have now limited yourself to being able to run tests after you have completed large changes, and not incrementally.
So the old wisdom on this is that unit tests should be able to run in 5 minutes or less. Just a silly Django crud app with a parallel test runner, you can do something like 50k-100k tests in 5 minutes (Most tests take about 1/10th of a second, with several going on in parallel). Past the 50-100k test mark it gets harder to do locally, but is trivial to do in CI (for example, there are several CI as a service services that split your tests between containers, so you can run any number of parallel containers).
> Also when a test breaks, you now need to determine if it is broken because your DB isn't configured correctly, or because something is wrong with your code.
Because database state and schema is stored in git, and applied programmatically, we already ruled out the DB. So if a test breaks, it is the code.
> You can help mitigate these problems by creating 2 sets of tests. Unit tests and integration test. Unit tests use mocks, and run quickly. Integration tests will create a environment that mimics what you have in prod, and will take a while to complete.
To me, you had a problem (Database not consistent), and rather solve it in a sane way (store schema in source control, apply automatically) - You solved it in a very roundabout way (worry about which class of a tests a given test is in, write two types of tests, still fight bad schema in prod issues because you didn't solve the root problem (schema not in source control) )