The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources, rather than a standalone file. In other words, this metadata then is the state.
Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?
That's still storing state, except instead of keeping it within a single file which you have full control over, you're instead sprinkling that state all over your infrastructure and then hoping it doesn't get mangled in between invocations.
And also hoping that service's API has all the tools needed to find your scattered state within a reasonable amount of time in order to diff any changes you make in your declarations
I built pretty much the system you’re describing for my company (a stateless terraform alternative) and this scan that you cite as a negative happens in… 100ms in parallel? Roughly the same amount of time it would take to download a state file? Dunno about you but ensuring state is always accurate and in-sync is well worth the trade-off to me.
I'm curious, and not antagonistically, about the size and variability of the infrastructure you're working with, on how many platforms it's running, and how many people are in charge of it.
Is this tool private or available for us to try out?
It’s small, under 50 servers running OpenBSD and Linux under a wide variety of configurations (including gpu), 6 databases, and Redis. Though the system is easily extensible to other clouds and resource types, it only needs to work on GCP right now. Been running it in prod for the past 3 or 4 years—no outages, no downtime, no surprises. Doubled infra over the holidays then scaled everything back with no issue.
The stateless approach has worked really well for us.
This is the reason we didn't get Terraform. In a windows shop the state is never the same because most people will fix it via the UI and then the state file becomes useless
At small sizes, this probably works reasonably well. However, as you scale, the performance will get worse and worse, and you run the risk of hitting rate limits and/or missing stuff.
So, fine for you in your personal small environment, but maybe not something suitable for Terraform in general.
I can’t imagine you would ever hit a rate limit, how often are you deploying large sums of infrastructure? For me it’s maybe a couple times a day and the surface area is small
AWS API Rate limits are very surprising. I've hit them on new accounts when trying to deploy terraform based code, and that's without having to literally scan the entire account.
If I have 3k machines running in an autoscaling group I think it would be ridiculous to have to hit each one of those with API calls to try and infer which were or weren't part of my state. Building a simple high availability VPC is about 72 resources just by itself.
I don't think people advocating for "the cloud is the state" realize how big even trivial environments can get- let alone complex ones.
Huh I've never hit rate limiting on GET with AWS, could just be my luck. This also might not be the answer for every problem, but could be nice for some
So the position is Terraform should have state, but rather than keep it in one place such as an S3 object, it should be spread across the metadata of many resources?
Some issues with that:
- Fetching the whole state would be hugely impractical due to the number of API calls
- The risk of losing state information by a resource being deleted outside of Terraform is greater
- Again, not all resources have metadata that could be used to store state
>- The risk of losing state information by a resource being deleted outside of Terraform is greater
This isn't losing state information though! That is the state. If state information were kept outside this it would now be wrong which means terraform *would do the wrong thing.
If some state information is stored using metadata then we need that metadata to know the total state. The contents of TF state is more than what can be read from the provider APIs often.
IAM Users are taggable, but to get the tags on a given user, you must request them one user at a time from a known list of users. The "List all users" call doesn't return their tags. Obviously this is less of an issue for the TF state use case, but does add to the API call overhead for any tag-based approach.
My favorite is that you can look up some resources by tag directly, but other resources have to use the special resource tagging api. The AWS API also silently fails, on purpose, for some things- try looking up quotas for Organizations as an example and you'll end up with an empty array instead of the actual quotas.
Cloud providers having bad APIs is definitely the default state.
If you look through the octoDNS providers there's a number of cases where extra info is stored for "dynamic" records. The metadata is often things like the pool name or rule number. In other cases it's details about the health check config/version. The extra info is sometimes stored into a "notes" field, other times it's encoded into the ID or similar.
It's true that nothing extra is needed for simple/standard records, but once you start doing GeoDNS, failover, health check, etc. it's required.
In all cases thus far we've been able to find a way to store/indicate whatever we need.
You can have multiple TXT records for a given domain name, so it would be possible to store an arbitrary amount of metadata for whatever systems you desire, and just loop through the TXT records to figure out which ones are for the current system's purpose.
> The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources
Except for the Cloud's and API's that don't provide them but we stil need configuration managed. This is the world we live in, and in that world Terraform is the solution to the problems that we seek. In an ideal world Terraform would not be needed, but we don't live in that world.
We use Terraform extensively at our organisation. Some examples come to mind that make this impractical:
- For services that do support tags, we are already reaching limits on the number of tags that can be associated with a single resource. For example, in Azure, some resources still only support 10 unique key/values
- Drift detection against write, but no read secrets mean that you cannot do drift detection over certificates, and secrets. Depending on your organisation and how they manage things like PKI, this may be impractical to track validity of the endpoint.
- Many services we manage don't have tags. For example, we use Terraform to manage Github Repositories, Actions, AzDo Pipelines, and Permissions
- Some object types simply don't have primary keys that are easily searchable by the provider, and requires some sort of composite key to be compiled and tracked.
State gives us a common schema and playing field to significantly simply the generation of dependency graphs and show drift. I imagine that even without a 'statefile', you would end up having to generate a similar graph in memory anyway.
> Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?
Tags do not solve the deletion issue. It would require two step deployments, for example by adding delete = true to the config, applying, then removing the resource from the config entirely and applying again. But I don't think that's too bad tbh.
But I do not believe leveraging tags and/or metadata is the right approach - configuration for these resources could potentially be large (e.g. GKE resources) and most providers will have a size constraint on their metadata and tag values. Creating a metadata/tag key for each configuration key would also get messy, but solves the value size problem.
Why wouldn’t it be possible to not store the previous state at all? Terraform’s job is to reconcile what exists with what is declared - we should be able to rely on the provider’s APIs to understand what exists, perform the diff, and reconcile the changes.
Let's say your TF config declares a database Foo. Your AWS account has databases Foo and Bar. How does TF know whether it's responsible for database Bar and whether to delete it?
Interesting. Disregarding for a moment that not all cloud resources supporting tags on all cloud providers (and let's not get started with all the non-cloud stuff Terraform is capable to manage), how does tf know that `stack x` is the thing that's currently running?
By the time the tag has enough information to be useful, you are storing a TF state file in a tag. Anyway, I don't think people complaining about a s3 bucket for tf state actually has experience creating tools like tf.
If you deleted a resource block from a previously applied terraform declaration and reapplied it, how would the provider know that the resource in the environment needed to be deleted verses accepting it as an object not managed by terraform?
Where would it store it's history to make the diff against it?
Well, you can just delete/remove/clean everything visible and create only the things based on the configuration. Then you can't store any state in your Terraform-managed infrastructure, but for people that don't want state anyways, that is a blessing I guess?
Same famous problem for any kind of make-based build system, removing targets usually leave old files in your build directory. IIRC Bazel can avoid it but don’t know how, either it must keep a list of old files or compare the entirety of build directory to the build-graph.
Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?