A few suggestions:

1. Application / service to service secrets should be handled either by Vault or AWS Param Store. Both are good. AWS Param Store is easier but less feature rich. Ideally, your Vault instance would have the database engine which allows for automatic rotation and password management of databases like Postgres.

2. Developers should only have access to AWS through SSO using something like Okta. Okta has MFA through it's own authenticator app (not through SMS).

3. Password sharing should be handled with something like sops or if you need a more lightweight solution - Password Pusher. Sops works really well for hiding credentials within code that needs to be deployed. A team managed 1Password instance works well too.

4. Terraform should inherit a role that gives permissions to run against infrastructure. That role should be short lived and limited to running in a CI/CD environment (preferably on your own runner if that's possible).

5. You should have a separate AWS environment for running test projects. You can do this virtually (localstack) or physically through having a separate AWS account. I prefer the later since it's the real thing and implements all the APIs. Localstack can be decent though. If I had to setup an organization now, I would have a minimum of 3 AWS accounts. One for production infrastructure, one for staging environments and one for test environments. I would also add in AWS Budget Alerts and Cost Anomaly detection to prevent huge spend on it.

This is pretty much how my startup is running things, with the exception that we're using credstash for #1.

Hashicorp Vault. Checks all your boxes, I think.

Or for something more lightweight (though it sounds like you're just about getting to the size where something more instrumented is worth it), you can have a git repo with secrets encrypted with individual keys. (git-secret and pass are two of the more popular ones here)

> 8. Bonus: How to manage non-technical secrets, e.g. credentials for web shops to order supplies? Multiple people would have to order something.

Probably using a separate system. Bitwarden, 1Password, KeepassXC.

Warning to anyone interested in Vault. Vault policies involve a learning curve. Maybe they have improved documentation and examples since I learned Vault management but I remember how complicated and time consuming it was.

This was the problem I found with Vault too - I ended up as the only person in the company that knew how to use it and therefore it became a burden very quickly. We switched to EnvKey which basically has zero learning curve.

For a small team I wouldn't go there. Except if the product requires a higher level of security and we have no money to spend on the SaaS or self-hosting is a requirement.

The problem isn't self-hosting. It's the ongoing setup and management through their DSL.

FWIW, If you don't require the single-pane management, some scenarios can be simplified by deploying additional individual clusters/instances with separate access rather than controlling only via policy.

Mentioning since I've def seen people get stuck and forget to consider the obvious.

BitWarden just announced secret management.

https://bitwarden.com/products/secrets-manager/

For AWS definitely use SSO with short lived credentials and proper landing zone setup.

1Password for all the logins, 1Password Secrets Automation for all the infrastructure stuff.

Each user has their own vault of personal logins and related items, then various groups can have shared vaults with shared logins.

Using it with almost 100 people company wide.. working well so far!

Haven't tried it out yet and not affiliated but Infisical (YC W23) is an "Open-source secrets manager for developers" that just launched on here:

https://news.ycombinator.com/item?id=34955699

One of the creators of Infisical (https://infisical.com) here. We indeed satisfy most of these points. Happy to answer any questions here or feel free to join our Slack of you need any help: https://infisical.com/slack

We use EnvKey - https://www.envkey.com - I can't recommend it highly enough - it's almost perfect. My only feature request would be for supporting scoped variables, even if it's just a veneer of grouping in the desktop app's UI - we end up with quite a lot of variables in there, some of it just for the CI to run, some for local dev only etc. and being able to thematically group variables would be really useful.

My experience is similar to the other comments here:

- A combination of sops[1] and AWS Parameter Store might be a good candidate for (4)

- (5) can be solved with AWS SSO / OIDC (also: platforms that offer OIDC, usually support that feature for free, and a variety of them can be configured via Terraform)

- For (6), there are solutions like Atlantis[2] for Terraform, but I am not familiar with alternatives for Pulumi and Ansible. Might be a good idea to spend some time and build a pipeline with whichever CI/CD platform you are using (bonus: GitHub Actions supports OIDC auth with AWS as an IAM role[3], no secrets nor credentials required!)

- For (7), you can create a separate AWS account and manage it with AWS Organizations (or not)

- For (8), 1pass[4] for any credentials that requires username/password combo (non-tech/breakglass account). Groups can be created to limit certain credentials to subsets of users

[1] https://github.com/mozilla/sops

[2] https://www.runatlantis.io/docs/repo-level-atlantis-yaml.htm...

[3] https://docs.github.com/en/actions/deployment/security-harde...

[4] https://support.1password.com/custom-groups/

Adding to existing suggestions.

- About 5, https://github.com/99designs/aws-vault is super handy to keep AWS credentials encrypted.

- Use full-disk encryption as a team policy.

- About 7, it is likely that this is solved by having preview environments so that developers don't need to manage infrastructure manually.

- About 8, 1password shared vault works nicely for us.

A few suggestion: - start with a password manager (I will suggest https://1password.com). For any service outside the cloud this should be fine

- Where possible enable SSO, even for AWS or other cloud services (https://blog.leapp.cloud/how-to-saml-federate-your-aws-accou...). Here an example on how to do it with GSuite and aws

- For all your employees I can advice you Leapp as open-source project (https://github.com/Noovolari/leapp). It solve mayor of the problem listed here:

. Responsable for the AWS infrastructure get the IAM Security standard on track (short-lived credentials, MFA, IAM Users secured, rotation of credentials and access to the console with a set of secured credentials, and generation of Azure ceredentials)

. temporary tokens are mandatory, so let a project like leapp locally manage IAC credentials for you.

- Is not too strictly to manage pipelines only after confirmation? You can create a specific role for managing a specific policy to deploy your pipelines. Here an article on how to start organizing your AWS account as a startup (https://blog.leapp.cloud/aws-multi-account-strategy-explaine...)

- "People work on their own devices.": that's why Leapp has been created, check it out.

- Optional: "It would be good if developers get credentials to setup small test environments in AWS." how anout creating a sandbox account for them? - "Bonus: How to manage non-technical secrets" even here, Leapp is a desktop app and is vastly used also for accessing to EC2 instances even for not technical people

I hope this guide can help you!

Hello,

At my current gig we're using doppler[^1] (no affiliation) for application secrets. We're using doppler with their kubernetes operator which supports auto-rotation on secrets. Secrets are set as "env variables". So far, doppler has not suffer big outages or we did not notice, because the operator will keep working even if their API is down - of course you won't get updates. Access control could be more _fine grained_ and they added secret auto-rotation option recently[^2]. We don't use that yet.

I've been a happy 1Password user (no affiliation) and we use it company wide to share user secrets. 1Password support CI/CD integration IIRC, so in theory should cover most use cases.

If you can pay for AWS Vault, the terraform integration comes out of the box. However if you're a small team running vault might be a daunting task and you're inserting another SPoF.

There are many open source application secrets tools that you could check out though. In the past I had great experience with credstash[^3]. Credstash is a really simple and secure open source solution that is based on AWS KMS, IAM and DynamoDB. Costs pennies to run for medium size deployments. Once you setup and document the way to use it, it's really easy. The downside is that as a tool is pretty _raw_ you have to build things like "secret generators", etc. But combined with a slack bot can be a really powerful, secure, cost-effective self-hosted solution that doesn't require maintenance (as opposed to solutions like Vault).

If you have specific questions about any of the above tools feel free to drop an email. I'll happy to answer questions.

[^1]: https://www.doppler.com/

[^2]: Auto-rotation is complicated because you need to integrate the auto-rotation with external tools yourself most of the times. There are Hashi-Vault modules for SQL DBs but not for Mailgun or CloudFlare for example.

[^3]: https://github.com/fugue/credstash

There are several tools available for managing secrets and credentials in a small startup. Some popular options include:

LastPass, 1Password, Vault and Bitwarden, all these are cloud-based password manager that offers secure storage and sharing of login credentials. they also includes features like MFA and the ability to generate temporary tokens.

As for how to do it, I recommend evaluating each of these tools based on your specific needs and choosing the one that best fits your requirements. In general, you should look for a tool that offers strong security features, easy integration with your existing tools, and a user-friendly interface.

LastPass has a lot of egg on their face from a recent data breach.

I moved off of their service because I could no longer trust it.

All deploys are done from a VPS host machine secured behind a firewall and only accessible via passphrase-protected ssh keys.

All secrets are in a file on there. I make it a file of exports I source before launching the app. Backup the file in 1pw.

Secrets get deployed along with the app to a server also secured with ssh and a firewall/VPC.

Maybe doesn't scale to an org chart, but a scrappy startup of 3-5 with some contractors and it works well. This method allows you to literally have a single VPS prod host and a single VPS deploy box. That's $10 and super simple and completely cloud-agnostic.

Would recommend one of two things: - 1Password/1Password Secrets Automation - Github/Gitlab CI/CD Secrets

1Password may be easier to setup with easy CLI/UI access for secrets as well as in your CI pipeline. It's paid but I love the ergonomics of the system. It's what I do for all my personal projects.

Gitlab CI/CD secrets is what we use at my startup. The most annoying thing with Gitlab secrets is that you need to give maintainer access to use secrets in local scripting, but we've been able to work around it for now

Just want to drop these 2 resources: - Sync environment variable tools: https://finddev.tools/category/sync-environment-var - Tool to send secret info: https://finddev.tools/category/send-secret-info

Check out Amazon KMS if you're on AWS - https://aws.amazon.com/kms/, theres really no way to get around using a secrets manager.

For non-technical secrets, most organizations have a password manager (1password has been the go to at my previous two orgs)

AWS Secrets Manager with strong a strong least-privilege, regularly-reviews IAM configuration has worked well for us

Building on this I’ve found https://github.com/segmentio/chamber to be super useful

This is what we use and it works exactly as you would expect. The only downside is that it seems to get very expensive very quickly.

Doppler is nice. Also AWS has Parameter Store (lower level, cheaper) and Secret Manager. You could also use built-in secret storage in Github / Gitlab for CI workflows.

An old Android phone in a safe.

1Password teams

I think a password manager like 1pass for all secrets (definitely not confluence) plus a cloud hosted secret system like Aws secretsmanager for secrets that need to be accessed programmatically e.g by various IAM roles.

Devs, once assumed a role, download the secret env files from secretsmanager.

You might see where I'm going with this, you have two systems to store secrets in, one for general use and one for programmatic use. It's not ideal but pragmatic and do consider if your in a security sensitive domain, then you would want to customise this process e.g. by only having one storage system and shorter access etc.

Someone of the two infra people would have the ability to assume a role/get a set of expiring "deploy" keys that they would paste into CI To start a deployment (still in ci just doing stuff that CI normally doesn't have permission to do)