I remember how one of our engineers had his docker daemon connected to production instead of his local one and casually did a docker rm -f $(docker ps -aq) .
Same thing happened to me but with CI, which felt bad enough already.
Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.
For a large or v large tech co you should probably be aggregating logs to a centralised location that doesn't require access to production systems in this way. Stack dumps should also be collected safely off-system if necessary.
Perhaps my industry is a little more security conscious (I don't know which industry you're talking about), but this doesn't seem like good practice.
Let me be clear, I agree it should not be normal to SSH into a prod box. Our logs are centrally aggregated. But it’s one thing to say it’s not normal, but quite another to say engineers shouldn't have access, because I totally disagree with that.
What normally (should) happens in that unusual case is that the engineer is issued a special short-lifetime credential to do what needs to be done. An audit trail is kept of when and to whom the credential was issued, for what purpose, when it was revoked, etc.
It can be efficient, particularly in smaller companies, but that's where exactly this rule should be applied.
In some industries, security and customer requirements will at times mandate that developer workstations have no access to production. Deployments must even be carried out using different accounts than those used to access internal services, for security and auditing purposes.
There are of course good reasons for this; accidents, malicious engineers, overzealous engineers, lost/stolen equipment, risk avoidance, etc.
When you apply this rule, it makes for more process and perhaps slower response times to problems, but accidents or other internal-related issues mentioned above drop to zero.
Given how easy it is to destroy things these days with a single misplaced Kubernetes or Docker command, safeguards need to be put in place.
Let me tell you a little story from my experience;
I built myself a custom keyboard from a Numpad kit. I had gotten tired of typing so many docker commands in every day and I had the desire to build something. I built this little numpad into a full blown Docker control centre using QMK. A single key-press could deploy or destroy entire systems.
One day, something slid off of something else on my desk, onto said keyboard, pressing several of the keys while I happened to have an SSH session to a remote server in focus.
Suffice it to say, that little keyboard has never been seen since. On an unrelated topic, I don't have SSH access to production systems.
This exactly. I have deleted database records from a production DB thinking I am executing on my development DB. I've kept separate credentials and revoked dev machine access to prod environment ever since.
Same thing happened to me but with CI, which felt bad enough already.