A few weeks before the time change that happened last fall, I had to debug an issue in our production environment related to something that got scheduled at, IIRC, 1:30 am "local time" and due to some date-handling code that didn't handle the time shift properly, I was getting "ambiguous time" errors. Anyway, I fixed the issue and in the process became intrigued by the interesting fact that there are certain times each year that happen twice.
A couple of weeks after that, on the night of the time change, I went to a party and afterwards a friend of mine and I went back to my house and started jamming and messing around with synthesizers and so on. As it neared 2:00 am, he told me he'd best be heading home, because it was getting rather late. I told him not to worry and that we were about to travel back in time...which we did. Then we spent another hour hanging out until it was almost 2:00 am again and he left.
I don't ever recall actually witnessing this happening before, in the past, I've always awakened in the morning to find I needed to change the clock on the stove. I really recommend staying up for the time change, because this is a pretty magical time of year. If you don't like the hour you experienced between 1:00 am and 2:00 am, you have just one opportunity per year where guess what — you get a do-over! Or if you really loved it, guess what - you can relive it!
A long while ago my friend and I released a Safari extension that shows a few clocks in the timezones that you choose. I can't recall the details, but we had an off by 1 error on a few zones during DST.
I ended up fixing it by hand changing the time, releasing a version every 6 months for years, otherwise we would get mails about it from the few users using it.
I think I could automate this or otherwise solve the issue, but it always felt nice to move the clocks of a few hundred people.
Are your production servers configured in something other than UTC, or is there a factor here I'm missing? I've never really seen a good reason for a live server not to be Z'd out
This is grand-parent's point IMO: running your cron on UTC is a decision to have it on that specific timezone.
So you're effectively deciding for instance to send your user newsletter at varying hours, sometimes at 1h00, sometimes at 3h00. Or accept that every scheduled event needs to be appropriately timezoned and adjusted as needed if DST change.
Or to have your logs timestamp be non obvious when comparing events separated by months, or have to convert all your business timestamps to a different timezone.
Those are all sensitive decisions, and we usually accept the trade-off, but it needs to be an explicit tradeoff.
Those of us of a certain age learned long ago never to schedule cron (etc.) jobs in a production environment between 01:00 and 03:00 local time.