Here's another one that I feel is often overlooked by traditional A/B testers: if you have multiple changes, don't simply test them independently. Learn about fractional factorial experiments and interactions, and design your experiment accordingly. You'll get a much more relevant result.

My impression is that companies like to add/test a lot of features separately - and individually these features are good, but together they form complex clutter and end up being a net negative.