We've actually just started having the pain of Mongo migrations; adding indexes to large collections. If you just create the index on the primary and let it propagate throughout the cluster, then it will block further oplog replication on the secondaries and the whole cluster will go down for majority writes. The proper way is to take each secondary out of the cluster in turn, add the index, then step-down the primary and migrate it.
It's just as much of a pain as migrating your favourite SQL database, although at least you don't need to do it for small collections and for adding columns (which is a big bonus).
Collections with tens of Gigabytes and hundreds of millions of records can take a few hours per-node to build an index. You'll need to think about index migration strategies before that point though. I'd guess that once you have gigabytes of data in a collection you'll probably have less downtime with an election than you will waiting for the index to build.
We've actually just started having the pain of Mongo migrations; adding indexes to large collections. If you just create the index on the primary and let it propagate throughout the cluster, then it will block further oplog replication on the secondaries and the whole cluster will go down for majority writes. The proper way is to take each secondary out of the cluster in turn, add the index, then step-down the primary and migrate it.
It's just as much of a pain as migrating your favourite SQL database, although at least you don't need to do it for small collections and for adding columns (which is a big bonus).