You end up hitting scaling issue the agency hasn't hit before, and they're learning on your dime. That scaling could be technical, but also organizational and procedural. Your org may be big enough to have specific legal/regulatory issues to contend with, but the agency has never dealt with those before, for example.

I understand people will almost always be 'learning' in some capacity on every project. "Hey, we're constantly learning! This is great!". I recognize it, but don't always think it's something to celebrate. You'll usually be better off working with an agency that's dealt with your size project/org before.