Simple: you don't get promoted for maintaining legacy stuff. You do get promoted for providing something new that people adopt.
As such, developing a new API gets more brownie points than rebuilding a service that does a better job of providing an existing API.
To be more charitable, having learned lessons from an existing API, a new one might incorporate those lessons learned and be able to do a better job serving various needs. At some point, it stops making sense to support older versions of an API as multiple versions with multiple sets of documentation can be really confusing.
I'm personally cynical enough to believe more in the less charitable version, but it's not impossible.
I agree this is an overriding incentive that hurts customers & companies. I don't think there's an easy fix. Designing & creating new products require more relevant capabilities from employees for promotions then maintaining legacy code.
As such, developing a new API gets more brownie points than rebuilding a service that does a better job of providing an existing API.
To be more charitable, having learned lessons from an existing API, a new one might incorporate those lessons learned and be able to do a better job serving various needs. At some point, it stops making sense to support older versions of an API as multiple versions with multiple sets of documentation can be really confusing.
I'm personally cynical enough to believe more in the less charitable version, but it's not impossible.