This is a longstanding problem in audio engineering, along the lines of "how does one un-bake a cake to get the eggs?" There's always going to be artefacts and distortion ranging from unpleasant to extreme, hence audio engineers when mixing from stems/channels will do a stereo instrumental mix and a mix with vocals.