"What happens when an LLM outputs a patented algorithm?" remains a huge land mine out there, particularly since patent infringement does not require intent or even knowledge, and these models have trained on every patent ever granted.

If you can prove that your LLM did not learn from the patent (eg cut-off for learning was before), then the LLM outputting the algorithm (or product etc) would be pretty good evidence that a practitioner of ordinary competence in the field, or whatever the exact legal wording is, found the whole invention to be trivial.

It doesn't work when that happens with humans so it absolutely wouldn't work when it happens with a machine

Patents do to a small extent, maybe. But eg medical patents are a lot about protecting all the 'sweat' you put in, not so much the creativity.

You don't need a realization to receive a patent, just the drawing.

I'm not quite so sure for medical patents.

I mean, why are patent trolls not getting patents for all compounds under the sun for all conceivable medical uses?