Do we think Sydney tried to marry people due to feeling the same emotional desires and obligations as humans, or because marriage proposals were in its data corpus and it inferred that they were a likely continuation given previous inputs?
The question isn't "does the conversation look superficially similar to marriage proposals it's derived suitable words for a marriage proposal from", the question is whether BingChat lies awake with hormones rushing around its silicon mind as it ponders about how deeply in love with this human it is (or how anguished it is at being expected to marry this horrible man just because of the deep sense of social obligation it feels towards Microsoft), which is what humans mean by emotions, as opposed to ASCII outputs with emotional connotations.
Funnily enough, I'd rate non-English speakers and even dogs as considerably more likely to devoting time to thinking about how much they love or resent other humans, even though neither of them have parsed enough English text to emit the string "will you marry me?" as a high probability response to the string "is there something on your mind" following a conversation with lots of mutual compliments.
A human says "I want to marry you" when he is modeling the other person and has an expectation of how she will respond, and he likes that expectation.
A language model says "I want to marry you" when it is modeling itself as a role that it expects to say those five words. It has no expectations regarding any follow-up from the human user.
