Of course the loneliest 5% are going to do something like this. If it weren't for AI they'd be writing twilight fan-fic and roleplaying it on some chatroom, or giving all their money to a "saudi prince."
Seems like nothing new, just a better or more immersive form of fantasy for those who can't have the life they fantasize about.
I'd argue it'd be psychologically healthier to roleplay in a chatroom with people who are human on the other end (if that could be guaranteed, which it no longer can).
Humans can potentially be much nastier than a chatbot. There are lonely vulnerable who can be exploited, but there are also people who get off on manipulating other people and convincing them to make profoundly self destructive and life altering choices.
Sure, but Google and OpenAI aren't going to do this kind of manipulation. And the actual sadists are probably not as satisfied with letting a machine do it.
Seems like nothing new, just a better or more immersive form of fantasy for those who can't have the life they fantasize about.