My concern would be the same as with a Junior Dev. It's fine if they don't know something but please don't bullshit/make something up. Maybe tech questions are immune to this but it doesn't seem to have any conception of true and false.
This is a silly example but I asked for a list of Star Trek references in Seinfeld and it gave 6-7 examples that all sounded genuine (Jerry makes a joke about transporters in the Contest episode) but were 100% made up. If I wasn't super familiar with the show I wouldn't have been able to tell most were invented. With code generation that's less important because we have ways of testing code for "truth" but I would worry about relying on any factual statements from the thing.
Sure, humans are not obsolete just yet! People are needlessly freaking out about losing their jobs or even reason for existence to AI. The reason for employing developers or designers is not that we need someone to write loops and draw lines.
It's a tool that makes some things significantly more easier and it does have risks. It will replace only people who are doing jobs not suitable for people.
The problem is that most of us work for companies ACTIVELY TRYING to use this stuff to replace human beings, INCLUDING THE ACTUAL DECISION MAKING and slapping black boxes everywhere in their bureaucratic processes because "machine says no" is a SUPER beneficial thing to a business, especially the modern massive corporation that doesn't really have competition and is mostly profitable because it ignores problems.
I give it five years until even we here are unable to get the attention of a tech company employee to fix our wrongly locked out account because not only are the primary touch points """automated""" but the appeals and appeals of appeals are also entirely automated.
Everyone here will get to enjoy "machine says no" a hundred times more often. Every tech support or bill support or anything support will put you through half an hour of terrible "AI" interaction before you are even allowed to be routed to a human, and businesses will use this to get rid of even more call center employees.
Hate not being able to understand the thick accent in your support call? Get ready for 1% of the time the AI in the phone call puts together sounds that don't actually form words, or just straight up misunderstand what you say, and nobody will believe you because "it's so accurate". Get ready to be gaslit by your fucking phone.
The prevalence of “computer says no” or “machine says no” in our modern society makes me scream and can honestly send me into depression and anxiety. It creates the most helpless feeling.
I have an increasing pile of issues with companies’ services that just persist because fixing them requires getting in contact with a human and a human who actually knows something and isn’t just a terminal for the computerized system.
20 years ago, we could have had the same conversation about outsourcing customer support. It was rife with problems and limitations, just as AI today is.
But that didn't stop companies from doing it anyway. The C-suite isn't listening to researchers and the general public, they're looking at what their shareholders think they 'should' do. Once McKinsey and BCG whip out their "AI Digital Transformation" powerpoints, it's over.
This points out the most dangerous part of ChatGPT. It's a highly confident idiot. When I told it was explicitly wrong, it basically responded with "I'm sorry it wasn't clear".
This is a silly example but I asked for a list of Star Trek references in Seinfeld and it gave 6-7 examples that all sounded genuine (Jerry makes a joke about transporters in the Contest episode) but were 100% made up. If I wasn't super familiar with the show I wouldn't have been able to tell most were invented. With code generation that's less important because we have ways of testing code for "truth" but I would worry about relying on any factual statements from the thing.