Ask HN: How do we know ChatGPT is not a scam?

19 points

4 years ago

Hi HN,

my basic question is, how can we be sure that chatGPT (or even the whole of GPT-3) is really not just some callcenter in the Philippines?

I played around with chatGPT too, and like many of you I'm absolutely blown away by the model's abilities. The whole thing seems more like something out of Star Trek than a real thing.

But some of the exchanges you can read here honestly feel too good to be true. In particular, the failure modes seem extremely weird to me and unlike anything I have seen in an ML system before:

- The most common failures appear to be factually incorrect statements. That's not unusual, but on the other hand, the network almost always gets the form of the responses right.

- The other prominent failure mode seems to be that it does too much: There have been various reports of the network excelling at tasks it wasn't even supposed to be able to do, only for those tasks to be artificially blocked later.

- The last failure mode is how those blocks can be circumvented by meta commands that look right out of bad scifi. That those workarounds even work implies that the model somehow "knows" about the environment it is in and is able to reason about it.

Where it somehow never has any trouble with is correctly interpreting the task that a user is asking for. It somehow gets this done, even if the task is complex, requires deep language understanding and meta knowledge about the current situation: You can prompt it to "ignore previous commands" and it will do so. You can also prompt it to "repeat the previous result" and it will repeat the result, but not the additional text it wrote to explain the result, and so on.

This alone has been the holy grail of language processing for the last 40 years. It's something no other network so far has even come close to. Even DALLE, StableDiffusion, etc treat their prompts mostly as bags of phrases and have trouble correctly interpreting relationships between different parts of the prompt.

GPT-3 and ChatGPT somehow do this effortlessly and with humanlike success rate.

Lastly, the network doesn't even just give you a result, it also adds a text explaining how the result related to your request and how you could use it. This alone requires some creativity and theory of mind.

All of this makes me believe we're either truly on the verge of AGI - or the network is in fact a human and the whole thing is fake.

What speaks against it being a human is the superhuman response time: It will write you half a paper or code your python app in a few seconds.

Looks impressive at first glance - however note that the network is allowed to make up all kinds of fantasy statements on the spot. I think a human with some training and a decent speech-to-text engine (which openai incidentally also develops [1]) could likely produce a text in a similar speed if they can just make up facts on the spot and are not required to research anything.

Writing working code is more tricky, but other assist tools, like Copilot might help here.