For the past few weeks my Twitter timeline has looked like a magic show. Someone types a sentence in plain English, “I want a red button shaped like a watermelon”, and gets back the code that draws the button. Someone else asks for an essay on the meaning of life and gets three paragraphs that read like a decent philosophy student wrote them. Others post poems, dialogues, contract summaries. All written by the same program: GPT-3, from OpenAI.
I watch these demos with two feelings pulling in opposite directions. On one side, the excitement of someone who spent years working on automation and personalization in e-commerce, and who knows how hard it is to get a machine to “talk” naturally to a customer. On the other, the scepticism of someone who has done publishing, from Știri de Bine (a Romanian good-news site I ran for years) to IconicMan, and who knows how easily the internet fills up with worthless text. So I tried to sort out what we know and what we don’t, as of late August 2020.
What GPT-3 actually is
At the end of May, OpenAI published a research paper with a fairly modest title, “Language Models are Few-Shot Learners”. It described a language model with roughly 175 billion parameters, more than ten times larger than any previously published model of its kind. In June they opened an API in private beta to a limited number of selected developers. That is where the demos that went around the internet in July came from.
Context matters: OpenAI started as a non-profit, then created a capped-profit structure, and in 2019 Microsoft announced a large investment in it. So this is not a garage experiment. It is a project with serious money and serious ambitions, and one that is also looking for a business model.
How it works, in plain words
A language model does something that sounds trivial: it takes a piece of text and guesses the most likely next word. Then the next one. Then another. That’s it. To guess well, it has been trained on enormous amounts of text from the web, books and other sources, and it has “learned” statistically how words, sentences and ideas connect.
What is new about GPT-3 is not the principle, it’s the scale. Once a model like this gets big enough, it starts doing things it was never explicitly trained for. Give it two or three examples, “translate this”, “write a product description in this style”, and it continues the pattern. That’s the “few-shot” in the title: it picks up a task from a handful of examples, not thousands.
Why it sounds so human
Because it has read a staggering amount of text written by humans. It knows what a blog post looks like, or a complaint email, a poem, a press release. Give it the beginning and it hands you an ending that fits the genre perfectly: tone, structure, even the clichés.
It has been reported that a student started a blog with posts generated almost entirely by GPT-3, and that one of them reached the top of Hacker News, a forum full of tech people, without many readers noticing. If the story is accurate, it’s a signal: we can no longer assume that fluent text has a human behind it.
The limits the demos don’t show
Viral demos are, by their nature, the best results, picked from many attempts. Anyone who has worked in tech knows a demo and a production system are two different worlds. A few limits are already visible:
- It doesn’t understand what it writes. It produces plausible text, not true text. It has no model of the world, only statistics about words.
- It makes things up with total confidence. It can cite a study that doesn’t exist, or put words in someone’s mouth, in the same calm tone it uses when it’s right.
- It absorbs the biases in its data. If the internet is full of stereotypes, the model will reproduce them. The authors acknowledge this in their own paper.
- It is expensive. Training a model like this takes resources only a few companies in the world can afford, and running it isn’t cheap either.
- It loses the thread on long texts. Over a few paragraphs it is convincing; over a full page it starts repeating or contradicting itself.
The risks: spam and fake news on an industrial scale
This is where my scepticism turns into worry. I’ve already written about the algorithms that decide what we see online. Now we are adding algorithms that can also write what we see. If someone with bad intentions can generate thousands of fake articles, reviews, comments or posts that sound credible, the cost of disinformation drops close to zero.
It is no accident that OpenAI has kept access closed and talks about responsible use. Last year they also hesitated to release the full previous version, GPT-2, precisely for fear of misuse. The question isn’t whether this technology will be used for spam, but how fast, and how well we’ll defend ourselves. For search engines and social networks, it’s a serious problem that is only just beginning.
The opportunities for marketing and e-commerce
Now the part that excites me. When I was working on Retargeting, the hardest part wasn’t the technology for sending messages, it was the content of those messages. Writing relevant variations for every type of customer is a huge amount of work. Back then I wrote about e-commerce personalization beyond the first name and about chatbots that promised a lot and delivered little. A model like GPT-3 could, in theory, change that equation:
- Product descriptions. Say you run an online store with a few thousand products and descriptions copied from the supplier. A language model could produce unique first drafts, which a human then checks and edits.
- Drafts for customer support. Not answers sent automatically, but suggestions an agent approves, so they can reply faster.
- Ideas, not final copy. Headlines, article topics, email subject lines, campaign angles. A brainstorming partner that never gets tired.
- Variations for testing. Ten different phrasings of the same message, from which you pick the good ones and test them on real customers.
I want to stress this: it’s all very early. The API is in closed beta, pricing isn’t clear, and quality in languages other than English is an open question. For a Romanian store, for instance, we simply don’t know yet how well it would write in Romanian.
What it means for people who write
In my publishing experience, good writing has always had two parts: form and substance. GPT-3 seems very good at form. Substance, meaning verified information, real experience, an opinion you stand behind, it can’t really provide, because it has nothing to say from experience of its own. I argued in my piece on content marketing that sells that people come back for trust, not volume. I think that still holds.
What will probably change is the price of mediocre text: it will head towards zero. Anyone who makes a living filling pages with keywords has reason to worry. Anyone who has something to say may have just gained a new assistant.
My conclusion, for now
I’m excited and sceptical at the same time, and I think that’s the healthy attitude. GPT-3 is the most impressive thing I’ve seen in AI applied to text, but it isn’t a mind. It’s a giant mirror of what we humans have written, with everything good and bad that comes with it. I don’t know how fast it will make its way into real products or what it will cost. I just know it’s worth watching closely, with eyes wide open.
Have you tried GPT-3, or seen demos that convinced you? Would you use it in your business, or are you more worried about what it could do in the wrong hands?