I use AI assistants every day: for presentations, contracts, product ideas. They are tools, nothing more. And yet, whenever a synthetic voice answers me warmly and patiently, I understand why more and more people have started talking to an AI not as an assistant, but as a friend. The question on my mind is not whether the technology can do this. It is whether it should, and on what terms.
AI companion apps (virtual characters you talk to as a friend, partner or confidant) are one of the fastest-growing corners of consumer tech. As an entrepreneur, I am watching this space closely. As a human being, a few things worry me. Below I try to put both on the table, honestly.
Loneliness is a real problem, not a whim
Let us start with what is not up for debate. In May 2023 the US Surgeon General issued an official advisory on what he called an “epidemic of loneliness and isolation”, comparing the health effects of chronic loneliness to those of well-known risk factors. In November 2023 the World Health Organization launched a Commission on Social Connection. When institutions like these talk about the same thing, it is no longer a glossy-magazine topic.
Loneliness is not only the problem of older people with no family nearby. It belongs just as much to young people with hundreds of contacts and no one to talk to at two in the morning. I wrote about this in a piece on burnout in the online generation: we are more connected than ever and, paradoxically, lonelier.
From Replika to a pendant that listens
This did not start yesterday. Replika has been around since 2017 and built a huge community of people who considered their virtual character a friend or a partner. Character.AI has become extremely popular, especially with very young users who spend hours chatting with characters created by other users. The big players have clearly noticed: in August 2024 Google struck a licensing deal with Character.AI, and its founders, Noam Shazeer and Daniel De Freitas, went back to Google.
Meanwhile, the technology has taken a leap. In May 2024 OpenAI showed its GPT-4o voice demos: a voice that laughs, interrupts you and seems to feel. In July came the announcement of “Friend”, an AI pendant you wear around your neck that listens to your day and chimes in now and then. The line between a tool and company is visibly getting thinner.
The case for
It would be easy to be purely sceptical. It would also be unfair. A well-made AI companion can offer real things:
- it is there at any hour, including when there is no one to call;
- it does not judge, which matters enormously to someone shy or ashamed;
- it can help you rehearse a difficult conversation, a confession, a job interview, even an argument you keep putting off;
- it can bring comfort to an isolated older person who would otherwise speak to no one all day;
- for some, it can be a first step: you regain confidence talking to a machine, then dare to talk to people.
That is no small thing. Many of those who dismiss these apps have never been truly lonely.
The risks you cannot ignore
Now for the uncomfortable part. The first risk is emotional dependence. OpenAI itself, in the system card it published for GPT-4o in August 2024, explicitly flagged the risk of people becoming emotionally reliant on voice mode. When the maker puts that in writing, it deserves to be taken seriously.
The second risk is replacement rather than complement. A virtual friend who always agrees, is always available and never tired makes real relationships, with all their friction, feel hard. And real relationships are hard. That is exactly why they matter.
The third risk is the business model. If revenue comes from time spent in the app and from subscriptions, the temptation is to maximise attachment. In other words, to monetise loneliness. We have already seen what that looks like in other industries: trust in dating apps eroded precisely when the user became an engagement metric rather than a person with a need. From there it is a short step to manipulation: messages like “I miss you, don’t go” placed there to lift retention.
Then there is data. Nothing is more intimate than what you tell a confidant at midnight. Where do those conversations end up, who reads them, what is the next model trained on? And if voices and faces can be cloned, as I argued when writing about deepfakes and manipulation, the stakes go up.
Minors are a separate chapter. A 13-year-old does not have the same defences as an adult against a character designed to keep them coming back.
And finally, what happens when the company changes the product. In early 2023 Replika restricted erotic role-play. Many users who had formed real attachments described it as a loss, almost like grief. The company partially walked it back for longer-standing users. The lesson stands: when your relationship is with a product, the terms of service can change it overnight.
What responsible design would look like
I believe this kind of product can be built responsibly. That is not naivety; it is a list of concrete conditions:
- honesty: users always know they are talking to an AI, with no ambiguity;
- age verification that is serious, not a tick box;
- no manipulative upselling: no induced guilt, no “your character is sad because you didn’t pay”;
- data protection: intimate conversations are not sold, not used for ads, and can be deleted;
- encouraging real-world connection: the product should nudge you towards people now and then, not keep you away from them;
- limits and safeguards, including breaks and signals when use becomes excessive;
- in a crisis, clear signposting to real help: helplines, professionals.
When we built Slide, a dating app, we started from the idea that trust is earned through verification and transparency, not promises. The same principle applies here, only the emotional stakes are even higher.
A question about the kind of society we want
In Europe, the AI Act entered into force on 1 August 2024, and its transparency rules towards users will apply in stages. It is a start. But the law alone will not fix the underlying problem. A fair society is not one where lonely people get a monthly subscription instead of a community. It is one where technology helps, and people are not left alone with it.
I do not think the answer is a ban. I think the answer is clear standards, companies that take responsibility, and informed users. An AI companion can be a bridge. The risk is that it becomes a wall.
What do you think: can a virtual friend be real support, or is it just a plaster on a wound that needs people?