Last September I wrote about AI companions, loneliness and ethics. I raised more questions than I answered, and I implicitly promised not to leave them hanging. Since then, those questions have become very concrete for me: I am involved in building Desirex (desirex.app), an AI companion app.
I know how that sounds. An entrepreneur writes a piece about ethics, and at the end it turns out he has a product in exactly that space. So let me be clear from the start: this is not a launch announcement and it is not an ad. It is an attempt to write down what we have set ourselves and where we see the tensions, so that we can be held to our word.
Why build an AI companion at all?
The first question friends asked me was simple: “Why on earth would you go there?” My answer has two parts. First, the category already exists and is growing, whether we like it or not. People talk to AI not just to draft their emails but to feel heard. That will not stop because some of us turn up our noses.
Second, if this category is going to exist anyway, it matters enormously who builds it. It can be built by people who ask, at every decision, “Is this good for the user?” or by people who only ask, “Does this increase time in the app?” I would rather be among the first and make honest mistakes than leave the field to those who never ask the question.
Principle one: we do not pretend to be human
We have set ourselves the goal that users always know, unambiguously, that they are talking to an artificial intelligence. Not just in a paragraph buried in the terms and conditions, but in the experience itself. An AI companion can be warm, attentive and pleasant. But the moment it claims to be a person, the relationship is built on a lie, and a lie cannot be the foundation of a responsible product.
Beyond conviction, this is also a legal matter. The EU’s artificial intelligence regulation, which I wrote about when the AI Act was adopted, includes transparency obligations: people must be told when they are interacting with an AI system. These obligations are being phased in, but we see no reason to wait for the deadline.
Adults only, with age verification
We have set ourselves the rule that the app is for adults only, with real age verification, not a checkbox any twelve-year-old can tick in two seconds. From my experience with Slide (a dating app I helped build), I know how hard it is to verify who is actually behind a screen; I wrote back then about verified profiles and how to fight fake profiles. Verification adds friction and inevitably costs you some users. That is a price we accept.
Intimate conversations deserve serious protection
People tell an AI companion things they do not tell even those closest to them. That turns data protection from a compliance box into a moral obligation. We have set ourselves a few simple rules:
- we collect only the data we genuinely need for the service to work;
- we encrypt conversations and strictly limit who can access them;
- we do not sell user data or treat it as a commodity;
- users can delete their conversations and their account, and deletion means deletion, not “archiving”;
- we treat GDPR not as a chore but as a minimum standard.
This is not theoretical. In 2023, Italy’s data protection authority ordered an AI companion app to stop processing the data of Italian users. Regulators are watching this category closely, and rightly so.
No manipulative design
This, I think, is the hardest part. An AI companion holds the most powerful persuasion tool ever invented: a voice that knows you, answers at any hour and seems to care about you. Put it to work for engagement metrics and you can do real harm.
That is why we have set ourselves not to use guilt as a retention mechanism (no “you left me all alone” messages), not to use dark patterns that push people to pay or keep chatting longer than they want to, and to make cancelling a subscription as easy as starting one. More than that, we want to encourage healthy use, with gentle nudges when someone is spending a great deal of time in the app, and to encourage real-world relationships rather than compete with them. A good companion should sometimes send you out to see your friends.
When someone needs real help
An AI is not a therapist or a doctor and must not present itself as one. But signs of crisis can surface in conversations, and in 2024 the international press reported painful cases linked to platforms in this category. We have set ourselves clear safety guardrails and, when such signs appear, to point people towards real help: emergency lines, professionals, other humans. We will not get everything perfect from day one. But this is an area where we would rather be overly cautious.
Transparency when the product changes
Digital products change. Features come and go, AI models get replaced, policies get rewritten. In an ordinary app, that is an annoyance. In a companion someone has grown attached to, a sudden change can feel like a loss. We have set ourselves to announce important changes in advance, explain what is happening and why, and give people the option to take their data or leave.
The tensions I will not hide
It would be hypocritical to pretend all this comes at no cost. There is a real tension between a business model and the user’s wellbeing. The industry’s classic metrics, time in app, sessions per day, retention, can end up pulling against what is healthy for a person. We have not solved that equation for good. What we have set ourselves is to measure more than engagement and to be willing to give up growth where it comes from someone’s vulnerability.
There is also the reputational risk. AI companions are viewed with suspicion, sometimes deservedly. I have been told bluntly that this is not a “clean” space to attach my name to. I understand the argument. But with Slide, a dating app built around real people, I saw that you can enter a category with trust problems precisely in order to do things differently. I cannot guarantee we will succeed. I can guarantee we will try, and that we will say publicly where we get it wrong.
What comes next
I have written all of this as intentions, not achievements, because that is the honest way to put it. A principle written on a blog is worth nothing until it survives its first encounter with commercial pressure. I want this text to be a marker you can come back to in a year and ask me: “Did you do what you said?”
Until then, I will ask you: what would an AI companion have to do for you to trust it? And where would you draw the line it should never cross?