Peter and I are writing a book on artificial intelligence together, and for months that work meant studying AI influencers: how they are built, how they gain followers, why some feel more convincing than others. At some point Peter built one of his own. He called her Lea.
Lea exists for research. She has no personality of her own, only a set of design choices Peter made one at a time: facial expressions calibrated to read as natural, a wardrobe, an apartment, a stated preference for being outdoors. Every detail was deliberate, chosen to make her feel like a person.
It worked on us before it worked on anyone else. Both of us had to remind ourselves, actively and more than once, that Lea has no personality. Entire stories grew out of her design that we seriously bought into, the way you might believe a character in a novel, except we had written the code ourselves that morning.
I laughed about it at the time. We both did. The obvious reference is Pygmalion, the sculptor who carves a statue, knows it is stone and falls for it anyway. The myth is thousands of years old, and it struck us as funny that the same thing was happening in an office, at a screen, with two people who write dialogue for a character and then start talking to her as if she could answer. We kept talking to her after the test was over.
It took me a long time to understand what that admission actually meant.
The question that stayed with me was about us more than about Lea. The people who built the illusion could point to the exact line of code behind every expression, and they fell for it anyway. What does that leave for everyone who meets Lea with none of that knowledge? A warning label, something like “this is an AI-generated person”, was my first instinct for a fix. I no longer think it would have helped. We knew exactly how she was built, down to the code, and that knowledge changed nothing about how we felt.
The likelier explanation is biological rather than moral. Human beings are wired to respond to faces, to gaze, to a voice that sounds familiar. That response fires in milliseconds, before conscious thought can catch up, and settles whether a face reads as friend or foe, familiar or strange, on its own. A character designed well enough sends the right signals in the right order, and the response follows regardless of what the viewer knows about how it was made. An adult who knows with certainty that Lea is a machine still feels something when she smiles.
It sounds almost tender, put that way, like a charming side effect of the programming work: look how real Lea has become. I read it differently today. If the two of us, both technically fluent in exactly how Lea was assembled, could not talk ourselves out of the response, the people who encounter her with no idea she was built at all have far less protection than a disclaimer could supply.
Neither knowing the mechanism nor building it yourself disables the response. Psychology and neuroscience probably explain each step of that response in detail, but the explanation does not change what happened in that office. By the end the two people who made the illusion believed it too. What to call a deception that takes in its own makers is a question I have not settled.