Five pages in Automatica
In 1983 the journal Automatica published Lisanne Bainbridge’s paper “Ironies of Automation”. Five pages in volume 19. It is about control in process industries, with some examples drawn from flight-deck automation. There is not a word in it about offices or customer chats.
The question it asks: what happens to the human when a machine takes over their work. The answer is awkward. The more advanced a control system is, the more crucial the operator’s contribution may turn out to be. Whatever was foreseen, the system handles by itself. The person is left with what nobody foresaw, and that happens rarely.
In 2021 Bainbridge put the full text on her own site. The quotations from her below are taken from there.
The bot takes the easy part
In the 2021 version of the paper it reads: “By taking away the easy parts of their task, automation can make the difficult parts of the human operator’s task more difficult” (Bainbridge, “Ironies of Automation”).
An operator who ran the process by hand knew how the plant behaved on an ordinary day and noticed a deviation from small signs. After automation, the same person watches instruments on which nothing happens. Bainbridge points to research on attention: even a highly motivated person cannot keep watching a source where very little happens for more than about half an hour.
Now swap the plant for the Telegram account of a shop or a clinic. The bot answers whatever repeats: how much is it, where are you, do you have this size, when is there a free slot. These questions go to the bot first precisely because they are all alike.
The manager gets the rest. A customer was sent the wrong item. Someone asks about something that is not on the price list. A regular wants a discount. Someone sends a third message in a row in capitals. These conversations used to arrive mixed in with a hundred simple ones, and the manager had a feel for the day: what people were asking about, what had run out, what mood customers were in. Now the manager opens a chat when it has already gone wrong.
The hard part stays, the skill goes
In the same paper Bainbridge names the second irony, which explains where the human’s job in an automated system comes from: “the designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate”.
So nobody designed the human’s job. It is whatever was left over. With a chatbot it looks like this: anything that did not fit into the script and the knowledge base goes “to the manager” by default. Leave it at that, and nobody works out in advance whether the manager can cope, or how many minutes they have for it.
Then there is skill. Bainbridge writes that physical skills deteriorate when they are not used. For plants she suggests giving the operator a short spell of hands-on control every shift, and simulator practice if that is not realistic.
We have no measurements for sales managers. Our assumption is this: someone who has not answered a simple question for six months remembers the price list, the delivery terms and the way ordinary happy customers write less well. A hard case needs all of that at once.
In 2024 researchers posted “Ironies of Generative AI” on arXiv, applying the same ironies to generative models. Forty-one years on, the argument was needed again.
A dead end with no human: DPD and Taco Bell
Two public stories show what happens when there is no hand-off at all.
January 2024, the UK parcel firm DPD. A customer, a London-based pianist and conductor, asked the chatbot to put him through to a person. The bot said it could not. So he started playing with it, and got it to swear, to write a poem about how useless DPD was and to call DPD the worst delivery firm in the world. According to TIME, the screenshots he posted on X drew 1.3 million views and more than 20,000 likes. DPD said the error appeared after a system update on 18 January, that the AI element was immediately disabled and was being updated, and that it had run in the chat successfully for a number of years.
The second story is from the US. Taco Bell put voice AI on order-taking at more than 500 drive-throughs. In August 2025 TechCrunch mentioned an order for 18,000 cups of water, placed to get past the AI and reach a human. Taco Bell’s chief digital and technology officer, Dane Mathews, told The Wall Street Journal about the system: “Sometimes it lets me down, but sometimes it really surprises me.” The chain has not dropped AI. Mathews said franchisees will be coached on which restaurants and which hours should use voice AI, and where staff should monitor it and step in as needed.
In both stories the customer started breaking the bot after failing to reach a person. When the door to a human is shut, customers find their own way round, and the company reads about it in the news.
A cord above the line
At Toyota the right to stop the machine is built into the production system itself. The principle is called jidoka. The official Toyota site describes it as “automation with a human touch”. When something goes wrong, whether a machine fault, a quality problem or a work delay, the machine detects it and stops on its own. Or the operator stops the line by pulling the cord. A stop here is a normal function, not a breakdown.
For a bot, that cord has two parts. The bot can stop by itself and call a person. And the person can take the conversation over at any moment without switching the whole system off.
The second issue is speed. In 1960, writing in Science, Norbert Wiener discussed machines that act so fast and so irreversibly that nobody can step in before the action is complete. With such machines, he wrote, we need to be quite sure that the purpose put into them is “the purpose which we really desire and not merely a colorful imitation of it” (Wiener, Science, 1960).
A poor bot reply in a chat can be put right with the next message. A mass mailing to the whole customer base cannot: by the time someone spots the wrong price, everyone has it. So the check on a mailing comes before it is sent.
How to design the hand-off
The conclusion for a first-line bot: the hand-off to a human is designed at the start, together with the script. Leave it to the end, and the human gets exactly the leftovers Bainbridge wrote about.
The reasons for a hand-off are written down in advance, as a list. For example:
- the customer asks for a person outright, or asks the same question for the third time;
- money that is not on the price list: a discount, a refund, instalments;
- a complaint about an order that has already been paid for;
- the customer is ready to pay or is in a hurry;
- a photo or a file arrives that the bot cannot read.
The bot tells the customer what is going on: that it is a bot, that the conversation has gone to a person, and when to expect a reply. If the bot simply goes quiet, the customer cannot tell whether to wait or to write somewhere else.
The manager receives the conversation with its context: what the customer asked and what the bot has already said. Otherwise the person’s first message asks again for something the customer already wrote. Who answers for the bot’s words is a question we covered separately.
A person takes over a chat in one action, and the bot goes quiet in that chat. The manager’s skill is maintained on purpose. The equivalent of Bainbridge’s hands-on control every shift: the manager regularly handles some routine requests personally, or reads a sample of the day’s bot replies.
Some of this already works in Valli, our AI auto-responder for Telegram Business, which is currently in pilot. If the owner writes in a chat themselves, Valli stays silent in that chat for two hours. The hot-lead card has a button that keeps Valli silent in that chat for 24 hours. When a customer agrees to the price, asks for an invoice or is in a hurry, Valli’s rules tell it to say the owner will join, and the owner gets a card with the customer’s message and Valli’s reply. Valli does not confirm meetings; the owner does. If a customer sends a photo or a file without a caption, Valli calls the owner, because it cannot yet answer from images or files. Asked whether it is a bot, its rules forbid it to lie.
What we don’t know
Bainbridge wrote about plant operators and pilots, not about language models. Carrying her conclusions over to chatbots is our analogy; the paper itself does not make it.
We have no measured data on how a first-line bot affects sales managers’ skills. Nor do we have hand-off statistics from the Valli pilot yet.
We did not open the DPD customer’s original post on X; the story and the view count are retold from TIME. Dane Mathews’s words come from TechCrunch (which spells his name Matthews), quoting his interview with The Wall Street Journal; the WSJ original is behind a paywall.
How we draw this line across a whole project — from choosing a process to handover — is on AI business automation.
Before you put a bot on incoming messages, write down which conversations it hands to a person. That is a line of its own in our free process breakdown: 48 hours, no intro call, and the boundary between bot and manager is written down before launch.
