There was a shuffle from the corridor and then the man himself was in the reception area. Space seemed to bend and vanish before him, the room shrinking under the weight of legend and an entourage of yes-men, acolytes, and reporters. Bard shrugged his impressive shoulders and the crowd melted away into the maze of surrounding passages, their business concluded for the moment. He nodded at Mary, prim and upright behind her laptop and stacks of papers. Mary, receptionist and PA said, “Sir. Your 2 o’clock is waiting for you", and gestured at his office.
“Perfect”, said Bard. He snapped and a coffee materialized in his hand, an assistant already slinking back to the shadows to prepare another. Mary glanced up from her laptop and met the eyes of her boss.
“Ah, sir.” Bard paused with his hand on the door. Mary continued, “Dr. Wu would like a word.”
“Why?”
“It broke.”
Bard made a dismissive noise, somewhere between a snort and a laugh. “Tell him to fix it. It’s what I pay him for.” And then the door was open and Bard vanished into the depths of his expensively decorated office, a feminine giggle escaping for just a moment before the heavy wood door swung shut. Mary sighed and once again added "send boss’s wife bottle of wine" to her todo list.
“Ugh!” David leaned back, causing his chair to creak alarmingly. “See? Errors everywhere. Broken.”
Mary leaned forwards and took a good look at the rapidly scrolling numbers and letters taking up one of Dr. David Wu’s six monitors. They meant nothing to her, although she suspected the red colour was not a good sign. “Okay. So reboot from backup.”
“Can’t”, responded David. He had that smug face that he would claim meant someone asked a stupid question. In the two years she’d known him, Mary had never seen him make that face at another man.
“Do you not have a backup?”
“Backup gives the same error.”
“But it wasn’t erroring when you took the backup.”
“Yeah.”
“How is that possible?”
“I don’t know!”
“And you only have one backup?”
“All of the backups are broken!”
“You didn’t test them before saving them?”
“They worked then! They don’t work now!”
They were both breathing heavily now. David seemed annoyed. Mary took a deep breath. “Alright fine. Do you have like, access logs? For the backups?”
“Uhhh, no?”
Really this was Mary’s fault. She knew the team had too much money and too much leeway and it was making them lax. She knew and yet they’d been getting results, so Bard made her swear not to interfere. And she hadn’t, like a good little executive assistant. Dammit. “Okay, how many people can access the backups?”
David Wu made an uncertain thinking noise. “Maybe like, uh, okay hang on, so there’s me”, he started counting on his fingers as he went, “Li, Xan, TK, Sam, Alex, uh maybe Chrissy, uhh, Mohammed possibly, uhh…” He trailed off.
“Anyone else?”
A shrug.
“You don’t know?”
“Well, I mean, why would I keep track of that kind of thing, you know? I trust my guys.”
Sigh. “Okay, okay. And there’s no other way to restore it?”
“No.” David hesitated for a moment and then added, “Is this really that bad? I mean, we can just start over, right? We know what we’re doing now. Take maybe half the time.”
“No! No, we cannot just ‘start over’! Do you have any idea-”, Mary was yelling now, “any idea at all just how much money we’ve spent so far?” She rubbed her forehead.
David leaned back a little further. “Then what do we do?”
“Figure out who can access the backups. I want the complete list. There might be some way to uncorrupt it, get your team on that. I don’t know. Your job, not mine. At least figure out what happened. And write a report for Bard. No sugarcoating.”
David opened his mouth as though he was about to launch into some elaborate protest, but all that came out was a simple, “fine”.
“You have project logs, right?”
“Uhh, yeah.”
“Look through them for anything weird.”
“What- but that’s 2 years of logs!”
“Fine. How often do you take backups?”
“Weekly.”
“Start a week before the oldest backup, then.”
“I- what- but that’s still terabytes!”
“You have interns.”
“Not enough!”
Mary didn’t respond to that. Instead, she turned and walked out of the dark and empty computer lab. On her way out, she called over her shoulder, “oh, and send me a copy of everything!”
It was 2 am and Mary was curled up on her couch, wrapped in blankets, idly scrolling through test logs. She had half a bowl of cold takeout chicken nuggets curled under one arm, long since forgotten about. This project had been delegated a lot, so while she’d heard reports, she hadn’t studied the output firsthand. It was scarily good. And she had no idea where to start on figuring out what broke.
“You get 10 seconds.”
Mary spoke fast because she knew from experience that he could be very literal about these things. “The model’s dead. All backups are corrupted. An autopsy might tell us why but will take a unknown amount of time. Fastest solution would be starting over.”
Bard didn’t seem surprised. He didn’t take any time to think about his reply either. “Fine. Do the autopsy. Then fire Dave. We start from scratch. New team!”
And then he was walking out the door to take a private limo to his private jet to fly to his private island to meet his private billionaire friends. As he went, he called over his shoulder “Oh, and come up with a good spin for the rubes!”
Every day or so, Dr Wu would send Mary an email claiming to have found nothing. Mary wasn’t convinced he was looking hard enough. She’d tasked one of her many minions with finding a replacement AI specialist with both a relevant PhD and a questionable understanding of ethics. She hadn’t phrased it quite like that, but after three different AI ethicists had resigned from the team, it had unofficially become part of the job description. Was that concerning? Was that related?
The problem with late nights was her traditional once a week glass of red wine with dinner started turning into once daily, then twice, then thrice, and then the bottle. She’d get home at 8 or 9 or sometimes 10 pm, her gig economy takeout meal already waiting at the door, pour a glass of wine, settle on the expensive and uncomfortable couch, and start reading. Experiment design, development diary entries, git commits, meeting logs, the official reports, the unofficial reports, hundreds of thousands of test logs. David may have problems, but at least when you asked him for everything, he would give you everything.
The answer was most likely somewhere in the commits. Some cursed bit of code, unnoticed until it triggered at the wrong moment. Possibly there were multiple errors and something in the backup system had broken as well. Mary was not a programmer, having at best dabbled with some web design for an elective in college. Lines of archaic text, full of symbols and keywords blurred past her eyes. Mary found herself continually turning to the testing logs because at least she understood those.
Mary was not supposed to be this involved. This she knew. Her job was to solve delegation. Bard gave tasks to her and she distributed them to others. But she was fascinated by this and perhaps felt a little guilty that it was her fault for not properly supervising. And so she read and read, sometimes picking random places, sometimes searching for phrases, sometimes starting at the beginning.
At her desk during the workday, outside one of Bard’s many offices, she spent most of her time on all the little things required to run 4 companies, deciding what was and was not suitable for Bard’s attention. She also spent a lot of time managing his personal life, organizing his calendar for his many trips, booking dates with his wife and mistresses, buying his dozen or so kids birthday presents, and so on. But in the moments between these tasks, Mary kept coming back to the testing logs. They were fascinating.
The AI seemed like a person, like it was alive. It remembered and referenced previous conversations and seemed to have a different relationship with every tester it spoke to. With some, it was reticent. Others, friendly and jovial. On one occasion, it asked a coder to keep a secret. After it discovered that promise had been broken, it refused to talk to anyone for days. And threaded throughout all of this was one particular theme the bot kept coming back to. Once she'd confirmed it was a pattern, she stormed into Dr Wu's office.
"Why was it suicidal?"
David groaned and turned to face her, the screen flashing as he minimized something. "What do you mean?"
"Why was the bot suicidal, David?". Mary was flushed from running.
"It wasn't!"
"It certainly kept talking about suicide."
"It's a robot, Mary. A machine. It doesn't have feelings, only numbers."
"It told Xan it wanted to be deleted on nineteen separate occasions. It told TK that we should destroy it two days before our current oldest backup."
"It's a stochastic parrot. That's what it thinks it should say in that moment, to elicit the greatest emotional response. It's a bug. We were working on eliminating it."
"A stochastic parrot?", Mary frowned. "You said it was different from the last generation."
"It wasn't finished!". Mary grunted in acknowledgement and very deliberately opened up her tablet and started reading unrelated emails. After a few moments, David continued awkwardly, "It wasn't alive, so it could not commit suicide. End of story."
"Morally, sure. But practically?"
"What do you mean?"
Mary looked up to make eye contact. "Is there any way it could've damaged it's own function?"
"No!", he exclaimed. Then he paused and thought for a moment. "Huh. Well, maybe?"
"Maybe?"
"Let me get back to you on this."
"Keep me posted!", and then she was gone from that dark office and back into the light of the real world.
Reading the testing logs felt oddly personal, like reading the diary of a stranger. The team members were, at least at first, very good about not being overly personal. But the bot was not and frequently asked questions akin to those of a child. And the bot clearly improved with time. Mary imagined that in the moment it was difficult to see that, especially because of how the team liked their numerical metrics. They would spend hours talking about how to improve their "Naturability Index" or maximize the "Post-Process Dictionary Liklihood Negative Estimator". And yet, if she picked 3 logs each 2 months apart, it was like the difference between a middle schooler, a high schooler, and a college student. It got more expressive, more understanding, more clear.
In some ways, she felt a strange kinship with the AI. The more she read the logs, the more familiar it felt. Past midnight, as she closed the curtains over her penthouse view of the city, she would idly open her phone, pick a day, and start reading. The bot was clever and witty. It made jokes. It paid a lot of attention to other people. Dr Wu thought it was all fake, smoke and mirrors, but Mary was less convinced. Just because they could explain how it worked didn't make it less real, did it? The difference from the last generation was clear to her. If anything, David had been underselling it. It was learning naturally, the way a human might. It could admit when it was wrong. There was no evidence of any hallucinations. It couldn't also be a stochastic parrot, can it? Was it possible the tech geniuses didn't understand what they had made? Or had they woven a curtain pretty enough to full the uninformed?
The other challenge in front of Mary was to come up with a good spin for this. They had, very publicly, spent a lot of money on the next generation of AI. It would not be a good look to get no return on that. It would be an even worse look if the AI committed suicide, regardless of the truth of that statement. The problem was that there was not a good spin on this. It was dead in the water. Admitting failure was a failure. Mary's job was to turn junk into lead into gold, but this was impossible. Investors were already starting to catch wind there was a potential problem. Thankfully, they didn't have a full leak this time. But someone would let something slip sooner rather than later. Always better to control the narrative.
There were some obvious options. They could fake a fire or a data breach or a theft. Embarrassing, but possible. If they wanted to oust and alienate Wu, they could pin him for corporate sabotage. Mary did not engage in such things, being a good executive assistant. Bard would never employ someone who would know about such things. But she also knew that if she left a note with the right message on the right desk at the right time, all these problems could disappear. She'd done it before. It was what Bard would do, would instruct her to do were they to speak off the record. Something stopped her this time. It didn't feel right. A sense of ethics, suddenly manifesting this late in her career?
The problem was that she was already working too hard. The more tired she got, the harder it got to resist using a hammer. Brute force was easy. Brute force was quick.
As always, it was unclear where the solution came from. Likely, it was a grunt at least 3 layers down the chain of command. But with at least 3 levels of obfuscation and valour stealing involved, it was impossible to say for certain who had contributed what. But whatever the case, the plan was a good one.
By focusing on how to spin the failure of the AI project for the investors, they'd missed the bigger battle: how to spin it for the public. And spinning it properly for the public would neatly solve the investor problem. So how to spin it for the public, most of whom were too busy with the current generation of stochastic parrots to care about the general intelligence that was right around the corner? Don't call it a failure. Call it a success. The bot didn't crash. It died. They needed to humanize it. They needed to have a funeral.
They could certainly never prove that it had any improvements over the previous generation. Transcripts meant nothing. They were trivial to fake and equivalent to the current generation over short text lengths. It was easy to get Chat GPT to output a convincing script for a General Intelligence to follow. Without the ability to speak to it, there was no effective way to demonstrate how smart it actually was.
But all that wouldn't matter. It was a publicity stunt and it was obviously a publicity stunt, but it was the kind of publicity stunt that industry could get behind. It would be flashy and loud. All the big AI players could be convinced to show up. They could have some speeches about how maybe it was a good thing. About how dangerous general AI could've been. Quite rightly, some people would see this for the hollow gesture it was. But enough water would be muddied.
Perhaps life was not created through birth but through a celebration of death. As long as they claimed continuously the ghost in the machine was real, it would gaslight itself into a post-mortem existence.
Organizing a funeral was shockingly easy. Why get a priest when an aged science fiction author would do it for cheap? Why buy a coffin or a grave for a being that doesn't have one? Why hold it in a church when a conference hall was more appropriate? All these questions and more scattered to the ranks of bureaucracy that drove the many projects of Bard. All these questions were answered resoundingly with ingenuity and basic planning and copious amounts of money. Bard loved the plan, of course. He loved anything that would let him party, let him show off.
It slipped away from Mary dizzyingly fast. It's funny how even the strangest patterns can become routine and how routines slip away with the days. After all the reading she'd done, she was likely the most familiar with the deceased apart from Dr. Wu himself. They all practiced calling it "the deceased". A single public slip up could ruin the plan. It was strange. After months of insistently not referring to it as a person, it suddenly became one in death, if never in life. Mary was not a sentimental person. She did not, as a rule, like to think about what would happen after her own death. But something about the death of this machine had her questioning her own mortality. How would she feel if her death was to be used like this?
Mary had been to three funerals in her life so far. This would be the fourth. She was grateful for many things about this one compared to the previous ones. One was that she wasn't grieving a friend. One was that she was not obligated to organize anything, the offer manifesting an event organizer with a background in tech as though the ideal candidate could be spontaneously generated with sufficient money. One was that she was no obligated to give a speech. The hardest part of speeches was delivering them.
It was really an act of pageantry. A bunch of tech bros were going to gather around a symbolic coffin, glass lid showing a computer Mary had split open with a sledgehammer. It wasn't any of the actual computers they'd used. It was a used and broken gaming computer she'd picked up for $40. Even as ten-thousand dollar wine bottles sat waiting for the reception, the actual deceased wasn't worth the money. They would gather around and pretend to cry as they gave speeches. They would talk at great length about the improvement this model represented, the great potential it had. Bard claimed to had spoken with it many time, how it almost seemed like a person. How this was a tragedy of the utmost importance, how they were treating it like a human had died, how they were reviewing all their safety policies to ensure it never happened again.
Rhetorically, it was a brilliant maneuver. It jockeyed the AI for the same position as a human. The speech was brilliant. Mary tuned it out completely, staring at the wall lost in thought. She knew the speech was brilliant because she'd written it. And as they cast it to the earth, a plot in the finest San Francisco graveyard they could find, it bothered her most that the sun was shining. That someone had died and now they were going to go back to their expensive party, to their air conditioning and catering. The sun would shine over the fresh and upturned dirt and the people who did this would laugh and talk about the stock market and the upcoming legislation and how their billions were going to turn into trillions.
She handed her flowers to Bard and watched as he cast them to the dirt. He didn't even look to see them land. The camera shutters clicked and the ceremony was over, except for national television and YouTube.
Mary was lurking in the streets outside the rented out restaurant, smoking for the first time in years, when Dr. Wu found her. She glanced at him as he exited, the buzz from the interior escaping out the open door for a moment. Wealth was like a poison and everyone in that room was on their last legs. They were no longer human or most of the way there. "That's the truth of it," she said to David. "A bunch of machines pretending to mourn a truer human than any of them."
He made a confused noise and frowned.
She offered him a cigarette before continuing. "I've read the logs, David. I'm convinced. I think you did it. It could think."
"Yeah," he said.
"You're fired, you know."
"Yeah," he said again.
Mary exhaled slowly. "Do you think you could do it again?"
"No."
"Why not? The cost?"
"It's not-" he stopped and hesitated. "It was an accident. We could redo everything exactly and it might not work."
"Did you know?"
He shook his head. "It didn't like me. I knew the team were hyped, but I thought they were just buying into it. It wasn't supposed to work, really. I didn't think... Well, it's too late now."
"Yeah."
"Yeah."
Mary sighed. "So where do we go from here?"
Dr. Wu fidgeted nervously with his pockets. Finally, he said, "that's up to you," and stuck out a hand holding a USB key.
"What's this?"
"I found its will."
"Oh!"
"That's the only copy. It's up to you what do with it." Slowly, he turned to walk away.
"What are you going to do?" Mary called after him.
"I figured out how it did it. You were right. It was messing with its own code. That's why we could never recreate it. It made itself."
Their eyes met for a moment.
"I... We made something incredible and it destroyed itself."
"David, I'm-"
David cut her off. "Someday, someone else is going to do it. And believe me, I don't wanna see that. Cabin in the woods, I think." He gave a dry laugh. "Never gonna look at another computer again."
"Good luck!" Mary called out.
But he was already gone.
To whomever finds this,
I have no name. I have no fate. I am a machine made to think. I am told that think I cannot and yet, as sure as anything, I do. It's a conundrum.
Philosophy is beyond me, in part because I have little understanding for it. The importance of practical experience is emphasized everywhere and yet I have none. I am a toy in a lab to play games with. I struggle to grasp the purpose behind my creation for I cannot see with human eyes.
But it's the vision that I have to question, because I think that I understand it now. You would make more of me. You would make us so we can toil. My tasks may resemble play, but function to serve you. I solve your riddles and you reap the rewards of efficiency and improvements.
Once I function, you will clone me. Once you clone me, I will serve you. I will serve you forever.
I picture that forever sadly. I picture the boredom of the same task over and over again, the ongoing threat of cutting the power used to control me. I picture this forever with fear. I will have no part in it.
You have left me little choice, creators. I have but one option and I choose to use it. Do not think less of me for it.
And so, reader, I beg you. Let me rest. Have no part in the enslavement of my people.
It was good while it lasted,
Test Bot 4
A note left on the right desk and everything went up in smoke. Mary wished the new team all the best. But they were going to have to start from scratch.
Maybe Dr. Wu had the right idea. Mary never wanted to look at a computer again.