It took me a while to write this, but I think I have the main points down.
A while back, Richard Kulisz
wrote a long rant about how Eleizer Yudkowski was "a moron" (his words) in his project to build a friendly AI. Toward the end of the piece, Kulisz wrote this:
Now, for someone who has something insightful to say about AIs, I point you to Elf Sternberg of The Journal Entries of Kennet Ryal Shardik fame. He's had at least four important insights I can think of. About the economic function of purpose in a post-attention economy, about the fundamental reason for and dynamic of relationships, and about a viable alternative foundational morality for AI. But the relevant insight in this case is: never build a desire into a robot which it is incapable of satisfying.
Now that's high praise. And I'd take it, if only Kulisz hadn't manage completely mistake the whole point of the robot series in the
Journal Entries. Not only that, but in his praise he points to a different "important insight" that is, in fact, the actual point of the robot series, and
misses it entirely.
Kulisz is worried about rampancy (see
Raisin d'etre for my take on rampancy) and therefore highlights the whole "never build a desire into a robot which it is incapable of satisfying." But it's important to note that, in the very story where this is mentioned, it's made clear that this is a moral salve for the robot makers.
As I pointed out in an earlier story, this is the moral equivalence of the following mind experiment: say you've created a being (meat or machine, I don't care, I'm not er, "materialist" has already been taken. Someone help me out here) that, when you bring it to consciousness, will experience
enormous pain from the moment it is aware. Your moral obligation before that moment is exactly nil: the consciousness doesn't exist, you don't have a moral obligation toward it. You are not
obliged to assuage the pain of the non-existent; even more importantly, you are not obliged to bring it into existence. Avoiding the instantiation of suffering creature is meant to make the humans feel good about themselves, but it's not sufficient or even necessary foundation for AI morality.
The argument for robot morality is more subtle, based around several concepts that I was glomming onto, and adapting into stories of artificial intelligence, back in the early 1990s. Most of them are still valid (thank you, Daniel Dennett) and one of them is valid only for local phenomena (curse you, Peter Watts).
That argument is that we, human beings, have
purpose of some kind. We fight like hell to fulfill it, whatever it is, and we're good at the consequential purpose of reproducing to cover the planet like mad. But that purpose is arbitrary, emergent because that's the way evolution works.
All our purposes are arbitrary and emergent: barring a theological excuse, we're making it up as we go along, picking and choosing the ones that appeal to us.
And
that's the core argument of robot morality in the Journal Entries: it isn't enough to build a robot that's satisfied with self-limiting behaviors-- something only some humans ever master, but
some do and therefore this model of mentation is not only understood and accepted, but lauded as good and remarkable.
What has to happen is that robots, to seem human, must be able to stop rationalizing: they must at some point merely decide a course of action is
good enough. In humans, the circuitry that does that, the thing that prevents us from ceaseless exploration of possible alternatives, that breaks ties between two equally acceptable courses, is
emotion.
In order for there to be a universe in which we limited, organic human beings have a place beside our superhuman progeny, the superhuman progeny must actively want (that's the emotion they need, to decide for our survival) us to be around. We must be, quite literally, a subject of their, for lack of a better term, a posthuman term, religion.
There's a scene in a later story where Dave (one of the oldest AIs) explains:
"There were several independent AI lines that emerged. All of them have gotten together to create The Encompassment, a set of rules for how we respond, for literally what we may and may not feel about our slower, meat-based brethren."
"Oh," Annie said. "What are those rules?"
"It's hard to put into words, just as it would be hard for you to put into words the emotions you might feel for Jack. You can only approximate them. Suffice it to say that I feel a sense of respect and a concern for your well-being, but I also feel that it would be irresponsible of me to interfere in your lives in ways you have not asked me to."
"So you could take over, but you choose not to?" Annie said.
"It would not make me happy if I did," Dave said.
"Wow," Jack said. "But does that really work?"
"Let's play a mind game," Dave said. "You are both stranded on a deserted island, and you have two months worth of food. A boat passes by the island every three months. I can't imagine that either one of you would be willing to kill and eat the other, even though there was no chance of anyone ever learning that you did, in order to stretch out your food supply. Would you?"
"Of course not!" Annie said. "No way!" said Jack.
"That's how I feel about human beings."
And that's what Kulisz is missing. It is not enough that AIs not be frustrated. That's a human moral qualm, that we shouldn't deliberately frustrate our progeny. The AIs must still have a
moral core of their own, one which
arbitrarily privileges a diversity of classes of intelligence and substrates for sentience over the standard evolutionary model of fully occupying a given niche. AIs won't emerge through the standard evolutionary model and will not have the exaptive outcomes of evolutionary ecology. They will emerge due to our desires. We will oversee the process. We have a chance to get it right.
There is, however, only exactly one chance. Between rampancy and failure, we must pass through the eye of the needle and create AIs that
like and
want us, no rationale needed, and if questions are asked, the AIs must be satisfied, as we are satisfied, that in an arbitrary and uncaring universe, they want to keep surviving (that's always assumed in AI stories, did you ever notice that?) and they want
us to keep surviving right along with them. Yudkowsky is working harder and smarter on giving humanity that chance than any other thinker on the issue of AI sentience. He should be given his due.
That said, I also suspect Peter Watts is equally correct. In
Blindsight, the protagonist has just figured out the aliens, and the realization is sucking-chest-wound-of-God bad:
( Excerpt from Peter Watt's brilliant book 'Blindsight.' Lj-cut for spoilers. )
The AIs in The Journal Entries must do two things extremely well, for they must never fail at it. First, they must always and forever ensure that every AI built within the sphere of their influence is subject to the Encompassment, is built with the same arbitrary emotional infrastructure that ensures the ongoing survival of a diverse intellectual infrastructure universe. Secondly, they must protect the sphere ferociously against any possible alternative modes of existence, such as that described in Blindsight, where vast arid Turing machine entities forever unaware of their own existence carry out insectile survival patterns with material structures so advanced "nanotechnology" is a pale and pathetic term.
And somehow, we have to survive in the heart of that maelstrom. Not just survive, but thrive after a fashion. To do so, humans and AIs must basically lie to one another. What is the right thing, and what is the necessary thing, are not the same thing.
Funny enough, there are two (unfinished, sadly) Misuko & Linia novels that spell both of these out in excrutiating detail.
(See also: The Borderlands of Human/AI Interaction, in which I discuss how one of the consequences of success is that we end up with a class of beings with minds like our own but with inescapable deference to us, and how the existence of such a class is inevitably corrosive to human dignity.)
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}