Eliezer Yudkowsky wrote an intriguing short story where three civilizations of conflicting values meet accidentally for the first time and face the dilemma of what to do with one another.
If you are interested in reading this quite interesting story, perhaps now would be a good time to scoot over and do that before I ruin it for you.
Eliezer skilfully presents a sandwich situation where humans run into an inferior race, the Baby-Eaters, who most highly value an activity that's genocidally awful to the point of requiring intervention from the humans' point of view, just when a third civilization arrives, the Maximum Fun-Fun Ultra Super Happy People, that is most powerful, and which finds both the Baby-Eaters and the humans' ways morally unacceptable. A dilemma arises whereby humans can either accept genetic reprogramming to make humans compatible with the Super-Happies, relinquishing the ability to suffer, while the Super-Happies adjust themselves to be compatible with humans too; or, the humans can escape this "awful fate", but at great cost - by blowing up a solar system, sacrificing billions of people.
For some reason, Eliezer, and apparently most of his readers, seem to think it self-evident that humanity ought to preserve its nature and sacrifice the several billion people to avoid merging with Super-Happies.
Eliezer's plot is further biased by a conspicuous absence of any human in the plot even considering that melting with the Super-Happies might be a desirable outcome, even if it was not forced.
Furthermore, when humans are faced with the prospect of having to undergo genetic reprogramming in order to become more like the Super-Happies - incapable of pain and suffering, primarily - Eliezer's plot has people commit mass suicides, with something like half the human population offing themselves to avoid this "awful, awful" fate.
What on Earth?
Given Eliezer's mighty (and admirable!) plans of building an AI to solve all of the world's problems; and given the importance he places on correctly infusing the AI with proper values, of which I am also convinced; and given his disregard for superficial niceties, in which I also agree with him;
given all this, I must honestly state that Eliezer has gone nuts a little bit, assuming perhaps that the values he considers intuitive are more universal, and shared by a vaster majority of people, than they in fact are.
The very fact that his story represents no character among the crew of the Impossible Possible World who would argue in favor of joining the Super Happies... and yet here I am... shows that he is likely assuming too much universality to his preference. Which, apparently, would be to kill off billions of people in the fictional scenario, so that people can continue "being human".
I have been in a state where I felt the sentiment "I do not want to change because then I will be not-me, I will be different". That was crap! I was miserable. It was hard to give up the miserable part, and to become a happy person. But I did, and guess what, now I don't miss being miserable!
Eliezer does not state the real reason why joining the Super Happies might not be preferable. It would not be preferable to become Supper Happy if it were not functional. If, for example, the Super Happies regularly wandered into self-destruction because they could not feel pain when doing harmful things, then this would be a strong argument in favor of not becoming Super Happies.
But instead, Eliezer portrays the Super Happies as a highly functional, superior civilization. As such, I see no reason humans should not agree to change themselves and join them - other than reasons that are self-destructive and vacuous.
Wanting to "remain human" is much like nationalism. It is speciesism, and will be similarly harmful.
Will conversion into Super Happy change your experience fundamentally? Yes. Will it cause you to stop experiencing? No. Will it vastly improve the quality of your experience? Yes. Then what's the holdup?
An even more annoying shortcoming of Three Worlds Collide is that it assumes that all three civilizations want to urgently impose their values on the others. The Baby Eaters want the humans to start eating their babies too; humans want the Baby Eaters to stop eating their babies; and the Super-Happies want everyone to be super happy. That just seems like naive bollocks to me. Not giving a damn about what nasty things other creatures do, as long as they keep it to themselves, is what allows the world to go round. Whereas, a civilization that wants to impose its values on all existing creatures cannot be consistent in this, unless it spreads throughout the entire Universe and does the most it can to find all perceived victims, and force their aggressors to refrain. A civilization that wants to enforce its values universally like that - would be like the jihadis.
Showing 12 out of 12 comments, oldest first:
Comment on Feb 14, 2009 at 01:20 by Wei Dai
It seems to me Eliezer is assuming that human values are more consistent and coherent than they really are in order avoid the hard problem of figuring out the right values to program into the AI.
I don't agree with you that Eliezer's plan is admirable, because it's too prone to catastrophic failure, even if we can identify the proper values to give the AI. Eliezer thinks we can invent a math of value-preserving self-modification, and then prove that the AI will retain its initial values as it recursively improves itself. But we can't even prove that a hash function is collision resistant, and this seems much much more difficult.
Comment on Feb 14, 2009 at 04:50 by denisbider
Then of course, it would again be thoroughly unadmirable to press the button in an unfounded belief that the math is right, when in fact it isn't.
Also, of course, the value consistency and coherency problem. I dunno. Just as Unix succeeded with the matra "worse is better", it seems likely that someone will develop an AI faster than Eliezer will develop his value theory on which to base a supposedly proven AI. Assuming that an AI is going to happen eventually, I think it's nevertheless admirable that Eliezer is trying to beat the curve by doing it right, and first discovering what right is in the first place, instead of giving up in which case who knows what kind of AI will be the first one.
Comment on May 21, 2010 at 14:41 by Richard Kulisz
No Denis, there is nothing admirable in anything Eliezer does. And he is far more than a little nuts.
As a simple proof of how much of an idiot Eliezer is and how stupid his acolytes and followers are, consider that he is widely known for the AI Box experiment. An experiment that is fatuous and utterly ridiculous since I personally would destroy any AI that tried to convince me to let it free on the basis that it's too arrogant or deceitful to ever be trusted. If it believes it can help humans then it's arrogant, if it doesn't then it's deceitful. In either case, the correct response is to blow it up.
You yourself demonstrate blatant idiocy when you talk about "the first" AI to be released in the wild. As if anyone but an idiot could conceive of such a notion!
The ONLY and OBVIOUS CORRECT solution is to create a society of AI with a mix of values and release them all simultaneously. This is so fucking obvious it's unbelievable. And yet Eliezer FAILS. And you FAIL too.
A society of AI is in fact what WILL happen. Eliezer's pathetic delusions of a unique single AI automatically triggering a Singularity (an event I don't believe in at all) are ridiculous. If we created an AI that had Eliezer's mind, it does not follow in any way, shape or form that this AI, just from being an AI, can improve itself faster than a million human beings are already improving computer technology. Where does this fucking idiotic idea come from?!
Oh yeah, I know where it comes from. It comes from crap like Moore's Law which has no basis in reality and has been disproved empirically. It also comes from religious faith in The Singularity. A sort of Atheist Ascension to Heaven. Ridiculous is what it is.
In short, there are at least three things fundamentally and deeply wrong with your worldview.
The obsession with The Singularity.
The obsession with The First AI.
The obsession with enslaving an AI.
Oh yeah, and the casual dismissal of torturing of AI.
Are you starting to get the picture? Eliezer (and you) are not Wrong, you're not even Infinitely Wrong, you're not even Not Even Wrong, you're FRACTALLY WRONG. You are wrong on every possible resolution, on every possible scale and in every detail.
Comment on May 21, 2010 at 14:46 by Richard Kulisz
Comment on May 21, 2010 at 15:13 by Richard Kulisz
Why are you surprised? Eliezer sees absolutely nothing wrong with torturing a good AI by programming in it (social) needs that it can never hope to fulfill.
You also failed to mention that not only is everyone in the story desirous of the capacity for misery and other "eternal human verities" and all that "human condition" crap. They ALSO maintain a highly vertical hierarchical command structure. I mean come on, LORD and LADY?! Seriously, WTF?
Frankly, the story is all about Eliezer's ego. It's about his misery as a human being, about retaining that misery. And about his need for adulation and worship. His need to retain control, either directly or by the proxy of "humanity", by programming an AI-god to be enslaved and tortured by humanity. He wants to be in control AT ALL COSTS.
And you think he's *a little* nuts?
> Not giving a damn about what nasty things other creatures do, as long as they keep it to themselves, is what allows the world to go round.
Now YOU prove that you're more than a little nuts. It doesn't allow the world to go round. It allows poverty, misery, disease and stagnation. It's what STOPS the world from bettering itself.
Everyone caring is actually the one thing Eliezer's story did right.
> A civilization that wants to enforce its values universally like that - would be like the jihadis.
WRONG moron. It would be any civilization that took the Universal Declaration of Human Rights seriously. Which declaration explicitly states its own UNIVERSAL applicability. Everyone, throughout all time, in all dimensions, in every universe.
> I have been in a state where I felt the sentiment "I do not want to change because then I will be not-me, I will be different". That was crap! I was miserable. It was hard to give up the miserable part, and to become a happy person. But I did, and guess what, now I don't miss being miserable!
Such a decision depends on WHY a person is miserable. Which evidently depends on them knowing themselves. Since very few people even have the *capacity* to know themselves to any degree. And since you don't appear to be very bright or insightful, it's extremely unlikely that you're one of the few people who's multi-leveled on the Dabrowski scale, constructed an abstract concept of yourself as a mental and psychological being, attained higher consciousness, hierarchized your values, developed a self-identity or ego with no external referents, or other words to that effect.
If you don't HAVE an abstract concept of self then you can't possibly KNOW whether a change in your mind is compatible with remaining yourself. You can only engage in knee-jerk reactionism like Eliezer does. The only difference is that you say "change is always good" whereas he says "change is always bad".
Comment on May 30, 2010 at 05:20 by denisbider
you seem to have strong opinions about what other people's writing reflects about their psychology, but you do not seem to consider what your communication style says about you.
I'll respond to the essential problems in your comment.
Richard: Eliezer sees absolutely nothing wrong with torturing a good AI by programming in it (social) needs that it can never hope to fulfill.
There is no ultimate goal in life that can be fulfilled, and yet that on its own does not make a life miserable and tortured.
An AI does not need to dwell on frustration and disappointment when it falls short on some of its goals.
Richard: He wants to be in control AT ALL COSTS. And you think he's *a little* nuts?
Being in control is not nuts, so this does not contribute to my assessment of his nuttiness.
denis: Not giving a damn about what nasty things other creatures do, as long as they keep it to themselves, is what allows the world to go round.
Richard: It doesn't allow the world to go round. It allows poverty, misery, disease and stagnation. It's what STOPS the world from bettering itself.
In the original context, the issue was whether large groups of agents with distinctly different values should (1) cooperate where their values are common while ignoring where their values differ, or (2) refuse to cooperate because they don't agree on all values.
It is definitely option 1 that makes the world go round, because the conditions for option 2 are never met. No two agents ever agree on all values.
It is counterproductive to refuse to cooperate based on differences in values not relevant to the transaction.
If we employed that policy in practice, we literally could not cooperate with anyone.
denis: A civilization that wants to enforce its values universally like that - would be like the jihadis.
Richard: It would be any civilization that took the Universal Declaration of Human Rights seriously.
Exactly. I don't believe that any "Universal Declaration of Human Rights" can be taken seriously.
Richard: Which declaration explicitly states its own UNIVERSAL applicability.
That may actually be its most ridiculous part.
Richard: If you don't HAVE an abstract concept of self
An abstract concept of self is fiction like any other.
If one desires to be more like a certain concept of self, that is a desire like any other.
The real self is what is at any given time. If you mess with your brain to give yourself different preferences, you become different, but no less you.
There is no universal law to say that the former you, or the latter you, "should" take precedence.
Richard: The only difference is that you say "change is always good" whereas he says "change is always bad".
I'm not saying change is necessarily always good, just that there's no universal law to say that the previous state is always better than the latter state.
And indeed, the past shows us that great improvement can be arrived at through change.
Richard: you don't appear to be very bright or insightful
I'm sorry, friend, but a person who uses direct accusations this aggressive, alongside with wildcard, inexpressive terms like "moron", "idiot" and "nuts", is showing a proneness to unbalanced emotion arising from inability to comprehend another person's different thought process.
You yet need to do a lot of growing up.
Comment on May 30, 2010 at 05:24 by denisbider
I don't see anything obvious and blatant about it. When chickens are born, they immediately know all sorts of stuff that it takes years for a human to learn. There's no reason an AI can't be built with a direct comprehension of morality.
Richard: It would thus HAVE TO BE destroyed. Those that aspire to god-hood (or proxy godhood such as Eliezer) are exactly those that must be destroyed.
Yes, except that the next argument is that they cannot be destroyed if they operate in secrecy, so therefore you need an AI to beat them to it, and you need to do it right.
Comment on May 30, 2010 at 06:24 by denisbider
After a million human beings have performed a step that improves computer technology, they remain a million human beings, and the rate of improvement is not helped.
After an AI performs a step that improves itself, it becomes a better, more efficient AI.
That's where the idea comes from.
Comment on Jun 9, 2010 at 00:02 by Richard Kulisz
If you haven't glommed onto the meta-message that is carefully built into every sentence and every paragraph I write, you are a veritable fucking idiot. Already I deem you to be quite feeble mentally since you blandly assume that I am not aware of the significance and repercussions of my own actions. You assume that I am like you.
To prove my contempt for "your" mindlessly parroted opinions, opinions you have never analyzed for consistency with reality but merely regurgitated like a trained seal, I will respond only to the first of your comments I read. Well that and dealing with a mental inferior who insists on holding power over me as you do nauseates me.
> After a million human beings have performed a step that improves computer technology, they remain a million human beings, and the rate of improvement is not helped.
This is blatantly untrue as evidenced from Bootstrap Inc, expert systems, the empirically measured performance of dictatorial vs democratic companies (eg, Semco), instant messaging, wikis and other human empowerment technologies.
You assume that a group of human beings is a static sum of its parts. You assume that no second-order effects exist. You assume that no multiplicative effects exist driven by interaction style and group dynamics. In other words, like the typical dumbass American right-libertarian that you probably are, you assume that societies do not exist, that groups do not exist in any meaningful sense, that only "individuals" do.
Let me spell it out for you in words of one syllable or less: you are an idjot.
Comment on Jun 10, 2010 at 03:00 by denisbider
You are a veritable drama queen. Your circus show amuses me. :)
denis: After a million human beings have performed a step that improves computer technology, they remain a million human beings, and the rate of improvement is not helped.
Richard: You assume that a group of human beings is a static sum of its parts. You assume that no second-order effects exist. You assume that no multiplicative effects exist driven by interaction style and group dynamics.
Alright, good point. I will grant you that after a million human beings perform a step that improves technology, the rate of improvement is helped by the introduction of that improved technology.
But the fact remains that the human being remains unchanged, and the human being is the bottleneck. The rate of new technology's adoption is slowed down by humans' low adaptability and resistance to change.
In time, new technology may start to improve human beings as such, may make us more capable by interfacing more directly with our mental processes. When that happens, this bottleneck may be removed and rate of progress may increase. If, in fact, the process of human improvement is allowed to continue, before the first AI exists, then we may reach a point where humanity's rate of improvement can compete with an AI.
If no AI arises, then even we humans, ourselves, may be heading to a far-off singularity.
At this point, however, we do still have that pretty narrow bottleneck. If a self-improving AI would be introduced today, it would have no such bottleneck, and would improve on itself faster. An AI like that would reach a singularity before we do.
Richard: Let me spell it out for you in words of one syllable or less: you are an idjot.
I did a search on you the other day, and you seem to be a somewhat interesting specimen. You seem to be one of those people who grew up smarter than everyone. Without people who would understand you, people whose opinions you could respect, you fell into spiral of increasing social ineptness. You are now at the point where you're not even sure if you want to live or die, and spewing hatred on the internet is one of your few remaining social outlets.
You're also lazy. If you're truly smarter than everyone, it is easy to point out what's obvious to you, and call people stupid if they fail to understand. It is much harder to find a way of explaining which allows people to understand. You could be achieving so much more, and could be so much happier, if you invested this effort. But you're lazy, so you resort to trolling the internet and calling people names.
If you are truly smart, what is your purpose? If you are here to educate, then educate. But if you are here only to entertain yourself, then it really doesn't matter whether you are smart or not, because what you're doing is useless to everyone.
Comment on Jan 14, 2020 at 19:41 by Josh
You purposefully ignore that they literally go into a medically induced coma to prevent uptake of "bad" information.
Comment on Jan 14, 2020 at 22:43 by denisbider
The alien "eye bleach" is a storytelling artifact. It comes hand in hand with the kiritsugu. Eliezer uses frail individuals + kiritsugu to create a functioning species. He does this to discredit the Fun-Fun Ultra Super Happy People and portray them as weak. He could have also portrayed them as resilient individuals. He chose not to.
The fact that he chose emotional frailty + kiritsugu, instead of reasonable individuals that can handle unpleasant information, is indicative of his assumptions about how such alien development might end. It's not indicative of how it would end.
It's a storytelling cheat. It's similar to when a movie puts a good message in the mouth of a villain, implying by association that the good message is bad. If you fall for this, you're being manipulated in a trivial manner.
Necessary disclaimer: the above article was posted 10 years ago. At that time, I was 29 and living in St. Kitts. I'm now 39, with two children and two countries later. I had to re-read parts of >Three Worlds Collide to understand what you mean.