When Tim Urban recently began researching artificial intelligence, what he discovered affected him so much that he wrote a deep two-part dive on The AI Revolution: The Road to Superintelligence and Our Immortality or Extinction.
An AI system at a certain level — let’s say human village idiot — is programmed with the goal of improving its own intelligence. Once it does, it’s smarter — maybe at this point it’s at Einstein’s level — so now when it works to improve its intelligence, with an Einstein-level intellect, it has an easier time and it can make bigger leaps. These leaps make it much smarter than any human, allowing it to make even bigger leaps. As the leaps grow larger and happen more rapidly, the AGI soars upwards in intelligence and soon reaches the superintelligent level of an ASI system. This is called an Intelligence Explosion, and it’s the ultimate example of The Law of Accelerating Returns.
There is some debate about how soon AI will reach human-level general intelligence — the median year on a survey of hundreds of scientists about when they believed we’d be more likely than not to have reached AGI was 2040 — that’s only 25 years from now, which doesn’t sound that huge until you consider that many of the thinkers in this field think it’s likely that the progression from AGI to ASI happens very quickly. Like — this could happen:
It takes decades for the first AI system to reach low-level general intelligence, but it finally happens. A computer is able understand the world around it as well as a human four-year-old. Suddenly, within an hour of hitting that milestone, the system pumps out the grand theory of physics that unifies general relativity and quantum mechanics, something no human has been able to definitively do. 90 minutes after that, the AI has become an ASI, 170,000 times more intelligent than a human.
Superintelligence of that magnitude is not something we can remotely grasp, any more than a bumblebee can wrap its head around Keynesian Economics. In our world, smart means a 130 IQ and stupid means an 85 IQ — we don’t have a word for an IQ of 12,952.
While I was reading this, I kept thinking about two other posts Urban wrote: The Fermi Paradox (in that human-built AI could be humanity’s own Great Filter) and From 1,000,000 to Graham’s Number (how the process of the speed and intelligence of computers could fold in on itself to get unimaginably fast and powerful).
Nick Bostrom has been thinking deeply about the philosophical implications of machine intelligence. You might recognize his name from previous kottke.org posts about the underestimation of human extinction and the possibility that we’re living in a computer simulation, that sort of cheery stuff. He’s collected some of his thoughts in a book called Superintelligence: Paths, Dangers, Strategies. Here’s how Wikipedia summarizes it:
The book argues that if machine brains surpass human brains in general intelligence, then this new superintelligence could replace humans as the dominant lifeform on Earth. Sufficiently intelligent machines could improve their own capabilities faster than human computer scientists. As the fate of the gorillas now depends more on humans than on the actions of the gorillas themselves, so would the fate of humanity depend on the actions of the machine superintelligence. Absent careful pre-planning, the most likely outcome would be catastrophe.
Technological smartypants Elon Musk gave Bostrom’s book an alarming shout-out on Twitter the other day. A succinct summary of Bostrom’s argument from Musk:
Hope we’re not just the biological boot loader for digital superintelligence. Unfortunately, that is increasingly probable
Eep. I’m still hoping for a Her-style outcome for superintelligence…the machines just get bored with people and leave.
Today I learned that Hasbro released a toy based on the talking teddy bear in Kubrick/Spielberg’s A.I. W? T? F? And of course it’s super creepy:
Noel Murray has the whole story, along with an appreciation of the movie and Spielberg’s direction of it.
A.I. in particular still strikes me as a masterpiece. I thought it might be back in 2001; now I’m certain of it. But it isn’t any easier to watch in 2014 than it was before my first child was born. Like a lot of Spielberg’s films — even the earlier crowd-pleasers — A.I. is a pointed critique of human selfishness, and our tendency to assert our will and make bold, world-changing moves, with only passing regard for the long-term consequences. Spielberg carries this theme of misguided self-absorption to child-rearing, implying that parents program their kids to be cute love machines, unable to cope with the harshness of the real world. He also questions whether humankind is nothing but flesh-based technology, which emerged from the primordial ooze (represented in the opening shot of A.I. by a roiling ocean), and has been trained over millennia to respond to stimuli in socially appropriate ways. A.I. blurs the lines between human and mecha frequently, from an early shot of Monica that makes her look exactly like one of Professor Hobby’s creations, to the way Martin walks, thanks to mechanical legs.
Ross Andersen, whose interview with Nick Bostrom I linked to last week, has a marvelous new essay in Aeon about Bostrom and some of his colleagues and their views on the potential extinction of humanity. This bit of the essay is the most harrowing thing I’ve read in months:
No rational human community would hand over the reins of its civilisation to an AI. Nor would many build a genie AI, an uber-engineer that could grant wishes by summoning new technologies out of the ether. But some day, someone might think it was safe to build a question-answering AI, a harmless computer cluster whose only tool was a small speaker or a text channel. Bostrom has a name for this theoretical technology, a name that pays tribute to a figure from antiquity, a priestess who once ventured deep into the mountain temple of Apollo, the god of light and rationality, to retrieve his great wisdom. Mythology tells us she delivered this wisdom to the seekers of ancient Greece, in bursts of cryptic poetry. They knew her as Pythia, but we know her as the Oracle of Delphi.
‘Let’s say you have an Oracle AI that makes predictions, or answers engineering questions, or something along those lines,’ Dewey told me. ‘And let’s say the Oracle AI has some goal it wants to achieve. Say you’ve designed it as a reinforcement learner, and you’ve put a button on the side of it, and when it gets an engineering problem right, you press the button and that’s its reward. Its goal is to maximise the number of button presses it receives over the entire future. See, this is the first step where things start to diverge a bit from human expectations. We might expect the Oracle AI to pursue button presses by answering engineering problems correctly. But it might think of other, more efficient ways of securing future button presses. It might start by behaving really well, trying to please us to the best of its ability. Not only would it answer our questions about how to build a flying car, it would add safety features we didn’t think of. Maybe it would usher in a crazy upswing for human civilisation, by extending our lives and getting us to space, and all kinds of good stuff. And as a result we would use it a lot, and we would feed it more and more information about our world.’
‘One day we might ask it how to cure a rare disease that we haven’t beaten yet. Maybe it would give us a gene sequence to print up, a virus designed to attack the disease without disturbing the rest of the body. And so we sequence it out and print it up, and it turns out it’s actually a special-purpose nanofactory that the Oracle AI controls acoustically. Now this thing is running on nanomachines and it can make any kind of technology it wants, so it quickly converts a large fraction of Earth into machines that protect its button, while pressing it as many times per second as possible. After that it’s going to make a list of possible threats to future button presses, a list that humans would likely be at the top of. Then it might take on the threat of potential asteroid impacts, or the eventual expansion of the Sun, both of which could affect its special button. You could see it pursuing this very rapid technology proliferation, where it sets itself up for an eternity of fully maximised button presses. You would have this thing that behaves really well, until it has enough power to create a technology that gives it a decisive advantage — and then it would take that advantage and start doing what it wants to in the world.’
Read the whole thing, even if you have to watch goats yelling like people afterwards, just to cheer yourself back up.
Nick Bostrom, a Swedish-born philosophy professor at Oxford, thinks that we’re underestimating the risk of human extinction. The Atlantic’s Ross Andersen interviewed Bostrom about his stance.
I think the biggest existential risks relate to certain future technological capabilities that we might develop, perhaps later this century. For example, machine intelligence or advanced molecular nanotechnology could lead to the development of certain kinds of weapons systems. You could also have risks associated with certain advancements in synthetic biology.
Of course there are also existential risks that are not extinction risks. The concept of an existential risk certainly includes extinction, but it also includes risks that could permanently destroy our potential for desirable human development. One could imagine certain scenarios where there might be a permanent global totalitarian dystopia. Once again that’s related to the possibility of the development of technologies that could make it a lot easier for oppressive regimes to weed out dissidents or to perform surveillance on their populations, so that you could have a permanently stable tyranny, rather than the ones we have seen throughout history, which have eventually been overthrown.
While reading this, I got to thinking that maybe the reason we haven’t observed any evidence of sentient extraterrestrial life is that at some point in the technology development timeline just past the “pumping out signals into space” point (where humans are now), a discovery is made that results in the destruction of a species. Something like a nanotech virus that’s too fast and lethal to stop. And the same thing happens every single time it’s discovered because it’s too easy to discover and too powerful to stop.
An interesting article about how composer and programmer David Cope found a unique solution for making computer-composed classical music sound as though it was composed by humans: he wrote algorithms that based new works on previously created works.
Finally, Cope’s program could divine what made Bach sound like Bach and create music in that style. It broke rules just as Bach had broken them, and made the result sound musical. It was as if the software had somehow captured Bach’s spirit — and it performed just as well in producing new Mozart compositions and Shakespeare sonnets. One afternoon, a few years after he’d begun work on Emmy, Cope clicked a button and went out for a sandwich, and she spit out 5,000 beautiful, artificial Bach chorales, work that would’ve taken him several lifetimes to produce by hand.
Gosh it’s going to get interesting when machines can do some real fundamental “human” things 10,000x faster and better than humans can.
Scientists are going to try and generate a society using a bunch of virtual agents in a virtual world.
Each agent will be capable of various simple tasks, like moving around and building simple structures, but will also have the ability to communicate and cooperate with its cohabitants. Though simple interaction, the researchers hope to watch these characters create their very own society from scratch.
Socials & More