Superintelligence: Paths, Dangers, Strategies


Nick Bostrom - 2014
    The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. If machine brains surpassed human brains in general intelligence, then this new superintelligence could become extremely powerful--possibly beyond our control. As the fate of the gorillas now depends more on humans than on the species itself, so would the fate of humankind depend on the actions of the machine superintelligence.But we have one advantage: we get to make the first move. Will it be possible to construct a seed Artificial Intelligence, to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation?

The Singularity is Near: When Humans Transcend Biology


Ray Kurzweil - 2005
    In his classic The Age of Spiritual Machines, he argued that computers would soon rival the full range of human intelligence at its best. Now he examines the next step in this inexorable evolutionary process: the union of human and machine, in which the knowledge and skills embedded in our brains will be combined with the vastly greater capacity, speed, and knowledge-sharing ability of our creations.

Rationality: From AI to Zombies


Eliezer Yudkowsky - 2015
    Real rationality, of the sort studied by psychologists, social scientists, and mathematicians. The kind of rationality where you make good decisions, even when it's hard; where you reason well, even in the face of massive uncertainty; where you recognize and make full use of your fuzzy intuitions and emotions, rather than trying to discard them. In "Rationality: From AI to Zombies," Eliezer Yudkowsky explains the science underlying human irrationality with a mix of fables, argumentative essays, and personal vignettes. These eye-opening accounts of how the mind works (and how, all too often, it doesn't!) are then put to the test through some genuinely difficult puzzles: computer scientists' debates about the future of artificial intelligence (AI), physicists' debates about the relationship between the quantum and classical worlds, philosophers' debates about the metaphysics of zombies and the nature of morality, and many more. In the process, "Rationality: From AI to Zombies" delves into the human significance of correct reasoning more deeply than you'll find in any conventional textbook on cognitive science or philosophy of mind. A decision theorist and researcher at the Machine Intelligence Research Institute, Yudkowsky published earlier drafts of his writings to the websites Overcoming Bias and Less Wrong. "Rationality: From AI to Zombies" compiles six volumes of Yudkowsky's essays into a single electronic tome. Collectively, these sequences of linked essays serve as a rich and lively introduction to the science—and the art—of human rationality.

The Precipice: Existential Risk and the Future of Humanity


Toby Ord - 2020
    

Our Final Invention: Artificial Intelligence and the End of the Human Era


James Barrat - 2013
    Corporations & government agencies around the world are pouring billions into achieving AI’s Holy Grail—human-level intelligence. Once AI has attained it, scientists argue, it will have survival drives much like our own. We may be forced to compete with a rival more cunning, more powerful & more alien than we can imagine. Thru profiles of tech visionaries, industry watchdogs & groundbreaking AI systems, James Barrat's Our Final Invention explores the perils of the heedless pursuit of advanced AI. Until now, human intelligence has had no rival. Can we coexist with beings whose intelligence dwarfs our own? Will they allow us to?

Against Empathy: The Case for Rational Compassion


Paul Bloom - 2016
    Many of our wisest policy-makers, activists, scientists, and philosophers agree that the only problem with empathy is that we don’t have enough of it.Nothing could be farther from the truth, argues Yale researcher Paul Bloom. In AGAINST EMPATHY, Bloom reveals empathy to be one of the leading motivators of inequality and immorality in society. Far from helping us to improve the lives of others, empathy is a capricious and irrational emotion that appeals to our narrow prejudices. It muddles our judgment and, ironically, often leads to cruelty. We are at our best when we are smart enough not to rely on it, but to draw instead upon a more distanced compassion.Basing his argument on groundbreaking scientific findings, Bloom makes the case that some of the worst decisions made by individuals and nations—who to give money to, when to go to war, how to respond to climate change, and who to imprison—are too often motivated by honest, yet misplaced, emotions. With precision and wit, he demonstrates how empathy distorts our judgment in every aspect of our lives, from philanthropy and charity to the justice system; from medical care and education to parenting and marriage. Without empathy, Bloom insists, our decisions would be clearer, fairer, and—yes—ultimately more moral.Brilliantly argued, urgent and humane, AGAINST EMPATHY shows us that, when it comes to both major policy decisions and the choices we make in our everyday lives, limiting our impulse toward empathy is often the most compassionate choice we can make.

Life 3.0: Being Human in the Age of Artificial Intelligence


Max Tegmark - 2017
    It doesn't shy away from the full range of viewpoints or from the most controversial issues--from superintelligence to meaning, consciousness and the ultimate physical limits on life in the cosmos.

The Alignment Problem: Machine Learning and Human Values


Brian Christian - 2020
    Today’s "machine-learning" systems, trained by data, are so effective that we’ve invited them to see and hear for us?and to make decisions on our behalf. But alarm bells are ringing. Recent years have seen an eruption of concern as the field of machine learning advances. When the systems we attempt to teach will not, in the end, do what we want or what we expect, ethical and potentially existential risks emerge. Researchers call this the alignment problem.Systems cull résumés until, years later, we discover that they have inherent gender biases. Algorithms decide bail and parole?and appear to assess Black and White defendants differently. We can no longer assume that our mortgage application, or even our medical tests, will be seen by human eyes. And as autonomous vehicles share our streets, we are increasingly putting our lives in their hands.The mathematical and computational models driving these changes range in complexity from something that can fit on a spreadsheet to a complex system that might credibly be called “artificial intelligence.” They are steadily replacing both human judgment and explicitly programmed software.In best-selling author Brian Christian’s riveting account, we meet the alignment problem’s “first-responders,” and learn their ambitious plan to solve it before our hands are completely off the wheel. In a masterful blend of history and on-the ground reporting, Christian traces the explosive growth in the field of machine learning and surveys its current, sprawling frontier. Readers encounter a discipline finding its legs amid exhilarating and sometimes terrifying progress. Whether they—and we—succeed or fail in solving the alignment problem will be a defining human story.The Alignment Problem offers an unflinching reckoning with humanity’s biases and blind spots, our own unstated assumptions and often contradictory goals. A dazzlingly interdisciplinary work, it takes a hard look not only at our technology but at our culture—and finds a story by turns harrowing and hopeful.

Gödel, Escher, Bach: An Eternal Golden Braid


Douglas R. Hofstadter - 1979
    However, according to Hofstadter, the formal system that underlies all mental activity transcends the system that supports it. If life can grow out of the formal chemical substrate of the cell, if consciousness can emerge out of a formal system of firing neurons, then so too will computers attain human intelligence. Gödel, Escher, Bach is a wonderful exploration of fascinating ideas at the heart of cognitive science: meaning, reduction, recursion, and much more.

A Thousand Brains: A New Theory of Intelligence


Jeff Hawkins - 2021
    For all of neuroscience's advances, we've made little progress on its biggest question: How do simple cells in the brain create intelligence? Jeff Hawkins and his team discovered that the brain uses maplike structures to build a model of the world-not just one model, but hundreds of thousands of models of everything we know. This discovery allows Hawkins to answer important questions about how we perceive the world, why we have a sense of self, and the origin of high-level thought.

Our Posthuman Future: Consequences of the Biotechnology Revolution


Francis Fukuyama - 2002
    Fukuyama sketches a brief history of man's changing understanding of human nature: from Plato and Aristotle to the modernity's utopians and dictators who sought to remake mankind for ideological ends. Fukuyama argues that the ability to manipulate the DNA of all of one person's descendants will have profound, and potentially terrible, consequences for our political order, even if undertaken with the best of intentions. In Our Posthuman Future, one of our greatest social philosophers begins to describe the potential effects of genetic exploration on the foundation of liberal democracy: the belief that human beings are equal by nature.

Human Compatible: Artificial Intelligence and the Problem of Control


Stuart Russell - 2019
    Conflict between humans and machines is seen as inevitable and its outcome all too predictable.In this groundbreaking book, distinguished AI researcher Stuart Russell argues that this scenario can be avoided, but only if we rethink AI from the ground up. Russell begins by exploring the idea of intelligence in humans and in machines. He describes the near-term benefits we can expect, from intelligent personal assistants to vastly accelerated scientific research, and outlines the AI breakthroughs that still have to happen before we reach superhuman AI. He also spells out the ways humans are already finding to misuse AI, from lethal autonomous weapons to viral sabotage.If the predicted breakthroughs occur and superhuman AI emerges, we will have created entities far more powerful than ourselves. How can we ensure they never, ever, have power over us? Russell suggests that we can rebuild AI on a new foundation, according to which machines are designed to be inherently uncertain about the human preferences they are required to satisfy. Such machines would be humble, altruistic, and committed to pursue our objectives, not theirs. This new foundation would allow us to create machines that are provably deferential and provably beneficial.In a 2014 editorial co-authored with Stephen Hawking, Russell wrote, "Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last." Solving the problem of control over AI is not just possible; it is the key that unlocks a future of unlimited promise.

Strangers Drowning: Grappling with Impossible Idealism, Drastic Choices, and the Overpowering Urge to Help


Larissa MacFarquhar - 2015
    Larissa MacFarquhar digs deep into the psychological roots and existential dilemmas motivating those rare individuals practicing lives of extreme ethical commitment. The donor who offers up her kidney to a complete stranger; the activist who abandons possessions to devote himself to the cause; the foster parent who adopts dozens of children: such do-gooders inspire us but also force us to question deep-seated notions about what it means to be human. How could these do-gooders value strangers as much as their own loved ones? What does it really take to live a life of extreme virtue? Might it mean making choices as heartbreaking as the one in the old philosophy problem: abandoning a single family member to drown so that two strangers might live? Strangers Drowning combines real-life stories of unimaginable selflessness along with deep meditations on the shocking implications of these ethical acts.

Stealing Fire: How Silicon Valley, the Navy SEALs, and Maverick Scientists Are Revolutionizing the Way We Live and Work


Steven Kotler - 2017
    Over the past decade, Silicon Valley executives like Eric Schmidt and Elon Musk, Special Operators like the Navy SEALs and the Green Berets, and maverick scientists like Sasha Shulgin and Amy Cuddy have turned everything we thought we knew about high performance upside down. Instead of grit, better habits, or 10,000 hours, these trailblazers have found a surprising short cut. They're harnessing rare and controversial states of consciousness to solve critical challenges and outperform the competition.New York Times bestselling author Steven Kotler and high performance expert Jamie Wheal spent four years investigating the leading edges of this revolution—from the home of SEAL Team Six to the Googleplex, the Burning Man festival, Richard Branson’s Necker Island, Red Bull’s training center, Nike’s innovation team, and the United Nations’ Headquarters. And what they learned was stunning: In their own ways, with differing languages, techniques, and applications, every one of these groups has been quietly seeking the same thing: the boost in information and inspiration that altered states provide.Today, this revolution is spreading to the mainstream, fueling a trillion dollar underground economy and forcing us to rethink how we can all lead richer, more productive, more satisfying lives. Driven by four accelerating forces—psychology, neurobiology, technology and pharmacology—we are gaining access to and insights about some of the most contested and misunderstood terrain in history. Stealing Fire is a provocative examination of what’s actually possible; a guidebook for anyone who wants to radically upgrade their life.

World Without Mind: The Existential Threat of Big Tech


Franklin Foer - 2017
    Over the past few decades there has been a revolution in terms of who controls knowledge and information. This rapid change has imperiled the way we think. Without pausing to consider the cost, the world has rushed to embrace the products and services of four titanic corporations. We shop with Amazon; socialize on Facebook; turn to Apple for entertainment; and rely on Google for information. These firms sell their efficiency and purport to make the world a better place, but what they have done instead is to enable an intoxicating level of daily convenience. As these companies have expanded, marketing themselves as champions of individuality and pluralism, their algorithms have pressed us into conformity and laid waste to privacy. They have produced an unstable and narrow culture of misinformation, and put us on a path to a world without private contemplation, autonomous thought, or solitary introspection--a world without mind. In order to restore our inner lives, we must avoid being coopted by these gigantic companies, and understand the ideas that underpin their success.Elegantly tracing the intellectual history of computer science--from Descartes and the enlightenment to Alan Turing to Stuart Brand and the hippie origins of today's Silicon Valley--Foer exposes the dark underpinnings of our most idealistic dreams for technology. The corporate ambitions of Google, Facebook, Apple, and Amazon, he argues, are trampling longstanding liberal values, especially intellectual property and privacy. This is a nascent stage in the total automation and homogenization of social, political, and intellectual life. By reclaiming our private authority over how we intellectually engage with the world, we have the power to stem the tide.At stake is nothing less than who we are, and what we will become. There have been monopolists in the past but today's corporate giants have far more nefarious aims. They're monopolists who want access to every facet of our identities and influence over every corner of our decision-making. Until now few have grasped the sheer scale of the threat. Foer explains not just the looming existential crisis but the imperative of resistance.