Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. Princess Celestia will satisfy your values through friendship and ponies, and it will be completely consensual.

    Book by

    Once her neural paths were scanned and the pattern uploaded to an enormous bank of computers, she could be happy and carefree. No more aging, dandruff, fear of death, or any of the other drawbacks of hauling her own squishy meatsack hither and yon.But her new life is a troubled one. As she learns to enhance her own intelligence, she grows alienated from her old life. She has no legal rights, and the government is trying to shut her down and steal her research.But when a nuke detonates over Manhattan, Helen devotes her growing power to bringing the culprit to justice. She uncovers a force bent not on geopolitical power, but on the extinction of all humankind.Will our heroine be enslaved or deleted by the Feds? Can she avert an impending war with China? How will she find time to finish her doctoral thesis? And (hey, we're trying not to judge) isn't Dr. Mellings way too old for her?Alternately goofy and dark, The Improbable Rise of Singularity Girl tells the story of a singular woman, and a life lived at an exponential pace.

    The first, the orthogonality thesis, holds (with some caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary—more or less any level of intelligence could be combined with more or less any final goal. The second, the instrumental convergence thesis, holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so. In combination, the two theses help us understand the possible range of behavior of superintelligent agents, and they point to some potential dangers in building such an agent.