But if you haven't seen the performance improvements you expected, or still don't feel confident enough to use Spark in production, this practical book is for you. Authors Holden Karau and Rachel Warren demonstrate performance optimizations to help your Spark queries run faster and handle larger data sizes, while using fewer resources.Ideal for software engineers, data engineers, developers, and system administrators working with large-scale data applications, this book describes techniques that can reduce data infrastructure costs and developer hours. Not only will you gain a more comprehensive understanding of Spark, you'll also learn how to make it sing.With this book, you'll explore:How Spark SQL's new interfaces improve performance over SQL's RDD data structureThe choice between data joins in Core Spark and Spark SQLTechniques for getting the most out of standard RDD transformationsHow to work around performance issues in Spark's key/value pair paradigmWriting high-performance Spark code without Scala or the JVMHow to test for functionality and performance when applying suggested improvementsUsing Spark MLlib and Spark ML machine learning librariesSpark's Streaming components and external community packages

    Weinberg adds new insights and highlights the similarities and differences between now and then. Using a conversational style that invites the reader to join him, Weinberg reunites with some of his most insightful writings on the human side of software engineering.Topics include egoless programming, intelligence, psychological measurement, personality factors, motivation, training, social problems on large projects, problem-solving ability, programming language design, team formation, the programming environment, and much more.Dorset House Publishing is proud to make this important text available to new generations of programmers -- and to encourage readers of the first edition to return to its valuable lessons.

    With Step By Step, you set the pace-building and practicing the skills you need, just when you need them! Topics include building a project plan and fine-tuning the details; scheduling tasks, assigning resources, and managing dependencies; monitoring progress and costs; keeping projects on track; communicating project data through Gantt charts and other views.

    You need to assume it will run in the most hostile environments imaginable -- and design, code, and test accordingly. Writing Secure Code, Second Edition shows you how. This edition draws on the lessons learned and taught throughout Microsoft during the firm s massive 2002 Windows Security Push. It s a huge upgrade to the respected First Edition, with new coverage across the board. Michael Howard and David LeBlanc first help you define what security means to your customers -- and implement a three-pronged strategy for securing design, defaults, and deployment. There s especially useful coverage of threat modeling -- decomposing your application, identifying threats, ranking them, and mitigating them. Then, it s on to in-depth coverage of today s key security issues from the developer s standpoint. Everyone knows buffer overruns are bad: Here s a full chapter on avoiding them. You ll learn how to establish appropriate access controls and default to running with least privilege. There s detailed coverage of overcoming attacks on cryptography (for example, avoiding poor random numbers and bit-flipping attacks). You ll learn countermeasures for virtually every form of user input attack, from malicious database updates to cross-site scripting. We ve just scratched the surface: There are authoritative techniques for securing sockets and RPC, protecting against DOS attacks, building safer .NET applications, reviewing and testing code, adding privacy features, and even writing high-quality security documentation. Following these techniques won t just improve security -- it ll dramatically improve robustness and reliability, too. Bill CamardaBill Camarda is a consultant, writer, and web/multimedia content developer. His 15 books include Special Edition Using Word 2000 and Upgrading & Fixing Networks For Dummies®, Second Edition.

    With a blend of software engineering facts and thought-provoking opinions, Fred Brooks offers insight for anyone managing complex projects. These essays draw from his experience as project manager for the IBM System/360 computer family and then for OS/360, its massive software system. Now, 45 years after the initial publication of his book, Brooks has revisited his original ideas and added new thoughts and advice, both for readers already familiar with his work and for readers discovering it for the first time.The added chapters contain (1) a crisp condensation of all the propositions asserted in the original book, including Brooks' central argument in The Mythical Man-Month: that large programming projects suffer management problems different from small ones due to the division of labor; that the conceptual integrity of the product is therefore critical; and that it is difficult but possible to achieve this unity; (2) Brooks' view of these propositions a generation later; (3) a reprint of his classic 1986 paper "No Silver Bullet"; and (4) today's thoughts on the 1986 assertion, "There will be no silver bullet within ten years."

    Expert software designers have specific habits, learned practices, and observed principles that they apply deliberately during their design work. This book offers sixty-six insights, distilled from years of studying experts at work, that capture what successful software designers actually do to create great software.The book presents these insights in a series of two-page illustrated spreads, with the principle and a short explanatory text on one page, and a drawing on the facing page. For example, "Experts generate alternatives" is illustrated by the same few balloons turned into a set of very different balloon animals. The text is engaging and accessible; the drawings are thought-provoking and often playful.Organized into such categories as "Experts reflect," "Experts are not afraid," and "Experts break the rules," the insights range from "Experts prefer simple solutions" to "Experts see error as opportunity." Readers learn that "Experts involve the user"; "Experts take inspiration from wherever they can"; "Experts design throughout the creation of software"; and "Experts draw the problem as much as they draw the solution."One habit for an aspiring expert software designer to develop would be to read and reread this entertaining but essential little book. The insights described offer a guide for the novice or a reference for the veteran--in software design or any design profession.A companion web site provides an annotated bibliography that compiles key underpinning literature, the opportunity to suggest additional insights, and more.

    This book is for anyone who has an interest in SQL Server 2012 and wants to understand its capabilities, including database administrators, application developers, and technical decision makers.

    Multicore machines and 64-bit operating systems are now standard even for casual users, and Java itself has introduced new features to manage applications. The base JVM has kept pace with those developments and offers a very different performance profile in its current versions. By guiding you through this changing landscape, Java Performance: The Definitive Guide helps you gain the best performance from your Java applications.You’ll explore JVM features that traditionally affected performance—including the just-in-time compiler, garbage collection, and language features—before diving in to aspects of Java 7 and 8 designed for maximum performance in today's applications. You’ll learn features such as the G1 garbage collector to maximize your application’s throughput without causing it to pause, and the Java Flight Recorder, which enables you to see application performance details without the need for separate, specialized profiling tools.Whether you’re new to Java and need to understand the basics of tuning the JVM, or a seasoned developer looking to eek out that last 10% of application performance, this is the book you want.

    Advocates of NoSQL databases claim they can be used to build systems that are more performant, scale better, and are easier to program." ""NoSQL Distilled" is a concise but thorough introduction to this rapidly emerging technology. Pramod J. Sadalage and Martin Fowler explain how NoSQL databases work and the ways that they may be a superior alternative to a traditional RDBMS. The authors provide a fast-paced guide to the concepts you need to know in order to evaluate whether NoSQL databases are right for your needs and, if so, which technologies you should explore further. The first part of the book concentrates on core concepts, including schemaless data models, aggregates, new distribution models, the CAP theorem, and map-reduce. In the second part, the authors explore architectural and design issues associated with implementing NoSQL. They also present realistic use cases that demonstrate NoSQL databases at work and feature representative examples using Riak, MongoDB, Cassandra, and Neo4j. In addition, by drawing on Pramod Sadalage's pioneering work, "NoSQL Distilled" shows how to implement evolutionary design with schema migration: an essential technique for applying NoSQL databases. The book concludes by describing how NoSQL is ushering in a new age of Polyglot Persistence, where multiple data-storage worlds coexist, and architects can choose the technology best optimized for each type of data access.

    Emphasis is placed on the basic principles and priorities of the software process. Sections are organized in a natural way for managers and professionals to implement improvement activities.

    Even for people not working with the Unified Process, the book is still of great use. UML 2 and the Unified Process, Second Edition is a must-read for every UML 2 beginner and a helpful guide and reference for the experienced practitioner."--Roland Leibundgut, Technical Director, Zuehlke Engineering Ltd."This book is a good starting point for organizations and individuals who are adopting UP and need to understand how to provide visualization of the different aspects needed to satisfy it. "--Eric Naiburg, Market Manager, Desktop Products, IBM Rational SoftwareThis thoroughly revised edition provides an indispensable and practical guide to the complex process of object-oriented analysis and design using UML 2. It describes how the process of OO analysis and design fits into the software development lifecycle as defined by the Unified Process (UP). UML 2 and the Unified Process contains a wealth of practical, powerful, and useful techniques that you can apply immediately. As you progress through the text, you will learn OO analysis and design techniques, UML syntax and semantics, and the relevant aspects of the UP. The book provides you with an accurate and succinct summary of both UML and UP from the point of view of the OO analyst and designer.This book providesChapter roadmaps, detailed diagrams, and margin notes allowing you to focus on your needs Outline summaries for each chapter, making it ideal for revision, and a comprehensive index that can be used as a reference New to this edition:Completely revised and updated for UML 2 syntax Easy to understand explanations of the new UML 2 semantics More real-world examples A new section on the Object Constraint Language (OCL) Introductory material on the OMG's Model Driven Architecture (MDA) The accompanying website providesA complete example of a simple e-commerce system Open source tools for requirements engineering and use case modeling Industrial-strength UML course materials based on the book

    So, why does conventional wisdom insist that software engineers focus primarily on the design and development of large-scale computing systems?In this collection of essays and articles, key members of Google's Site Reliability Team explain how and why their commitment to the entire lifecycle has enabled the company to successfully build, deploy, monitor, and maintain some of the largest software systems in the world. You'll learn the principles and practices that enable Google engineers to make systems more scalable, reliable, and efficient--lessons directly applicable to your organization.This book is divided into four sections: Introduction--Learn what site reliability engineering is and why it differs from conventional IT industry practicesPrinciples--Examine the patterns, behaviors, and areas of concern that influence the work of a site reliability engineer (SRE)Practices--Understand the theory and practice of an SRE's day-to-day work: building and operating large distributed computing systemsManagement--Explore Google's best practices for training, communication, and meetings that your organization can use

    It covers topics ranging from personal responsibility and career development to architectural techniques for keeping your code flexible and easy to adapt and reuse. Read this book, and you'll learn how toFight software rot; Avoid the trap of duplicating knowledge; Write flexible, dynamic, and adaptable code; Avoid programming by coincidence; Bullet-proof your code with contracts, assertions, and exceptions; Capture real requirements; Test ruthlessly and effectively; Delight your users; Build teams of pragmatic programmers; and Make your developments more precise with automation. Written as a series of self-contained sections and filled with entertaining anecdotes, thoughtful examples, and interesting analogies, The Pragmatic Programmer illustrates the best practices and major pitfalls of many different aspects of software development. Whether you're a new coder, an experienced programmer, or a manager responsible for software projects, use these lessons daily, and you'll quickly see improvements in personal productivity, accuracy, and job satisfaction. You'll learn skills and develop habits and attitudes that form the foundation for long-term success in your career. You'll become a Pragmatic Programmer.

    Test-driven development (TDD) is a modern software development practice that can dramatically reduce the number of defects in systems, produce more maintainable code, and give you the confidence to change your software to meet changing needs. But C++ programmers have been ignored by those promoting TDD--until now. In this book, Jeff Langr gives you hands-on lessons in the challenges and rewards of doing TDD in C++.Modern C++ Programming With Test-Driven Development, the only comprehensive treatment on TDD in C++ provides you with everything you need to know about TDD, and the challenges and benefits of implementing it in your C++ systems. Its many detailed code examples take you step-by-step from TDD basics to advanced concepts. As a veteran C++ programmer, you're already writing high-quality code, and you work hard to maintain code quality. It doesn't have to be that hard. In this book, you'll learn: how to use TDD to improve legacy C++ systems how to identify and deal with troublesome system dependencies how to do dependency injection, which is particularly tricky in C++ how to use testing tools for C++ that aid TDD new C++11 features that facilitate TDD As you grow in TDD mastery, you'll discover how to keep a massive C++ system from becoming a design mess over time, as well as particular C++ trouble spots to avoid. You'll find out how to prevent your tests from being a maintenance burden and how to think in TDD without giving up your hard-won C++ skills. Finally, you'll see how to grow and sustain TDD in your team. Whether you're a complete unit-testing novice or an experienced tester, this book will lead you to mastery of test-driven development in C++.What You Need A C++ compiler running under Windows or Linux, preferably one that supports C++11. Examples presented in the book were built under gcc 4.7.2. Google Mock 1.6 (downloadable for free; it contains Google Test as well) or an alternate C++ unit testing tool. Most examples in the book are written for Google Mock, but it isn't difficult to translate them to your tool of choice. A good programmer's editor or IDE. cmake, preferably. Of course, you can use your own preferred make too. CMakeLists.txt files are provided for each project. Examples provided were built using cmake version 2.8.9. Various freely-available third-party libraries are used as the basis for examples in the book. These include: - cURL - JsonCpp - Boost (filesystem, date_time/gregorian, algorithm, assign) Several examples use the boost headers/libraries. Only one example uses cURL and JsonCpp.

    I'm going to share that mindset with you — along with hundreds of actionable techniques and proven habits — so you can shortcut those years.Introducing The Effective Engineer — the only book designed specifically for today's software engineers, based on extensive interviews with engineering leaders at top tech companies, and packed with hundreds of techniques to accelerate your career.For two years, I embarked on a quest seeking an answer to one question:How do the most effective engineers make their efforts, their teams, and their careers more successful?I interviewed and collected stories from engineering VPs, directors, managers, and other leaders at today's top software companies: established, household names like Google, Facebook, Twitter, and LinkedIn; rapidly growing mid-sized companies like Dropbox, Square, Box, Airbnb, and Etsy; and startups like Reddit, Stripe, Instagram, and Lyft.These leaders shared stories about the most valuable insights they've learned and the most common and costly mistakes that they've seen engineers — sometimes themselves — make.This is just a small sampling of the hard questions I posed to them:- What engineering qualities correlate with future success?- What have you done that has paid off the highest returns?- What separates the most effective engineers you've worked with from everyone else?- What's the most valuable lesson your team has learned in the past year?- What advice do you give to new engineers on your team? Everyone's story is different, but many of the lessons share common themes.You'll get to hear stories like:- How did Instagram's team of 5 engineers build and support a service that grew to over 40 million users by the time the company was acquired?- How and why did Quora deploy code to production 40 to 50 times per day?- How did the team behind Google Docs become the fastest acquisition to rewrite its software to run on Google's infrastructure?- How does Etsy use continuous experimentation to design features that are guaranteed to increase revenue at launch?- How did Facebook's small infrastructure team effectively operate thousands of database servers?- How did Dropbox go from barely hiring any new engineers to nearly tripling its team size year-over-year? What's more, I've distilled their stories into actionable habits and lessons that you can follow step-by-step to make your career and your team more successful.The skills used by effective engineers are all learnable.And I'll teach them to you. With The Effective Engineer, I'll teach you a unifying framework called leverage — the value produced per unit of time invested — that you can use to identify the activities that produce disproportionate results.Here's a sneak peek at some of the lessons you'll learn. You'll learn how to:- Prioritize the right projects and tasks to increase your impact.- Earn more leeway from your peers and managers on your projects.- Spend less time maintaining and fixing software and more time building and shipping new features.- Produce more accurate software estimates.- Validate your ideas cheaply to reduce wasted work.- Navigate organizational and people-related bottlenecks.- Find the appropriate level of code reviews, testing, abstraction, and technical debt to balance speed and quality.- Shorten your debugging workflow to increase your iteration speed.