With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once. Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice. Learn how to parse complicated HTML pages Traverse multiple pages and sites Get a general overview of APIs and how they work Learn several methods for storing the data you scrape Download, read, and extract data from documents Use tools and techniques to clean badly formatted data Read and write natural languages Crawl through forms and logins Understand how to scrape JavaScript Learn image processing and text recognition

    The pytest testing framework helps you write tests quickly and keep them readable and maintainable - with no boilerplate code. Using a robust yet simple fixture model, it's just as easy to write small tests with pytest as it is to scale up to complex functional testing for applications, packages, and libraries. This book shows you how.For Python-based projects, pytest is the undeniable choice to test your code if you're looking for a full-featured, API-independent, flexible, and extensible testing framework. With a full-bodied fixture model that is unmatched in any other tool, the pytest framework gives you powerful features such as assert rewriting and plug-in capability - with no boilerplate code.With simple step-by-step instructions and sample code, this book gets you up to speed quickly on this easy-to-learn and robust tool. Write short, maintainable tests that elegantly express what you're testing. Add powerful testing features and still speed up test times by distributing tests across multiple processors and running tests in parallel. Use the built-in assert statements to reduce false test failures by separating setup and test failures. Test error conditions and corner cases with expected exception testing, and use one test to run many test cases with parameterized testing. Extend pytest with plugins, connect it to continuous integration systems, and use it in tandem with tox, mock, coverage, unittest, and doctest.Write simple, maintainable tests that elegantly express what you're testing and why.What You Need: The examples in this book are written using Python 3.6 and pytest 3.0. However, pytest 3.0 supports Python 2.6, 2.7, and Python 3.3-3.6.

    Web Development With Clojure shows you how to apply Clojure programming fundamentals to build real-world solutions. You'll develop all the pieces of a full web application in this powerful language. If you already have some familiarity with Clojure, you'll learn how to put it to serious practical use. If you're new to the language, the book provides just enough Clojure to get down to business.You'll learn the full process of web development using Clojure while getting hands-on experience with current tools, libraries, and best practices in the language. You'll develop Clojure apps with both the Light Table and Eclipse development environments. Rather than frameworks, Clojure development builds on rich libraries. You'll acquire expertise in the popular Ring/Compojure stack, and you'll learn to use the Liberator library to quickly develop RESTful services. Plus, you'll find out how to use ClojureScript to work in one language on the client and server sides.Throughout the book, you'll develop key components of web applications, including multiple approaches to database access. You'll create a simple guestbook app and an app to serve resources to users. By the end, you will have developed a rich Picture Gallery web application from conception to packaging and deployment.This book is for anyone interested in taking the next step in web development.Q&A with Dmitri SotnikovWhy did you write Web Development with Clojure?When I started using Clojure, I found that it took a lot of work to find all the pieces needed to put together a working application. There was very little documentation available on how to organize the code, what libraries to use, or how to package the application for deployment. Having gone through the process of figuring out what works, I thought that it would be nice to make it easier for others to get started.What are the advantages of using a functional language?Over the course of my career, I have developed a great appreciation for functional programming. I find that it addresses a number of shortcomings present in the imperative paradigm. For example, in a functional language any changes to the data are created via revisions to the existing data. So they only exist in the local scope. This fact allows us to safely reason about individual parts of the program in isolation, which is critical for writing and supporting large applications.Why use Clojure specifically?Clojure is a simple and pragmatic language that is designed for real-world usage. It combines the productivity of a high-level language with the excellent performance seen in languages like C# or Java. It's also very easy to learn because it allows you to use a small number of concepts to solve a large variety of problems.If I already have a preferred web development platform, what might I get out of this book?If you're using an imperative language, you'll get to see a very different approach to writing code. Even if you're not going to use Clojure as your primary language, the concepts you'll learn will provide you with new ways to approach problems.Is the material in the book accessible to somebody who is not familiar with Clojure?Absolutely. The book targets developers who are already familiar with the basics of web development and are interested in learning Clojure in this context. The book introduces just enough of the language to get you productive and allows you to learn by example.

    Each chapter gives you the complete source code for a new game and teaches the programming concepts from these examples. The book is available under a Creative Commons license and can be downloaded in full for free from http: //inventwithpython.com/pygame This book was written to be understandable by kids as young as 10 to 12 years old, although it is great for anyone of any age who has some familiarity with Python.

    It provides a high-productivity programming model that promotes cleaner code architecture, test-driven development, and powerful extensibility, combined with all the benefits of ASP.NET.ASP.NET MVC 4 contains a number of significant advances over previous versions. New mobile and desktop templates (employing adaptive rendering) are included together with support for jQuery Mobile for the first time. New display modes allow your application to select views based on the browser that's making the request while Code Generation Recipes for Visual Studio help you auto-generate project-specific code for a wide variety of situtations including NuGet support.In this fourth edition, the core model-view-controller (MVC) architectural concepts are not simply explained or discussed in isolation, but are demonstrated in action. You'll work through an extended tutorial to create a working e-commerce web application that combines ASP.NET MVC with the latest C# language features and unit-testing best practices. By gaining this invaluable, practical experience, you'll discover MVC's strengths and weaknesses for yourself--and put your best-learned theory into practice.The book's authors, Steve Sanderson and Adam Freeman, have both watched the growth of ASP.NET MVC since its first release. Steve is a well-known blogger on the MVC Framework and a member of the Microsoft Web Platform and Tools team. Adam started designing and building web applications 15 years ago and has been responsible for some of the world's largest and most ambitious projects. You can be sure you are in safe hands.

    You'll learn how to confidently and securely build and deploy OAuth on both the client and server sides. Foreword by Ian Glazer.Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.About the TechnologyThink of OAuth 2 as the web version of a valet key. It is an HTTP-based security protocol that allows users of a service to enable applications to use that service on their behalf without handing over full control. And OAuth is used everywhere, from Facebook and Google, to startups and cloud services.About the BookOAuth 2 in Action teaches you practical use and deployment of OAuth 2 from the perspectives of a client, an authorization server, and a resource server. You'll begin with an overview of OAuth and its components and interactions. Next, you'll get hands-on and build an OAuth client, an authorization server, and a protected resource. Then you'll dig into tokens, dynamic client registration, and more advanced topics. By the end, you'll be able to confidently and securely build and deploy OAuth on both the client and server sides.What's InsideCovers OAuth 2 protocol and designAuthorization with OAuth 2OpenID Connect and User-Managed AccessImplementation risksJOSE, introspection, revocation, and registrationProtecting and accessing REST APIsAbout the ReaderReaders need basic programming skills and knowledge of HTTP and JSON.About the AuthorJustin Richer is a systems architect and software engineer. Antonio Sanso is a security software engineer and a security researcher. Both authors contribute to open standards and open source.Table of ContentsPart 1 - First stepsWhat is OAuth 2.0 and why should you care?The OAuth dance Part 2 - Building an OAuth 2 environmentBuilding a simple OAuth clientBuilding a simple OAuth protected resourceBuilding a simple OAuth authorization serverOAuth 2.0 in the real world Part 3 - OAuth 2 implementation and vulnerabilitiesCommon client vulnerabilitiesCommon protected resources vulnerabilitiesCommon authorization server vulnerabilitiesCommon OAuth token vulnerabilities Part 4 - Taking OAuth furtherOAuth tokensDynamic client registrationUser authentication with OAuth 2.0Protocols and profiles using OAuth 2.0Beyond bearer tokensSummary and conclusions

    However, Python’s unique strengths, charms, and expressiveness can be hard to grasp, and there are hidden pitfalls that can easily trip you up. This second edition of Effective Python will help you master a truly “Pythonic” approach to programming, harnessing Python’s full power to write exceptionally robust and well-performing code. Using the concise, scenario-driven style pioneered in Scott Meyers’ best-selling Effective C++, Brett Slatkin brings together 90 Python best practices, tips, and shortcuts, and explains them with realistic code examples so that you can embrace Python with confidence. Drawing on years of experience building Python infrastructure at Google, Slatkin uncovers little-known quirks and idioms that powerfully impact code behavior and performance. You’ll understand the best way to accomplish key tasks so you can write code that’s easier to understand, maintain, and improve. In addition to even more advice, this new edition substantially revises all items from the first edition to reflect how best practices have evolved. Key features include 30 new actionable guidelines for all major areas of Python Detailed explanations and examples of statements, expressions, and built-in types Best practices for writing functions that clarify intention, promote reuse, and avoid bugs Better techniques and idioms for using comprehensions and generator functions Coverage of how to accurately express behaviors with classes and interfaces Guidance on how to avoid pitfalls with metaclasses and dynamic attributes More efficient and clear approaches to concurrency and parallelism Solutions for optimizing and hardening to maximize performance and quality Techniques and built-in modules that aid in debugging and testing Tools and best practices for collaborative development   Effective Python will prepare growing programmers to make a big impact using Python.

    You may still be drawing a paycheck, but the job you were hired to do no longer exists. Your company has changed, the technology has changed, the economy has changed, and the ways you can add value have changed. Have you adapted to these changes? Or are you at risk? Architect your career Economic downturn. Job cuts. Outsourcing. The ever-changing tech landscape. The threats abound. Chad Fowler is here to offer 52 ways to keep your job, despite the vagaries of the market.It's all about making the right choices. Choosing which technologies to focus on and which business domains to master have at least as much impact on your success as your technical knowledge--don't let those choices be accidental. Chad shows you all aspects of the decision-making process so you can ensure that you're investing your time and energy in the right areas.It's all about skills. You'll develop a structured plan for keeping your skills up-to-date so that you can compete with both the growing stable of developers in so-called low-cost countries as well as your higher-priced local peers. You'll learn how to shift your skillset up the value chain, from an offshore-ready commodity to one in high demand.It's all about marketing. As with any product or service, if nobody knows what you're selling, nobody will buy. Chad shows you how to create a plan for marketing yourself both inside your company and to the industry in general.Like it or not, the IT career landscape has changed. This handbook will teach you what you need to do to avoid being left behind. About the author Chad Fowler has been a software developer and manager for some of the world's largest corporations. He recently lived and worked in India, setting up and leading an offshore software development center for a large multinational company.

    This third edition has been updated toreflect current best practices andthe abundance of changes introduced by the latest version of thelanguage, Python 2.5.Whether you're a novice or an advancedpractitioner, you'll find thisrefreshed book more than lives up to its reputation. "ProgrammingPython," 3rd Edition, teaches you the rightway to code. It explains Python language syntax and programmingtechniques in a clear and concisemanner, with numerous examples that illustrate both correct usage andcommon idioms. By reading thiscomprehensive guide, you'll learn how to apply Python in real-worldproblem domains such as: GUI programmingInternet scriptingParallel processingDatabase managementNetworked applications"Programming Python," Third Edition coverseach of thesetarget domainsgradually, beginning with in-depth discussions of core concepts andthen progressing toward completeprograms. Large examples do appear, but only after you've learnedenough to understand their techniques andcode.Along the way, you'll also learn how to use the Python language inrealistically scaled programs--concepts such as Object-Oriented Programming (OOP) and code reuseare recurring side themes throughout thistext. If you're interested in Python programming, then this O'Reillyclassic needs to be within arm's reach. Thewealth of practical advice, snippets of code, and patterns of programdesign can all be put into use on adaily basis--making your life easier and more productive.Reviews of the second edition:.".".about as comprehensive as any book can be.""--Dr. Dobb's Journal""If the language had manuals, they would undoubtedlybe the texts from O'Reilly...'Learning Python' and 'Programming Python'are definitive treatments.""--SD Times

    If programming intrigues you (for whatever reason), Beginning Programming All-In-One Desk Reference For Dummies is like having a starter programming library all in one handy, if hefty, book.In this practical guide, you'll find out about algorithms, best practices, compiling, debugging your programs, and much more. The concepts are illustrated in several different programming languages, so you'll get a feel for the variety of languages and the needs they fill.Inside you'll discover seven minibooks:Getting Started: From learning methods for writing programs to becoming familiar with types of programming languages, you'll lay the foundation for your programming adventure with this minibook. Programming Basics: Here you'll dive into how programs work, variables, data types, branching, looping, subprograms, objects, and more. Data Structures: From structures, arrays, sets, linked lists, and collections, to stacks, queues, graphs, and trees, you'll dig deeply into the data. Algorithms: This minibook shows you how to sort and search algorithms, how to use string searching, and gets into data compression and encryption. Web Programming: Learn everything you need to know about coding for the web: HyperText. Markup Language (better known simply as HTML), CSS, JavaScript, PHP, and Ruby. Programming Language Syntax: Introduces you to the syntax of various languages - C, C++, Java, C#, Perl, Python, Pascal, Delphi, Visual Basic, REALbasic - so you know when to use which one. Applications: This is the fun part where you put your newly developed programming skills to work in practical ways. Additionally, Beginning Programming All-In-One Desk Reference For Dummies shows you how to decide what you want your program to do, turn your instructions into "machine language" that the computer understands, use programming best practices, explore the "how" and "why" of data structuring, and more. And you'll get a look into various applications like database management, bioinformatics, computer security, and artificial intelligence. After you get this book and start coding, you'll soon realize that -- wow! You're a programmer!

    With the sophisticated algorithms in this book, you can write smart programs to access interesting datasets from other web sites, collect data from users of your own applications, and analyze and understand the data once you've found it.Programming Collective Intelligence takes you into the world of machine learning and statistics, and explains how to draw conclusions about user experience, marketing, personal tastes, and human behavior in general -- all from information that you and others collect every day. Each algorithm is described clearly and concisely with code that can immediately be used on your web site, blog, Wiki, or specialized application. This book explains:Collaborative filtering techniques that enable online retailers to recommend products or media Methods of clustering to detect groups of similar items in a large dataset Search engine features -- crawlers, indexers, query engines, and the PageRank algorithm Optimization algorithms that search millions of possible solutions to a problem and choose the best one Bayesian filtering, used in spam filters for classifying documents based on word types and other features Using decision trees not only to make predictions, but to model the way decisions are made Predicting numerical values rather than classifications to build price models Support vector machines to match people in online dating sites Non-negative matrix factorization to find the independent features in a dataset Evolving intelligence for problem solving -- how a computer develops its skill by improving its own code the more it plays a game Each chapter includes exercises for extending the algorithms to make them more powerful. Go beyond simple database-backed applications and put the wealth of Internet data to work for you. "Bravo! I cannot think of a better way for a developer to first learn these algorithms and methods, nor can I think of a better way for me (an old AI dog) to reinvigorate my knowledge of the details."-- Dan Russell, Google "Toby's book does a great job of breaking down the complex subject matter of machine-learning algorithms into practical, easy-to-understand examples that can be directly applied to analysis of social interaction across the Web today. If I had this book two years ago, it would have saved precious time going down some fruitless paths."-- Tim Wolters, CTO, Collective Intellect

    because the language seems like a good way to accomplish programming tasks that don't require the low-level bit handling power of C.-- Richard Bejtlich, TaoSecurityPython is a new and innovative scripting language. It is set to replace Perl as the programming language of choice for shell scripters, and for serious application developers who want a feature-rich, yet simple language to deploy their products.Dive Into Python is a hands-on guide to the Python language. Each chapter starts with a real, complete code sample, proceeds to pick it apart and explain the pieces, and then puts it all back together in a summary at the end.This is the perfect resource for you if you like to jump into languages fast and get going right away. If you're just starting to learn Python, first pick up a copy of Magnus Lie Hetland's Practical Python.

    After a year of self-study, I learned to program well enough to land a job as a software engineer II at eBay. Once I got there, I realized I was severely under-prepared. I was overwhelmed by the amount of things I needed to know but hadn't learned yet. My journey learning to program, and my experience at my first job as a software engineer were the inspiration for this book. This book is not just about learning to program; although you will learn to code. If you want to program professionally, it is not enough to learn to code; that is why, in addition to helping you learn to program, I also cover the rest of the things you need to know to program professionally that classes and books don't teach you. "The Self-taught Programmer" is a roadmap, a guide to take you from writing your first Python program, to passing your first technical interview. I divided the book into five sections: 1. Start to program in Python 3 and build your first program.2. Learn Object-oriented programming and create a powerful Python program to get you hooked.3. Learn to use tools like Git, Bash, and regular expressions. Then use your new coding skills to build a web scraper.4. Study Computer Science fundamentals like data structures and algorithms.5. Finish with best coding practices, tips for working with a team, and advice on landing a programming job.You CAN learn to program professionally. The path is there. Will you take it?

    Uses progressive enhancement techniques to ensure graceful degradation (which makes sites usable in all browsers). Shows readers how to write their own Ajax scripts instead of relying on third-party libraries. Web site designers love the idea of Ajax--of creating Web pages in which information can be updated without refreshing the entire page. But for those who aren't hard-core programmers, enhancing pages using Ajax can be a challenge. Even more of a challenge is making sure those pages work for all users. In Bulletproof Ajax, author Jeremy Keith demonstrates how developers comfortable with CSS and (X)HTML can build Ajax functionality without frameworks, using the ideas of graceful degradation and progressive enhancement to ensure that the pages work for all users. Throughout this step-by-step guide, his emphasis is on best practices with an approach to building Ajax pages called Hijax, which improves flexibility and avoids worst-case scenarios.

    This cookbook includes more than 100 recipes to help you take advantage of REST, HTTP, and the infrastructure of the Web. You'll learn ways to design RESTful web services for client and server applications that meet performance, scalability, reliability, and security goals, no matter what programming language and development framework you use.Each recipe includes one or two problem statements, with easy-to-follow, step-by-step instructions for solving them, as well as examples using HTTP requests and responses, and XML, JSON, and Atom snippets. You'll also get implementation guidelines, and a discussion of the pros, cons, and trade-offs that come with each solution.Learn how to design resources to meet various application scenariosSuccessfully design representations and URIsImplement the hypertext constraint using links and link headersUnderstand when and how to use Atom and AtomPubKnow what and what not to do to support cachingLearn how to implement concurrency controlDeal with advanced use cases involving copying, merging, transactions, batch processing, and partial updatesSecure web services and support OAuth