Table of Contents
Preface xi
Chapter 1: Getting Started – One Environment per Project 1
Creating a virtual Python environment using venv 2
Creating your frst venv 3
venv arguments 4
Differences between virtualenv and venv 6
Bootstrapping pip using ensurepip 7
ensurepip usage 7
Manual pip install 7
Installing C/C++ packages 8
Debian and Ubuntu 9
Red Hat, CentOS, and Fedora 9
OS X 9
Windows 10
Summary 11
Chapter 2: Pythonic Syntax, Common Pitfalls, and Style Guide 13
Code style – or what is Pythonic code? 14
Formatting strings – printf-style or str.format? 15
PEP20, the Zen of Python 15
Beautiful is better than ugly 16
Explicit is better than implicit 17
Simple is better than complex 18
Flat is better than nested 20
Sparse is better than dense 20
Readability counts 21
Practicality beats purity 21
Errors should never pass silently 22
In the face of ambiguity, refuse the temptation to guess 24
One obvious way to do it 24
Now is better than never 25
Hard to explain, easy to explain 25
Namespaces are one honking great idea 25
Conclusion 26
Explaining PEP8 27
Duck typing 27
Differences between value and identity comparisons 29
Loops 30
Maximum line length 31
Verifying code quality, pep8, pyflakes, and more 32
flake8 32
Pylint 35
Common pitfalls 35
Scope matters! 36
Function arguments 36
Class properties 37
Modifying variables in the global scope 38
Overwriting and/or creating extra built-ins 39
Modifying while iterating 41
Catching exceptions – differences between Python 2 and 3 42
Late binding – be careful with closures 44
Circular imports 45
Import collisions 47
Summary 48
Chapter 3: Containers and Collections – Storing Data the
Right Way 49
Time complexity – the big O notation 50
Core collections 51
list – a mutable list of items 52
dict – unsorted but a fast map of items 55
set – like a dict without values 57
tuple – the immutable list 59
Advanced collections 62
ChainMap – the list of dictionaries 62
counter – keeping track of the most occurring elements 64
deque – the double ended queue 66
defaultdict – dictionary with a default value 68
namedtuple – tuples with feld names 71
enum – a group of constants 72
OrderedDict – a dictionary where the insertion order matters 74
heapq – the ordered list 75
bisect – the sorted list 76
Summary 79
Chapter 4: Functional Programming – Readability Versus Brevity 81
Functional programming 82
list comprehensions 82
dict comprehensions 85
set comprehensions 86
lambda functions 86
The Y combinator 87
functools 89
partial – no need to repeat all arguments every time 90
reduce – combining pairs into a single result 91
Implementing a factorial function 91
Processing trees 93
itertools 95
accumulate – reduce with intermediate results 95
chain – combining multiple results 95
combinations – combinatorics in Python 96
permutations – combinations where the order matters 97
compress – selecting items using a list of Booleans 98
dropwhile/takewhile – selecting items using a function 98
count – infnite range with decimal steps 98
groupby – grouping your sorted iterable 100
islice – slicing any iterable 101
Summary 102
Chapter 5: Decorators – Enabling Code Reuse by Decorating 103
Decorating functions 104
Why functools.wraps is important 105
How are decorators useful? 107
Memoization using decorators 109
Decorators with (optional) arguments 111
Creating decorators using classes 115
Decorating class functions 116
Skipping the instance – classmethod and staticmethod 116
Properties – smart descriptor usage 121
Decorating classes 125
Singletons – classes with a single instance 125
Total ordering – sortable classes the easy way 126
Useful decorators 130
Single dispatch – polymorphism in Python 130
Contextmanager, with statements made easy 133
Validation, type checks, and conversions 135
Useless warnings – how to ignore them 138
Summary 140
Chapter 6: Generators and Coroutines – Infnity, One Step
at a Time 141
What are generators? 142
Advantages and disadvantages of generators 145
Pipelines – an effective use of generators 146
tee – using an output multiple times 148
Generating from generators 149
Context managers 151
Coroutines 154
A basic example 154
Priming 155
Closing and throwing exceptions 156
Bidirectional pipelines 158
Using the state 162
Summary 166
Chapter 7: Async IO – Multithreading without Threads 167
Introducing the asyncio library 168
The async and await statements 168
Python 3.4 169
Python 3.5 169
Choosing between the 3.4 and 3.5 syntax 170
A simple example of single-threaded parallel processing 171
Concepts of asyncio 172
Futures and tasks 172
Event loops 174
Processes 180
Asynchronous servers and clients 185
Basic echo server 185
Summary 188
Chapter 8: Metaclasses – Making Classes (Not Instances)
Smarter 189
Dynamically creating classes 190
A basic metaclass 191
Arguments to metaclasses 193
Accessing metaclass attributes through classes 193
Abstract classes using collections.abc 194
Internal workings of the abstract classes 195
Custom type checks 199
Using abc.ABC before Python 3.4 201
Automatically registering a plugin system 201
Importing plugins on-demand 204
Importing plugins through confguration 205
Importing plugins through the fle system 206
Order of operations when instantiating classes 207
Finding the metaclass 208
Preparing the namespace 208
Executing the class body 208
Creating the class object (not instance) 209
Executing the class decorators 209
Creating the class instance 209
Example 209
Storing class attributes in defnition order 212
The classic solution without metaclasses 212
Using metaclasses to get a sorted namespace 213
Summary 215
Chapter 9: Documentation – How to Use Sphinx and
reStructuredText 217
The reStructuredText syntax 218
Getting started with reStructuredText 219
Inline markup 219
Headers 221
Lists 223
Enumerated list 223
Bulleted list 224
Option list 225
Defnition list 225
Nested lists 226
Links, references, and labels 227
Images 229
Substitutions 231
Blocks, code, math, comments, and quotes 232
Conclusion 233
The Sphinx documentation generator 233
Getting started with Sphinx 234
Using sphinx-quickstart 234
Using sphinx-apidoc 239
Sphinx directives 243
The table of contents tree directive (toctree) 243
Autodoc, documenting Python modules, classes, and functions 244
Sphinx roles 247
Documenting code 249
Documenting a class with the Sphinx style 250
Documenting a class with the Google style 252
Documenting a class with the NumPy style 253
Which style to choose 254
Summary 255
Chapter 10: Testing and Logging – Preparing for Bugs 257
Using examples as tests with doctest 258
A simple doctest example 258
Writing doctests 263
Testing with pure documentation 264
The doctest flags 267
True and False versus 1 and 0 268
Normalizing whitespace 269
Ellipsis 270
Doctest quirks 271
Testing dictionaries 271
Testing floating-point numbers 273
Times and durations 273
Testing with py.test 274
The difference between the unittest and py.test output 275
The difference between unittest and py.test tests 280
Simplifying assertions 281
Parameterizing tests 286
Automatic arguments using fxtures 287
Print statements and logging 291
Plugins 293
Mock objects 302
Using unittest.mock 302
Using py.test monkeypatch 304
Logging 305
Confguration 306
Basic logging confguration 306
Dictionary confguration 307
JSON confguration 308
Ini fle confguration 309
The network confguration 310
Logger 314
Usage 315
Summary 317
Chapter 11: Debugging – Solving the Bugs 319
Non-interactive debugging 320
Inspecting your script using trace 321
Table of Contents
[ vii ]
Debugging using logging 325
Showing call stack without exceptions 327
Debugging asyncio 329
Handling crashes using faulthandler 331
Interactive debugging 332
Console on demand 332
Debugging using pdb 333
Breakpoints 335
Catching exceptions 338
Commands 340
Debugging using ipdb 341
Other debuggers 343
Debugging services 343
Summary 344
Chapter 12: Performance – Tracking and Reducing Your
Memory and CPU Usage 345
What is performance? 346
Timeit – comparing code snippet performance 347
cProfle – fnding the slowest components 351
First profling run 351
Calibrating your profler 353
Selective profling using decorators 356
Using profle statistics 358
Line profler 361
Improving performance 363
Using the right algorithm 363
Global interpreter lock 363
Try versus if 364
Lists versus generators 364
String concatenation 364
Addition versus generators 365
Map versus generators and list comprehensions 366
Caching 366
Lazy imports 367
Using optimized libraries 367
Just-in-time compiling 368
Converting parts of your code to C 368
Memory usage 369
Tracemalloc 369
Memory profler 370
Memory leaks 372
Reducing memory usage 378
Generators versus lists 379
Recreating collections versus removing items 380
Using slots 380
Performance monitoring 382
Summary 383
Chapter 13: Multiprocessing – When a Single CPU Core Is
not Enough 385
Multithreading versus multiprocessing 385
Hyper-threading versus physical CPU cores 388
Creating a pool of workers 390
Sharing data between processes 392
Remote processes 393
Distributed processing using multiprocessing 393
Distributed processing using IPyparallel 396
ipython_confg.py 397
ipython_kernel_confg.py 397
ipcontroller_confg.py 397
ipengine_confg.py 399
ipcluster_confg.py 399
Summary 402
Chapter 14: Extensions in C/C++, System Calls, and
C/C++ Libraries 403
Introduction 403
Do you need C/C++ modules? 404
Windows 404
OS X 404
Linux/Unix 405
Calling C/C++ with ctypes 406
Platform-specifc libraries 406
Windows 406
Linux/Unix 407
OS X 407
Making it easy 408
Calling functions and native types 408
Complex data structures 410
Arrays 410
Gotchas with memory management 412
CFFI 413
Complex data structures 414
Arrays 415
ABI or API? 415
CFFI or ctypes? 415
Native C/C++ extensions 416
A basic example 416
C is not Python – size matters 419
The example explained 421
static 421
PyObject* 422
Parsing arguments 422
C is not Python – errors are silent or lethal 424
Calling Python from C – handling complex types 425
Summary 427
Chapter 15: Packaging – Creating Your Own Libraries
or Applications 429
Installing packages 429
Setup parameters 430
Packages 434
Entry points 434
Creating global commands 434
Custom setup.py commands 435
Package data 438
Testing packages 439
Unittest 439
py.test 440
Nosetests 442
C/C++ extensions 443
Regular extensions 444
Cython extensions 444
Wheels – the new eggs 446
Distributing to the Python Package Index 447
Summary 449
Index 451
- Log in to post comments