Category Archives: Tutorials

Tutorial materials

Why import * is not allowed outside the global scope

So today I was looking at import machinery in Python due to this question about import styles in Persephone, an open source natural language processing library I’ve been contributing to.

Looking around at this more I’ve found a couple of things that I didn’t realize before:

Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo():
...     from math import *
...     return sqrt(4)
...
<stdin>:1: SyntaxWarning: import * only allowed at module level
>>> foo()
2.0

This warning was introduced in Python 2.1 via PEP 227 the 2.1 release was the same release that introduced the Warnings framework itself. Nested scopes can break the import * so the language specification specifically prohibits this. However as you can see the CPython 2.x implementation doesn’t enforce it, so this produces a warning but works when you try to use it.

Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo():
...    from math import *
...    return sqrt(4)
...
File "<stdin>", line 1
SyntaxError: import * only allowed at module level

This produces a syntax error. The Python3 approach here is much nicer considering that the code is prohibited by the spec. Why this happens is a lot more interesting though, and ultimately comes down to efficiency.

Basically a lot of efficiency gains can be had if the local variables are known at function parsing time.

So lets look at what a recent CPython implementation does. First lets look at FrameObject in include/frameobject.h:

typedef struct _frame {
    PyObject_VAR_HEAD
    struct _frame *f_back;      /* previous frame, or NULL */
    PyCodeObject *f_code;       /* code segment */
    PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */
    PyObject *f_globals;        /* global symbol table (PyDictObject) */
    PyObject *f_locals;         /* local symbol table (any mapping) */
    PyObject **f_valuestack;    /* points after the last local */
    /* Next free slot in f_valuestack.  Frame creation sets to f_valuestack.
       Frame evaluation usually NULLs it, but a frame that yields sets it
       to the current stack top. */
    PyObject **f_stacktop;
    PyObject *f_trace;          /* Trace function */
    char f_trace_lines;         /* Emit per-line trace events? */
    char f_trace_opcodes;       /* Emit per-opcode trace events? */

    /* Borrowed reference to a generator, or NULL */
    PyObject *f_gen;

    int f_lasti;                /* Last instruction if called */
    /* Call PyFrame_GetLineNumber() instead of reading this field
       directly.  As of 2.3 f_lineno is only valid when tracing is
       active (i.e. when f_trace is set).  At other times we use
       PyCode_Addr2Line to calculate the line from the current
       bytecode index. */
    int f_lineno;               /* Current line number */
    int f_iblock;               /* index in f_blockstack */
    char f_executing;           /* whether the frame is still executing */
    PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */
    PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */
} PyFrameObject;

This represents a Python stack frame.

Note we have f_localsplus which stores the locals variables.

Now how these frames are evaluated is interesting, because stack frames are frequently created and destroyed there is an optimization used to make function calls faster. Each code object has a “zombie frame” associated with it that allows a faster call because the frame itself doesn’t need to have memory allocated for it during the call. This frame gets created at the same time as the code object and has its memory allocated at creation time, so when the call happens this memory just gets reused and the required values are filled into this pre-existing frame directly.

There’s a particularly informative comment here in Objects/frameobject.c

 /* Stack frames are allocated and deallocated at a considerable rate.
   In an attempt to improve the speed of function calls, we:
   1. Hold a single "zombie" frame on each code object. This retains
   the allocated and initialised frame object from an invocation of
   the code object. The zombie is reanimated the next time we need a
   frame object for that code object. Doing this saves the malloc/
   realloc required when using a free_list frame that isn't the
   correct size. It also saves some field initialisation.
   In zombie mode, no field of PyFrameObject holds a reference, but
   the following fields are still valid:
     * ob_type, ob_size, f_code, f_valuestack;
     * f_locals, f_trace are NULL;
     * f_localsplus does not require re-allocation and
       the local variables in f_localsplus are NULL.

What this design means is that the struct member f_localsplus is created at the time that the code_object is created. (The code object is created when the function is first encountered) So it means that all local variable names must be defined at that point in time the function is defined so that these variables can be stored in this part of the frameobject structure. Not needing to constantly calculate the number of local variables to allocate and deallocate the memory needed for the function call is a significant win. This way the function when called doesn’t need to find the local variables, they are already there and they just need to be populated with the values. As a result of this optimization from mod import * must be made illegal within the function scope and hence this is why the syntax error is raised.

For an example of some implementation code that requires this in Objects/frameobject.c:

 static void _Py_HOT_FUNCTION
frame_dealloc(PyFrameObject *f)
{
    PyObject **p, **valuestack;
    PyCodeObject *co;

    if (_PyObject_GC_IS_TRACKED(f))
        _PyObject_GC_UNTRACK(f);

    Py_TRASHCAN_SAFE_BEGIN(f)
    /* Kill all local variables */
    valuestack = f->f_valuestack;
    for (p = f->f_localsplus; p < valuestack; p++)
        Py_CLEAR(*p);

This fast clearing of locals to enable stack frame reuse can only work if f_localsplus is always the same size. If the size could be different between stack frames a lot more checks would have to be inserted, but as discussed above this would be a performance hit in a very critical part of the implementation.

How a web page request makes it down to the metal

The other day I was interested in how many steps occur between sending a POST or GET request from a website to the actual processing that happens on the CPU of the server. I figured that I knew bits and pieces of the puzzle but I wanted to see the complete path from the highest levels of abstraction all the way to the lowest without missing anything too big in-between. It turns out that in a modern web system there are a lot of steps. I have been really fascinated by this much like the explorer that wants to find a path from one known place to another. If you are interested in better understanding how your computer works you might find walking along this path with your tech stack helpful.

Continue reading

Interfacing Python with C or C++

One of the main downsides of writing applications in Python is that they can run slow. When you run into an insurmountable runtime speed issue the commonly accepted wisdom is to either use a more efficient Python implementation such as PyPy or to write the performance critical code in a more efficient compiled language (usually C or C++).

In this article I’m going to focus on ways to integrate fast compiled code for use by Python code, I usually make a point of consider the feasibility of PyPy vs the costs of using a different implementation as it can give you great performance increases with less added complexity.

If you want to use C or C++ compiled code in your Python project here’s a quick summary of the options available to you:

If there’s an important option missing please leave a comment with information on it!

Examples

A good way to demonstrate some of these options is via a simple example.

Lets say we want to compute a factorial recursively and we are using a fairly naive recursive approach. (Note that function calls in Python are slow which is a non-accidental part of this example, sometimes we could choose a better algorithm, but from the point of view of an example this is a fairly simple function where choosing a better language implementation would result in a win.)

Here’s the python code equivalent:

def factorial(n):
    """Compute a factorial of n"""
    if n <= 0:
        return 1
    else:
        return n * factorial(n - 1)

Use the Python C interface

If you are using the cPython implementation you can use the C interface to call out to C code. Here’s an example:

#include <Python.h>

/* Computes the factorial value we need for the example */
int
compute_it(int x){
    if(x <= 0){
        return 1;
    }else{
        return compute_it(x - 1) * x;
    }
}

/* Python binding for our compute_it function */
static PyObject *
expensive_computations_compute_it(PyObject *self, PyObject *args){
    int input, result;

    if(!PyArg_ParseTuple(args, "i", &input)){
        return NULL;
    }
    result = compute_it(input);
    return Py_BuildValue("i", result);
}

/* Methods for our expensive_computations object*/
static PyMethodDef ExpensiveComputationMethods[] = {
    {"compute_it", expensive_computations_compute_it, METH_VARARGS, "Computes our test factorial function"},
    {NULL, NULL, 0, NULL} /* End of methods sential value */
};


/* define the module */
static struct PyModuleDef expensive_computations_module = {
    PyModuleDef_HEAD_INIT,
    "demo_module",
    -1,
    ExpensiveComputationMethods
};

/* Init code */
PyMODINIT_FUNC
PyInit_demo_module(void){
    return PyModule_Create(&expensive_computations_module);
}

int main(int argv, char* argc[]){
    /*Init the python interpreter */
    Py_Initialize();

    return 0;
}

More can be found about this way of going about extending Python in the docs.

Note that if your use case is just calling C library functions or system calls this is a poor choice, if you aren’t writing any custom code CFFI is a lot better in that case (not to mention more portable). One other thing that is worth mentioning is that there’s a lot of boilerplate here to convert between Python and C types. This is something that is handled better with some of the other libraries.

Interface Python with C++

Previously the easiest way to do this was with Boost.Python as the library greatly reduced the boilerplate. However boost is a huge dependency to introduce to a project and can be a massive pain to compile. (This is a fairly substantial downside if you weren’t already using boost with your c++ code.) Thankfully there’s another option now that lets you get interoperability without introducing a heavy dependency on boost: pybind11

This library is header only which makes it much easier to compile as part of your c++ project.See the pybind11 tutorial on here for more about using this option.

Roughly it will look like this:

#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;
/**
 * computes factorials
 * @param n   The input number
 */
int compute_factorial(int n){
    if(x <= 0){
        return 1;
    }else{
        return compute_factorial(x - 1) * x;
    }
}

/**
 * Generate the python bindings for this C++ function
 */
PYBIND11_PLUGIN(factorials_example) {
    py::module m("factorial_example", "Computing factorials in c++ with python bindings using pybind11");
    m.def("factorial_from_cpp", &compute_factorial, "A function which computes factorials in C++");
    return m.ptr();
}

Use CFFI

This is one of the biggest contributions from the PyPy ecosystem to the broader Python universe, it works on PyPy and cPython. The main idea behind CFFI is to generate an API to the C code without needing to worry about ABI issues. This gives you a rather large amount of efficiency in writing the code because a lot of the nasty details are hidden away from you. If you only need to make a call to existing C you don’t even need a C compiler run as part of the process, CFFI takes care of that for you.

Essentially you can write out your C code from within Python and CFFI will then go and compile the code and automatically create an API for you to use the C code from within Python. Honestly the whole idea is brilliant as it cuts down the time and effort required to consume existing C code. It also makes porting your code between different python implementations substantially easier because you don’t end up tied to the implementation specific way of creating extensions. You still need a C compiler and there’s a few limitations but overall this approach tends to work well. Going with the factorial example again it looks something like this:

# file "cffi_factorials.py"

from cffi import FFI
ffibuilder = FFI()

ffibuilder.cdef("int factorial_fast(int);")

ffibuilder.set_source("_example",
r"""
    static int factorial_fast(int n){
        if(x <= 0){
            return 1;
        }else{
            return compute_it(x - 1) * x;
        }
    }
""")

if __name__ == "__main__":
    ffibuilder.compile(verbose=True)

We then need to deal with actually building this as part of our build script because we need to call the C compiler to build our code before we can call it from python.

# file "setup.py"
from setuptools import setup

setup(
    ...
    setup_requires=["cffi>=1.0.0"],
    cffi_modules=["cffi_factorials.py:ffibuilder"],
    install_requires=["cffi>=1.0.0"],
)

As you can see this is much less verbose in the code than using the python C interface.

Embedding Python in another language

If the majority of your project is in another language but you need some sort of scripting ability embedding Python into your project can be a good option. This can be substantially easier overall than needing to write an interpreter in the main language. This is more of a high level decision but might be the right way for your project depending on what you are doing.

Resolving CFFI install issues on Ubuntu

Some of my Python projects rely on a variety of packages, most of which install flawlessly via PIP. Recently I’ve been setting up a new machine running Ubuntu 16.04 and I ran into a few issues installing CFFI via PIP. Essentially CFFI requires some packages that are not installed by default on Ubuntu 16.04, the following should help you get past those issues.

When running pip install cffi the first failure you might see is something like this:

building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/c
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.5m -I/home/janis/.virtualenvs/virt_env/include/python3.5m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.5/c/_cffi_backend.o
c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Failed building wheel for cffi

This happens because Python development headers are not installed (as hinted at by the missing Python.h include file), to fix that we must install:

sudo apt-get install python3-dev

Or if for some reason you are running Python2 and the packages were not already installed:

sudo apt-get install python-dev

After installing that there’s a second fail you might encounter:

running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/c
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.5m -I/home/janis/.virtualenvs/virt_env/include/python3.5m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.5/c/_cffi_backend.o
c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

This is again a result of missing packages, this time libffi-dev:

sudo apt-get install libffi-dev

After installing that I got a successful CFFI install via PIP.

Generating python bindings from C++11 with pybind11

Code for this tutorial found here: https://github.com/shuttle1987/pybind11_tutorial_code

Recently I was involved in some discussion sparked by this blog post where “rdrewd” was talking about benchmarking a simple program that generated prime numbers in Python. The thing about this that I like is that it’s a very good example of a situation where making pragmatic decisions about implementation is highly relevant. I like that the author is focusing on the pragmatics of solving the underlying problem.

There’s a page on the python wiki about performance in python that is a good summary of some key issues. One passage I think is highly relevant (emphasis mine):

Some people are inappropriately obsessed with speed and think that just because C can provide better performance for certain types of problem, it must therefore be a better language for all purposes. Other people think that speed of development is far more important, and choose Python even for those applications where it will run slower. Often, they are surprised to find Python code can run at quite acceptable speeds, and in some cases even faster than what they could get from C/C++ with a similar amount of development time invested.

Usually it is not the absolute speed that is important, you should think about what would be an acceptable speed of execution. Optimizations beyond achieving this acceptable speed are wasteful of resources (usually: your time. And thus: money.).

The general philosophy of cPython is to write the bottleneck code in a faster language. The original blog post got my attention because it’s a very good example of a time where calling out to a faster language makes sense. Personally if I’m faced with a situation where the code is running in cPython I’ll try PyPy first because the speedups due to the tracing JIT are quite remarkable. Many times in my career switching to PyPy was sufficient to make the implementation of a project have feasible run times. The times it has not been enough I’ve used various FFI techniques to get the benefit of fast development in Python while running performance critical code in a faster language.

If there’s one thing I’d like people to take away from this is that choosing implementations is not a binary choice, sometimes mixing and matching is best.

Generation of prime numbers

I mostly want to talk about programming implementation details but first I think it’s good to go over some algorithms.

Even the best implementation in the world can’t overcome a bad algorithm choice in a problem like this. Efficiently generating prime numbers is actually quite involved.

Naive algorithms like the Sieve of Eratosthenes are conceptually easy to understand but allow for almost no locality of reference. Modern CPUs are very sophisticated and are often explicitly designed to leverage locality of reference via memory caching. Efficiently using the CPU cache can have huge impacts on the performance of otherwise similar looking code.

The best algorithms for generating primes allow for good locality of reference. For example the Sieve of Atkin has a better asymptotic complexity than the Sieve of Eratosthenes, but it also has much better locality of reference compared to a naive implementation of the Sieve of Eratosthenes. (Upon further investigation even though the asymptotic complexity for an idealized operation is lower for some algorithms the actual underlying operations on the hardware can have a significantly large constant factor. In a practical setting this cannot be overlooked, benchmarking as always is very useful)

A good implementation of a prime number generation algorithm will take locality into account. This will outperform naive implementations because the algorithms with better locality of reference allows for much better utilization of cache-backed CPU hardware.

All this applies for any language, when it matters good algorithms choice along with good implementation makes a huge difference.

EDIT: As Dana Jacobsen has pointed out in the comments the choice of of algorithm here is quite involved, and optimizing these algorithms makes a substantial difference, thanks for pointing out some considerations I overlooked.

Practical implementation

Lets say we have chosen the right algorithm and that algorithm intrinsically takes some serious computing power to run. How do we actually get that into code in a way we can easily use in our applications?

The reference code that sparked this off looks like this in Python3:

primes=[2]
cand=3
while len(primes) < 20:
    for factor in primes:
        if cand % factor == 0:
            # Not a prime
            break
    else:
       primes.append(cand)
    cand += 2
print(primes)

Original Python2 code on pastebin: http://pastebin.com/DHQtzN0s

For the sake of the article lets assume that this algorithm is the one we have decided to use. The blog post talked of the relative merits of using different languages, a C implementation was made and people benchmarked various different implementations of the same algorithm in different languages. Many people in the comments tried various languages and the inevitable discussions of the relative merits of languages came up. The author expressed regret nobody wrote a c++ implementation, so here’s a naive one I threw together today:

#include <iostream>
#include <vector>

/**
 * Generates the a sequence of prime numbers
 * @param how_many   how many primes to generate
 */
std::vector<int> generate_primes(unsigned int how_many){
    std::vector<int> primes;
    primes.reserve(how_many);
    primes.push_back(2); //2 is the first prime number
    unsigned int candidate = 3;
    bool potential_prime = true;
    while(primes.size() < how_many){
        potential_prime = true;
        for(unsigned int prime: primes){
            if(candidate % prime == 0){
                candidate += 2;
                potential_prime = false;
                break;
            }
        }
        if(potential_prime){
            primes.push_back(candidate);
        }
    }
    return primes;
}

int main(){
    std::vector<int> first_20_primes = generate_primes(20);
    for(int prime : first_20_primes){
        std::cout << prime << std::endl;
    }
    return 0;
}

This C++11 solution is fairly close to the original python code.

Comparatively speaking you can see that Python makes it really easy to implement something quickly. There’s more that can be done with c++ as well but this is about the limit of quickly putting something together. As is fairly obvious C++ is just a more verbose language.

Benchmarking

Decisions about performance benefit enormously from benchmarking. Because the impact of performance is very much non-uniform finding where the performance matters is a crucial step. If you are ever trying to decide if you need to use a foreign function interface out or any other means of writing code in a different language due to performance reasons you really need to be benchmarking your code to find out where the real bottlenecks are.

Some benchmarks were provided in the blog post that were on extremely small sets of data. The problem size was small enough where the IO completely eclipsed the run time. But that was the authors entire point. In a throwaway project this small the run time difference did not actually matter. In the particular case of generating 20 primes, just write it in Python, you will have the solution with the least amount of developer hours spent that way. The computational time is going to be small enough that the trade-off of time that would be required to write it in another language is just not worth it.

However what happens when you want to generate the first thousand primes? How about the first 10 thousand? What happens if you put this code on a popular web server where you are getting a lot of hits and you are charged for the amount of CPU time used?

We should seriously consider another implementation when the balance shifts away from ease of implementation being the main factor. Don’t make the decision lightly, if you are writing high quality code to begin with then using higher performance code that complicates the building of your project involves a real trade-off in the form of introducing additional complexity.

First we would benchmark and see where the bottlenecks are as this enables us to make a judgment call about the costs of improving the performance of the code by re-implementing those bottlenecks.

FFI and python

So lets say you have benchmarked and you have found that your python implementation of this function is such a bottleneck that you can’t satisfy your requirements, what do you do next?

One of the big things that has changed in recently is how much easier it is to make python bindings now. You don’t even need to know the python C-api, you can use something like pybind11 and just use that to generate your python modules.

Here’s how we would expose that C++ function to Python:

#include <vector>

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

namespace py = pybind11;

/**
 * Generates the a sequence of prime numbers
 * @param how_many   how many primes to generate
 */
std::vector<int> generate_primes(unsigned int how_many){
    std::vector<int> primes;
    primes.reserve(how_many);
    primes.push_back(2); //2 is the first prime number
    unsigned int candidate = 3;
    bool potential_prime = true;
    while(primes.size() < how_many){
        potential_prime = true;
        for(unsigned int prime: primes){
            if(candidate % prime == 0){
                candidate += 2;
                potential_prime = false;
                break;
            }
        }
        if(potential_prime){
            primes.push_back(candidate);
        }
    }
    return primes;
}

/**
 * Generate the python bindings for this C++ function
 */
PYBIND11_PLUGIN(primes_py3) {
    py::module m("example", "Generating primes in c++ with python bindings using pybind11");

    m.def("generate_primes", &generate_primes, "A function which generates a list of primes. The length of this list is specified by the user");

    return m.ptr();
}

Overall the amount of boilerplate is remarkably small. Also the pybind11 library is header only, which is great for keeping builds simple as we don’t need to link to it. However the build is still complex enough that making a script is worthwhile:

#!/bin/bash
g++ -O3 -fPIC -shared -std=c++11 -I /home/janis/code_snippets/prime_gen/pybind11-1.8.1/include `python3-config --cflags --ldflags` prime_gen_python3_bindings.cpp -o primes_py3.so

This compiles the C++ into a form that can be imported from python directly:

janis@janis-laptop:~/code_snippets/prime_gen (master)$ python3
Python 3.4.3+ (default, Oct 14 2015, 16:03:50)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import primes_py3
>>> primes_py3.__doc__
'Generating primes in c++ with python bindings using pybind11'
>>> primes_py3.generate_primes.__doc__
'generate_primes(int) -> list<int>\n\nA function which generates a list of primes. The length of the list is specified by the user\n'
>>> primes_py3.generate_primes(20)
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71]

As you can see from the python side this is really easy to use, you don’t even know that the C++ is there.

When working on a team this can be a very powerful technique to have in your arsenal. By making performance critical modules in this way you can keep the ease of use for the consumers of the code while getting performance improvements.

Summary

Ease of development in smaller projects is one of the biggest wins that Python delivers, this is because speed of development is one of Python’s strongest points. C++ has very made very different decisions because the philosophy of the language is fundamentally different. C++ is designed to be powerful, it isn’t designed for ease of use in smaller projects, though it has got a lot easier to use in the more modern revisions of the language. Where C++ really shines is in the certain situations where the power it gives you is actually important, such as high performance requirements situations.

As this tutorial shows when there’s some performance bottlenecks you have options other than complete re-writes. Specifically you can just deal with the parts your profiling/benchmarking have shown to be problematic.

If you are writing your performance critical code in c++ using pybind11 is a convenient way to consume that code from Python. It is by no means the only way, you could use subprocesses or message queues like ZeroMQ, each of these has their own set of trade-offs. If you have C code then I’d recommend looking into CFFI, it might make your life a lot easier. I used it in my MMAP backed array library and saved myself a ton of hassles.

Essentially you want to use the right tool for the job, and sometimes this involves using more than one language.

If you have any questions about code implementation decisions and trade-offs please leave a comment below! Improving the performance of code is something we specialize in, contact us if you have a project you are looking for help with.

 

Deprecation warnings in python

When you are maintaining a package you want to be able to deprecate functionality and be able to give your users advance warning when you know that something is going away in the future. Because we wish to warn users without breaking the code we want to have the ability to give information without changing the flow of execution. Luckily Python has standard library support for warnings in the warnings module.

One thing you’ll notice is that warnings are actually in the standard exception hierarchy. Within this there’s some subclasses of Warning that provide more specific categories. Just like exceptions choosing the most specific correct category is a good practice.

When you want to warn your users that something is going away in the future there’s a PendingDeprecationWarning. Then when you actually deprecate it you can use DeprecationWarning.

The full list of categories can be found here: https://docs.python.org/3/library/warnings.html#warning-categories

The benefit of this approach is that by using convention you make your intentions to the users of your code unambiguous.

Here’s an example of how this works with methods:

#deprecation_example.py
import warnings

class Example:
    def __init__(self):
        self.items = [3,4,6,2,1]

    def slow_sort(self):
        warnings.warn(
            "slow_sort is deprecated, use fast_sort instead",
            DeprecationWarning
        )
        import time
        time.sleep(5)
        return sorted(self.items)

    def gone_in_future_version_sort(self):
        warnings.warn(
            "gone_in_future_version_sort will be deprecated in version 2, use fast_sort instead",
             PendingDeprecationWarning
        )
        return sorted(self.items)

    def fast_sort(self):
        return sorted(self.items)

Now when we run this:

janis:~/deprecation_tutorial$ python3
Python 3.4.3+ (default, Oct 14 2015, 16:03:50)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from deprecation_example import Example
>>> test = Example()
>>> test.slow_sort()
[1, 2, 3, 4, 6]
>>> test.gone_in_future_version_sort()
[1, 2, 3, 4, 6]
>>> test.fast_sort()
[1, 2, 3, 4, 6]

Now notice that nothing appeared to happen here. This is good! Warnings are for the sake of the developers, we do not want the execution for the users to change when we introduce a warning.

So in order to get the information displayed we have to start the interpreter with a flag  -Wd

janis:~/deprecation_tutorial$ python3 -Wd
Python 3.4.3+ (default, Oct 14 2015, 16:03:50)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from deprecation_example import Example
>>> test = Example()
>>> test.slow_sort()
/home/janis/deprecation_tutorial/deprecation_example.py:11: DeprecationWarning: slow_sort is deprecated, use fast_sort instead
  DeprecationWarning
[1, 2, 3, 4, 6]
>>> test.gone_in_future_version_sort()
/home/janis/deprecation_tutorial/deprecation_example.py:20: PendingDeprecationWarning: gone_in_future_version_sort will be deprecated in version 2, use fast_sort instead
  PendingDeprecationWarning
[1, 2, 3, 4, 6]
>>> test.fast_sort()
[1, 2, 3, 4, 6]

Now we see the warnings!

Say we want to write some code in a script:

from deprecation_example import Example
test_instance = Example()
test_instance.slow_sort()
test_instance.gone_in_future_version_sort()
test_instance.fast_sort()

Just like the REPL example running this with no flags to python will not display any warnings output.

janis:~/deprecation_tutorial$ python3 test_script.py
janis:~/deprecation_tutorial$

Invoking the interpreter with the  -Wd flag will however give us info:

janis:~/deprecation_tutorial$ python3 -Wd test_script.py
/home/janis/deprecation_tutorial/deprecation_example.py:11: DeprecationWarning: slow_sort is deprecated, use fast_sort instead
  DeprecationWarning
/home/janis/deprecation_tutorial/deprecation_example.py:20: PendingDeprecationWarning: gone_in_future_version_sort will be deprecated in version 2, use fast_sort instead
  PendingDeprecationWarning

Now say we wanted to do that via code:

import warnings
warnings.simplefilter('default')
from deprecation_example import Example
test_instance = Example()
test_instance.slow_sort()
test_instance.gone_in_future_version_sort()
test_instance.fast_sort()

Now we have set it up we get the warning reporting even without the interpreter flag:

janis:~/deprecation_tutorial$ python3 test_script.py
/home/janis/deprecation_tutorial/deprecation_example.py:11: DeprecationWarning: slow_sort is deprecated, use fast_sort instead
  DeprecationWarning
/home/janis/deprecation_tutorial/deprecation_example.py:20: PendingDeprecationWarning: gone_in_future_version_sort will be deprecated in version 2, use fast_sort instead
  PendingDeprecationWarning

Note that if you are using the standard library logging you can set it up to log all warnings with logging.captureWarnings . In production systems this is a very good idea, as you want to gather as much information as you can about things that will impact your code.

If you have any questions please feel free to post a comment below!

Testing for exceptions with Jasmine

I just made a PR to the KaTeX library to help fix some cryptic error messages.

To do this I had a few places raise exceptions early instead of propagating deeply into the code before breaking in strange places with cryptic error messages. The library can only deal with expressions that are strings so I added in this validation:

var render = function(expression, baseNode, options) {
    if (!(typeof expression === 'string' || expression instanceof String)) {
        throw new TypeError('render requires string typed expression');
    }

Naturally I wanted to make a test for this. Jasmine lets you check if an exception was thrown with the toThrowError. However I made a mistake when I first did this:

it("should throw TypeError when the expression is of the wrong type", function() {
    expect(katex.renderToString({badInputType: "yes"})).toThrowError(TypeError);
    expect(katex.renderToString(["foo", {badInputType: "yes"}])).toThrowError(TypeError);
});

See what went wrong?

The problem here is that the exception is thrown before  expect can actually deal with it, therefore it slips out and fails the test case. To fix this you have to wrap your call in a function which Jasmine will invoke from within  expect :

it("should throw TypeError when the expression is of the wrong type", function() {
    expect(function() {
        katex.renderToString({badInputType: "yes"});
    }).toThrowError(TypeError);
    expect(function() {
        katex.renderToString(["foo", {badInputType: "yes"}]);
    }).toThrowError(TypeError);
});

Markup languages with Git, a document workflow

Currently I’m writing this entry in a text editor that is a WYSIWYG type of editor. This is entirely common, to the point where people are just conditioned to thinking that editing text and WYSIWYG go together. There are however a number of document markup languages that are not WYSIWYG, lightweight ones such as Markdown and ReStructuredText are popular along with more powerful languages such as LaTeX.

All these markup languages store your documents in plain text files, for example LaTeX stores these as “.tex” files which then get compiled into whatever final output type you end up using (in my case this is usually pdf). At first I remember being put off by the way in which you needed to compile your documents with LaTeX, for someone conditioned to using packages like open office this extra step felt alien and weird. So strong was the sense strangeness of using a markup language for my regular documents that I abandoned using LaTeX the first time I encountered it. I didn’t come to a favourable cost vs benefit breakdown for learning LaTeX. This was because I overestimated the learning curve and I didn’t see how big the benefits were until later. The benefits of using a markup language very clearly outweigh the negatives for a variety of situations and document types. Hopefully after reading this some of these benefits will be more obvious.

It is important to choose the right tool for the job. For complex typesetting problems LaTeX is extremely powerful, for example if you need to typeset mathematics formulae or have extreme control over your formatting it’s just about the only game in town. However you don’t always need this power, if you are just dealing with simple documents with basic formatting you probably will do better using something like Markdown. This is especially the case if you are trying to convince other team-mates to also use a markup language for their documents. The benefits have to be clear to people for them to make the jump from whatever it is that they are currently using to create their documents. For me the tipping point was when I got into using version control software effectively, at this point the power of using documents with a markup language became really clear. Because I was working on a mathematics paper LaTeX ended up being the best choice for that project.

Collaboration

Have you ever had to collaborate on a sizeable document with a team? If so how did you manage merging all the individual changes into the final master document? I did this once for a technical report with open office writer and the process was particularly painful, combining all the changes was slow and error prone. There was a proliferation of similar files as well to deal with with people making a lot of temporary files, questions such as which files are most recent and which changes have to be migrated came up again and again. A couple of times we lost changes too because of documents being simultaneously overwritten. All these problems are hard to solve in an ad-hoc manner, thankfully this is mostly a solved problem though. The fact that LaTeX documents are stored in plain text is extremely useful when you consider that you can be using version control software on your documents. As the documents are stored in plain text this allows you to use all the software out there that can deal with maintain changes to text.

If you are not familiar with version control you really should stop and take some time to find out more. One of the big wins version control creates is when you need to collaborate on a document. The LaTeX + Git combination really shines through because it is now extremely straightforward to share your changes via making commits and merging your changes. (Note that any other markup language for documents will get this same benefit). Because Git is decentralized you don’t need to set up any servers, in fact you don’t even need a network or internet access, you can just push changes with a USB key if you need. This makes backing up your files really easy and saves the massive hassle of dealing with a ton of similarly named backup directories or zip files.

Benefits of using Git

From the point where you have got Git up and running onwards you can just edit your documents and save the changes in git as you go along. Aside from easier collaboration you get a lot of other major benefits essentially for free by doing it this way including:

  • A reliable way to back up your files (via git push to another location)
  • A full history of changes people have made.
  • Ability to check differences between revisions of the file.
  • Find out who wrote what lines by using things such as “git blame”.
  • Find things like the breakdown of how many lines people wrote in files.
  • Ability to integrate scripting tools into your document workflow (for example you could use git hooks to say build your document and place it in a common directory every time someone makes a push)

Note that some of those benefits have nothing at all to do with collaboration, even when I’m working on a project by myself I find Git to be worth using because of the way it provides me backups and a convenient way to look at what changes were made over time.

Here’s an example some of those things from looking at a README markdown file from a project I worked on. Here’s the log of commits changes for a file:

git log README.md

Which give the following:

commit d6a32350c3261c00f761eb3e125d025186e04104
Author: Tim <tim@example.com>
Date:   Thu Oct 4 14:22:35 2012 +1000

    Made a simple addition to the readme about git usage

commit b38dcd7b7f2e03248d85d4fe9f6fbb51d44c816a
Author: Janis <janis@example.com>
Date:   Wed Oct 3 16:24:27 2012 -0400

    Created a TODO file to track feature requests and added some more information to the README

commit 1bff657c061d89545fbce9e2cacf4aa99f9f1f73
Author: Janis <janis@example.com>
Date:   Tue Oct 2 21:23:23 2012 -0400

    added a little text into the readme file

commit 40e58627a20f7cd14c06e3db5865e0d995fbedf4
Author: Bob <bob@example.com>
Date:   Tue Oct 2 18:02:07 2012 -0700

    Initial commit

You can also do things like get a quick breakdown of who wrote which lines:

# count the number of lines attributed to each author
git blame --line-porcelain file_name |
sed -n 's/^author //p' |
sort | uniq -c | sort -rn

For example running that on README.md:

16 Janis
 3 Bob
 2 Tim

This just scratches the surface of the power that writing your documents in a markup language along with version control affords you.

Misc

Not every file that is generated in the process of making a document should be kept in source control. For example with LaTeX in general I’d make the following .gitignore file types:

*.aux
*.backup
*.dvi
*.log
*.pdf
*.ps
*.tex~

Using LaTeX for fast document generation

Many systems have some sort of report generation component. This is often some variation on extracting data from a database (or other sources) then doing some analysis on that data and outputting it in some readable form. Sometimes a requirement is for reports to be available in PDF format. I use a lot of Python for small tasks and many in-house report generation tasks fall into the category where developer time is much more expensive than processor time. Being able to make these reports quickly AND have the eventual typesetting look good is a big win, even if it’s not the most performant code in the world. This is especially the case if the report is a one-off report.

If the reason for creating PDFs is because it will be printed I find using LaTeX to be especially useful because it handles many of the annoying details of typesetting printed materials. There are a ton of little typesetting things that LaTeX does, for example it deals with excessive rivers in the text, I didn’t even realize it did this automatically because I didn’t notice any of these in the documents it generated. So given that LaTeX does a good job of automated typesetting it seemed like a natural candidate to make PDF files. The only tricky thing is automating the generation and compilation of the LaTeX documents from within code, which is the thing the rest of this tutorial covers.

Example problem

For the sake of this tutorial we look at a fairly common situation: We have some graphs with some along with some descriptive text describing when the data was generated. For the sake of the example the data we wish to plot is generated by the following:

def create_data():
    """Example data for JaggedVerge LaTeX tutorial"""
    x_vals = list(range(0,10))
    y_vals = [x**2 for x in x_vals]
    return x_vals, y_vals

Using PyLaTeX

There happens to be a library specifically designed to generate LaTeX from Python called PyLaTeX. For going direct to PDF this library solves a lot of problems. Specifically you don’t need to have an intermediate LaTeX file, you can go direct from Python code to PDF, the benefit of which is that you have fewer steps required in building your PDF.

First we have to set up our python virtual environment. Given that we are using python 3.3+ for this tutorial we can use the virtual environments that the language supports. (If you are using a different version then you have to use the virtualenv wrapper.)

pip install pylatex

PyLaTeX makes extensive use of context managers to handle the various LaTeX commands. Let’s start with a really simple example of creating a PDF:

from pylatex import Document, Section

doc = Document()
with doc.create(Section("Our section title")):
    doc.append("Simple example")

doc.generate_pdf('pylatex_example_output')

That’s ALL the code we need to generate a PDF. Without any more boilerplate to deal with lets get to satisfying the rest of the requirements. Because PyLaTeX has support for TikZ we can create some simple graphs without needing any extra dependencies:

with doc.create(TikZ()):
    plot_options= 'height=10cm, width=10cm, grid=major'
    with doc.create(Axis(options=plot_options)) as plot:
        x_coords, y_coords = create_data()
        coordinates = zip(x_coords, y_coords)

        plot.append(Plot(name="Our data", coordinates=coordinates))

That gets us our plot. Now all we need to do is handle the time stamping and a few miscellaneous document issues. First lets add a title to the document:

doc.preamble.append(Command('title', 'PyLaTeX example'))
doc.preamble.append(Command('author', 'JaggedVerge'))
doc.append(NoEscape(r'\maketitle'))

Note that PyLaTeX is a wrapper around LaTeX code so just like in LaTeX if you miss the  \maketitle command the title will not be generated. We can create the time stamp with regular python code:

formatted_timestamp = time.strftime("%a, %d %b %Y %H:%M:%S +0000", data_creation_time)
doc.append("The data in this plot example was created on {}".format(formatted_timestamp))

Once again we can fairly easily get from python to LaTeX whenever we are dealing with text.

At this point we have the following:

from doc_gen import create_data
from pylatex import (
    Axis,
    Command,
    Document,
    Plot,
    Section,
    TikZ,
    NoEscape,
)

import time
data_creation_time = time.gmtime()

doc = Document()
doc.preamble.append(Command('title', 'PyLaTeX example'))
doc.preamble.append(Command('author', 'JaggedVerge'))
doc.append(NoEscape(r'\maketitle'))


with doc.create(Section("Data report")):
    formatted_timestamp = time.strftime("%a, %d %b %Y %H:%M:%S +0000", data_creation_time)
    doc.append("The data in this plot example was created on {}".format(formatted_timestamp))
    with doc.create(TikZ()):
        plot_options= 'height=10cm, width=10cm, grid=major'
        with doc.create(Axis(options=plot_options)) as plot:
            x_coords, y_coords = create_data()
            coordinates = zip(x_coords, y_coords)

            plot.append(Plot(name="Our data", coordinates=coordinates))

doc.generate_pdf('pylatex_example_output')

Which generates the following document:

Rendered PyLaTeX output

Example of PyLaTeX generated PDF document

Just as with LaTeX it’s probably a good idea to put this plot into a figure environment. If you know LaTeX already it’s fairly straightforward to generate documents using PyLaTeX. In the future I’ll write about manipulating existing LaTeX documents if there is interest. Please leave a comment if you have any question or are interested in seeing more content like this in the future.

Python comprehension syntax for Java developers

Recently I’ve been working with both Java and Python. One of the jarring differences between Java and Python is the relative brevity of Python code that’s enabled by the various Python language features. In all languages there’s a decision about how much syntactic sugar is used and how succinct the language is, with regards to this the Python and Java languages happen to be close to the opposite extremes. Something that Java developers who are new to Python frequently have trouble with is the Python comprehensions syntax as there’s no direct analogue of this in Java. Part of the difficulty of learning this part of Python syntax is that it’s hard to search for as there’s no searchable keywords involved.

Here are a few examples of some of the syntax that you can see in Python source code:

[x**2 for x in range(5)] #list comprehension
{x**2 for x in range(5)} #set comprehension
(x**2 for x in range(5)) #generator expression

All of these generate a collection of items based on the same rule. While the types are different these all share a common form of syntax which can be broken down into 2 constituent parts:

  1. A rule that generates a sequence of items
  2. The data type that the sequence of items is stored as

Part 1: sequence generation

The general structure for the comprehension is as follows:

foo(x) for x in iterable_data_type

The equivalent of this in pseudocode is:

foreach x in iterable_data_dype:
    foo(x)

Essentially you loop over every value in the given iterable variable and you do something with it. (Note that you don’t have to call a function with x as a parameter, more on that a bit later.)

For example say we had:

x**2 for x in range(10)

This is essentially shorthand for:

for x in range(10):
     x**2

As you can see this generates a sequence of values where we square each value.

Similar code in Java would be:

//Similar to the range construct in Python
List<Integer> makeSequence(int begin, int end) {
    List<Integer> ret = new ArrayList(end - begin + 1);
    for(int i = begin; i <= end; i++, ret.add(i));
    return ret;
}

//Calling code
List<Integer> range = makeSequence(0, 10);
foreach(Integer item: range){
    Math.pow(item, 2.0);
}

Note that we haven’t actually done anything with the result of squaring the numbers yet, which leads us nicely to part 2.

Part 2: data types

Compared to Java there’s a lot of syntactic sugar involved in the idiomatic usage of Python. You can see an example of this when creating variables, while Python is strongly typed you do not supply a type when creating a variable, the type is determined by the contents of what the variable refers to. With Java you must supply the type when creating a variable and these types are keyword based, such as the  List<Integer>  type we had in the examples before.

For example in Python creating a list from a comprehension looks like this:

[x**2 for x in range(10)]

Which like earlier is shorthand for:

ret = []
for x in range(10):
     ret.append(x**2)
return ret

Running this in the interpreter in interactive mode you can see what this does:

>>> a = [x**2 for x in range(10)]
>>> type(a)
<class 'list'>
>>> a
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Note that the variable  a here is of the list type because we assigned a list to it. In Java a rough equivalent of this code is:

List<Integer> range = makeSequence(0, 10);
List<Integer> ret = new ArrayList<Integer>();
foreach(Integer item: range){
     ret.add(Math.pow(item, 2.0));
}
return ret;

In Python when you have the form  [sequence_generation_expression] you get a list. If it were {sequence_generation_expression} you would get a set, or if it was  (sequence_generation_expression)  you would get a generator expression. More on this in the following examples.

Examples

Here’s a few examples of how these work with various types by running through some code in the interpreter in interactive mode. If you are trying out this code while reading then you might also want to try running the type and help commands on the various variables you create in the process.

List comprehension

>>> a = [x**2 for x in range(10)]
>>> type(a)
<type 'list'>
>>> a
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

This list comprehension creates a list like the one in the earlier example.

Set comprehension

>>> b = {x**2 for x in range(10)}
>>> type(b)
<class 'set'>
>>> b
{0, 1, 64, 4, 36, 9, 16, 49, 81, 25}

This creates a set with the same elements as those found by applying the rule. Note that the order of the elements were created in the set is not preserved with the contents of the set. This is because the set uses a hashing function to add and lookup items. Overall this is similar to Java’s HashSet data type.

Dictionary comprehension

>>> c = {x: x**2 for x in range(10)}
>>> type(c)
<class 'dict'>
>>> c
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

This creates a dictionary with the x’s themselves as keys and the x squared as values. This is similar to Java’s HashMap datatype.

Generator expression

>>> d = (x**2 for x in range(10))
>>> type(d)
<class 'generator'>
>>> d
<generator object <genexpr> at 0x7f78bea3eb40>

Essentially a generator lazily creates the same sequence of items that the list comprehension does but it will only compute the values as they are needed. This is a fairly substantial departure from what you would see in the core Java language which is basically eager-evaluation everywhere, you could create a class that emulated this but it would take some effort. Note that it only generates the list once, so once you consume the values you need to re-instantiate the generator. This makes generator expressions useful in much the same way as pipelines are, you can run some processes without having to store the entire dataset in memory at once. We can feed a generator expression into the constructor of other types:

>>> d = (x**2 for x in range(10))
>>> list(d)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> list(d)
[]

Again note that the generator can only be used once.

Creating a tuple

To create a tuple we have to use the explicit tuple keyword because there’s no other way of making these:

tuple(x**2 for x in range(5))

As mentioned earlier you don’t need to do anything with the value from the iterable in a comprehension. For example say we had a function  foo() that we wanted to call 3 times, we could do this as follows:

[foo() for _ in range(3)]

This uses the Python convention of the underscore representing an unused variable.