Probing External C++ Functions with a Cython Wrapper in Jupyter

This is a practical demonstration of how cython can be used to wrap a C++ function. This has already been discussed here, as well as here.

But there is a problem with the methon discussed therein: it requires the user to include the C++ source files (which is not difficult if you're using distutils from a setup.py), which is currently not supported by the %cython magic. A workaround is documented here, but I feel this is unsuitable for two reasons:

1. It still requires access to the C++ sources (which can be unwieldy for large projects)
2. It uses jupyter to generate a whole lot of termporary (garbade) files, which can clutter up an applications/results dir

Especially point 1 gives me pause, because I'm looking for a way to interact with parts of a larger project (without having to rebuild half of it). Hence we're going to try something else here...

Setup

We want to interact with an external C++ file: addone.cpp which is part of another project:

.
├── External\ Wrapper.ipynb
└── main_proj
    ├── Makefile
    ├── addone.cpp
    ├── addone.h
    └── main.cpp

The main project (creatively main_proj/main.cpp) calls add_one from addone.cpp. This object is

Let's imagine that we've only got addone.h and addone.o. Then we can still build a cython wrapper in a %cython magic!

Convert .o files to Linkable Libraries

First we convert addone.o to libaddone.so. Note the name change: adding -lX to the linker args, makes the linker look for libX.so in the linker's search path (specified by the -L argument):

clang++ -fpic -shared -Wl,-all_load addone.o -o libaddone.so

Using the Makefile

The Makefile contains a helpful target: libaddone.so which does the work above for us ;):

CC=clang++
SO=-fpic -shared -Wl,-all_load

libaddone.so: addone.o
    $(CC) $(SO) $< -o $@

Build a Cython Wrapper

For example, our add_one function has the signature: double add_one(double) and is defined in addone.h so the cython wrapper code is simply:

cdef extern from "addone.h":
    cdef float add_one(float)

followed by a python function to wrap the cython cdef function if you want to access it throughout the rest of the notebook.

Use the right %%cython magic arguments

We need to tell cython to use C++ mode: --cplus, and we need to tell it where to look for libraries and which ones to link. Remember that we've converted our C++ object file into a shared library. In our case, this library lives in the ./main_proj/ directory (relative to this notebook), so the compiler arguments are pretty simple -I./main_proj -L./main_proj -laddone. All of these parameters are documented here.

Dyncamic Linkers Beware

If you're using a dynamic linker (for example macOS' dyld) you'll need to include ./main_proj in the linker's search path. For example, if we're using dyld, launch jupyter using:

env DYLD_FALLBACK_LIBRARY_PATH=./main_proj jupyter lab

And we're done!

Example External C++ Function

Now we need an example C++ function contained in an external code project (main_proj). This example is pretty dumb, but it serves the purpose of an external function. The contents of addone.cpp is:

#include <iostream>


double add_one(double a) {
    std::cout << "Adding 1 to: " << a << std::endl;
    return a + 1;
}

with the header file: addone.h:

#ifndef __ADDONE_H__
#define __ADDONE_H__

double add_one(double a);

#endif

An Interactive Wrapper in Jupyter

Having put all of this together, we've got ourselves an interactive wrapper:

%%cython --cplus -I./main_proj -L./main_proj -f -laddone

#cython wrapper code

cdef extern from "addone.h":
    cdef float add_one(float)

# python wrapper function for c-function above
def py_add_one(x):
    return add_one(x)

We now have a python wrapper for our c-function (which happens to be wrapped in a cython cdef function). Running:

py_add_one(10)

should output:

Adding 1 to: 10

in the stdout.

Resources

1. A demo containing an jupyter notebook, as well as the external C++ code outlined above.