Theodore Tucker: A reproducible approach to scientific Python

A reproducible approach to scientific Python: Git, CI and remote execution with Jupyter notebooks Theodore Tucker Jupyter notebooks are a mainstay of the scientific computing toolkit, elegantly presenting Python code alongside its output and accompanying text and providing an excellent interactive developer experience. When they are used to analyse data pertinent to a functional safety assessment, however, more guardrails are desirable to guarantee the provenance and validity of both input and output. This talk will show how we applied standard practices in software engineering – version control with Git and continuous integration (CI) – and techniques from OS build engineering to a repository of Jupyter notebooks to increase their rigour, making them suitable for presentation to safety assessors of a Linux operating system. The key to the operation is the Jupyter suite’s nbconvert program. We can use this locally to ensure output cells of Jupyter notebooks are never committed to Git, just as we would never commit a binary in a software project. We can then use it in CI pipelines to execute the notebooks in a controlled and isolated environment, providing a review-ready HTML rendering for each notebook in each change request; reviewers can then be sure that the results of analysis are truly reproducible from the input data, and not just on one engineer’s laptop. This extends to the CI environment, as we can build bit-for-bit reproducible container images containing Python’s powerful scientific computing libraries (including pandas, numpy, scipy and matplotlib) and Jupyter on top of the definitions in the Freedesktop SDK project: all with Python tooling and without access to the Internet. Executing hundreds of notebooks in CI for each change request can slow engineers down. However, the same caching infrastructure used for distributed operating system builds – the open-source Buildbox and Buildgrid implementations of a Remote Execution API (REAPI) client and server respectively – can also be leveraged to speed up notebook execution, shaving hours from CI pipeline time without sacrificing reproducibility. We are excited to share how our new, reproducible, and flexible approach to Jupyter notebooks can be achieved with open-source technologies, bringing together the fields of scientific computing and build engineering and hardening interactive computing for the demands of safety in industry. Talk room 2 (200 people) Sun 3:40 pm - 4:10 pm