Developer tooling¶

The milk v2 documentation is build from markdown files using mkdocs.

For nontrivial documentation work, it is quite useful to do a full website render locally on your development machine. Here's how:

1.1 Installing mkdocs¶

# Optional: create an environment
mamba env create -n mkdocsenv python=3.13
mamba activate mkdocsenv

# install mkdocs
mamba install mkdocs

# install extensions used herein -- some not available on conda-forge, so using pip here.
pip install pymdown-extensions \
    mkdocs-material[plugins] \
    mkdocs-glightbox \
    mkdocs-git-revision-date-localized-plugin \
    mkdocs-minify-plugin \
    mkdocs-macros-plugin

1.2 Build and serve the documentation¶

mamba run -n mkdocsenv mkdocs build --strict
#                      ^^^^^^^^^^^^^^^^^^^^^-- only this part if `mamba activate mkdocsenv` is still active

# Build and serve the website (here, called directly in my mamba session)
mamba run -n mkdocsenv mkdocs serve
# Access -- follow the link or:
<browser> http://127.0.0.1:8000/

The Cacao v1 documentation is build from markdown files using jekyll.

# Into the docs repository
cd cacao-docs # or the cloning folder from cacao-org/docs.git
bundle install
bundle exec jekyll serve

# Access
<browser> http://127.0.0.1:4000/docs/

Formatting is enforced by pre-commit hooks, configured in .pre-commit-config.yaml, and re-checked identically in CI (.github/workflows/pre-commit.yaml).

We don't worry too much about defining a style guide in plain text; what the formatting hooks outputs is the source of truth.

3.1 Setup¶

# Use your existing python environment
pip install pre-commit
# Or pre-commit is now available through apt
sudo apt install pre-commit

# with cwd == $MILK_ROOT
cd $MILK_ROOT
pre-commit install

3.2 Usage¶

The typical usage is the one that is forced upon the developer. Upon running git commit, the hook invokes pre-commit run. By default this applies the pre-commit hooks on all the staged files.

pre-commit is also pretty versatile to run specific hooks on specific files at anytime, or reformat the entire repository if we want to change our preferred formatting.

3.3 Example: fixing a failing hook¶

Formatting hooks fix the files in place and fail the commit; nothing is lost, you just stage the result and commit again:

$ git commit -m "fix: bounds check in image_crop2D"
clang-format.............................................................Failed
- hook id: clang-format
- files were modified by this hook

$ git diff                      # review what the hook changed
$ git add -u && git commit -m "fix: bounds check in image_crop2D"

Hooks that do not auto-fix (e.g. check-yaml, check-merge-conflict) print the offending file and line: fix it by hand and commit again.

git add .             # stage everything
pre-commit run        # run the hooks on everything
git restore --staged . # unstage everything, now format-compliant.
# Now cherry-pick your commit contents.

3.4 Some more checking...¶

Ideally, we want a 1-1 mapping between the developer's local check with pre-commit and the CI stack once the commit or PR reaches github. Sometimes that's not practical, plus, there can be no actual enforcement of what one actually commits!

Style rules (for C/C++) and best practices that clang-format cannot enforce (column-aligned parameters, Kernel-Doc, scope minimization) are covered in Coding standards.

Documentation changes are additionally checked in CI (.github/workflows/docs-lint.yml) with markdownlint (.markdownlint.yml), a link checker and a typos checker (_typos.toml).

The Python test suite lives in python/tests/ and exercises the installed milk binaries and shared libraries through the milk and pyMilk Python packages. pytest runs it against an existing install; nox builds and installs milk in isolated environments, then runs pytest.

We're trying to move away from scattered ctest and bash testing scripts.

mkdir -p _build && cd build_
cmake .. # Pass your preferred build options
make -j 20 && sudo make install
ctest

mkdir -p _build && cd build_
cmake .. -DCMAKE_INSTALL_PREFIX=. # install configured to _build/milk-1.03.00
make -j 20 && make install
PATH="$PWD/milk-1.03.00/bin:$PATH" ctest

The execution versatility provided by pytest is convenient for checking regressions, validating new code, debugging one or few tests efficiently, and even deploying temporary, reproducible work sessions.

nox takes virtualization a step further: it creates autonomous sessions (a bit like docker but with narrower reach) in which we can build milk with different options, then run the tests; or use sessions to run completely different things, e.g. a coverage report, etc. The nox sessions are configured to be completely isolated from the production build on the same machine.

4.1 Setup¶

export PYMILK_ROOT=~/src/pyMilk
export MILK_INSTALLDIR=<path to your milk install>

cd python
pip install -e "$PYMILK_ROOT" # pyMilk
pip install pytest pytest-timeout nox

4.2 pytest¶

pytest runs a series of predefined tests, located in *_test.py files in the python/tests directory. Running from the python/ directory:

cd $MILK_ROOT/python

# Everything
pytest
# Machine readable
pytest -q --color=no --tb=short

# One file, e.g. check that no standalone executable links CLIcore
pytest tests/trivial/build_sanity_test.py

# Verbose
pytest -v tests/trivial/build_sanity_test.py

# Stop at first failure, drop into debugger
pytest -x --pdb

# Filter by test name
pytest -k

# Or any combination thereof !

source $MILK_ROOT/python/tests/env.bash

def test_some_env():
    deploy_something()

    input('Test environment is ready')

    undeploy_something()

@pytest.fixture
def my_fixture():
    setup_something()
    yield None # <-- this is what's passed to the test function.
    teardown_something()

def test_some_env(my_fixture):
    input('Test environment is ready')

pytest -s mains/some_file.py -k test_some_env
# -s provides stdin/stdout for the test and is needed to hold on at the input() call.

source $MILK_ROOT/python/tests/env.bash

4.3 nox¶

nox creates clean, isolated python environment, that can be used for any relevant purpose that requires an isolated tree and a clean set of packages.

We use it to run the milk tests with various build options, or to run coverage reports. The default sessions install milk and pyMilk, configures and builds milk in a temporary directory, installs it there, and runs pytest. The build variants are:

cd $MILK_ROOT/python

nox -l                              # list sessions
nox -s build_and_pytest_nocli_nolto # one session, including those excluded by default

nox -s tests_run_coverage           # run the coverage session, which is not in the defaults; See #5 below.

UV_OFFLINE=1 nox ...

nox -N [-s <session>]

Coverage tells you which C code the Python test suite did not execute. It is a validation tool for the tests, not a pass/fail gate.

5.1 Build, test, report¶

It is possible to run the coverage build directly in the main shell environment (build with coverage flags, install, run the tests, etc), but inconvenient due to managing the gcc tracking and counter files.

We packaged all of it in a nox session, that produces two html reports, for python and for C/C++.

nox -s tests_run_coverage           # run the coverage session, which is not in the defaults; See #5 below.

5.2 Reading the report¶

The python coverage report:

<browser> $MILK_ROOT/cov_py/index.html

<browser> $MILK_ROOT/cov_c/index.html

Building and testing never requires root: install into a staging directory under _build/, and point your environment at it.

6.1 Install and use¶

cd _build
cmake --install . --prefix _install

export MILK_INSTALLDIR="$PWD/_install"
export PATH="$MILK_INSTALLDIR/bin:$PATH"

milk-check        # verifies MILK_ROOT, MILK_INSTALLDIR, MILK_SHM_DIR and PATH

To remove it: rm -rf _build/_install, and unset the modified variables.

6.2 Global install without sudo¶

Follow the process for installing milk system-wide: the default recommendation is an install under /usr/local/milk, which is a symbolic link to /usr/local/milk-<version>.

By changing ownership of that folder:

sudo chown -R <user>:<user> /usr/local/milk-<version>

For attaching to running processes, tmux log inspection and core dumps, see Debugging.

7.1 gdb¶

gdb --args milk-fpsexec-mymodule exec <args...>
(gdb) run
# ... crash ...
(gdb) bt full                    # backtrace with local variables
(gdb) frame 2                    # select a frame from the backtrace
(gdb) print img.im->md->size[0]  # print info on one variable
(gdb) info locals                # show local variables in the frame
(gdb) info threads
(gdb) thread apply all bt

For an already-running loop, attach instead (gdb -p <PID>, from milk-procinfo-list).

7.2 valgrind¶

valgrind is the tool of choice for invalid reads and writes, uninitialized values and leaks, at a 10 to 50 times slowdown. Use few iterations.

# The interactive CLI, with a ready-made wrapper: output goes to milk-cli.memcheck.log
milk-debug
tail -f milk-cli.memcheck.log

# A standalone executable
valgrind --leak-check=full --track-origins=yes --max-stackframe=4442392 \
    milk-fpsexec-mymodule exec <arg1> <arg2>

In the output, read the first error of each block first: the "Invalid write of size 4" stack is the faulting access, the "Address ... is N bytes after a block of size M alloc'd" stack is where the buffer came from.

The project follows the REUSE specification: every file carries its copyright and license as SPDX tags, and the license texts are in LICENSES/.

8.1 Usage¶

# Validation: reports files lacking copyright/license info, unused or missing license texts
reuse lint

# Add the standard header to a new file
reuse annotate --copyright "Olivier Guyon et al" --year 2026 \
    --license LGPL-3.0-or-later src/my_module/my_func.c

The result matches the existing sources:

// SPDX-FileCopyrightText: 2026 Olivier Guyon et al
//
// SPDX-License-Identifier: LGPL-3.0-or-later

Run reuse lint before opening a PR that adds files. For files that cannot hold a header (binaries, FITS files), declare them in a REUSE.toml at the repository root.