Development Guidelines

This section provides details on how to configure and/or develop this codebase. For details specific to:

maintaining the template served by this codebase, see Maintaining This Template.
configuring the services used by this codebase, see Configuring Services.

The development workflow for this project follows a standard collaborative pattern: all work happens on feature branches, changes are proposed via Pull Requests (PRs) on GitHub, automated checks run on every PR, and code only reaches the main branch once those checks pass. This means contributors don’t need to manually remember every quality standard — the automation handles verification, and everyone can focus on the work itself.

Continuous Integration/Continuous Deployment (CI/CD) Workflow

Continuous Integration (CI) means that every proposed change is automatically built and tested before it can be merged — catching problems early, when they are cheap to fix. Continuous Deployment (CD) means that once a change is accepted, publishing a new release is a single button press rather than a manual multi-step process.

This project uses GitHub Workflows (scripts that GitHub runs automatically in response to events such as opening a PR or creating a release) to automate these tasks. The workflow files live in ./.github/workflows/ and cover three situations:

pull_request.yml — runs whenever a Pull Request (a proposal to merge a branch into main) is opened or updated. It enforces:
- linting (automated checks that flag common code mistakes and style issues) and code formatting standards
- proper maintenance of the Poetry project
- successful building of the project
- successful building of the documentation
- successful running of tests
bump.yml — runs whenever code is merged to main. It automatically increments the project version number (see Versioning below).
publish.yml — runs whenever a new release is created through the GitHub UI. It rebuilds the documentation on Read the Docs (if configured) and publishes a new package version to the Python Package Index (PyPI; if configured).

Setting-up the Code

A local development copy of the code base can be obtained and configured as follows:

Navigate to the GitHub page hosting the project
If you want to fork the code so that you work on your own version of the repository (not generally needed or recommended):
- Click on the fork button at the top of the page;
- Edit the details you want to have for the new repository; and
- Press Create fork.
Obtain the URL for the repository you’re going to use (denoted <url>) by clicking on the green Code button on the repository GitHub page
On your local machine, navigate with your terminal to the location where you want to place the code
Generate a local copy using git clone <url>;

Note

Although not strictly necessary, it is recommended that you configure the branch permissions of any forked repositories as detailed in the GitHub configuration section.

Poetry and Python environments for development

Poetry is used to manage this project (see here for an introduction). It simplifies & helps with managing the following:

Creation and activation of a Python environment for the project

Python development should always be managed using a Python environment. Poetry creates and manages one for you automatically. To activate it, run the following from within the project and then execute the printed command:
```
$ poetry env activate
```
This prints the shell command needed to activate the environment (e.g. source /path/to/venv/bin/activate). Run that command to enter the environment, after which python and installed tools will be from the project’s venv. Alternatively, prefix individual commands with poetry run to run them inside the environment without activating it globally.
Note

You don’t have to use Poetry to manage your Python environment if you would rather not. You can instruct Poetry to respect your existing Python environments (e.g. created with pyenv) by setting the following option:
```
$ poetry config virtualenvs.prefer-active-python true
```
Dependency management

Poetry manages a “lock file” (which should be committed and maintained within the code repository) ensuring repeatible installs for all versions.
Publication of the project to the Python Package Index (PyPI) so that people can easily install it for themselves

Once properly configured, publishing to PyPI with Poetry is extremely easy. This is generally managed by the CI/CD workflow for the project though, and developers should never have to manually do this.

Installing Development Dependencies

Once the code is locally installed, dependencies should be installed by moving to the project’s root directory and executing the following:

$ poetry install

In what follows, it will be assumed that this has been done.

Guidelines

In the following, we lay-out some important guidelines for developing on this codebase.

Branches

Development should never be conducted on the main branch. Instead, create a new branch for each piece of work (git checkout -b my-feature), then open a Pull Request (PR) — a GitHub mechanism for proposing that your branch be merged into main. GitHub will run the automated checks on your PR; once they pass and a reviewer approves, the branch can be merged. If GitHub has been properly configured (see here), direct pushes to main are blocked, so this workflow is enforced automatically.

Versioning

Semantic versioning (i.e. a scheme that follows a vMAJOR.MINOR.PATCH format; see https://semver.org for details) is used for this project. The single point of truth for the current production version is the last git tag on the main branch with a v[0-9]* format. When developing locally, the reported version will appear as v0.0.0-dev.

Changes are handled by a GitHub Workflow which increments the version and creates a new tag whenever a push occurs to the main branch. This ensures that every commit on the main branch is assigned a unique version. The logic by which it modifies the version is as follows:

if the message of the PR’s head commit contains the text [version:major], then MAJOR is incremented;
else if it contains the text [version:minor], then MINOR is incremented;
else PATCH is incremented.

A MAJOR version change should be indicated if the PR introduces a breaking change. A MINOR version change should be indicated if the PR introduces new functionality.

Note

Make sure you think carefully about what type of changes you are committing. If you are adding functionality, make sure you bump the MINOR version; if you are making breaking changes, make sure you bump the MAJOR version. Users will be very thankful that you did!

Tests

PyTest is used to run tests for this codebase. Make sure you run them before submitting any code to a PR by executing the following from the project root directory:

$ pytest

When testing, temporary projects get created and need to be installed into an environment before tests can be run, documentation builds checked, etc. Be warned that this can both pollute your development environment and lead to unreliable tests due to the bleeding of state from past test runs to new runs.

To address this, a Dockerfile and Makefile are provided for running tests in a container. Run the following to build the container:

make docker-build

Run the following to run the tests in the container:

make docker-tests

Note

Running the tests in a container on MacOS is slow: 10 mins on an M1 Mac vs 1 min in the CI/CD. Optimisation suggestions welcome.

Some further comments about how testing has been configured for projects rendered bt the ADACS Base Template:

Coverage

Code coverage measures what fraction of the codebase is executed by the test suite — a line that is never run by any test cannot be verified to be correct. PyTest has been configured to produce a coverage report after each run, listing the percentage covered per file and the specific line numbers that no test reached.

While not strictly enforced, we encourage developers to make sure that anything they do to the codebase does not reduce this metric. The report is a useful guide to where additional tests would add the most value.

Type Hints

Type hints are used in this codebase but presently not configured to be enforced. Developers are encouraged to use them and use mypy (which has been added to the list of developer dependencies to this project) to check for a host of errors that this tool can efficiently identify. This can be done by running mypy, passing it the path to the code you want to check as follows:

$ mypy  --explicit-package-bases <path>

This can be a specific file or a path underwhich all code is checked. To run mypy on the whole project codebase, simply run the following from the code’s root directory:

$ mypy  --explicit-package-bases python

Git Hooks

This project has been set-up with pre-configured git hooks. They should be used as a means for developers to quickly check that (at least some) of the code standards of the project are being met by commited code. Ultimately, all standards are actually enforced by the continuous integration pipeline (see below). Running quick checks (like linting) at the point of commiting code can save time that might otherwise be lost later (for example) at the PR or release stage when testing needs to be rigorous and policy enforcement generally fails slower. Developers can choose to either:

use the git hooks defined by this project (recommended, for the reasons given above; see below for instructions),
not to use them, and rely purely on the CI workflow to enforce all project policies, or
configure their IDE of choice to manage things, in which case it is up to them to make sure that this aligns with the policies being enforced by the CI.

If developers would like to utilise the git hooks provided by this project they just need to run the following command from within the project:

$ pre-commit install

Some of these hooks require internet access to work. If you are trying to commit to the repository locally and are being prevented from doing so because you are working offline, the hooks can be ignored by using the --no-verify flag when running git commit, like so:

$ git commit --no-verify

Alternatively, you can disable them by running:

$ pre-commit uninstall

They can subsequently be re-enabled by reinstalling them.

Maintaining Git Hooks

The git hooks are defined in the .pre-commit-config.yaml file. Specific revisions for many of the tools listed should be managed with Poetry, with syncing managed with the sync_with_poetry hook. Developers should take care not to use git hooks to enforce any project policies. That should all be done within the continuous integration workflows. Instead: these should just be quality-of-life checks that fix minor issues or prevent the propagation of quick-and-easy-to-detect problems which would otherwise be caught by the CI later with considerably more latency. Furthermore, ensure that the checks performed here are consistant between the hooks and the CI. For example: make sure that any linting/code quality checks are executed with the same tools and options.

Releases

Releases are generated through the GitHub UI. A GitHub Workflow has been configured to do the following when a new release is produced:

Run the tests for the project,
Ensure that the project builds,
Rebuild the documentation on Read the Docs, and
Publish a new version of the code on PyPI.

Note

If a release is flagged as a “pre-release” through the GitHub interface, then documentation will not be built and the project will be published on test.PyPI.org (if configured; see instructions here) instead.

Generating a new release

To generate a new release, do the following:

Navigate to the project’s GitHub page,
Click on Releases in the sidebar,
Click on Create a new release (if this is the first release you have generated) or Draft release if this is a subsequent release,
Click on Choose a tag and select the most recent version listed,
Write some text describing the nature of the release to prospective users, and
Click Publish Release.

Documentation

Documentation for this project is generated using Sphinx and is hosted on Read the Docs for the latest release version. Sphinx is configured here in the following ways:

Content can be managed with Markdown (.md) rather than Restructured Text (.rst)

Developers are mostly spared the pain of direcly editing .rst files (the usual way of generating content for Sphinx) in the following ways:
- default .rst files are generated by sphinx-apidoc from a Jinja2 template placed in the docs/_templates directory of the project.
- MyST-Parser is used to source all content from Markdown files. MyST-Parser also offers several optional Markdown extensions enabling the rendering of richer content (e.g. Latex equations). Several of these extensions have been enabled by default, but not all. This can be managed by editing the myst_enable_extensions list in docs/conf.py.
A single point of truth for high-level aspects of the documentation

The project README.md is utilised, creating a single point of truth for the main high-level aspects of the documentation for both this documentation and all the homepages associated with the services used by this project (see above).

Note

MyST-Parser directives will not render properly on GitHub. Similarly: GitHub Markdown directives will not work on Read The Docs.
As much content as possible is generated from the code itself

sphinx-autodoc is used to extract documentation from the docstrings in your codebase.

Generating the Documentation

Documentation can be generated locally by running the following from the root directory of the project:

$ make docs

This will generate an html version of the documentation in docs/_build/html which can be opened in your browser. On a Mac (for example), this can be done by running the following:

$ open docs/_build/html/index.html

Editing the Documentation

The majority of documentation changes can be managed in one of the following 5 ways:

Edits to README.md:

Most high-level content should be presented in the README.md file. This content gets used by the project documentation and is shared by the GitHub project page and the PyPI page.
Project Docstrings:

Documentation for code changes specifying the codebase’s API, implementation details, etc. should be managed directly in the Docstrings of the project’s .py files. This content will automatically be harvested by sphinx-apidoc.
Existing Markdown files in the docs directory:

Examine the Markdown files in the docs/content directory. Does the content that you want to add fit naturally within one of those files? If so: add it there.
Add a new Markdown file:

Otherwise, create a new .md file in the docs/content directory and add it to the list of Markdown files referenced in docs/index.rst. Note that these files will be added to the documentation in the order specified, so place it in that list where you want it to appear in the final documentation. This new .md file should start with a top-level title (marked-up by starting a line with a single #; see the top of this file for an example).
Extend the MyST-Parser support:

New MyST-Parser extensions can be enabled in docs/conf.py by extending the myst_enable_extensions list. See the MyST-Parser documentation for a list of available extensions and instructions on how to use them.

Adding images, etc.

While not strictly required, it is best practice to place any images, plots, etc. used in the documentation in the docs/assets directory.