Publishing to PyPI

This document describes the prerequisites and workflow to publish Spine Toolbox to The Python Package Index (PyPI).

Versioning of Spine Toolbox packages

Spine Toolbox packages use the latest Git tag to dynamically generate the version number. During the build process Git tags of the form X.Y.Z are sorted and the latest is used to generate the package version. If the tip of the current branch (HEAD) is at a tag, the version number is the tag. However, if there have been commits since the latest tag, the next version is guessed and a dev??-* component is included (e.g. '0.7.0.dev77+gf9538fee.d20230816'). Note that the dev* component also includes an indication of the number of commits since the last tag.

Under this scheme, the release process is simply to create a new Git tag, and publish it. However since the different Spine packages depend on each other, you need to update the different version number requirements in their respective pyproject.toml files. This can be done conveniently by using the CLI tools available in the spine-conductor repo.

Creating Git tags and publishing to PyPI

  1. Check out the spine-conductor repo, and install it, either in a virtual environment or using pipx.

  2. You can create a TOML configuration file as mentioned in the README of the repo; say release.toml. Something like the sample below should work.

    release.toml
    [tool.conductor]
packagename_regex = "spine(toolbox|(db){0,1}[_-][a-z]+)"  # package name on PyPI

[tool.conductor.dependency_graph]
spinetoolbox = ["spine_items", "spine_engine", "spinedb_api"]
spine_items  = ["spinetoolbox", "spine_engine", "spinedb_api"]
spine_engine = ["spinedb_api"]
spinedb_api  = []

[tool.conductor.repos]
spinetoolbox = "."
spine_items  = "venv/src/spine-items"
spine_engine = "venv/src/spine-engine"
spinedb_api  = "venv/src/spinedb-api"

# # default
# [tool.conductor.branches]
# spinetoolbox = "master"
# spine_items  = "master"
# spine_engine = "master"
# spinedb_api  = "master"

  3. Now you can create a release by calling the conduct release -c release.toml command with the TOML file as config. This starts a guided session where the spine-conductor CLI tool deduces the next version numbers from existing Git tags, updates the corresponding pyproject.toml files in all the repos to reflect the new package versions, and finally prompts you to add any edited files, and create the new Git tag. A typical session would like this:

    A typical release session; note the JSON summary in the end.
    $ cd /path/to/repo/Spine-Toolbox
$ conduct release --bump patch -c release.toml  # or include in pyproject.toml
Repository: /path/to/repo/Spine-Toolbox
## master...origin/master
 M pyproject.toml (1)
Select the files to add (comma/space separated list): 1
Creating tag: 0.6.19 @ 034fb4b
Repository: /path/to/repo/venv/src/spine-items
## master...origin/master
 M pyproject.toml (1)
Select the files to add (comma/space separated list): 1
Creating tag: 0.20.1 @ 5848e25
Repository: /path/to/repo/venv/src/spine-engine
## master...origin/master
 M pyproject.toml (1)
Select the files to add (comma/space separated list): 1
Creating tag: 0.22.1 @ e312db2
Repository: /path/to/repo/venv/src/spinedb-api
## master...origin/master
Select the files to add (comma/space separated list):
Creating tag: 0.29.1 @ d9ed86e

Package Tags summary  💾 ➡ 'pkgtags.json':
{
  "Spine-Toolbox": "0.6.19",
  "spine-items": "0.20.1",
  "spine-engine": "0.22.1",
  "Spine-Database-API": "0.29.1"
}

    If the session completes successfully, you will see a session summary with the newest Git tags that were created for each package.

  4. Push the newly created tags to GitHub. On sh-like shells like: bash, zsh, or git-bash (Windows):

    for repo in . venv/src/{spinedb-api,spine-{items,engine}}; do
    pushd $repo;
    git push origin master --tags;
    popd
done

    With Powershell on Windows, something like this should work:

    "." , "venv/src/spinedb-api", "venv/src/spine-items", "venv/src/spine-engine" | % {
  pushd $_;
  git push origin master --tags;
  popd;
}

  5. Now you can trigger the workflow to publish the packages to PyPI either by using GitHub CLI, or from the workflow dispatch menu in the spine-conductor repo.

    cat pkgtags.json | gh workflow run --repo spine-tools/spine-conductor test-n-publish.yml --json

    If you are using the workflow dispatch menu, make sure you input the exact same package versions as shown in the summary.

Done! Note: Soon, (4) & (5) will be wrapped in a separate command provided by spine-conductor.

The release.toml file

The config file is a standard TOML file conformant with pyproject.toml, meaning all configuration goes under the section tool.conductor. The configuration is split into 4 sections: a regex to identify our packages, dependency graph between our packages, path to the repos to be used for the release, and the branches to be used (optional).

  1. You can specify a regular expression that will be used to identify “our” packages. Something like the following should work:

    Spine package name regular expression
    [tool.conductor]
packagename_regex = "spine(toolbox|(db){0,1}[_-][a-z]+)"  # package name on PyPI

    Note that PyPI treats - (hyphen) and _ (underscore) as equivalent in package names; i.e. spinedb_api and spinedb-api are equivalent, the regex should accomodate that.

  2. The dependency graph between our packages should be specified under the dependency_graph section:

    Spine package dependency graph
    [tool.conductor.dependency_graph]
spinetoolbox = ["spine_items", "spine_engine", "spinedb_api"]
spine_items  = ["spinetoolbox", "spine_engine", "spinedb_api"]
spine_engine = ["spinedb_api"]
spinedb_api  = []

    Essentially it is a mapping of the “primary” package, and a list of its Spine dependencies.

  3. Point to the repository directories relative to your current working directory. The following example would be valid if you are preparing the release from the Toolbox repo, and the other Spine package repos are in the virtual environment.

    Repository paths
    [tool.conductor.repos]
spinetoolbox = "."
spine_items  = "venv/src/spine-items"
spine_engine = "venv/src/spine-engine"
spinedb_api  = "venv/src/spinedb-api"

  4. You can also specify the branches for each repository that should be used for the release. This section is optional, and if left unspecified, the branch name is assumed to be master.

    Release branches on Spine repositories
    # default: master
[tool.conductor.branches]
spinetoolbox = "release"
spine_items  = "release"
spine_engine = "release"
spinedb_api  = "release"

Manual release (in case of emergency)

This section documents what the spine-conductor CLI tool does under the hood. It is here in case of an emergency (e.g. there’s a bug), and the release has to be done manually.

As mentioned earlier, the package version is now derived from Git tags. However, because of the internal dependency between the Spine packages, the versions of the dependencies have to synchronised with the new version. The steps are as follows:

  1. Determine the next version for each Spine package. This can be done manually with Git, or you can use setuptools_scm in a Python REPL.

    • You can run git describe --tags in the repo. This will print out the latest tag followed by a trailer with metadata on distance from the tag; something like this: 0.6.18-100-g411c13e1. If you want to make a patch release, the next version would be 0.6.19 and a minor release would be 0.7.0. Repeat this process for all 4 Spine repos.

    • If using a Python REPL, you can do the following for a minor release:

      >>> from setuptools_scm import get_version
>>> get_version(".", version_scheme="release-branch-semver")
'0.7.0.dev100+g411c13e1.d20230823'

      For a patch release, do the following:

      >>> get_version(".", version_scheme="guess-next-dev")
'0.6.19.dev100+g411c13e1.d20230823'

      Note the first argument to get_version is the path to the repository. The above examples assume the repository is your current directory. If it’s not, you can provide the path as the first argument.

  2. Once the new package versions are determined, you need to edit the pyproject.toml files in all 4 repositories with the correct version numbers. For example, in the Spine-Toolbox repo if you were to do a minor release, i.e. 0.6.18 → 0.7.0, the following change would be sufficient:

    Example edit to pyproject.toml for Spine-Toolbox
    diff --git a/pyproject.toml b/pyproject.toml
index bd38a2b7..dd9c228e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,8 +20,8 @@ dependencies = [
     "jupyter-client >=6.0",
     "qtconsole >=5.1",
     "sqlalchemy >=1.3",
-    "spinedb_api >=0.29.0",
-    "spine_engine >=0.22.0",
+    "spinedb_api >=0.30.0",
+    "spine_engine >=0.23.0",
     "numpy >=1.20.2",
     "matplotlib >= 3.5",
     "scipy >=1.7.1",
@@ -30,7 +30,7 @@ dependencies = [
     "pygments >=2.8",
     "jill >=0.9.2",
     "pyzmq >=21.0",
-    "spine-items >= 0.20.0",
+    "spine-items >= 0.21.0",
 ]

 [project.urls]

  3. After updating the pyproject.toml file for all 4 Spine repos as above, add and commit the changes in all repos:

    git commit -i pyproject.toml -m "Release 0.7.0"

  4. Create a Git tag on the latest commit:

    git tag 0.7.0 HEAD

  5. Push the tags to GitHub. On sh-like shells like: bash, zsh, or git-bash (Windows):

    Recipe to push Git tags to GitHub on sh-like shells (bash, zsh, git-bash)
    for repo in . venv/src/{spinedb-api,spine-{items,engine}}; do
    pushd $repo;
    git push origin master --tags;
    popd
done

    With Powershell on Windows:

    Recipe to push Git tags to GitHub on Powershell
    "." , "venv/src/spinedb-api", "venv/src/spine-items", "venv/src/spine-engine" | % {
  pushd $_;
  git push origin master --tags;
  popd;
}

  6. Now you can trigger the workflow to publish the packages to PyPI from the workflow dispatch menu in the spine-conductor repo. Ensure you input the exact same package versions as in the tags.

  7. In case the workflow above also fails, you have to build the source distribution archive and wheels locally and upload to PyPI manually.

    To build, ensure you have build installed. The build backend ensures build isolation, and reproducibility of the wheels given a source distribution.

    Build distribution archives and wheels
    python -m pip install build
python -m build

    Once the build completes, you can find the source tarball and the wheel in dist/. Now you may upload these files to PyPI.

    It is good practise to first test using TestPyPI before uploading to PyPI, since releases on PyPI are read-only. You want to avoid mistakes.

    Register an account and ask some of the owners of the Spine Toolbox package (or other relevant package) to add you as a maintainer.

    Upload the distribution using

    twine upload --repository testpypi dist/*

    See Using TestPyPI for more information. To avoid entering your username and password every time, see Keyring support in twine documentation or generate an API key. If everything went smoothly, you are ready to upload the real index. Again, you need to register to PyPI and ask to become a maintainer of the package you want to upload to. Upload the distribution using

    $ twine upload dist/*

Done! Now fix the bug that forced you to do the manual release ;)