This document provides notes for contributing to or modifying the Carrot Transform tool.

If you only need to use carrot-transform - not develop it - check out the official documentation.

Development Notes

Using uv

We use uv for managing dependencies and running Python scripts.

What is uv?

uv is a small command line program that invokes other python tools. There’s also uvx, a companion to uv, which works like npx or pipx to run packages without installing them to a project.

Installing uv

Follow the installation guide. It’s surprisingly fast to set up.

Running Tests

We use pytest for testing. You can run all tests with:

uv run pytest -m "not skipping"

This requires Docker as tests are run against MinIO, Trino and PostGreSQL database engines that need to be running in the background. This will take a long time.

A shorter selection of tests can be run with just …

uv run pytest

… which won’t use Docker and but will still perform some database tests with SQLite.

Running from Source

You can run the CLI directly like this:

uv run -m carrottransform.cli.subcommands.run mapstream \
    --inputs carrottransform/examples/test/inputs \
    --person Demographics \
    --rules-file carrottransform/examples/test/rules/rules_14June2021.json \
    --output build

⚠️ On Windows, you would need to replace / with ^ for line continuation in the terminal.

Or just paste it into a text editor and hit END END BACKSPACE DELETE until you’ve reformatted it into one line, then, paste that into the command prompt

Deploying to PyPI and CI / GitHub Actions

The CI file uses GitHub actions to test the project and (when appropriate) deploy to PyPI. When the CI job runs, it tests and assembles the project. If the commit being examined is against the main branch, then, the project will be deployed to PyPI Test. If the commit has been tagged with a tag starting with v... then the project will be deployed to PyPI Main.

Release

The Carrot Transform release procedure is;

• update version in pyproject.toml
• merge that into main
• tag it with v?.?.? and await the CI job

This can be done as follows …

• λ git checkout main to checkout the main branch
• λ git pull origin main to ensure your local is up to date
• λ uvx poetry version minor this will increment the minor version
  • you can change major . minor . patch with this command
  • you can also do the edits by hand if you like
• λ NEW_VERSION=$(uvx poetry version -s) to read the version
  • this will be different on Windows
  • again, you can set the envar by hand
  • … or substitue the values below by hand …
• λ git checkout -b release/v$NEW_VERSION
  • Windows cmd uses %NAME% for shell substitution; you’ll need to do something different
• λ git add pyproject.toml
• λ git commit -m "Bump version to $NEW_VERSION"
• λ git push --set-upstream origin release/v$NEW_VERSION
• submit the PR and merge it with main
• λ git checkout main
• λ git pull origin main
• λ git tag v$NEW_VERSION
• λ git push origin v$NEW_VERSION