Carrot MapperDevelopersQuickstart

Developer Quickstart

This page will guide you through getting Carrot running locally for development.

In development, docker compose only runs the supporting services (Postgres, blob storage, OMOP Lite, Keycloak and Airflow). The API (Django) and the frontend (Next.js) are run from source, so you get fast reload and a normal debugger.

Start by cloning the Carrot Mapper repository.

For how to contribute (issues, code style, pull requests and releases), see the repository’s CONTRIBUTING.md.

Prerequisites

  • Docker Desktop
  • uv for the Python API (the API requires Python 3.11, which uv will install for you)
  • Node.js and npm for the frontend
  • (Optional) Vocabularies downloaded from Athena, only if you need real vocabularies (see below).

Services

The docker-compose.yml in the root directory defines the following services:

ServiceDescriptionPort(s)
dbPostgres 18. Holds Carrot’s data, the omop schema, and separate airflow and keycloak databases.5432
omop-liteOne-shot container that creates the omop schema and loads vocabularies, then exits.-
azuriteAzure Blob/Queue storage emulator (profile azure, the default for local development).10000, 10001, 10002
rustfsS3-compatible storage (profile main), used instead of Azurite when STORAGE_TYPE=s3.9000 (API), 9001 (console)
keycloakIdentity provider used by Airflow for authentication. Imports the CARROT-MAPPER realm from realm-config/.8085
airflow-apiAirflow 3 API server (replaces the old airflow-webserver). Serves the Airflow UI and REST API.8080
schedulerAirflow scheduler, which also runs the DAG processor.-

The airflow-api and scheduler images are built locally from app/airflow, and app/airflow/dags is mounted into them, so changes to the DAGs are picked up without rebuilding.

Getting Started

For convenience, use three separate terminals:

  • Terminal 1 at the root directory (Docker services)
  • Terminal 2 at the app/api directory (Django API)
  • Terminal 3 at the app/next-client-app directory (Next.js frontend)

OMOP schema and vocabs setting

Carrot needs an OMOP CDM schema with vocabularies inside its database. This is created by the OMOP Lite container (omop-lite) when the stack starts.

By default (SYNTHETIC_VOCABS=true), OMOP Lite loads a small synthetic set of vocabularies, which is enough for most development work and requires no download.

If you need the real Athena vocabularies (for example, to test the auto-mapping functionality), do the following before starting the services:

  1. Set SYNTHETIC_VOCABS=false in the root .env file (created in the next step).
  2. Create a folder named vocabs in the root directory and put the vocabulary files downloaded from Athena there. The guide for downloading can be found here.

We don’t recommend downloading all of the available vocabularies on Athena, as this makes loading much slower. The minimum required vocabularies are: SNOMED CT, RxNorm, RxNorm Extension, LOINC, ICD10, ICD10CM, Gender, Ethnicity, OMOP Extension, NHS Ethnic Category, OPCS4 and UK BioBank.

Start the database and load vocabularies

At terminal 1 (root directory), create the .env file and start Postgres, OMOP Lite and the storage emulator:

cp .env.example .env
docker compose up -d db omop-lite azurite

Wait for omop-lite to finish and exit before continuing. It is a one-shot container, and its run time depends on the size of the vocabularies you are loading. You can follow its progress with:

docker compose logs -f omop-lite

On first start, Postgres runs init-pg/init-pg.sql, which creates the separate airflow and keycloak databases. No manual schema creation is needed for Airflow.

Bootstrap and run the API

At terminal 2 (app/api), install dependencies, set up the database and start the Django dev server:

uv sync
uv run manage.py migrate
uv run manage.py automatic_seeding_data
uv run manage.py default_super_user
uv run manage.py automatic_queue_and_containers_creation
uv run manage.py runserver
  • migrate creates Carrot’s tables.
  • automatic_seeding_data seeds the reference data Carrot needs.
  • default_super_user creates the superuser from SUPERUSER_DEFAULT_USERNAME, SUPERUSER_DEFAULT_PASSWORD and SUPERUSER_DEFAULT_EMAIL in .env (admin / admin by default).
  • automatic_queue_and_containers_creation creates the storage queues and blob containers.

There is no separate app/api/.env. python-dotenv walks up from app/api looking for a .env, so the root .env covers the API too.

Bring up the rest of the stack

At terminal 1, start the remaining services (Keycloak and Airflow):

docker compose up -d

The .env file sets COMPOSE_PROFILES=azure, so a plain docker compose up starts Azurite as the storage service. Airflow waits for both db and keycloak to be healthy before starting, so this can take a minute or two on first run.

Run the frontend

At terminal 3 (app/next-client-app), create the frontend .env and start the Next.js dev server:

cp .env.example .env
npm install
npm run dev

The frontend .env points at the API running from source on http://localhost:8000.

First instances creation

Create some first instances in Carrot by logging in to the Django Admin panel at http://localhost:8000/admin/ with the default credentials.

Following this order, fill out the required information to create new instances of Data Partner, Dataset and Project.

Check-in

You should now be able to log in to Carrot at http://localhost:3000/ with the superuser credentials. After logging in, you will be presented with the Projects page.

Try using Carrot by following the user quickstart. If you can upload a new scan report, create mapping rules, and export the mapping rules JSON, Carrot is running correctly.

🎉
Congratulations on your first Carrot running!

Default login details

ServiceURLUsernamePasswordNotes
Django Admin (API)http://localhost:8000/adminadminadminDefault superuser. Add more users from here.
Frontendhttp://localhost:3000adminadminSame default superuser as the API.
Airflowhttp://localhost:8080carrot-adminpassword123Airflow authenticates through Keycloak.
Keycloakhttp://localhost:8085adminadminKeycloak admin. Add more users here to give Airflow access.

Airflow 3 uses Keycloak (KeycloakAuthManager) instead of basic auth, so Airflow users and roles are managed in Keycloak, in the CARROT-MAPPER realm defined in realm-config/Carrot-Mapper.json. Keycloak is reached at http://keycloak.localtest.me:8085, a hostname that always resolves to 127.0.0.1, so no hosts file changes are needed.

Day-to-day development

Once the schema and migrations already exist in the db volume, the start-up order no longer matters. Just run:

# Terminal 1 (root)
docker compose up -d
 
# Terminal 2 (app/api)
uv run manage.py runserver
 
# Terminal 3 (app/next-client-app)
npm run dev

To stop the Docker services, run at the root directory:

docker compose down
⚠️

docker compose down -v also removes the db_data_pg18 volume, which deletes Carrot’s data and the loaded vocabularies. You will need to repeat the setup steps afterwards.

Using S3 storage instead of Azurite

Local development uses Azurite by default. To use the S3-compatible rustfs service instead, instead of commands in step 2, run these commands at the root directory:

cp .env.s3.example .env
docker compose up -d db omop-lite rustfs

Then restart the stack. The rustfs console is available at http://localhost:9001 (s3admin / s3admin).

⚠️

Switching storage type on an existing setup is not recommended, as files already uploaded to one storage service won’t be available in the other.

Upgrading from Airflow 2 setups

If you set up Carrot locally before the upgrade to Airflow 3, note the following breaking changes:

  • airflow-webserver has been replaced by airflow-api, and the scheduler now also runs the DAG processor.
  • Airflow authentication moved from basic auth to Keycloak. Airflow’s admin user is no longer created at start-up.
  • Airflow metadata now lives in a separate airflow database rather than an airflow schema in the postgres database.
  • AIRFLOW_BASE_URL is now the bare host (e.g. http://localhost:8080), without the /api/v1/ suffix.
  • Storage connections are provided through AIRFLOW_CONN_WASB_CONN / AIRFLOW_CONN_S3_CONN environment variables. The AIRFLOW_VAR_WASB_* / AIRFLOW_VAR_S3_* variables are no longer used.
  • New variables are required in .env: AIRFLOW_OPENID_CLIENT_SECRET, AIRFLOW_API_SECRET_KEY, AIRFLOW_API_AUTH_JWT_SECRET, KEYCLOAK_HOSTNAME, KEYCLOAK_BOOTSTRAP_ADMIN_PASSWORD and KEYCLOAK_STARTUP_COMMAND.
  • The database volume is now db_data_pg18 (Postgres 18).

The simplest way to migrate is to re-copy .env.example to .env (re-applying any custom values) and follow this guide from the start.

What next? Explore and try using Carrot with the user quickstart.