Azure Deployment Steps

This page will guide you through some high-level steps required to deploy Carrot Mapper on Azure.

Resources Creation

Let’s start by create some Azure resource for Carrot:

Resource group

If you didn’t do so, create a resource group which holds related resources for an Azure solution.

Choose a suitable Subscription, Region and a name for your Resource group.

Web apps

Two web apps will need creating: One for Backend and another for Frontend.

Basics

Choose a suitable Subscription, Resource group, Region, Pricing plan, and a name for your Web app.

Publish mode and Operating system should be Container and Linux, respectively.

Container

Choose Other container registries as Image source.

For Docker hub options, setup as below:

Access type: Public

Registry server URL:

ghcr.io/health-informatics-uon

Image and tag:

carrot/frontend:<tag>

You can choose a version of this container by using the tag for the image. Look for available tags here.

Networking: As default.

Monitor + Secure: We recommend Application Insights to be enabled.

Tags: Give name and value for the web app’s tag, if needed.

Function app

Choose App service when prompted.

Basics

Choose a suitable Subscription, Resource group, Region, Pricing plan, and a name for your Function app.

Select Container Image as the mode of deploying.

Storage: Choose or create an Azure storage where the Function app will store its Blobs and Queues.

Networking: As default.

Monitor + Secure: We recommend Application Insights to be enabled.

Tags: Give name and value for the web app’s tag, if needed.

Container Image: A default image will be used as you create a function app. Therefore, after the app being deployed you must change this image by doing so:

Access the newly created Function app
Under tab Deployment, choose Deployment center
Choose Private Registry as Registry source
Change Server URL to: https://ghcr.io
Change Full Image Name and Tag to: health-informatics-uon/carrot/workers:<tag>. Look for available tags here.
Save the setting

PostgreSQL Database

If you didn’t have one, create a PostgreSQL Database on Azure.

Follow prompted steps to create a suitable Database for Carrot.

Data Factory

Data factory is useful when creating workflows/scripts to load and manage data on Azure storage and Azure database.

Create one following the prompted steps.

Key vault

Key vault is highly recommended to keep secrets and keys for functioning Carrot safe. If you didn’t have one, create a Key Vault following the prompted steps on Azure.

Preparation

You will need to prepare your database and storage before configure other apps.

Database

Using Data factory or other ways to create a schema named omop in your database, then load there the vocabularies, which you would like to be in Carrot, downloaded from Athena. The guide for downloading can be found here.

Storage

Navigate to the Storage account linked with you Function app, then Data storage. Add these Queues and Containers:

Queue for Rules actions triggers, e.g., `rules`
Queue for Mapping rules exports, e.g., `rules-exports`
Queue for Scan reports uploads, e.g., `uploadreports`

Note: Queues names should match with RULES_QUEUE_NAME RULES_FILE_QUEUE_NAME WORKERS_UPLOAD_NAME in Azure functions.

Configuration

Below are some important steps to configure created apps making them working together smoothly. More details and Azure examples can be found on Configuration section.

Backend app

Navigate to Environment variables section under Settings of your Backend web app, then add the variables below:

Key	Description
`FRONTEND_URL`	The URL of your Frontend app.
`ALLOWED_HOSTS`	A list of strings representing the host/domain names that this Django site can serve. Put here the URL of your Backend app and `localhost`.
`DB_ENGINE`	The database backend to use. Put here `django.db.backends.postgresql`.
`DB_HOST` `DB_PORT` `DB_NAME` `DB_USER` `DB_PASSWORD`	These settings (`port`, `host`, `name`, `user`, `password`) are required for PostgreSQL database connection. Put the details based on the setting of your Database. `DB_PASSWORD` is recommended to be put in the Key Vault.
`WORKERS_URL`	The URL of `Workers` (Azure functions) service which `Backend` should connect to.
`WORKERS_UPLOAD_NAME` `WORKERS_RULES_EXPORT_NAME`	Name of queues in Azurite that `Workers` send messages about ScanReport uploads and Mapping Rules exports, respectively. These should match with the queues for ScanReport uploads and Mapping rules file exports in Azure storage.
`WORKERS_RULES_KEY`	The key to authorise the request sent to `Workers` from `Backend`. Get this value by navigating to the Function app, then inside `App keys` under `Functions`, copy the `_master` key. It is recommended to put this in the Key Vault.
`STORAGE_CONN_STRING`	The key to connect Backend and Azure local storage. Get this value by navigating to the Azure storage, then inside `Access keys` under `Security + Networking`, copy the `Connection string`. It is recommended to put this in the Key Vault.
`SECRET_KEY`	A secret key for a particular Django installation. Choose a complex key for this. It is recommended to put this in the Key Vault.
`SIGNING_KEY`	A key required in JWT token generation process for Next Auth. Choose a complex key for this. It is recommended to put this in the Key Vault.

Restart your app for the environment variables to be applied.

After that, you need to seed the Carrot app database with the OMOP table and field names. For now, this can be done by connecting directly to your database either by local machine or through an environment, then run the command:

python manage.py loaddata mapping filetypes

A Django superuser also need creating by connecting to the Backend app and run the command:

python manage.py createsuperuser

Some first instances of Carrot also need creating after the Backend app up and running. Doing this be access the URL: <backend-URL>/admin/, log in using the super user credentials, then create first Data partner, Dataset, and Project.

Frontend app

Navigate to Environment variables section under Settings of your Frontend web app, then add the variables below:

Key	Description
`BACKEND_URL`	The URL of the Backend web app.
`NEXTAUTH_BACKEND_URL`	The backend base endpoint URL where `Frontend` send auth requests to. Put here: `<backendUrl>/api/`
`NEXTAUTH_SECRET`	Used to encrypt the NextAuth.js JWT, and to hash email verification tokens. Choose a complex key for this. It is recommended to put this in the Key Vault.
`NEXTAUTH_URL`	The base URL where login and logout processes point to. Put here the URL of the Frontend web app

Function app

Navigate to Environment variables section under Settings of your Function app, then add the variables below:

Key	Description
`FUNCTIONS_WORKER_RUNTIME`	The language or language stack of the worker runtime to load in the function app. Put here `python`.
`DB_ENGINE`	The database backend to use. Put here `django.db.backends.postgresql`.
`DB_HOST` `DB_PORT` `DB_NAME` `DB_USER` `DB_PASSWORD`	These settings (`port`, `host`, `name`, `user`, `password`) are required for PostgreSQL database connection. Put the details based on the setting of your Database. `DB_PASSWORD` is recommended to be put in the Key Vault.
`RULES_QUEUE_NAME` `RULES_FILE_QUEUE_NAME` `WORKERS_UPLOAD_NAME`	Matching name of queues in Azurite that `Workers` send messages about Rules actions triggers, Mapping Rules exports and ScanReport uploads, respectively.
`STORAGE_CONN_STRING`	The key to connect Backend and Azure local storage. Get this value by navigating to the Azure storage, then inside `Access keys` under `Security + Networking`, copy the `Connection string`. It is recommended to put this in the Key Vault.
`AzureWebJobsStorage`	Same value with the above.
`SECRET_KEY`	Same value with `SECRET_KEY` in Backend app. It is recommended to put this in the Key Vault.

Key vault

Set up your key vault by generate Keys/Secrets under Objects of your Key vault app. Give your secret a name and add there the value of the secret.

You would also need to set up the Access policies of your Key vault: Add there Applications and Users, along with their permissions to access the vault. If you find it hard to search for a specific object while creating access policy, search by the object ID which can be generated by Identity under Settings of the application.

To use secrets in Key vault, change the value of the important environment variables to:

@Microsoft.KeyVault(VaultName=<key-vault-name>;SecretName=<secret-name>)

After that, you need to click Pull reference values which is located on top of the Environment variables page to apply the values in the Key vault to your environment.

Testing

To test if Carrot is deployed successfully on Azure, open the URL of the Frontend app, then log in using admin credentials. If you can upload a scan report, add some manual and automatic Concepts, and download Mapping rules files successfuly, that’s a sign that Carrot now can be used.

🎉

Congratulations on your first Carrot running on Azure!

Requirements Internal data structures