Aller au contenu

Set up a development environment

Ce contenu n’est pas encore disponible dans votre langue.

This tutorial runs madbot from source on your machine: the API with its services (Vault, Redis, Celery), then the web client. At the end, you can sign in to your local madbot at http://localhost:3000.

Install:

  • Git;
  • pixi, which installs Python and every dependency of the API;
  • Node.js 22.23.4, the version required by the client, with Corepack, which provides the right version of pnpm.
Terminal window
# macOS and Linux
curl -fsSL https://pixi.sh/install.sh | sh
# Windows
powershell -ExecutionPolicy ByPass -c "irm -useb https://pixi.sh/install.ps1 | iex"

Docker is only needed for the optional parts: debugging emails with Mailhog and the Docker Compose stack.

Terminal window
git clone https://gitlab.com/ifb-elixirfr/madbot/madbot.git
cd madbot

The repository holds the API (api/), the web client (client/), the plugins_api package (plugin_api/) and this documentation (documentation/).

Terminal window
cd api
pixi install
pixi install -e dev

The default environment holds what madbot needs to run; the dev environment adds the test, lint and security tools. This also installs plugins-api from the package registry: you only need the plugin_api/ folder to change plugins_api itself.

Run a madbot command with pixi run madbot <command>, from api/:

Terminal window
pixi run madbot migrate

Or open a shell in the environment and call madbot directly:

Terminal window
pixi shell -e dev
madbot migrate

The project also defines shortcuts such as pixi run test or pixi run lint: see Tests and quality checks.

Copy the example configuration:

Terminal window
cp .env.example .env

.env.example lists every setting with an example value and explains the syntax the file must follow. For a first run, fill in:

  • SECRET_KEY: required, and the same for the API and the Celery workers. madbot refuses to start with an empty or placeholder value. Generate one:

    Terminal window
    python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
  • MADBOT_OIDC_RSA_PRIVATE_KEY: the private key with which madbot signs its OpenID Connect tokens, on a single line. Generate it and copy it:

    Terminal window
    openssl genrsa -out oidc.key 4096
    # macOS
    awk -v ORS='\\n' '1' oidc.key | pbcopy
    # Linux
    awk -v ORS='\\n' '1' oidc.key | xclip -selection clipboard
  • PUBLIC_BACKEND_DOMAIN: the address of the API, as host:port without the scheme (default localhost:8000).

madbot keeps the credentials of the connections in Vault. In a new terminal, from api/:

Terminal window
pixi run vault

The script starts a local Vault and prints the settings madbot needs:

Terminal window
MADBOT_VAULT_URL=http://localhost:8200
MADBOT_VAULT_MOUNT_POINT=madbot
MADBOT_VAULT_ROLEID=<role id>
MADBOT_VAULT_SECRETID=<secret id>

Copy them into api/.env. Keep the terminal open.

Redis carries the Celery tasks and the WebSocket messages. In a new terminal, from api/:

Terminal window
pixi run redis

pixi run redis uses REDIS_PORT from api/.env (default 6379).

The global workspace search is backed by Meilisearch. It is off by default (SEARCH_ENABLED=false): no indexing task is dispatched, /api/search answers 503 and the client hides its entry point. Skip this section unless you work on the search itself.

Meilisearch ships as a single binary (the meilisearch package on conda-forge is stale, so it is not a pixi dependency). In a new terminal, from api/, install it with the official script and start it:

Terminal window
curl -L https://install.meilisearch.com | sh
./meilisearch --master-key="<a-dev-master-key>"

Running from api/ keeps the binary and the data.ms/ data directory where api/.gitignore already ignores them.

By default it listens on http://localhost:7700. The master key is a secret you choose (Meilisearch recommends at least 16 bytes). Locally, reuse that same value as MEILISEARCH_API_KEY.

In api/.env:

Terminal window
SEARCH_ENABLED=true
MEILISEARCH_HOST=http://localhost:7700
MEILISEARCH_API_KEY=<the master key used above>

Enabling the search also registers a Meilisearch check on /health-check as a non-critical service: an unreachable Meilisearch shows as degraded there without failing the endpoint.

You build the index once the database is created (see below). Day-to-day updates then go through Celery, so keep a worker running (pixi run madbot celery). Whenever you turn SEARCH_ENABLED back on later, run pixi run madbot rebuild_search_index again: nothing written while the search was off is indexed automatically.

To enable the search on a deployed instance, see Enable the global search.

Back in the first terminal, from api/:

Terminal window
pixi run madbot migrate
pixi run madbot collectstatic
pixi run madbot initialise_madbot_metadata_fields
pixi run madbot create_periodic_tasks
pixi run madbot reload_plugins

If you enabled the search, build its index:

Terminal window
pixi run madbot rebuild_search_index

The management commands reference explains each madbot command.

Create an administrator account. Adapt the username, email, password and names; the ROR identifier of the institution is optional here:

Terminal window
echo "from django.contrib.auth import get_user_model; User = get_user_model(); User.objects.create_superuser('admin', 'admin@example.org', '<password>', id='9da4931b-e89e-48f1-988d-0aeb6a2ea1bd', first_name='Madbot', last_name='Admin', ror='https://ror.org/<ror id>', is_approved=True)" | pixi run madbot shell

Register the web client as an OpenID Connect application of the API:

Terminal window
pixi run madbot createapplication --client-id=madbot --client-secret=madbot --name="madbot" --algorithm=RS256 --redirect-uris=http://localhost:3000/api/auth/callback/madbot --user="9da4931b-e89e-48f1-988d-0aeb6a2ea1bd" public authorization-code

If the client does not run on port 3000, adapt --redirect-uris, and CORS_ALLOWED_ORIGINS and CSRF_TRUSTED_ORIGINS in api/.env.

Celery runs the long tasks: submissions, copies, imports. In a new terminal, from api/:

Terminal window
pixi run madbot celery

When DEBUG is on, the worker restarts whenever the code changes. In another terminal, start the scheduler of the periodic tasks:

Terminal window
pixi run madbot beat

To follow the tasks in a web interface, start Flower and open http://localhost:5555:

Terminal window
pixi run celery --broker=redis://localhost flower --port=5555 --basic_auth=$CELERY_FLOWER_OPTIONS
Terminal window
pixi run runserver

The API answers at the address of PUBLIC_BACKEND_DOMAIN, http://localhost:8000 by default. Its interactive reference is at http://localhost:8000/api/schema/swagger-ui/.

madbot alone has no source to link data from: plugins bring them. List the plugins to use in api/.env:

Terminal window
MADBOT_INSTALLED_PLUGINS=madbot_sshfs,madbot_zenodo

Then install them, from api/:

Terminal window
pixi run install-plugins

The script has two modes:

  • editable (default, or -e): it clones each plugin into MADBOT_PLUGINS_DIR (default madbot-plugins, a path relative to the folder that holds madbot/) and installs it with pip install -e. Your changes to the plugin code apply without reinstalling. When a plugin is already cloned, the script updates it with git pull if it is on main or dev without local changes, and leaves it as it is otherwise; --no-update-check skips this.
  • package (-p, or pixi run install-plugins-package): it installs each plugin from its Git repository, as a regular package.

Add -y to skip the confirmation. Then reload the plugins and check what is installed:

Terminal window
pixi run madbot reload_plugins
pixi run list-plugins

The plugins reference lists the available plugins.

To install a plugin by hand, install it in both environments:

Terminal window
pixi run -e default pip install -e /path/to/plugin
pixi run -e dev pip install -e /path/to/plugin

In a new terminal, from the root of the repository:

Terminal window
cd client
node -v

The client requires Node.js 22.23.4. Then install the dependencies, with the version of pnpm pinned by the project:

Terminal window
corepack enable
pnpm install

If the dependencies look corrupted, pnpm clean removes node_modules; run pnpm install again.

Terminal window
cp .env.example .env

Generate NUXT_SECRET:

Terminal window
openssl rand -base64 48 | tr -dc 'A-Za-z0-9' | head -c 32; echo

AUTH_ORIGIN and NEXTAUTH_URL are the full address of the authentication routes of the client, for example http://localhost:3000/api/auth. Keep them in line with NITRO_PORT: without NEXTAUTH_URL, the sign-in returns to port 3000.

Terminal window
pnpm dev

Open http://localhost:3000 (or the port of NITRO_PORT) and sign in with the administrator account.

SymptomCheck
The API refuses to start and mentions SECRET_KEYSECRET_KEY is set in api/.env, and is not a placeholder.
Errors about VaultThe terminal of pixi run vault is still open, and its settings are in api/.env.
Errors about Redis, tasks that never runRedis and the Celery worker are running.
The client stays on “Loading madbot…”The API answers, and its migrations are applied: pixi run madbot migrate.
A plugin is missingIt is listed in MADBOT_INSTALLED_PLUGINS, installed, and reload_plugins ran after the installation.