This is the full developer documentation for madbot # madbot documentation > Organise a research project, link its data where they are stored, describe them and submit them to repositories. [Use madbot](/use/concepts/)Workspaces, nodes, data, samples, metadata and submissions: the concepts of madbot. [Run an instance](/run/reference/commands/)The management commands of a madbot instance. [Automate with the API](/api/getting-started/)Drive madbot from your scripts and tools through its REST API. [Extend madbot](/extend/plugins/development-guide/)Connect a new data source or repository with a plugin. [Contribute](/contribute/)Report a problem, suggest an idea, or change the code. ## What madbot is [Section titled “What madbot is”](#what-madbot-is) madbot (Metadata And Data Brokering Online Tool) helps research teams manage their data and metadata. It does not store the data: it links them where they already are, a computing cluster, a storage server, an analysis platform, a repository, through plugins. Around these links, a team organises its project, describes its data with metadata, and submits them to public repositories such as ENA, Zenodo or Recherche Data Gouv. To discover madbot in a guided session, follow the [madbot training](https://ifb-elixirfr.gitlab.io/madbot/madbot-training/). To learn about the project, visit the [madbot website](https://madbot.france-bioinformatique.fr/). # Get started with the REST API > Authenticate with a personal access token, choose a workspace, try the endpoints in Swagger UI, read paginated lists and errors, then run a first script against madbot. Every madbot instance serves a REST API. The web client uses it, and so can your scripts. The [reference](/api/reference/) describes each endpoint for the latest release of `madbot-api`. This page explains what all the endpoints have in common, then runs a first script. Tip The reference does not call an API. To try an endpoint, use the Swagger UI of an instance: see [Try the endpoints in Swagger UI](#try-the-endpoints-in-swagger-ui). ## Authenticate with a personal access token [Section titled “Authenticate with a personal access token”](#authenticate-with-a-personal-access-token) A script authenticates with a personal access token, created in the web client: 1. Open the user menu, then **Account settings**, then **Tokens**. 2. Select **Create a new token**, name it and choose its scopes: * `read`: read the resources; * `write`: create, change and delete them; * `*`: full access, alone. 3. Copy the token. It starts with `pat_`, and the web client shows it only once. Send it in the `Authorization` header of each request: ```http Authorization: Bearer pat_… ``` The account of the token must be approved by an administrator of the instance. Tokens are managed in the web client only: the API does not create a token from another token. ## Name the workspace [Section titled “Name the workspace”](#name-the-workspace) Many endpoints work inside one workspace: nodes and their samples, data, connections, submissions and tasks. Name it in the `X-Workspace` header, with its identifier: ```http X-Workspace: 143fbb1c-a639-4c33-8c41-2c1c02d7b8c2 ``` The reference lists the header among the parameters of each endpoint that needs it. [`GET /api/workspaces`](/api/reference/operations/workspaces_list/) lists the workspaces of the account, with their identifiers. ## Try the endpoints in Swagger UI [Section titled “Try the endpoints in Swagger UI”](#try-the-endpoints-in-swagger-ui) Every instance serves an interactive reference of its API, for the version it runs, and its OpenAPI schema: | What | Path on the API of the instance | | ------------------------------------------------------------------- | ------------------------------- | | Interactive reference (Swagger UI), where you can try the endpoints | `/api/schema/swagger-ui/` | | OpenAPI schema, to generate a client or import it into a tool | `/api/schema/` | For example, on the national instance run by the French Institute of Bioinformatics: [api.madbot.france-bioinformatique.fr/api/schema/swagger-ui/](https://api.madbot.france-bioinformatique.fr/api/schema/swagger-ui/). On a local instance: . To call an endpoint from Swagger UI: 1. Create a personal access token (see above), and copy it with **Copy token (with Bearer)**. 2. Select **Authorize**. In the `ApiKeyAuth` box, paste the token in **Value**, as `Bearer pat_…`, then select **Authorize** and **Close**. 3. Open an endpoint, select **Try it out**, fill in its parameters, `X-Workspace` among them when it needs one, then select **Execute**. Caution Swagger UI calls the instance for real: a `POST`, `PUT`, `PATCH` or `DELETE` changes its data. Try them on a local instance, or in a workspace made for the purpose. ## Read a list, page by page [Section titled “Read a list, page by page”](#read-a-list-page-by-page) An endpoint that lists resources returns them page by page: | Query parameter | Meaning | | --------------- | ----------------------------------------------------------- | | `p` | The page, from 1. | | `page_size` | The number of results per page: 10 by default, 500 at most. | The response gives the results of the page and where to go next (here with the results shortened): ```json { "count": 2, "next": "http://localhost:8000/api/workspaces?p=2&page_size=1", "previous": null, "results": [{ "id": "143fbb1c-a639-4c33-8c41-2c1c02d7b8c2", "name": "Coastal erosion survey" }], "current_page": 1, "total_pages": 2 } ``` To read the whole list, follow `next` until it is `null`. Many lists also take `search` and `ordering`: the reference gives the parameters of each one. ## Read an error [Section titled “Read an error”](#read-an-error) An error comes with an HTTP status and a body of the same shape: ```json { "code": "header_validation_error", "message": "Invalid header. Check the `details` property for more details.", "details": [{ "parameter": "x-workspace", "code": "required", "message": "This field is required." }] } ``` * `code` names the kind of error, for a script to test. * `message` describes it for a person. * `details` lists the causes: the parameter at fault when there is one, a code and a message. | Status | `code` | When | | ------ | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | 400 | `body_validation_error`, `query_validation_error`, `header_validation_error` | The body, a query parameter or a header is missing or invalid. | | 401 | `unauthorized` | The token is missing, invalid, expired or revoked. | | 403 | `forbidden` | The token lacks the scope, the account is not approved, or it is not a member of the resource. | | 404 | `object_not_found` | The resource does not exist. | | 500 | `internal_server_error` | An unexpected error. `details` gives an identifier: send it to the administrators of the instance. | ## A first script [Section titled “A first script”](#a-first-script) This script creates a workspace and two nodes in it, reads them back one page at a time, and shows an error. It runs on a local instance (see [Set up a development environment](/contribute/development-setup/)), with `curl` and [`jq`](https://jqlang.org/). Create a token with the `read` and `write` scopes in your local web client, then set the address of the API and the token: ```bash export MADBOT_API=http://localhost:8000 export MADBOT_TOKEN=pat_… ``` Check that the API answers. This endpoint needs no token: ```bash curl -s "$MADBOT_API/api/version" ``` ```json {"version":"2.1.0","plugins_api_version":"2.0.1"} ``` Create a workspace, and keep its identifier: ```bash WORKSPACE=$(curl -s -X POST "$MADBOT_API/api/workspaces" \ -H "Authorization: Bearer $MADBOT_TOKEN" \ -H "Content-Type: application/json" \ -d '{"name": "Coastal erosion survey"}' | jq -r .id) echo "$WORKSPACE" ``` Create two nodes in it: ```bash for title in "Spring campaign" "Autumn campaign"; do curl -s -X POST "$MADBOT_API/api/nodes" \ -H "Authorization: Bearer $MADBOT_TOKEN" \ -H "X-Workspace: $WORKSPACE" \ -H "Content-Type: application/json" \ -d "{\"title\": \"$title\"}" | jq -r '.id + " " + .title' done ``` Read them back, one per page, from the first page until `next` is `null`: ```bash page="$MADBOT_API/api/nodes?page_size=1" while [ "$page" != "null" ]; do response=$(curl -s "$page" \ -H "Authorization: Bearer $MADBOT_TOKEN" \ -H "X-Workspace: $WORKSPACE") echo "$response" | jq -r '"page \(.current_page)/\(.total_pages): \(.results[0].title)"' page=$(echo "$response" | jq -r .next) done ``` ```text page 1/2: Spring campaign page 2/2: Autumn campaign ``` Without the `X-Workspace` header, the API answers with an error: ```bash curl -s "$MADBOT_API/api/nodes" -H "Authorization: Bearer $MADBOT_TOKEN" | jq ``` ```json { "code": "header_validation_error", "message": "Invalid header. Check the `details` property for more details.", "details": [ { "parameter": "x-workspace", "code": "required", "message": "This field is required." } ] } ``` The workspace and its nodes are in your local web client. Delete the workspace from there, or with `DELETE /api/workspaces/{id}`. # Contribute to madbot > Ask a question, report a bug, suggest an improvement or propose a change to madbot. Thank you for taking the time to contribute. Every contribution is welcome: a question, a bug report, an idea, a fix in the documentation or in the code. Everyone taking part in the project follows the [code of conduct](/contribute/code-of-conduct/). ## Ask a question [Section titled “Ask a question”](#ask-a-question) 1. Search the [existing issues](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/issues): someone may have asked the same question. If an issue answers it only partly, ask in that issue. 2. Otherwise, [open an issue](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/issues/new) and give as much context as you can: what you were trying to do, what happened, and the versions involved (madbot, browser, operating system). ## Report a bug [Section titled “Report a bug”](#report-a-bug) Security issues Never report a security problem, a vulnerability or a bug that exposes sensitive information in a public issue. Send it by email to . A good bug report lets someone else reproduce the problem without asking you more questions. Before you write it: * check that you use the latest version of madbot; * check that the problem does not come from your own setup, such as an incompatible version of a dependency; * search the [open bugs](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/issues/?state=opened\&label_name%5B%5D=bug) to see if it is already known; * collect what describes the problem: the error message or traceback, your operating system and platform, the versions of the tools involved, the input and the output. Then [open an issue](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/issues/new) with the **Bug** template: * describe what you expected and what happened instead; * list the steps that reproduce the problem, as short as possible; * add the information you collected. ## Suggest an improvement [Section titled “Suggest an improvement”](#suggest-an-improvement) Before you suggest a new feature or a change to an existing one: * check that you use the latest version of madbot, and that this documentation does not already cover it; * search the [issues of the madbot group](https://gitlab.com/groups/ifb-elixirfr/madbot/-/issues): if someone already suggested it, add your use case in a comment; * think about who would use it. madbot aims at features that serve most of its users; a connection to one specific system may fit better in a [plugin](/extend/plugins/development-guide/). Then [open an issue](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/issues/new) with the **Suggestion** template: * give it a clear and descriptive title; * describe the improvement step by step; * describe the current behaviour, the behaviour you expect instead, and why; * add screenshots if they help; * explain who would benefit from it. ## Propose a change [Section titled “Propose a change”](#propose-a-change) The code and this documentation live in the [madbot repository](https://gitlab.com/ifb-elixirfr/madbot/madbot). 1. [Set up a development environment](/contribute/development-setup/). 2. Create a branch and make your change, following the [branch and commit conventions](/contribute/branches-commits-releases/). 3. Run the [tests and quality checks](/contribute/tests-and-quality/), and add or update the tests of your change. 4. Update the documentation when your change is visible to users, people who run an instance, plugin developers or API users: see [Write documentation](/contribute/write-documentation/). 5. Open a merge request with a description that explains the context and the reason for the change. ### Use of AI [Section titled “Use of AI”](#use-of-ai) You may use an AI assistant to contribute, as the team does. The [AI policy](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/blob/main/AI_POLICY.md) of the repository gives the rules; in short: * you have read, understood and tested everything you submit, and you can explain it during the review; * your merge request goes through the same tests, quality checks and review as any other, and a member of the team reviews and approves it: an AI review can help the reviewer, it never replaces them; * your commits carry your own name and email: an AI assistant is never the author or the co-author of a commit; * a commit made with the help of an AI assistant ends with an `Assisted-by: AI` line (see [Commit messages](/contribute/branches-commits-releases/#commit-messages)), and so does the description of a merge request that contains such commits; * you never share secrets, personal data or the content of an instance with an AI tool, and you only use the tools your institution allows. The [AGENTS.md](https://gitlab.com/ifb-elixirfr/madbot/madbot/-/blob/main/AGENTS.md) file at the root of the repository gives the same rules, the commit conventions and the checks to run to the AI assistant you work with. ### Legal notice [Section titled “Legal notice”](#legal-notice) By contributing, you confirm that you are the author of the content you submit and answer for all of it, including the parts written with the help of an AI assistant, that you have the rights to it, and that it can be distributed under the project licence. # Architecture > How the madbot API, web client, workers and plugins fit together, and where to make a change. madbot is a web client that talks to a REST API. The API keeps its data in a database, delegates the long operations to Celery workers, pushes live events over WebSockets, and reaches the outside world through plugins. ```text Browser │ ┌────────▼─────────┐ │ Web client │ Nuxt, Vue, Vuetify, Pinia └────────┬─────────┘ │ REST (/api/), WebSocket (/ws/) ┌────────▼─────────┐ ┌───────────────┐ │ API │◀──────▶│ Redis │◀──┐ │ Django, DRF, │ └───────────────┘ │ tasks │ Channels │ │ └──┬─────┬─────┬───┘ ┌───────────────┐ │ │ │ │ │ Celery worker │───┘ │ │ │ │ and beat │ │ │ │ └───────┬───────┘ │ │ └──── plugins ───────┤ │ │ (sources and repositories) │ └── Vault (connection credentials) └──── Database (SQLite in development, PostgreSQL in production) ``` ## The repository [Section titled “The repository”](#the-repository) The repository holds three packages, released separately, and this documentation: | Folder | What it holds | | ---------------- | ------------------------------------------------------------------------------------------------------ | | `api/` | The API, the Django project `madbot_api`. | | `client/` | The web client, a Nuxt application. | | `plugin_api/` | The `plugins_api` package: the contract that plugins implement, installed by the API as `plugins-api`. | | `documentation/` | This site. | ## The API [Section titled “The API”](#the-api) The API is a Django application built on Django REST Framework. Its domain logic lives in the `madbot_api.core` app; analytics have their own app, `madbot_api.analytics`. ### From the request to the database [Section titled “From the request to the database”](#from-the-request-to-the-database) | Layer | Where | Role | | ------------ | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Routing | `madbot_api/urls.py`, `core/url_api.py` | Mounts the REST API under `api/`, OAuth under `o/`, the health checks; nested routers follow the resources (workspaces → members, nodes → samples, submissions → metadata objects…). | | Views | `core/views/` | ViewSets and API views. Most derive from `MadbotViewSet` (`core/views/utils/drf.py`), which brings authentication, approval of the user, scopes, pagination and error format. | | Access rules | `core/views/utils/mixin.py` | Resolve the workspace of the request and apply the workspace, node and submission memberships. | | Models | `core/models.py` | The entities: see the [data models](/contribute/reference/data-models/). | | Tasks | `core/tasks.py` | The Celery tasks. | The client names the workspace of a request in the `X-Workspace` header. ### Authentication [Section titled “Authentication”](#authentication) | Mechanism | Used by | | ---------------------------------------------------------- | --------------------------------------------------------------------------- | | OAuth 2 with OpenID Connect (`django-oauth-toolkit`, PKCE) | The web client. madbot signs its tokens with `MADBOT_OIDC_RSA_PRIVATE_KEY`. | | Personal access tokens | Scripts and tools that call the API. | | Sessions | The Django administration. | The OAuth scopes are `read`, `write`, `openid`, `email` and `profile`. An account must be approved before it can use the API. ### Long operations and live events [Section titled “Long operations and live events”](#long-operations-and-live-events) The API hands the long operations to **Celery** workers, through Redis: creating the metadata objects of a submission, submitting, checking the connections and the access to data, copying a node and its subtree. **Celery Beat** runs the periodic tasks (see [`create_periodic_tasks`](/run/reference/commands/#create_periodic_tasks)). While they run, the API pushes events to the client over WebSockets (Django Channels, served by Daphne): | Path | Events | | --------------------------------- | ----------------------------------------- | | `ws/notifications/` | Notifications of the user. | | `ws/workspaces//` | Events of a workspace: connections, data. | | `ws/submissions//` | Progress of a submission. | ### Plugins [Section titled “Plugins”](#plugins) The API loads the plugins listed in `MADBOT_INSTALLED_PLUGINS` and adds their Django apps to `INSTALLED_APPS`. `plugins_api` defines the base classes they implement: data plugins, submission plugins and metadata field definition plugins. The connection and submission views call the plugin of the connection to browse a source, check credentials or send a submission. The credentials of the connections are kept in Vault, not in the database. See the [plugin development guide](/extend/plugins/development-guide/). ## The web client [Section titled “The web client”](#the-web-client) The client is a Nuxt application, rendered in the browser only (`ssr: false`), written with Vue’s Composition API, Vuetify components and Pinia stores. | Folder | Role | | -------------- | ---------------------------------------------------------------------------------------------------------------------- | | `pages/` | Routes, by file. The workspace pages are under `pages/[w]/[wslug]/`. | | `components/` | Components, by domain: `Workspace/`, `Node/`, `Connection/`, `DataObject/`, `Sample/`, `Submission/`, `Metadata/`… | | `stores/` | One Pinia store per domain: workspaces, nodes, connections, samples, metadata, submissions, notifications, WebSockets. | | `composables/` | Shared logic, including the calls to the API and the WebSockets. | | `server/` | Server routes, for the authentication. | | `locales/` | The interface texts, in English and French. | * **Calls to the API** go through `useAPIFetch`, which adds the access token of the session and sends the user to a waiting page when the API does not answer. * **Authentication** uses `@sidebase/nuxt-auth`: the client signs in with the OpenID Connect provider of the API. * **Metadata forms** are generated from the JSON schemas of the fields with [JSON Forms](https://jsonforms.io/), and madbot’s own renderers in `components/JsonFormRenderers/`. * **WebSockets** are opened by `MadbotWebSocket`, managed per workspace by `useWebsocketStore`. ## Where to make a change [Section titled “Where to make a change”](#where-to-make-a-change) | Change | Where | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A model | `api/madbot_api/core/models.py`, then `pixi run madbot makemigrations core`. Use a UUID primary key, the audit fields and, where history matters, `HistoricalRecords`. | | An endpoint | A view in `core/views/`, its route in `core/url_api.py`, following the existing filtering, pagination and permissions. | | A long operation | A task in `core/tasks.py`. | | A screen | A page in `client/pages/`, components in `client/components/`, the state in a store. | | An interface text | `client/locales/en.json` and `fr.json`. | | A connection to another system | A [plugin](/extend/plugins/development-guide/), in its own repository. | Write the tests with the change: see [Tests and quality checks](/contribute/tests-and-quality/). # Branches, commits and releases > Which branch to start from, how to write a commit message, and how madbot is released. ## Branches [Section titled “Branches”](#branches) The repository has two long-lived branches: * **`dev`**, the default branch: the next release is prepared there. New features and most fixes go to `dev`. * **`main`**: what is released. The packages and the Docker images are published from `main`, and so is this documentation. | Your change | Start from | Merge request to | | --------------------------------------------------- | ------------------------------ | --------------------------------------------- | | A feature, a fix that can wait for the next release | `dev` | `dev` | | A fix that cannot wait (hotfix) | `main` | `main`, then `main` is merged back into `dev` | | Documentation of what is already released | `main` | `main`, then `main` is merged back into `dev` | | Documentation of a feature that is not released yet | `dev`, in the feature’s branch | `dev`, with the feature | [Write documentation](/contribute/write-documentation/) explains why a documentation change can start from `main`. Name your branch after the merge request, in lowercase words separated by hyphens: `workspace-report-pdf`, or `fix-…` for a bug fix: `fix-datalink-created-by`. Before you open the merge request, update your branch with its target branch, and write a description that gives the context and the reason for the change. ## Commit messages [Section titled “Commit messages”](#commit-messages) Commit messages follow [gitmoji](https://gitmoji.dev/) with a scope, then a short subject in the imperative: ```text ()