Skip to content

Architecture

madbot is a web client that talks to a REST API. The API keeps its data in a database, delegates the long operations to Celery workers, pushes live events over WebSockets, and reaches the outside world through plugins.

Browser
│
┌────────▼─────────┐
│ Web client │ Nuxt, Vue, Vuetify, Pinia
└────────┬─────────┘
│ REST (/api/), WebSocket (/ws/)
┌────────▼─────────┐ ┌───────────────┐
│ API │◀──────▶│ Redis │◀──┐
│ Django, DRF, │ └───────────────┘ │ tasks
│ Channels │ │
└──┬─────┬─────┬───┘ ┌───────────────┐ │
│ │ │ │ Celery worker │───┘
│ │ │ │ and beat │
│ │ │ └───────┬───────┘
│ │ └──── plugins ───────┤
│ │ (sources and repositories)
│ └── Vault (connection credentials)
└──── Database (SQLite in development, PostgreSQL in production)

The repository holds three packages, released separately, and this documentation:

FolderWhat it holds
api/The API, the Django project madbot_api.
client/The web client, a Nuxt application.
plugin_api/The plugins_api package: the contract that plugins implement, installed by the API as plugins-api.
documentation/This site.

The API is a Django application built on Django REST Framework. Its domain logic lives in the madbot_api.core app; analytics have their own app, madbot_api.analytics.

LayerWhereRole
Routingmadbot_api/urls.py, core/url_api.pyMounts the REST API under api/, OAuth under o/, the health checks; nested routers follow the resources (workspaces → members, nodes → samples, submissions → metadata objects…).
Viewscore/views/ViewSets and API views. Most derive from MadbotViewSet (core/views/utils/drf.py), which brings authentication, approval of the user, scopes, pagination and error format.
Access rulescore/views/utils/mixin.pyResolve the workspace of the request and apply the workspace, node and submission memberships.
Modelscore/models.pyThe entities: see the data models.
Taskscore/tasks.pyThe Celery tasks.

The client names the workspace of a request in the X-Workspace header.

MechanismUsed by
OAuth 2 with OpenID Connect (django-oauth-toolkit, PKCE)The web client. madbot signs its tokens with MADBOT_OIDC_RSA_PRIVATE_KEY.
Personal access tokensScripts and tools that call the API.
SessionsThe Django administration.

The OAuth scopes are read, write, openid, email and profile. An account must be approved before it can use the API.

The API hands the long operations to Celery workers, through Redis: creating the metadata objects of a submission, submitting, checking the connections and the access to data, copying a node and its subtree. Celery Beat runs the periodic tasks (see create_periodic_tasks).

While they run, the API pushes events to the client over WebSockets (Django Channels, served by Daphne):

PathEvents
ws/notifications/Notifications of the user.
ws/workspaces/<workspace id>/Events of a workspace: connections, data.
ws/submissions/<submission id>/Progress of a submission.

The API loads the plugins listed in MADBOT_INSTALLED_PLUGINS and adds their Django apps to INSTALLED_APPS. plugins_api defines the base classes they implement: data plugins, submission plugins and metadata field definition plugins. The connection and submission views call the plugin of the connection to browse a source, check credentials or send a submission. The credentials of the connections are kept in Vault, not in the database.

See the plugin development guide.

The client is a Nuxt application, rendered in the browser only (ssr: false), written with Vue’s Composition API, Vuetify components and Pinia stores.

FolderRole
pages/Routes, by file. The workspace pages are under pages/[w]/[wslug]/.
components/Components, by domain: Workspace/, Node/, Connection/, DataObject/, Sample/, Submission/, Metadata/…
stores/One Pinia store per domain: workspaces, nodes, connections, samples, metadata, submissions, notifications, WebSockets.
composables/Shared logic, including the calls to the API and the WebSockets.
server/Server routes, for the authentication.
locales/The interface texts, in English and French.
  • Calls to the API go through useAPIFetch, which adds the access token of the session and sends the user to a waiting page when the API does not answer.
  • Authentication uses @sidebase/nuxt-auth: the client signs in with the OpenID Connect provider of the API.
  • Metadata forms are generated from the JSON schemas of the fields with JSON Forms, and madbot’s own renderers in components/JsonFormRenderers/.
  • WebSockets are opened by MadbotWebSocket, managed per workspace by useWebsocketStore.
ChangeWhere
A modelapi/madbot_api/core/models.py, then pixi run madbot makemigrations core. Use a UUID primary key, the audit fields and, where history matters, HistoricalRecords.
An endpointA view in core/views/, its route in core/url_api.py, following the existing filtering, pagination and permissions.
A long operationA task in core/tasks.py.
A screenA page in client/pages/, components in client/components/, the state in a store.
An interface textclient/locales/en.json and fr.json.
A connection to another systemA plugin, in its own repository.

Write the tests with the change: see Tests and quality checks.