Performance and debugging
Ce contenu n’est pas encore disponible dans votre langue.
These tools help when you work on the speed of madbot, or on what it sends by email. They need the development setup.
Profile the API with Silk
Section titled “Profile the API with Silk”Django Silk records each request with its time and its database queries. Enable it in api/.env:
SILKY_PYTHON_PROFILER=TrueThen start the API in the dev environment, which holds Silk:
pixi run -e dev madbot runserverOpen http://localhost:8000/silk/ to read the requests. Silk is always off while the tests run: it adds queries that the tests would count.
Measure on a large workspace
Section titled “Measure on a large workspace”create_load_workspace generates a large workspace — deep node trees, data, samples, metadata and members with heterogeneous rights — to measure the API and the client on a realistic volume, identical from one machine and one measure to the next.
cd apipixi run madbot create_load_workspace --username <username> --profile medium --seed 42
# Print the planned volumetry, write nothingpixi run madbot create_load_workspace --username <username> --profile large --dry-run| Option | Role |
|---|---|
--username | Owner of the workspace (required, must exist) |
--profile {small,medium,large} | Predefined volumetry (default: small) |
--root-nodes, --depth, --node-per-node | Shape of the tree: every non-leaf node has --node-per-node children |
--data-per-node, --samples-per-node | Density of every node of the tree |
--metadata-per-object | Size of each root tree’s field set: every node and sample of the tree gets all of its fields |
--metadata-per-data | Metadata of each data: the first fields of its tree’s set |
--wide-nodes, --wide-node-size | Extra nodes, under the roots, each with that many samples and data — to load the batch edit tables |
--plugin-fields [PLUGIN ...] | Plugins whose metadata fields are used besides Madbot’s (default: ena); pass it without a value for Madbot fields only |
--members | Members invited besides the owner |
--seed | Seed of the generation; drawn and printed when omitted |
--name | Workspace name (default: Load <profile> seed <seed>) |
--plugin | Data plugin of the connection holding the data (default: the first installed one) |
--force | Delete the owner’s workspace of the same name, then recreate it |
--dry-run | Print the planned volumetry without writing anything |
Every volumetry option overrides the profile, one by one: to check a problem that only shows at 5,000 nodes, push --root-nodes or --depth instead of adding a profile.
| Profile | Nodes (wide) | Depth | Datalinks per node | Data | Samples | Metadata per node and sample | Metadata per data | Metadata | Members |
|---|---|---|---|---|---|---|---|---|---|
small | 21 (none) | 3 | 2 | 42 | 21 | 5 | 5 | 420 | 3 |
medium | 106 (1 × 100) | 4 | 5 | 625 | 310 | 30 | 30 | 31,230 | 5 |
large | 506 (2 × 500) | 6 | 50 | 26,200 | 2,008 | 100 | 20 | 775,400 | 10 |
Each node has its data (one datalink each) and its samples, each sample bound to one data of its node. Wide nodes (titled Wide node 1: …) hang under the root nodes and carry --wide-node-size samples and data each: open one and switch its samples or data to the batch edit view to measure a large table. small backs the automated test of the command and stays fast rather than realistic.
Metadata. Each root tree draws its own set of --metadata-per-object fields, and every node and sample of the tree carries all of them — as a project following one checklist would. Data carry only the first --metadata-per-data fields of the set (20 in large), which keeps its 26,000 data affordable: at 100 each, they alone would make 2.6 million metadata. A batch edit table therefore shows as many columns as the set has fields, even with the samples inherited from ancestors; 100 is about the size of the largest ENA checklist. The relation flags of the fields (node_related…) are informational and not followed.
Values are generated from each field’s JSON schema. At start-up the command draws 40 values per field and checks them with the same validation as Metadata.save(), offline, keeping the fields where at least half pass. The values of the fields where some fail, and of every field whose values are objects — the only ones seen failing, such as a few ENA fields refusing some of their “missing” values — are checked again when written, and redrawn until valid. Fields cross-checked against external services (taxon, protocol, topic, DOI…), patterns and a few formats are left out. With ENA loaded, 812 fields are usable (40 Madbot, 772 ENA); this probe takes about 15 seconds and the command prints how many fields it kept. Without the ENA fields, the metadata per object are capped at the Madbot fields available.
Cost of large. large writes 775,400 metadata; the command prints how long the writing took. The metadata are generated and written chunk by chunk, so memory does not grow with --metadata-per-object.
Rights. The owner owns every node. The first member co-owns the workspace, which reads every node without any node membership. The other members are plain workspace members, invited to some root trees as a whole, to a sub-tree only or to nothing, with mixed node roles. As through the API, a member of a node is a member of its parent: a member invited to a sub-tree collaborates on its ancestors, and the rest of the tree stays unreadable to them. Member users are named load-member-01, load-member-02… with an unusable password, and are reused from one run to the next. Each gets a made-up first and last name that depends on its number only, so it stays the same from one run to the next; a reused member without a name is given one.
Reproducibility. The same seed and options give the same structure: titles, tree, data, metadata values and memberships. Identifiers and timestamps differ.
What it bypasses — keep it in mind when reading a measure.
- Objects are written with
bulk_create, which skips modelsave()and signals: nosimple_historyrecord is created, and nothing would be indexed by a search engine. The history tables therefore do not grow as they would with real use. - Metadata are written directly, without the Celery task: the command needs neither a Celery worker nor a search engine.
- The workspace is the same from one machine to the next only if they have the same metadata fields: the same Madbot version, and the same ENA plugin version loaded (
pixi run madbot reload_plugins). The command warns when a plugin of--plugin-fieldshas no field. - The Madbot metadata fields must exist: run
pixi run madbot initialise_madbot_metadata_fieldsfirst, the command says so otherwise. - Data have made-up external identifiers and an access right for the owner only. Anything that asks the plugin about them (refresh, access verification) will fail — the connection exists to hold them, not to reach a real service.
--forcedeletes with history disabled, so recreating a large workspace stays quick.
Debug emails
Section titled “Debug emails”In development, madbot writes the emails it sends to files instead of sending them: they land in api/sent_emails/.
To read them in a web interface instead, use Mailhog, which needs Docker:
-
In
api/.env, withDEBUGon:Terminal window USE_MAILHOG=true -
Start the Mailhog container, from
api/:Terminal window sh ./scripts/start_mailhog.sh -
Restart the API, then open http://localhost:8025.

