Skip to content

Performance and debugging

These tools help when you work on the speed of madbot, or on what it sends by email. They need the development setup.

Django Silk records each request with its time and its database queries. Enable it in api/.env:

Terminal window
SILKY_PYTHON_PROFILER=True

Then start the API in the dev environment, which holds Silk:

Terminal window
pixi run -e dev madbot runserver

Open http://localhost:8000/silk/ to read the requests. Silk is always off while the tests run: it adds queries that the tests would count.

create_load_workspace generates a large workspace — deep node trees, data, samples, metadata and members with heterogeneous rights — to measure the API and the client on a realistic volume, identical from one machine and one measure to the next.

Terminal window
cd api
pixi run madbot create_load_workspace --username <username> --profile medium --seed 42
# Print the planned volumetry, write nothing
pixi run madbot create_load_workspace --username <username> --profile large --dry-run
OptionRole
--usernameOwner of the workspace (required, must exist)
--profile {small,medium,large}Predefined volumetry (default: small)
--root-nodes, --depth, --node-per-nodeShape of the tree: every non-leaf node has --node-per-node children
--data-per-node, --samples-per-nodeDensity of every node of the tree
--metadata-per-objectSize of each root tree’s field set: every node and sample of the tree gets all of its fields
--metadata-per-dataMetadata of each data: the first fields of its tree’s set
--wide-nodes, --wide-node-sizeExtra nodes, under the roots, each with that many samples and data — to load the batch edit tables
--plugin-fields [PLUGIN ...]Plugins whose metadata fields are used besides Madbot’s (default: ena); pass it without a value for Madbot fields only
--membersMembers invited besides the owner
--seedSeed of the generation; drawn and printed when omitted
--nameWorkspace name (default: Load <profile> seed <seed>)
--pluginData plugin of the connection holding the data (default: the first installed one)
--forceDelete the owner’s workspace of the same name, then recreate it
--dry-runPrint the planned volumetry without writing anything

Every volumetry option overrides the profile, one by one: to check a problem that only shows at 5,000 nodes, push --root-nodes or --depth instead of adding a profile.

ProfileNodes (wide)DepthDatalinks per nodeDataSamplesMetadata per node and sampleMetadata per dataMetadataMembers
small21 (none)324221554203
medium106 (1 × 100)45625310303031,2305
large506 (2 × 500)65026,2002,00810020775,40010

Each node has its data (one datalink each) and its samples, each sample bound to one data of its node. Wide nodes (titled Wide node 1: …) hang under the root nodes and carry --wide-node-size samples and data each: open one and switch its samples or data to the batch edit view to measure a large table. small backs the automated test of the command and stays fast rather than realistic.

Metadata. Each root tree draws its own set of --metadata-per-object fields, and every node and sample of the tree carries all of them — as a project following one checklist would. Data carry only the first --metadata-per-data fields of the set (20 in large), which keeps its 26,000 data affordable: at 100 each, they alone would make 2.6 million metadata. A batch edit table therefore shows as many columns as the set has fields, even with the samples inherited from ancestors; 100 is about the size of the largest ENA checklist. The relation flags of the fields (node_related…) are informational and not followed.

Values are generated from each field’s JSON schema. At start-up the command draws 40 values per field and checks them with the same validation as Metadata.save(), offline, keeping the fields where at least half pass. The values of the fields where some fail, and of every field whose values are objects — the only ones seen failing, such as a few ENA fields refusing some of their “missing” values — are checked again when written, and redrawn until valid. Fields cross-checked against external services (taxon, protocol, topic, DOI…), patterns and a few formats are left out. With ENA loaded, 812 fields are usable (40 Madbot, 772 ENA); this probe takes about 15 seconds and the command prints how many fields it kept. Without the ENA fields, the metadata per object are capped at the Madbot fields available.

Cost of large. large writes 775,400 metadata; the command prints how long the writing took. The metadata are generated and written chunk by chunk, so memory does not grow with --metadata-per-object.

Rights. The owner owns every node. The first member co-owns the workspace, which reads every node without any node membership. The other members are plain workspace members, invited to some root trees as a whole, to a sub-tree only or to nothing, with mixed node roles. As through the API, a member of a node is a member of its parent: a member invited to a sub-tree collaborates on its ancestors, and the rest of the tree stays unreadable to them. Member users are named load-member-01, load-member-02… with an unusable password, and are reused from one run to the next. Each gets a made-up first and last name that depends on its number only, so it stays the same from one run to the next; a reused member without a name is given one.

Reproducibility. The same seed and options give the same structure: titles, tree, data, metadata values and memberships. Identifiers and timestamps differ.

What it bypasses — keep it in mind when reading a measure.

  • Objects are written with bulk_create, which skips model save() and signals: no simple_history record is created, and nothing would be indexed by a search engine. The history tables therefore do not grow as they would with real use.
  • Metadata are written directly, without the Celery task: the command needs neither a Celery worker nor a search engine.
  • The workspace is the same from one machine to the next only if they have the same metadata fields: the same Madbot version, and the same ENA plugin version loaded (pixi run madbot reload_plugins). The command warns when a plugin of --plugin-fields has no field.
  • The Madbot metadata fields must exist: run pixi run madbot initialise_madbot_metadata_fields first, the command says so otherwise.
  • Data have made-up external identifiers and an access right for the owner only. Anything that asks the plugin about them (refresh, access verification) will fail — the connection exists to hold them, not to reach a real service.
  • --force deletes with history disabled, so recreating a large workspace stays quick.

In development, madbot writes the emails it sends to files instead of sending them: they land in api/sent_emails/.

To read them in a web interface instead, use Mailhog, which needs Docker:

  1. In api/.env, with DEBUG on:

    Terminal window
    USE_MAILHOG=true
  2. Start the Mailhog container, from api/:

    Terminal window
    sh ./scripts/start_mailhog.sh
  3. Restart the API, then open http://localhost:8025.