Aller au contenu

Plugin development guide

Ce contenu n’est pas encore disponible dans votre langue.

This comprehensive guide will walk you through creating a madbot plugin from scratch. We’ll build a simple file system data plugin as an example to demonstrate the complete development process.

Madbot’s plugin system provides a flexible architecture for integrating with external systems. Plugins are Python classes that inherit from specific base classes and implement required interfaces. The system supports three main types of plugins:

  • Data Plugins: Connect to external data sources and provide access to files and directories
  • Submission Plugins: Handle data submission to external repositories and services
  • Metadata Field Definition Plugins: Define metadata schemas to extend the madbot metadata referential

In this tutorial, we’ll create a Data Plugin that connects to a remote file system and allows users to browse and download files from remote servers.

Every class, method and exception of plugins_api is described in the plugins_api reference.

If you write your plugin with a coding assistant, give it the documentation as plain Markdown: llms-full.txt holds every page of this site and the plugins_api reference of the current version, and llms.txt lists what is available.

Before starting, ensure you have:

  • Python 3.11
  • Basic knowledge of Python classes and inheritance
  • Access to a madbot development environment

Let’s create a new plugin project called madbot_filesystem. This will be a data plugin that allows browsing remote directories.

Create the following directory structure:

madbot_filesystem_plugin/
├── pyproject.toml # Project configuration and dependencies
├── README.md # Plugin documentation
├── LICENSE # License terms for the plugin
├── CONTRIBUTING.md # Contribution guidelines (recommended)
├── code_of_conduct.md # Community behavior standards (recommended)
├── .gitlab-ci.yml # GitLab CI/CD pipeline configuration (recommended)
└── madbot_filesystem/
├── __init__.py
├── plugins.py # Your plugin class
├── app.py # The app that registers the plugin
└── static/
├── json/
│ └── schema/ # JSON schema files
│ └── uischema/ # Optional JSON Forms UI schema files (if they exist)
└── images/
└── filesystem.png # Your plugin logo

pyproject.toml:

[project]
name = "madbot-filesystem"
version = "1.0.0"
description = "A remote filesystem data plugin for madbot"
authors = [
{name = "Your Name", email = "your.email@example.com"}
]
readme = "README.md"
license = "BSD-3-Clause"
license-files = ["LICEN[CS]E*"]
requires-python = ">=3.11,<3.12"
keywords = ["madbot"]
dependencies = [
"plugins-api >=3.0.0,<4",
]
[project.optional-dependencies]
dev = [
'ruff',
'bandit[toml]',
'safety >= 2.4',
]
[build-system]
requires = ["setuptools>=45", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
include = ["madbot-filesystem*"]
exclude = ["logs*", "tests*"]

For more information about the pyproject.toml file, see this guide.

This file is enough to write the plugin. The repository around it also needs the registry that publishes plugins-api, floors on the build tools, and its own CI with the quality and security jobs: Set up a plugin repository lists them.

README.md:

# Madbot Filesystem Plugin
A madbot data plugin that provides access to remote file systems.
## Features
- Browse remote directories
- List files and subdirectories on remote servers
- Download files with integrity verification
## Installation
This plugin is designed to be installed as a madbot plugin. See the madbot documentation for installation instructions.

Step 2: Understanding Data Plugin Requirements

Section titled “Step 2: Understanding Data Plugin Requirements”

Before implementing our plugin, let’s understand what we need to build. A Data Plugin inherits from DataPlugin which inherits from ConnectionPlugin which inherits from MadbotPlugin. This means we need to implement methods from all three base classes:

  1. get_name(): Return the plugin name
  2. get_slug(): Return the plugin slug
  3. get_description(): Return plugin description
  4. get_author(): Return plugin author
  5. get_logo(): Return the path of the plugin logo
  6. get_color(): Return the color of the plugin
  7. get_status(): Return maturity status keyed by plugin type (e.g. {"data_plugin": PluginStatus.STABLE})

2.2 From ConnectionPlugin (Connection Management)

Section titled “2.2 From ConnectionPlugin (Connection Management)”
  1. get_instance_name(): Return the instance name of the connection plugin
  2. get_connection_fields(): Return connection fields (shared and private)
  3. get_shared_parameters(): Return all shared parameters in a cleaned form
  1. __init__(connection_parameters): Initialize the plugin with connection parameters
  2. get_root_name(): Return the name of the root data object
  3. get_root_id(): Return the unique identifier of the root data object
  4. browse_data_objects(parent_id): Return the direct children of a parent (navigation / listing). When parent_id is None, browse from the root of the source. Distinct from reading objects by id.
  5. get_data_objects_by_ids(object_ids): Return detailed information for a list of data object ids. Return a dict keyed by id: either a PluginDataObject or a PluginError (DataNotFoundError, PluginPermissionError, …). Raise source-level errors (SOURCE_ERRORS: UnreachableError, HostConnectionRefusedError, AuthenticationError) instead of storing them. Empty list returns {}. Duplicate ids appear once. Use self._read_each(object_ids, read_one) when the source has no batch API.
  6. get_batch_read_size() (optional): Max ids madbot sends per batch-read call. Default DEFAULT_BATCH_READ_SIZE (100). Return None to disable chunking.
  7. get_metadata_for_data_objects(object_ids) (optional): Return metadata for each id (dict of lists or per-id errors). Defaults to [] per id.
  8. get_data_associations_for_data_objects(object_ids) (optional): Return associations for each id. Defaults to [] per id.
  9. download_data(external_dataobject_id, destination_path): Download a data object to local storage

See the DataPlugin reference for full signatures.

Let’s create our filesystem plugin step by step.

Create madbot_filesystem/plugins.py:

import os
from pathlib import Path
from typing import Any
import requests
from plugins_api import (
AuthenticationError,
ConnectionField,
DataInvalidError,
DataNotFoundError,
DataPlugin,
DownloadDataError,
PluginDataObject,
PluginError,
PluginStatus,
UnreachableError,
)
class FilesystemDataPlugin(DataPlugin):
"""
A data plugin that provides access to remote file system directories.
This example demonstrates the plugin architecture without being tied to a
specific transport library.
"""
def __init__(self, connection_parameters: dict[str, Any]):
"""
Initialize the filesystem plugin.
Args:
connection_parameters: Dictionary containing connection parameters
"""
self.host = connection_parameters.get("host", "")
self.port = int(connection_parameters.get("port", 22))
self.username = connection_parameters.get("username", "")
self.password = connection_parameters.get("password", "")
# Example connection check, adapt to your protocol / client:
try:
response = requests.get(
f"https://{self.host}:{self.port}",
auth=(self.username, self.password),
timeout=20,
)
response.raise_for_status()
except requests.HTTPError:
raise AuthenticationError()
except requests.RequestException:
raise UnreachableError(self.host)
# ===== MadbotPlugin Methods =====
@classmethod
def get_name(cls) -> str:
"""Return the plugin name."""
return "Remote Filesystem Plugin"
@classmethod
def get_slug(cls) -> str:
"""Return the plugin slug."""
return "remote_filesystem_plugin"
@classmethod
def get_description(cls) -> str:
"""Return plugin description."""
return "A data plugin that provides access to remote file systems for browsing and downloading files."
@classmethod
def get_author(cls) -> str:
"""Return plugin author."""
return "John Doe"
@classmethod
def get_logo(cls) -> str:
"""Return the path of the plugin logo."""
return "images/filesystem.png"
@classmethod
def get_color(cls) -> str:
"""Return the color of the plugin."""
return "#4CAF50"
@classmethod
def get_status(cls) -> dict[str, PluginStatus]:
return {"data_plugin": PluginStatus.STABLE}
# ===== ConnectionPlugin Methods =====
def get_instance_name(self) -> str:
"""Return the instance name of the connection plugin."""
return f"Filesystem-{self.host}-{self.username}"
@classmethod
def get_connection_fields(
cls,
) -> tuple[list[ConnectionField], list[ConnectionField]]:
"""
Retrieve the connection fields for the connection plugin.
Returns:
Tuple containing shared and private connection fields
"""
shared_fields = [
ConnectionField(
slug="host",
schema_content={
"type": "string",
"title": "Host",
"description": "Remote server hostname or IP address",
},
),
ConnectionField(
slug="port",
schema_content={
"type": "integer",
"title": "Port",
"description": "Remote server port",
"default": 22,
},
),
]
private_fields = [
ConnectionField(
slug="username",
schema_content={
"type": "string",
"title": "Username",
"description": "Remote server username",
},
),
ConnectionField(
slug="password",
schema_content={
"type": "string",
"title": "Password",
"description": "Remote server password",
},
),
]
return shared_fields, private_fields
def get_shared_parameters(self) -> dict[str, Any]:
"""Return all shared parameters."""
return {"host": self.host, "port": self.port}
# ===== DataPlugin Methods =====
@staticmethod
def get_root_name() -> str:
"""Return the name of the root data object."""
return "Remote Filesystem Root"
@staticmethod
def get_root_id() -> str:
"""Return the unique identifier of the root data object."""
return "/"
def browse_data_objects(self, parent_id: str | None) -> list[PluginDataObject]:
"""Return the direct children of a parent; ``None`` starts at the root."""
parent_id = self.get_root_id() if parent_id is None else parent_id
path = parent_id if parent_id != self.get_root_id() else ""
items = get_directory_contents_from_remote_system(path)
if items is None:
return []
objects = [
PluginDataObject(
id=item_data["path"],
name=item_data["name"],
type="directory" if item_data["is_directory"] else "file",
size=item_data["size"],
has_children=item_data["is_directory"],
file_extension=(
Path(item_data["path"]).suffix.lstrip(".")
if not item_data["is_directory"]
else None
),
)
for item_data in items
]
return sorted(objects, key=lambda obj: obj.name)
def _read_data_object(self, object_id: str) -> PluginDataObject:
"""Read one object. Used via `_read_each` when the source has no batch API."""
if object_id == "/":
return PluginDataObject(
id="/",
name="Remote Filesystem Root",
type="directory",
size=0,
has_children=True,
)
data = get_file_data_from_remote_system(object_id)
if not data:
raise DataNotFoundError(object_id)
return PluginDataObject(
id=object_id,
name=Path(object_id).name,
type="directory" if data["is_directory"] else "file",
size=data["size"],
has_children=data["is_directory"],
file_extension=(
Path(object_id).suffix.lstrip(".")
if not data["is_directory"]
else None
),
)
def get_data_objects_by_ids(
self, object_ids: list[str]
) -> dict[str, PluginDataObject | PluginError]:
"""
Return detailed information for each requested id.
Prefer one remote call when the source allows it; otherwise wrap a
one-id reader with `_read_each` as below.
"""
return self._read_each(object_ids, self._read_data_object)
def download_data(self, external_dataobject_id: str, destination_path: str) -> str:
"""Download a data object to local storage and return the download report."""
data_object = self._read_data_object(external_dataobject_id)
if data_object.type == "directory":
raise DataInvalidError(external_dataobject_id)
content = download_file_from_remote_system(external_dataobject_id)
with open(destination_path, "wb") as handle:
handle.write(content)
local_size = os.path.getsize(destination_path)
if data_object.size is not None and local_size != data_object.size:
raise DownloadDataError(
f"Size mismatch: expected {data_object.size}, got {local_size}"
)
return destination_path

get_metadata_for_data_objects and get_data_associations_for_data_objects are optional. Override them when your source exposes metadata or associations. Otherwise, the base class returns an empty list for each id.

When the remote API can fetch several ids in one call, implement the batch method directly. It must show the three rules: one entry per distinct id, per-id errors stored in the result, source errors raised:

def get_data_objects_by_ids(self, object_ids):
if not object_ids:
return {}
# One remote call; connection or auth failures raise
# UnreachableError / AuthenticationError.
found = self.client.get_many(object_ids)
return {
object_id: (
self._to_data_object(found[object_id])
if object_id in found
else DataNotFoundError(object_id)
)
for object_id in dict.fromkeys(object_ids)
}

Create madbot_filesystem/app.py:

from django.apps import AppConfig
class MadbotFilesystemPluginConfig(AppConfig):
"""Django app configuration for the madbot filesystem plugin."""
default_auto_field = 'django.db.models.BigAutoField'
name = 'madbot_filesystem'
verbose_name = 'Madbot Filesystem Data Plugin'
def ready(self):
"""Called when Django starts up."""
# Import here to avoid circular imports
from . import plugins # noqa

Our plugin implementation includes several important aspects:

  1. Plugin identity: required MadbotPlugin methods, including get_status()
  2. Connection management: shared/private ConnectionField DTOs (slug + JSON Schema)
  3. Navigation vs read: browse_data_objects lists children; get_data_objects_by_ids reads by id (batch)
  4. Batch reads: use _read_each when there is no batch API, or one remote call when there is; result keys must match the requested ids exactly. Override get_batch_read_size() when the source has a different id-list limit (or return None for no chunking).
  5. Error handling: store per-id PluginError; raise SOURCE_ERRORS (UnreachableError, HostConnectionRefusedError, AuthenticationError)

Madbot relies on JSON Schema to describe plugin configuration, connection parameters, and metadata fields. This section explains what schemas are expected, where they live, and how they should be structured.

JSON Schemas in madbot serve three distinct purposes:

  1. Plugin settings schemas (optional and depending on the plugin type)
  • Global configuration of a plugin instance
  1. Connection field schemas
  • Describe connection parameters (host, credentials, tokens, etc.)
  1. Metadata field schemas
  • Describe metadata fields exposed by metadata plugins

All schemas follow JSON Schema Draft 7+ conventions.

Plugin settings schemas define instance-level configuration for a plugin.

Terminal window
madbot_filesystem/static/json/schema/<plugin_field_slug>_schema.json
  • Root type must be object
  • Each setting is defined in properties
  • Defaults are strongly recommended
  • Validation rules (minimum, enum, pattern, etc.) should be explicit
{
"type": "object",
"title": "Filesystem Plugin Configuration",
"properties": {
"max_file_size": {
"type": "integer",
"title": "Maximum File Size (MB)",
"description": "Maximum file size to allow for download",
"minimum": 1,
"maximum": 1000,
"default": 100
},
"allowed_extensions": {
"type": "string",
"title": "Allowed File Extensions",
"description": "Comma-separated list of allowed file extensions",
"pattern": "^(\\.\\w+)(,\\s*\\.\\w+)*$"
}
}
}

If your plugin needs to validate metadata values against JSON Schema, you can use the following implementation:

from jsonschema import Draft201909Validator
from referencing import Registry, Resource
def validate_metadata_schema(schema, value):
"""
Validate a metadata value against a JSON Schema.
This function ensures that the provided metadata value conforms to the structure
and constraints defined in the schema. It uses the Draft 2019-09 version
of JSON Schema for validation.
Parameters
----------
schema : dict
The JSON Schema to validate against.
value : dict
The metadata object to be validated.
Raises
------
jsonschema.exceptions.ValidationError
If value does not conform to schema.
Examples
--------
>>> schema = {"type": "object", "properties": {"name": {"type": "string"}}}
>>> value = {"name": "test"}
>>> validate_metadata_schema(schema, value) # No exception raised
"""
schema["$schema"] = "https://json-schema.org/draft/2019-09/schema"
registry = Registry(retrieve=Resource.from_contents(schema))
validator = Draft201909Validator(
schema,
registry=registry,
)
validator.validate(value)

Dependencies: You’ll need to add jsonschema >= 4.0.0 to your plugin’s pyproject.toml dependencies.

If you want to provide a better user experience, you can provide a UI schema to control the appearance and behavior of the metadata fields.

Create madbot_filesystem/static/json/uischema/filesystem_uischema.json:

{
"type": "VerticalLayout",
"elements": [
{
"type": "Control",
"scope": "#/properties/max_file_size",
"options": {
"slider": true
}
},
{
"type": "Control",
"scope": "#/properties/allowed_extensions",
"options": {
"placeholder": ".txt,.pdf,.jpg"
}
}
]
}

EditorJS is a block-styled editor used in madbot for rich text editing of metadata fields. It provides a structured JSON format for content that includes paragraphs, headers, lists, tables, code blocks, and various other formatting options.

In madbot, EditorJS is used as the primary rich text editor for metadata fields that require formatted content, such as descriptions, abstracts, and notes. The content is stored as JSON in a structured format that can be converted to different output formats (HTML, Markdown, plain text) depending on the needs of the plugin or the target system.

The EditorJS JSON structure follows this general format:

{
"blocks": [
{
"type": "paragraph",
"data": {
"text": "This is a paragraph with <b>bold</b> and <i>italic</i> text."
},
"tunes": {
"textVariant": "default"
}
},
{
"type": "header",
"data": {
"level": 2,
"text": "Section Title"
}
}
]
}

When your plugin receives metadata values from madbot, some fields may contain EditorJS JSON content. Depending on your target system’s requirements, you may need to convert this content to:

  • HTML (e.g., Zenodo)
  • Markdown (e.g., GitHub, GitLab)
  • Plain Text (e.g., ENA)

If your plugin needs to convert EditorJS content to HTML, Markdown, or plain text, you can use the following reference implementations. Simply copy these functions into your plugin code:

Helper Function: deep_json_load
import html
import json
import re
def deep_json_load(obj):
"""
Parse recursively a JSON-like structure.
Parameters
----------
obj : Any
A JSON string, list, dict, or primitive type.
Returns
-------
Any
A Python object with all nested JSON strings parsed into native types.
"""
if isinstance(obj, str):
s = obj.strip()
if s.startswith(("{", "[")):
try:
parsed = json.loads(obj)
return deep_json_load(parsed)
except (json.JSONDecodeError, TypeError):
return obj
return obj
if isinstance(obj, dict):
return {k: deep_json_load(v) for k, v in obj.items()}
if isinstance(obj, list):
return [deep_json_load(elem) for elem in obj]
return obj
Convert to HTML: convert_editorjs_to_html
import json
import re
def convert_editorjs_to_html(json_to_convert):
"""
Convert EditorJS content to HTML format.
This function converts EditorJS JSON content to HTML format,
supporting various block types like paragraphs, headers, lists, tables, and code blocks.
Parameters
----------
json_to_convert : dict or str
JSON from EditorJS to convert to HTML.
Returns
-------
str
HTML formatted description.
"""
if not json_to_convert:
return ""
html_parts = []
def process_list_items(items, style, level=0):
if style == "unordered":
tag = "ul"
elif style == "ordered":
tag = "ol"
elif style == "checklist":
tag = "ul"
else:
return ""
result = [f"<{tag}>"]
for item in items:
content = item.get("content", "")
if style == "checklist":
checked = item.get("meta", {}).get("checked", False)
checkbox = "☑" if checked else "☐"
result.append(f"<li>{checkbox} {content}</li>")
else:
result.append(f"<li>{content}</li>")
nested_items = item.get("items", [])
if nested_items:
result.append(process_list_items(nested_items, style, level + 1))
result.append(f"</{tag}>")
return "\n".join(result)
if not isinstance(json_to_convert, dict):
try:
json_to_convert = deep_json_load(json_to_convert)
except (json.JSONDecodeError, TypeError) as e:
raise ValueError(
{
"error": "Invalid JSON",
"original": json_to_convert,
}
) from e
for block in json_to_convert.get("blocks", []):
block_type = block.get("type")
block_data = block.get("data", {})
variant = block.get("tunes", {}).get("textVariant", "")
if block_type == "paragraph":
text = block_data.get("text", "")
# Handle inline formatting
text = (
text.replace("<i>", "<em>")
.replace("</i>", "</em>")
.replace("<b>", "<strong>")
.replace("</b>", "</strong>")
.replace('<code class="inline-code">', "<code>")
.replace("</code>", "</code>")
.replace('<u class="cdx-underline">', "<u>")
.replace("</u>", "</u>")
.replace('<mark class="cdx-marker">', "<mark>")
.replace("</mark>", "</mark>")
)
# Handle URLs
url_pattern = r'<a href="([^"]+)">([^<]+)</a>'
text = re.sub(url_pattern, r'<a href="\1" target="_blank">\2</a>', text)
# Handle blockquote styles
if variant in ["citation", "call-out", "details"]:
prefix = ""
if variant == "call-out":
prefix = "💡 "
elif variant == "details":
prefix = "ℹ️ "
html_parts.append(f"<blockquote>{prefix}{text}</blockquote>")
else:
html_parts.append(f"<p>{text}</p>")
elif block_type == "header":
level = block_data.get("level", 1)
text = block_data.get("text", "")
html_parts.append(f"<h{level}>{text}</h{level}>")
elif block_type == "list":
items = block_data.get("items", [])
style = block_data.get("style")
html_parts.append(process_list_items(items, style))
elif block_type == "table":
content = block_data.get("content", [])
with_headings = block_data.get("withHeadings", False)
if content:
html_parts.append("<table border='1'>")
if with_headings:
header_row = "".join(f"<th>{col}</th>" for col in content.pop(0))
html_parts.append(f"<tr>{header_row}</tr>")
for row in content:
data_row = "".join(f"<td>{col}</td>" for col in row)
html_parts.append(f"<tr>{data_row}</tr>")
html_parts.append("</table>")
elif block_type == "code":
code = block_data.get("code", "")
html_parts.append(f"<pre><code>{code}</code></pre>")
return "\n".join(html_parts)
Convert to Markdown: convert_editorjs_to_markdown
import json
import re
def convert_editorjs_to_markdown(json_to_convert):
"""
Convert EditorJS content to Markdown format.
This function converts EditorJS JSON content to Markdown format,
supporting various block types like paragraphs, headers, lists, tables, and code blocks.
Parameters
----------
json_to_convert : dict or str
JSON from EditorJS to convert to Markdown.
Returns
-------
str
Markdown formatted description.
"""
if not json_to_convert:
return ""
markdown_parts = []
def process_list_items(items, style, level=0, counter=None):
result = []
if counter is None:
counter = {}
if level not in counter:
counter[level] = 1
for item in items:
content = item.get("content", "")
indent = " " * level
if style == "unordered":
result.append(f"{indent}- {content}")
elif style == "ordered":
result.append(f"{indent}{counter[level]}. {content}")
counter[level] += 1
elif style == "checklist":
checked = item.get("meta", {}).get("checked", False)
checkbox = "x" if checked else " "
result.append(f"{indent}- [{checkbox}] {content}")
nested_items = item.get("items", [])
if nested_items:
result.extend(
process_list_items(nested_items, style, level + 1, counter)
)
return result
if not isinstance(json_to_convert, dict):
try:
json_to_convert = deep_json_load(json_to_convert)
except (json.JSONDecodeError, TypeError) as e:
raise ValueError(
{
"error": "Invalid JSON",
"original": json_to_convert,
}
) from e
for block in json_to_convert.get("blocks", []):
block_type = block.get("type")
block_data = block.get("data", {})
variant = block.get("tunes", {}).get("textVariant", "")
if block_type == "paragraph":
text = block_data.get("text", "")
text = (
text.replace("<i>", "*")
.replace("</i>", "*")
.replace("<b>", "**")
.replace("</b>", "**")
.replace('<code class="inline-code">', "`")
.replace("</code>", "`")
.replace('<u class="cdx-underline">', "")
.replace("</u>", "")
.replace('<mark class="cdx-marker">', "")
.replace("</mark>", "")
)
url_pattern = r'<a href="([^"]+)">([^<]+)</a>'
text = re.sub(url_pattern, r"[\2](\1)", text)
if variant in ["citation", "call-out", "details"]:
prefix = ""
if variant == "call-out":
prefix = "💡 "
elif variant == "details":
prefix = "ℹ️ "
markdown_parts.append(f"> {prefix}{text}\n")
else:
markdown_parts.append(f"{text}\n")
elif block_type == "header":
level = block_data.get("level", 1)
text = block_data.get("text", "")
markdown_parts.append(f"{'#' * level} {text}\n")
elif block_type == "list":
items = block_data.get("items", [])
style = block_data.get("style")
list_items = process_list_items(items, style)
markdown_parts.extend(item + "\n" for item in list_items)
elif block_type == "table":
content = block_data.get("content", [])
with_headings = block_data.get("withHeadings", False)
if content:
if with_headings:
heading = content.pop(0)
header_row = " | ".join(heading)
markdown_parts.append(f"| {header_row} |")
else:
header_row = " | ".join(["<!-- -->"] * len(content[0]))
markdown_parts.append(f"| {header_row} |")
separator = " | ".join(["---"] * len(content[0]))
markdown_parts.append(f"| {separator} |")
for row in content:
data_row = " | ".join(row)
markdown_parts.append(f"| {data_row} |")
markdown_parts.append("\n")
elif block_type == "code":
code = block_data.get("code", "")
markdown_parts.append(f"```\n{code}\n```\n")
return "\n".join(markdown_parts)
Convert to Plain Text: convert_editorjs_to_plain
import json
import re
def convert_editorjs_to_plain(json_to_convert):
"""
Convert EditorJS content to plain text format.
This function converts EditorJS JSON content to plain text format,
removing all formatting and supporting various block types.
Parameters
----------
json_to_convert : dict or str
JSON from EditorJS to convert to plain text.
Returns
-------
str
Plain text formatted description.
"""
if not json_to_convert:
return ""
block_texts = []
def normalize_text(text):
"""Normalize text for plain-text export while preserving line breaks."""
text = html.unescape(text or "")
text = text.replace("\u00A0", " ")
text = text.replace("\r\n", "\n").replace("\r", "\n")
normalized_lines = []
for line in text.split("\n"):
normalized_line = re.sub(r"[ \t]+", " ", line).strip()
normalized_lines.append(normalized_line)
return "\n".join(normalized_lines).strip()
def process_list_items(items, level=0):
result = []
for item in items:
content = normalize_text(item.get("content", ""))
indent = " " * level
result.append(f"{indent}- {content}")
nested_items = item.get("items", [])
if nested_items:
result.extend(process_list_items(nested_items, level + 1))
return result
if not isinstance(json_to_convert, dict):
try:
json_to_convert = deep_json_load(json_to_convert)
except (json.JSONDecodeError, TypeError) as e:
raise ValueError(
{
"error": "Invalid JSON",
"original": json_to_convert,
}
) from e
for block in json_to_convert.get("blocks", []):
block_type = block.get("type")
block_data = block.get("data", {})
if block_type == "paragraph":
text = block_data.get("text", "")
text = (
text.replace("<br>", "\n")
.replace("<i>", "")
.replace("</i>", "")
.replace("<b>", "")
.replace("</b>", "")
.replace('<code class="inline-code">', "")
.replace("</code>", "")
.replace('<u class="cdx-underline">', "")
.replace("</u>", "")
.replace('<mark class="cdx-marker">', "")
.replace("</mark>", "")
)
# Preserve links as plain text with URL
def replace_link(match):
href = match.group(1).strip()
label = match.group(2).strip()
if not href:
return label
if not label or label == href:
return href
return f"{label} ({href})"
text = re.sub(r'<a href="([^"]+)">([^<]*)</a>', replace_link, text)
block_texts.append(normalize_text(text))
elif block_type == "header":
text = block_data.get("text", "")
block_texts.append(normalize_text(text))
elif block_type == "list":
items = block_data.get("items", [])
list_items = process_list_items(items)
block_texts.append("\n".join(list_items).strip())
elif block_type == "table":
content = block_data.get("content", [])
table_rows = [
normalize_text(" ".join(str(cell) for cell in row)) for row in content
]
block_texts.append("\n".join(table_rows).strip())
elif block_type == "code":
code = block_data.get("code", "")
block_texts.append(normalize_text(code))
return "\n\n".join(part for part in block_texts if part)

Note: These EditorJS conversion functions use the deep_json_load helper function shown above. Make sure to include it in your plugin if you use any of the conversion functions.

Each plugin can define metadata fields that are uniquely identified using a slug — a structured, machine-readable identifier.

The general form of a slug is:

<plugin_name>__[<metadata_object_type>__]<field_name>
  1. <plugin_name>

    • The first part of the slug must always be the plugin name.
    • It identifies which plugin the field belongs to.
    • Example: zenodo, darwin_core
  2. <metadata_object_type> (optional)

    • Used when the plugin defines fields for a specific metadata object type.
    • If included, it appears between double underscores after the plugin name.
    • Example: dwc in darwin_core__dwc__vertical_datum
  3. <field_name>

    • The actual name of the metadata field.
    • Should be descriptive, lowercase, and use underscores instead of spaces.
    • Example: title, vertical_datum
  • zenodo__title → Plugin: zenodo, Field: title
  • darwin_core__dwc__vertical_datum → Plugin: darwin_core, Metadata Object Type: dwc, Field: vertical_datum
  • myplugin__fieldname → Plugin: myplugin, Field: fieldname
  • myplugin__object_type__fieldname → Plugin: myplugin, Metadata Object Type: object_type, Field: fieldname
  • Use only lowercase letters, digits, underscores, or hyphens.
  • The slug must start with the plugin name followed by __.
  • Only one or two double-underscore sections are allowed:
    • plugin__field
    • plugin__metadata_object_type__field
  • Example of invalid slugs:
    • Plugin__Title (uppercase letters)
    • plugin__metadata__object__field (too many sections)
    • plugin-field (missing double underscores)

This ensures that all plugin metadata fields are uniquely and predictably identified across madbot.

The ENA (European Nucleotide Archive) plugin provides submission support for biological sequence data. Its metadata field schemas are auto-generated from ENA’s public checklist API rather than written by hand, ensuring they always reflect the official ENA specification.

Each ENA sample field has a single canonical slug shared across all checklists that include it:

ena__sample__elevation
ena__sample__collection_date

The checklist code (erc000029, erc000011, etc.) is not part of the slug. Which checklists use a field and at which requirement level (mandatory / recommended / optional) is stored separately in madbot_ena/checklists.py.

The generator script fetches every ENA sample checklist XML from the EBI API, deduplicates fields that appear in multiple checklists, and writes one JSON schema file per unique field.

What the script does, step by step:

  1. Fetches all ENA checklist XMLs from https://www.ebi.ac.uk/ena/browser/api/xml/ERC0000<N> for N = 11 to 59.
  2. Parses each field inside every <FIELD_GROUP> block — name, description, format, restriction pattern, allowed values, and units.
  3. Deduplicates fields by structural fingerprint. Two fields with the same name and the same validation rules share one output file (e.g. elevation.json) with one canonical slug (ena__sample__elevation).
  4. Writes one JSON schema file per unique field under:
    madbot_ena/static/json/schema/sample/
  5. Updates patterns.json an internal registry that tracks which regex patterns are used across fields and which checklists reference them, used to generate human-readable help text.
  6. Writes madbot_ena/checklists.py a plain Python dict mapping each checklist code to its title, description, and its fields pre-bucketed by requirement level.
  7. Writes madbot_ena/static/json/schema/checklist.json the JSON schema for the checklist selector field shown in the submission UI.

Note: Checklist names in patterns.json may appear as either a human-readable label (e.g. GSC MIxS host associated) or as a bare ERC code (e.g. ERC000013). This reflects what ENA’s API returns in the <LABEL> element for each checklist, some checklists have a descriptive label registered, others only expose their accession code.

Run the script from the root of the madbot_ena plugin:

Terminal window
pixi run python madbot_ena/utils/generateMetadataJsonSchema.py

Re-run it whenever ENA updates its checklists (new fields, new allowed values, new checklists). The output files should be committed to the repository.

Requirements: the script needs network access to reach https://www.ebi.ac.uk. It has no other external dependencies beyond the packages already declared in pyproject.toml.

Each generated file in madbot_ena/static/json/schema/sample/ is a standard JSON Schema fragment. The level (mandatory / recommended / optional) is not stored in the schema file, it is a property of the relationship between a field and a checklist, and lives exclusively in checklists.py. Depending on the field type, the schema takes one of the following shapes:

Text field with a regex restriction:

{
"title": "Elevation",
"originalTitle": "elevation",
"description": "The elevation of the sampling site as measured by the vertical distance from mean sea level.",
"oneOf": [
{
"title": "Value available",
"type": "object",
"properties": {
"status": { "const": "Value available" },
"value": {
"type": "string",
"pattern": "-?[0-9]+(\\.[0-9]+)?"
}
}
},
{
"title": "Value unavailable",
"type": "object",
"properties": {
"status": { "const": "Value unavailable" }
}
}
]
}

Text choice field (controlled vocabulary):

{
"title": "Library layout",
"originalTitle": "library_layout",
"description": "Sequencing library layout.",
"type": "string",
"enum": ["SINGLE", "PAIRED"]
}

Field with a unit:

{
"title": "Sample volume or weight for DNA extraction",
"originalTitle": "sample_volume_or_weight_for_dna_extraction",
"description": "Volume (ml) or mass (g) of total collected sample processed for DNA extraction.",
"type": "object",
"properties": {
"value": {
"type": "string",
"pattern": "[0-9]+(\\.[0-9]+)?"
},
"unit": {
"type": "string",
"enum": ["g", "ml", "mg", "ng", "l"]
}
},
"required": ["value", "unit"]
}

The key properties present in every schema file are:

KeyDescription
titleHuman-readable field name (derived from the ENA LABEL)
originalTitleRaw field name exactly as returned by ENA’s API
descriptionField description from ENA
type / oneOfJSON Schema validation rules for the field value
patternRegex constraint on the value, when applicable
enumAllowed values for controlled vocabulary fields
fieldgroupENA field group the field belongs to (e.g. Organism)

Note: There is no level key in individual schema files. Requirement levels (mandatory / recommended / optional) are a checklist-scoped property and are stored in checklists.py only.

madbot_ena/checklists.py is the single source of truth for which fields each checklist requires, and at which level. It is auto-generated, do not edit manually.

ENA_CHECKLIST_CONFIGS = {
"erc000011": {
"title": "ERC000011 - GSC MIxS plant associated",
"description": "Minimum information about a MIxS plant-associated sample.",
"fields": {
"mandatory": ["ena__sample__collection_date", "ena__sample__lat_lon", ...],
"recommended": ["ena__sample__elevation", ...],
"optional": ["ena__sample__depth", ...],
},
},
...
}

At runtime, plugins.py uses two utility classes from madbot_ena/utils/checklist_handler.py to convert this dict into fully typed metadata object types:

ChecklistFieldAssociation

Couples a FieldRepresentation (resolved from the slug at runtime after add_fields() has loaded all JSON schema files) to a requirement level string. It is the explicit, typed representation of the statement “this field is mandatory for this checklist”.

@dataclass
class ChecklistFieldAssociation:
field: FieldRepresentation # resolved from slug at runtime
level: Literal["mandatory", "recommended", "optional"]

ChecklistMetadataObjectType

Subclass of MetadataObjectType that takes a checklist_code and a list of ChecklistFieldAssociation objects. It derives the three bucket lists (mandatory_fields, recommended_fields, optional_fields) automatically in model_post_init, preserving full backward compatibility with all existing consumers.

class ChecklistMetadataObjectType(MetadataObjectType):
checklist_code: str
field_associations: list[ChecklistFieldAssociation]
def model_post_init(self, __context: Any) -> None:
self.mandatory_fields = [a.field for a in self.field_associations if a.level == "mandatory"]
self.recommended_fields = [a.field for a in self.field_associations if a.level == "recommended"]
self.optional_fields = [a.field for a in self.field_associations if a.level == "optional"]

build_metadata_object_types() in plugins.py

Iterates over ENA_CHECKLIST_CONFIGS and produces one ChecklistMetadataObjectType per checklist. No manual field bucketing is needed, adding a new checklist only requires re-running the generator.

Whenever plugin field schemas are added, removed, or modified, including after re-running the ENA schema generator, you must reload the plugins into the database:

Terminal window
pixi run madbot reload_plugins

This command synchronises the PluginRawSchema and PluginMetadataField tables with the current state of all installed plugins. Without it, any new or changed fields will not be visible in the application.