Plugin development guide
This comprehensive guide will walk you through creating a madbot plugin from scratch. We’ll build a simple file system data plugin as an example to demonstrate the complete development process.
Overview
Section titled “Overview”Madbot’s plugin system provides a flexible architecture for integrating with external systems. Plugins are Python classes that inherit from specific base classes and implement required interfaces. The system supports three main types of plugins:
- Data Plugins: Connect to external data sources and provide access to files and directories
- Submission Plugins: Handle data submission to external repositories and services
- Metadata Field Definition Plugins: Define metadata schemas to extend the madbot metadata referential
In this tutorial, we’ll create a Data Plugin that connects to a remote file system and allows users to browse and download files from remote servers.
Every class, method and exception of plugins_api is described in the plugins_api reference.
If you write your plugin with a coding assistant, give it the documentation as plain Markdown: llms-full.txt holds every page of this site and the plugins_api reference of the current version, and llms.txt lists what is available.
Prerequisites
Section titled “Prerequisites”Before starting, ensure you have:
- Python 3.11
- Basic knowledge of Python classes and inheritance
- Access to a madbot development environment
Step 1: Project Setup
Section titled “Step 1: Project Setup”Let’s create a new plugin project called madbot_filesystem. This will be a data plugin that allows browsing remote directories.
1.1 Create Project Structure
Section titled “1.1 Create Project Structure”Create the following directory structure:
madbot_filesystem_plugin/├── pyproject.toml # Project configuration and dependencies├── README.md # Plugin documentation├── LICENSE # License terms for the plugin├── CONTRIBUTING.md # Contribution guidelines (recommended)├── code_of_conduct.md # Community behavior standards (recommended)├── .gitlab-ci.yml # GitLab CI/CD pipeline configuration (recommended)└── madbot_filesystem/ ├── __init__.py ├── plugins.py # Your plugin class ├── app.py # The app that registers the plugin └── static/ ├── json/ │ └── schema/ # JSON schema files │ └── uischema/ # Optional JSON Forms UI schema files (if they exist) └── images/ └── filesystem.png # Your plugin logo1.2 Initialize Project Files
Section titled “1.2 Initialize Project Files”pyproject.toml:
[project]name = "madbot-filesystem"version = "1.0.0"description = "A remote filesystem data plugin for madbot"authors = [ {name = "Your Name", email = "your.email@example.com"}]readme = "README.md"license = "BSD-3-Clause"license-files = ["LICEN[CS]E*"]requires-python = ">=3.11,<3.12"keywords = ["madbot"]dependencies = [ "plugins-api >=3.0.0,<4",]
[project.optional-dependencies]dev = [ 'ruff', 'bandit[toml]', 'safety >= 2.4',]
[build-system]requires = ["setuptools>=45", "wheel"]build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]include = ["madbot-filesystem*"]exclude = ["logs*", "tests*"]For more information about the pyproject.toml file, see this guide.
This file is enough to write the plugin. The repository around it also needs the registry that publishes plugins-api, floors on the build tools, and its own CI with the quality and security jobs: Set up a plugin repository lists them.
README.md:
# Madbot Filesystem Plugin
A madbot data plugin that provides access to remote file systems.
## Features
- Browse remote directories- List files and subdirectories on remote servers- Download files with integrity verification
## Installation
This plugin is designed to be installed as a madbot plugin. See the madbot documentation for installation instructions.Step 2: Understanding Data Plugin Requirements
Section titled “Step 2: Understanding Data Plugin Requirements”Before implementing our plugin, let’s understand what we need to build. A Data Plugin inherits from DataPlugin which inherits from ConnectionPlugin which inherits from MadbotPlugin. This means we need to implement methods from all three base classes:
2.1 From MadbotPlugin (Base Plugin Class)
Section titled “2.1 From MadbotPlugin (Base Plugin Class)”get_name(): Return the plugin nameget_slug(): Return the plugin slugget_description(): Return plugin descriptionget_author(): Return plugin authorget_logo(): Return the path of the plugin logoget_color(): Return the color of the pluginget_status(): Return maturity status keyed by plugin type (e.g.{"data_plugin": PluginStatus.STABLE})
2.2 From ConnectionPlugin (Connection Management)
Section titled “2.2 From ConnectionPlugin (Connection Management)”get_instance_name(): Return the instance name of the connection pluginget_connection_fields(): Return connection fields (shared and private)get_shared_parameters(): Return all shared parameters in a cleaned form
2.3 From DataPlugin (Data Operations)
Section titled “2.3 From DataPlugin (Data Operations)”__init__(connection_parameters): Initialize the plugin with connection parametersget_root_name(): Return the name of the root data objectget_root_id(): Return the unique identifier of the root data objectbrowse_data_objects(parent_id): Return the direct children of a parent (navigation / listing). Whenparent_idisNone, browse from the root of the source. Distinct from reading objects by id.get_data_objects_by_ids(object_ids): Return detailed information for a list of data object ids. Return adictkeyed by id: either aPluginDataObjector aPluginError(DataNotFoundError,PluginPermissionError, …). Raise source-level errors (SOURCE_ERRORS:UnreachableError,HostConnectionRefusedError,AuthenticationError) instead of storing them. Empty list returns{}. Duplicate ids appear once. Useself._read_each(object_ids, read_one)when the source has no batch API.get_batch_read_size()(optional): Max ids madbot sends per batch-read call. DefaultDEFAULT_BATCH_READ_SIZE(100). ReturnNoneto disable chunking.get_metadata_for_data_objects(object_ids)(optional): Return metadata for each id (dictof lists or per-id errors). Defaults to[]per id.get_data_associations_for_data_objects(object_ids)(optional): Return associations for each id. Defaults to[]per id.download_data(external_dataobject_id, destination_path): Download a data object to local storage
See the DataPlugin reference for full signatures.
Step 3: Creating Our Example Plugin
Section titled “Step 3: Creating Our Example Plugin”Let’s create our filesystem plugin step by step.
3.1 Basic Plugin Structure
Section titled “3.1 Basic Plugin Structure”Create madbot_filesystem/plugins.py:
import osfrom pathlib import Pathfrom typing import Any
import requestsfrom plugins_api import ( AuthenticationError, ConnectionField, DataInvalidError, DataNotFoundError, DataPlugin, DownloadDataError, PluginDataObject, PluginError, PluginStatus, UnreachableError,)
class FilesystemDataPlugin(DataPlugin): """ A data plugin that provides access to remote file system directories.
This example demonstrates the plugin architecture without being tied to a specific transport library. """
def __init__(self, connection_parameters: dict[str, Any]): """ Initialize the filesystem plugin.
Args: connection_parameters: Dictionary containing connection parameters """ self.host = connection_parameters.get("host", "") self.port = int(connection_parameters.get("port", 22)) self.username = connection_parameters.get("username", "") self.password = connection_parameters.get("password", "")
# Example connection check, adapt to your protocol / client: try: response = requests.get( f"https://{self.host}:{self.port}", auth=(self.username, self.password), timeout=20, ) response.raise_for_status() except requests.HTTPError: raise AuthenticationError() except requests.RequestException: raise UnreachableError(self.host)
# ===== MadbotPlugin Methods =====
@classmethod def get_name(cls) -> str: """Return the plugin name.""" return "Remote Filesystem Plugin"
@classmethod def get_slug(cls) -> str: """Return the plugin slug.""" return "remote_filesystem_plugin"
@classmethod def get_description(cls) -> str: """Return plugin description.""" return "A data plugin that provides access to remote file systems for browsing and downloading files."
@classmethod def get_author(cls) -> str: """Return plugin author.""" return "John Doe"
@classmethod def get_logo(cls) -> str: """Return the path of the plugin logo.""" return "images/filesystem.png"
@classmethod def get_color(cls) -> str: """Return the color of the plugin.""" return "#4CAF50"
@classmethod def get_status(cls) -> dict[str, PluginStatus]: return {"data_plugin": PluginStatus.STABLE}
# ===== ConnectionPlugin Methods =====
def get_instance_name(self) -> str: """Return the instance name of the connection plugin.""" return f"Filesystem-{self.host}-{self.username}"
@classmethod def get_connection_fields( cls, ) -> tuple[list[ConnectionField], list[ConnectionField]]: """ Retrieve the connection fields for the connection plugin.
Returns: Tuple containing shared and private connection fields """ shared_fields = [ ConnectionField( slug="host", schema_content={ "type": "string", "title": "Host", "description": "Remote server hostname or IP address", }, ), ConnectionField( slug="port", schema_content={ "type": "integer", "title": "Port", "description": "Remote server port", "default": 22, }, ), ]
private_fields = [ ConnectionField( slug="username", schema_content={ "type": "string", "title": "Username", "description": "Remote server username", }, ), ConnectionField( slug="password", schema_content={ "type": "string", "title": "Password", "description": "Remote server password", }, ), ]
return shared_fields, private_fields
def get_shared_parameters(self) -> dict[str, Any]: """Return all shared parameters.""" return {"host": self.host, "port": self.port}
# ===== DataPlugin Methods =====
@staticmethod def get_root_name() -> str: """Return the name of the root data object.""" return "Remote Filesystem Root"
@staticmethod def get_root_id() -> str: """Return the unique identifier of the root data object.""" return "/"
def browse_data_objects(self, parent_id: str | None) -> list[PluginDataObject]: """Return the direct children of a parent; ``None`` starts at the root.""" parent_id = self.get_root_id() if parent_id is None else parent_id path = parent_id if parent_id != self.get_root_id() else ""
items = get_directory_contents_from_remote_system(path) if items is None: return []
objects = [ PluginDataObject( id=item_data["path"], name=item_data["name"], type="directory" if item_data["is_directory"] else "file", size=item_data["size"], has_children=item_data["is_directory"], file_extension=( Path(item_data["path"]).suffix.lstrip(".") if not item_data["is_directory"] else None ), ) for item_data in items ] return sorted(objects, key=lambda obj: obj.name)
def _read_data_object(self, object_id: str) -> PluginDataObject: """Read one object. Used via `_read_each` when the source has no batch API.""" if object_id == "/": return PluginDataObject( id="/", name="Remote Filesystem Root", type="directory", size=0, has_children=True, )
data = get_file_data_from_remote_system(object_id) if not data: raise DataNotFoundError(object_id)
return PluginDataObject( id=object_id, name=Path(object_id).name, type="directory" if data["is_directory"] else "file", size=data["size"], has_children=data["is_directory"], file_extension=( Path(object_id).suffix.lstrip(".") if not data["is_directory"] else None ), )
def get_data_objects_by_ids( self, object_ids: list[str] ) -> dict[str, PluginDataObject | PluginError]: """ Return detailed information for each requested id.
Prefer one remote call when the source allows it; otherwise wrap a one-id reader with `_read_each` as below. """ return self._read_each(object_ids, self._read_data_object)
def download_data(self, external_dataobject_id: str, destination_path: str) -> str: """Download a data object to local storage and return the download report.""" data_object = self._read_data_object(external_dataobject_id)
if data_object.type == "directory": raise DataInvalidError(external_dataobject_id)
content = download_file_from_remote_system(external_dataobject_id) with open(destination_path, "wb") as handle: handle.write(content)
local_size = os.path.getsize(destination_path) if data_object.size is not None and local_size != data_object.size: raise DownloadDataError( f"Size mismatch: expected {data_object.size}, got {local_size}" )
return destination_pathget_metadata_for_data_objects and get_data_associations_for_data_objects are optional. Override them when your source exposes metadata or associations. Otherwise, the base class returns an empty list for each id.
When the remote API can fetch several ids in one call, implement the batch method directly. It must show the three rules: one entry per distinct id, per-id errors stored in the result, source errors raised:
def get_data_objects_by_ids(self, object_ids): if not object_ids: return {} # One remote call; connection or auth failures raise # UnreachableError / AuthenticationError. found = self.client.get_many(object_ids) return { object_id: ( self._to_data_object(found[object_id]) if object_id in found else DataNotFoundError(object_id) ) for object_id in dict.fromkeys(object_ids) }3.2 Plugin Registration
Section titled “3.2 Plugin Registration”Create madbot_filesystem/app.py:
from django.apps import AppConfig
class MadbotFilesystemPluginConfig(AppConfig): """Django app configuration for the madbot filesystem plugin."""
default_auto_field = 'django.db.models.BigAutoField' name = 'madbot_filesystem' verbose_name = 'Madbot Filesystem Data Plugin'
def ready(self): """Called when Django starts up.""" # Import here to avoid circular imports from . import plugins # noqaStep 4: Implementing the Plugin Class
Section titled “Step 4: Implementing the Plugin Class”4.1 Key Implementation Details
Section titled “4.1 Key Implementation Details”Our plugin implementation includes several important aspects:
- Plugin identity: required
MadbotPluginmethods, includingget_status() - Connection management: shared/private
ConnectionFieldDTOs (slug+ JSON Schema) - Navigation vs read:
browse_data_objectslists children;get_data_objects_by_idsreads by id (batch) - Batch reads: use
_read_eachwhen there is no batch API, or one remote call when there is; result keys must match the requested ids exactly. Overrideget_batch_read_size()when the source has a different id-list limit (or returnNonefor no chunking). - Error handling: store per-id
PluginError; raiseSOURCE_ERRORS(UnreachableError,HostConnectionRefusedError,AuthenticationError)
Advanced Features
Section titled “Advanced Features”JSON Schema Support
Section titled “JSON Schema Support”Madbot relies on JSON Schema to describe plugin configuration, connection parameters, and metadata fields. This section explains what schemas are expected, where they live, and how they should be structured.
What JSON Schemas Are Used For
Section titled “What JSON Schemas Are Used For”JSON Schemas in madbot serve three distinct purposes:
- Plugin settings schemas (optional and depending on the plugin type)
- Global configuration of a plugin instance
- Connection field schemas
- Describe connection parameters (host, credentials, tokens, etc.)
- Metadata field schemas
- Describe metadata fields exposed by metadata plugins
All schemas follow JSON Schema Draft 7+ conventions.
Plugin Settings Schema
Section titled “Plugin Settings Schema”Plugin settings schemas define instance-level configuration for a plugin.
Location
Section titled “Location”madbot_filesystem/static/json/schema/<plugin_field_slug>_schema.jsonCharacteristics
Section titled “Characteristics”- Root type must be object
- Each setting is defined in properties
- Defaults are strongly recommended
- Validation rules (minimum, enum, pattern, etc.) should be explicit
Example
Section titled “Example”{ "type": "object", "title": "Filesystem Plugin Configuration", "properties": { "max_file_size": { "type": "integer", "title": "Maximum File Size (MB)", "description": "Maximum file size to allow for download", "minimum": 1, "maximum": 1000, "default": 100 }, "allowed_extensions": { "type": "string", "title": "Allowed File Extensions", "description": "Comma-separated list of allowed file extensions", "pattern": "^(\\.\\w+)(,\\s*\\.\\w+)*$" } }}Validating Metadata Against JSON Schema
Section titled “Validating Metadata Against JSON Schema”If your plugin needs to validate metadata values against JSON Schema, you can use the following implementation:
from jsonschema import Draft201909Validatorfrom referencing import Registry, Resource
def validate_metadata_schema(schema, value): """ Validate a metadata value against a JSON Schema.
This function ensures that the provided metadata value conforms to the structure and constraints defined in the schema. It uses the Draft 2019-09 version of JSON Schema for validation.
Parameters ---------- schema : dict The JSON Schema to validate against. value : dict The metadata object to be validated.
Raises ------ jsonschema.exceptions.ValidationError If value does not conform to schema.
Examples -------- >>> schema = {"type": "object", "properties": {"name": {"type": "string"}}} >>> value = {"name": "test"} >>> validate_metadata_schema(schema, value) # No exception raised """ schema["$schema"] = "https://json-schema.org/draft/2019-09/schema" registry = Registry(retrieve=Resource.from_contents(schema)) validator = Draft201909Validator( schema, registry=registry, ) validator.validate(value)Dependencies: You’ll need to add jsonschema >= 4.0.0 to your plugin’s pyproject.toml dependencies.
UI Schema for Better User Experience
Section titled “UI Schema for Better User Experience”If you want to provide a better user experience, you can provide a UI schema to control the appearance and behavior of the metadata fields.
Create madbot_filesystem/static/json/uischema/filesystem_uischema.json:
{ "type": "VerticalLayout", "elements": [ { "type": "Control", "scope": "#/properties/max_file_size", "options": { "slider": true } }, { "type": "Control", "scope": "#/properties/allowed_extensions", "options": { "placeholder": ".txt,.pdf,.jpg" } } ]}EditorJS Content Support
Section titled “EditorJS Content Support”EditorJS is a block-styled editor used in madbot for rich text editing of metadata fields. It provides a structured JSON format for content that includes paragraphs, headers, lists, tables, code blocks, and various other formatting options.
How EditorJS is Used in madbot
Section titled “How EditorJS is Used in madbot”In madbot, EditorJS is used as the primary rich text editor for metadata fields that require formatted content, such as descriptions, abstracts, and notes. The content is stored as JSON in a structured format that can be converted to different output formats (HTML, Markdown, plain text) depending on the needs of the plugin or the target system.
The EditorJS JSON structure follows this general format:
{ "blocks": [ { "type": "paragraph", "data": { "text": "This is a paragraph with <b>bold</b> and <i>italic</i> text." }, "tunes": { "textVariant": "default" } }, { "type": "header", "data": { "level": 2, "text": "Section Title" } } ]}Using EditorJS Content in Plugins
Section titled “Using EditorJS Content in Plugins”When your plugin receives metadata values from madbot, some fields may contain EditorJS JSON content. Depending on your target system’s requirements, you may need to convert this content to:
- HTML (e.g., Zenodo)
- Markdown (e.g., GitHub, GitLab)
- Plain Text (e.g., ENA)
Converting EditorJS Content
Section titled “Converting EditorJS Content”If your plugin needs to convert EditorJS content to HTML, Markdown, or plain text, you can use the following reference implementations. Simply copy these functions into your plugin code:
Helper Function: deep_json_load
import htmlimport jsonimport re
def deep_json_load(obj): """ Parse recursively a JSON-like structure.
Parameters ---------- obj : Any A JSON string, list, dict, or primitive type.
Returns ------- Any A Python object with all nested JSON strings parsed into native types. """ if isinstance(obj, str): s = obj.strip() if s.startswith(("{", "[")): try: parsed = json.loads(obj) return deep_json_load(parsed) except (json.JSONDecodeError, TypeError): return obj return obj if isinstance(obj, dict): return {k: deep_json_load(v) for k, v in obj.items()} if isinstance(obj, list): return [deep_json_load(elem) for elem in obj] return objConvert to HTML: convert_editorjs_to_html
import jsonimport re
def convert_editorjs_to_html(json_to_convert): """ Convert EditorJS content to HTML format.
This function converts EditorJS JSON content to HTML format, supporting various block types like paragraphs, headers, lists, tables, and code blocks.
Parameters ---------- json_to_convert : dict or str JSON from EditorJS to convert to HTML.
Returns ------- str HTML formatted description. """ if not json_to_convert: return ""
html_parts = []
def process_list_items(items, style, level=0): if style == "unordered": tag = "ul" elif style == "ordered": tag = "ol" elif style == "checklist": tag = "ul" else: return ""
result = [f"<{tag}>"]
for item in items: content = item.get("content", "")
if style == "checklist": checked = item.get("meta", {}).get("checked", False) checkbox = "☑" if checked else "☐" result.append(f"<li>{checkbox} {content}</li>") else: result.append(f"<li>{content}</li>")
nested_items = item.get("items", []) if nested_items: result.append(process_list_items(nested_items, style, level + 1))
result.append(f"</{tag}>") return "\n".join(result)
if not isinstance(json_to_convert, dict): try: json_to_convert = deep_json_load(json_to_convert) except (json.JSONDecodeError, TypeError) as e: raise ValueError( { "error": "Invalid JSON", "original": json_to_convert, } ) from e
for block in json_to_convert.get("blocks", []): block_type = block.get("type") block_data = block.get("data", {}) variant = block.get("tunes", {}).get("textVariant", "")
if block_type == "paragraph": text = block_data.get("text", "")
# Handle inline formatting text = ( text.replace("<i>", "<em>") .replace("</i>", "</em>") .replace("<b>", "<strong>") .replace("</b>", "</strong>") .replace('<code class="inline-code">', "<code>") .replace("</code>", "</code>") .replace('<u class="cdx-underline">', "<u>") .replace("</u>", "</u>") .replace('<mark class="cdx-marker">', "<mark>") .replace("</mark>", "</mark>") )
# Handle URLs url_pattern = r'<a href="([^"]+)">([^<]+)</a>' text = re.sub(url_pattern, r'<a href="\1" target="_blank">\2</a>', text)
# Handle blockquote styles if variant in ["citation", "call-out", "details"]: prefix = "" if variant == "call-out": prefix = "💡 " elif variant == "details": prefix = "ℹ️ "
html_parts.append(f"<blockquote>{prefix}{text}</blockquote>") else: html_parts.append(f"<p>{text}</p>")
elif block_type == "header": level = block_data.get("level", 1) text = block_data.get("text", "") html_parts.append(f"<h{level}>{text}</h{level}>")
elif block_type == "list": items = block_data.get("items", []) style = block_data.get("style") html_parts.append(process_list_items(items, style))
elif block_type == "table": content = block_data.get("content", []) with_headings = block_data.get("withHeadings", False)
if content: html_parts.append("<table border='1'>") if with_headings: header_row = "".join(f"<th>{col}</th>" for col in content.pop(0)) html_parts.append(f"<tr>{header_row}</tr>")
for row in content: data_row = "".join(f"<td>{col}</td>" for col in row) html_parts.append(f"<tr>{data_row}</tr>")
html_parts.append("</table>")
elif block_type == "code": code = block_data.get("code", "") html_parts.append(f"<pre><code>{code}</code></pre>")
return "\n".join(html_parts)Convert to Markdown: convert_editorjs_to_markdown
import jsonimport re
def convert_editorjs_to_markdown(json_to_convert): """ Convert EditorJS content to Markdown format.
This function converts EditorJS JSON content to Markdown format, supporting various block types like paragraphs, headers, lists, tables, and code blocks.
Parameters ---------- json_to_convert : dict or str JSON from EditorJS to convert to Markdown.
Returns ------- str Markdown formatted description. """ if not json_to_convert: return ""
markdown_parts = []
def process_list_items(items, style, level=0, counter=None): result = [] if counter is None: counter = {}
if level not in counter: counter[level] = 1
for item in items: content = item.get("content", "") indent = " " * level
if style == "unordered": result.append(f"{indent}- {content}") elif style == "ordered": result.append(f"{indent}{counter[level]}. {content}") counter[level] += 1 elif style == "checklist": checked = item.get("meta", {}).get("checked", False) checkbox = "x" if checked else " " result.append(f"{indent}- [{checkbox}] {content}")
nested_items = item.get("items", []) if nested_items: result.extend( process_list_items(nested_items, style, level + 1, counter) )
return result
if not isinstance(json_to_convert, dict): try: json_to_convert = deep_json_load(json_to_convert) except (json.JSONDecodeError, TypeError) as e: raise ValueError( { "error": "Invalid JSON", "original": json_to_convert, } ) from e
for block in json_to_convert.get("blocks", []): block_type = block.get("type") block_data = block.get("data", {}) variant = block.get("tunes", {}).get("textVariant", "")
if block_type == "paragraph": text = block_data.get("text", "") text = ( text.replace("<i>", "*") .replace("</i>", "*") .replace("<b>", "**") .replace("</b>", "**") .replace('<code class="inline-code">', "`") .replace("</code>", "`") .replace('<u class="cdx-underline">', "") .replace("</u>", "") .replace('<mark class="cdx-marker">', "") .replace("</mark>", "") )
url_pattern = r'<a href="([^"]+)">([^<]+)</a>' text = re.sub(url_pattern, r"[\2](\1)", text)
if variant in ["citation", "call-out", "details"]: prefix = "" if variant == "call-out": prefix = "💡 " elif variant == "details": prefix = "ℹ️ "
markdown_parts.append(f"> {prefix}{text}\n") else: markdown_parts.append(f"{text}\n")
elif block_type == "header": level = block_data.get("level", 1) text = block_data.get("text", "") markdown_parts.append(f"{'#' * level} {text}\n")
elif block_type == "list": items = block_data.get("items", []) style = block_data.get("style") list_items = process_list_items(items, style) markdown_parts.extend(item + "\n" for item in list_items)
elif block_type == "table": content = block_data.get("content", []) with_headings = block_data.get("withHeadings", False)
if content: if with_headings: heading = content.pop(0) header_row = " | ".join(heading) markdown_parts.append(f"| {header_row} |") else: header_row = " | ".join(["<!-- -->"] * len(content[0])) markdown_parts.append(f"| {header_row} |")
separator = " | ".join(["---"] * len(content[0])) markdown_parts.append(f"| {separator} |")
for row in content: data_row = " | ".join(row) markdown_parts.append(f"| {data_row} |")
markdown_parts.append("\n")
elif block_type == "code": code = block_data.get("code", "") markdown_parts.append(f"```\n{code}\n```\n")
return "\n".join(markdown_parts)Convert to Plain Text: convert_editorjs_to_plain
import jsonimport re
def convert_editorjs_to_plain(json_to_convert): """ Convert EditorJS content to plain text format.
This function converts EditorJS JSON content to plain text format, removing all formatting and supporting various block types.
Parameters ---------- json_to_convert : dict or str JSON from EditorJS to convert to plain text.
Returns ------- str Plain text formatted description. """ if not json_to_convert: return ""
block_texts = []
def normalize_text(text): """Normalize text for plain-text export while preserving line breaks.""" text = html.unescape(text or "") text = text.replace("\u00A0", " ") text = text.replace("\r\n", "\n").replace("\r", "\n")
normalized_lines = [] for line in text.split("\n"): normalized_line = re.sub(r"[ \t]+", " ", line).strip() normalized_lines.append(normalized_line)
return "\n".join(normalized_lines).strip()
def process_list_items(items, level=0): result = [] for item in items: content = normalize_text(item.get("content", "")) indent = " " * level result.append(f"{indent}- {content}")
nested_items = item.get("items", []) if nested_items: result.extend(process_list_items(nested_items, level + 1)) return result
if not isinstance(json_to_convert, dict): try: json_to_convert = deep_json_load(json_to_convert) except (json.JSONDecodeError, TypeError) as e: raise ValueError( { "error": "Invalid JSON", "original": json_to_convert, } ) from e
for block in json_to_convert.get("blocks", []): block_type = block.get("type") block_data = block.get("data", {})
if block_type == "paragraph": text = block_data.get("text", "") text = ( text.replace("<br>", "\n") .replace("<i>", "") .replace("</i>", "") .replace("<b>", "") .replace("</b>", "") .replace('<code class="inline-code">', "") .replace("</code>", "") .replace('<u class="cdx-underline">', "") .replace("</u>", "") .replace('<mark class="cdx-marker">', "") .replace("</mark>", "") )
# Preserve links as plain text with URL def replace_link(match): href = match.group(1).strip() label = match.group(2).strip() if not href: return label if not label or label == href: return href return f"{label} ({href})"
text = re.sub(r'<a href="([^"]+)">([^<]*)</a>', replace_link, text) block_texts.append(normalize_text(text))
elif block_type == "header": text = block_data.get("text", "") block_texts.append(normalize_text(text))
elif block_type == "list": items = block_data.get("items", []) list_items = process_list_items(items) block_texts.append("\n".join(list_items).strip())
elif block_type == "table": content = block_data.get("content", []) table_rows = [ normalize_text(" ".join(str(cell) for cell in row)) for row in content ] block_texts.append("\n".join(table_rows).strip())
elif block_type == "code": code = block_data.get("code", "") block_texts.append(normalize_text(code))
return "\n\n".join(part for part in block_texts if part)Note: These EditorJS conversion functions use the deep_json_load helper function shown above. Make sure to include it in your plugin if you use any of the conversion functions.
Metadata Field Slug Structure in Plugins
Section titled “Metadata Field Slug Structure in Plugins”Each plugin can define metadata fields that are uniquely identified using a slug — a structured, machine-readable identifier.
The general form of a slug is:
<plugin_name>__[<metadata_object_type>__]<field_name>Components
Section titled “Components”-
<plugin_name>- The first part of the slug must always be the plugin name.
- It identifies which plugin the field belongs to.
- Example:
zenodo,darwin_core
-
<metadata_object_type>(optional)- Used when the plugin defines fields for a specific metadata object type.
- If included, it appears between double underscores after the plugin name.
- Example:
dwcindarwin_core__dwc__vertical_datum
-
<field_name>- The actual name of the metadata field.
- Should be descriptive, lowercase, and use underscores instead of spaces.
- Example:
title,vertical_datum
Examples of Valid Slugs
Section titled “Examples of Valid Slugs”zenodo__title→ Plugin:zenodo, Field:titledarwin_core__dwc__vertical_datum→ Plugin:darwin_core, Metadata Object Type:dwc, Field:vertical_datummyplugin__fieldname→ Plugin:myplugin, Field:fieldnamemyplugin__object_type__fieldname→ Plugin:myplugin, Metadata Object Type:object_type, Field:fieldname
- Use only lowercase letters, digits, underscores, or hyphens.
- The slug must start with the plugin name followed by
__. - Only one or two double-underscore sections are allowed:
plugin__fieldplugin__metadata_object_type__field
- Example of invalid slugs:
Plugin__Title(uppercase letters)plugin__metadata__object__field(too many sections)plugin-field(missing double underscores)
This ensures that all plugin metadata fields are uniquely and predictably identified across madbot.
ENA Plugin
Section titled “ENA Plugin”ENA Overview
Section titled “ENA Overview”The ENA (European Nucleotide Archive) plugin provides submission support for biological sequence data. Its metadata field schemas are auto-generated from ENA’s public checklist API rather than written by hand, ensuring they always reflect the official ENA specification.
Each ENA sample field has a single canonical slug shared across all checklists that include it:
ena__sample__elevationena__sample__collection_dateThe checklist code (erc000029, erc000011, etc.) is not part of the slug. Which checklists use a field and at which requirement level (mandatory / recommended / optional) is stored separately in madbot_ena/checklists.py.
Schema Generation
Section titled “Schema Generation”The generator script fetches every ENA sample checklist XML from the EBI API, deduplicates fields that appear in multiple checklists, and writes one JSON schema file per unique field.
What the script does, step by step:
- Fetches all ENA checklist XMLs from
https://www.ebi.ac.uk/ena/browser/api/xml/ERC0000<N>for N = 11 to 59. - Parses each field inside every
<FIELD_GROUP>block — name, description, format, restriction pattern, allowed values, and units. - Deduplicates fields by structural fingerprint. Two fields with the same name and the same validation rules share one output file (e.g.
elevation.json) with one canonical slug (ena__sample__elevation). - Writes one JSON schema file per unique field under:
madbot_ena/static/json/schema/sample/
- Updates
patterns.jsonan internal registry that tracks which regex patterns are used across fields and which checklists reference them, used to generate human-readable help text. - Writes
madbot_ena/checklists.pya plain Python dict mapping each checklist code to its title, description, and its fields pre-bucketed by requirement level. - Writes
madbot_ena/static/json/schema/checklist.jsonthe JSON schema for the checklist selector field shown in the submission UI.
Note: Checklist names in
patterns.jsonmay appear as either a human-readable label (e.g.GSC MIxS host associated) or as a bare ERC code (e.g.ERC000013). This reflects what ENA’s API returns in the<LABEL>element for each checklist, some checklists have a descriptive label registered, others only expose their accession code.
Running the Generator
Section titled “Running the Generator”Run the script from the root of the madbot_ena plugin:
pixi run python madbot_ena/utils/generateMetadataJsonSchema.pyRe-run it whenever ENA updates its checklists (new fields, new allowed values, new checklists). The output files should be committed to the repository.
Requirements: the script needs network access to reach
https://www.ebi.ac.uk. It has no other external dependencies beyond the packages already declared inpyproject.toml.
Schema File Structure
Section titled “Schema File Structure”Each generated file in madbot_ena/static/json/schema/sample/ is a standard JSON Schema fragment. The level (mandatory / recommended / optional) is not stored in the schema file, it is a property of the relationship between a field and a checklist, and lives exclusively in checklists.py. Depending on the field type, the schema takes one of the following shapes:
Text field with a regex restriction:
{ "title": "Elevation", "originalTitle": "elevation", "description": "The elevation of the sampling site as measured by the vertical distance from mean sea level.", "oneOf": [ { "title": "Value available", "type": "object", "properties": { "status": { "const": "Value available" }, "value": { "type": "string", "pattern": "-?[0-9]+(\\.[0-9]+)?" } } }, { "title": "Value unavailable", "type": "object", "properties": { "status": { "const": "Value unavailable" } } } ]}Text choice field (controlled vocabulary):
{ "title": "Library layout", "originalTitle": "library_layout", "description": "Sequencing library layout.", "type": "string", "enum": ["SINGLE", "PAIRED"]}Field with a unit:
{ "title": "Sample volume or weight for DNA extraction", "originalTitle": "sample_volume_or_weight_for_dna_extraction", "description": "Volume (ml) or mass (g) of total collected sample processed for DNA extraction.", "type": "object", "properties": { "value": { "type": "string", "pattern": "[0-9]+(\\.[0-9]+)?" }, "unit": { "type": "string", "enum": ["g", "ml", "mg", "ng", "l"] } }, "required": ["value", "unit"]}The key properties present in every schema file are:
| Key | Description |
|---|---|
title | Human-readable field name (derived from the ENA LABEL) |
originalTitle | Raw field name exactly as returned by ENA’s API |
description | Field description from ENA |
type / oneOf | JSON Schema validation rules for the field value |
pattern | Regex constraint on the value, when applicable |
enum | Allowed values for controlled vocabulary fields |
fieldgroup | ENA field group the field belongs to (e.g. Organism) |
Note: There is no
levelkey in individual schema files. Requirement levels (mandatory / recommended / optional) are a checklist-scoped property and are stored inchecklists.pyonly.
Checklist Configuration
Section titled “Checklist Configuration”madbot_ena/checklists.py is the single source of truth for which fields each checklist requires, and at which level. It is auto-generated, do not edit manually.
ENA_CHECKLIST_CONFIGS = { "erc000011": { "title": "ERC000011 - GSC MIxS plant associated", "description": "Minimum information about a MIxS plant-associated sample.", "fields": { "mandatory": ["ena__sample__collection_date", "ena__sample__lat_lon", ...], "recommended": ["ena__sample__elevation", ...], "optional": ["ena__sample__depth", ...], }, }, ...}At runtime, plugins.py uses two utility classes from madbot_ena/utils/checklist_handler.py to convert this dict into fully typed metadata object types:
ChecklistFieldAssociation
Couples a FieldRepresentation (resolved from the slug at runtime after add_fields() has loaded all JSON schema files) to a requirement level string. It is the explicit, typed representation of the statement “this field is mandatory for this checklist”.
@dataclassclass ChecklistFieldAssociation: field: FieldRepresentation # resolved from slug at runtime level: Literal["mandatory", "recommended", "optional"]ChecklistMetadataObjectType
Subclass of MetadataObjectType that takes a checklist_code and a list of ChecklistFieldAssociation objects. It derives the three bucket lists (mandatory_fields, recommended_fields, optional_fields) automatically in model_post_init, preserving full backward compatibility with all existing consumers.
class ChecklistMetadataObjectType(MetadataObjectType): checklist_code: str field_associations: list[ChecklistFieldAssociation]
def model_post_init(self, __context: Any) -> None: self.mandatory_fields = [a.field for a in self.field_associations if a.level == "mandatory"] self.recommended_fields = [a.field for a in self.field_associations if a.level == "recommended"] self.optional_fields = [a.field for a in self.field_associations if a.level == "optional"]build_metadata_object_types() in plugins.py
Iterates over ENA_CHECKLIST_CONFIGS and produces one ChecklistMetadataObjectType per checklist. No manual field bucketing is needed, adding a new checklist only requires re-running the generator.
Keeping Plugins Up to Date
Section titled “Keeping Plugins Up to Date”Whenever plugin field schemas are added, removed, or modified, including after re-running the ENA schema generator, you must reload the plugins into the database:
pixi run madbot reload_pluginsThis command synchronises the PluginRawSchema and PluginMetadataField tables with the current state of all installed plugins. Without it, any new or changed fields will not be visible in the application.

