plugins_api.data_plugins
Data plugins.
This module contains plugin classes for data management operations.
Attributes
| Name | Description |
|---|---|
DEFAULT_BATCH_READ_SIZE | Default maximum number of ids madbot sends per batch-read call. |
Classes
| Name | Description |
|---|---|
DataPlugin | Abstract class defining the minimum API for a data plugin. |
DEFAULT_BATCH_READ_SIZEattributemodule attribute#
DEFAULT_BATCH_READ_SIZE = 100Default maximum number of ids madbot sends per batch-read call.
Plugins can override DataPlugin.get_batch_read_size when their source has
a different limit, or return None to disable chunking.
DataPluginclass#
class DataPlugin(connection_parameters: dict[str, Any])Bases: ConnectionPlugin
Abstract class defining the minimum API for a data plugin.
Data plugins connect to external data sources and provide access to files and directories. They are used to browse, list, and download data from external systems.
browse_data_objectsmethodabstract#
def browse_data_objects(parent_id: str | None) -> list[PluginDataObject]Return the direct children of a parent data object.
Parameters
Returns
List[PluginDataObject]- Direct children of
parent_id, or of the root whenparent_idisNone.
download_datamethodabstract#
def download_data(external_dataobject_id: str, destination_path: str) -> strDownload a data object to local storage and returns an download report.
It is strongly recommended to implement data integrity verification in this method. This may include, but is not limited to:
- Checksum validation (e.g., MD5, SHA256) if available.
- File size comparison with the source.
- Any other consistency checks relevant to the data source.
Implementing integrity checks helps prevent data corruption or incomplete downloads, ensuring reliable processing of the imported data.
Parameters
Returns
str- Path to the local file.
Raises
NotImplementedError- This method must be implemented in a subclass.
get_batch_read_sizemethodclassmethod#
def get_batch_read_size() -> int | NoneReturn how many ids madbot should send per batch-read call.
Madbot uses this when calling get_data_objects_by_ids,
get_metadata_for_data_objects and
get_data_associations_for_data_objects for a long list of ids
on one connection.
get_data_associations_for_data_objectsmethod#
def get_data_associations_for_data_objects(object_ids: list[str]) -> dict[str, list[PluginDataAssociation] | PluginError]Return data associations for each requested data object.
Each association represents a group of data objects that should be associated together. A data object can participate in multiple associations, each returned as a separate PluginDataAssociation.
Associations should be returned bidirectionally - if data objects A, B, and C are associated together, then when querying A, B, or C, they should all return the same association group containing all three.
Returns
dict[str, list[PluginDataAssociation] | PluginError]- Mapping from object id to its association list, or to a per-id
PluginError. Returns{}whenobject_idsis empty. The default implementation returns an empty list for each distinct id, making this method optional for plugins that have no associations. Only requested ids may appear in the result.
Examples
If data object A is associated with B and C in one group:
>>> {"A": [PluginDataAssociation(associated_external_ids=["A", "B", "C"])]}If data object A has two separate associations (one with B, one with C):
>>> {... "A": [... PluginDataAssociation(associated_external_ids=["A", "B"]),... PluginDataAssociation(associated_external_ids=["A", "C"]),... ]... }With metadata:
>>> {... "A": [... PluginDataAssociation(... associated_external_ids=["A", "B", "C"],... metadata=[... PluginMetadata(... field=MetadataFieldID(... slug="association_type",... source="referential",... name="Association Type",... ),... value=MetadataValue(value="paired"),... )... ],... )... ]... }get_data_objects_by_idsmethodabstract#
def get_data_objects_by_ids(object_ids: list[str]) -> dict[str, PluginDataObject | PluginError]Return detailed information for each requested data object.
Every distinct id from object_ids must appear in the result, either
as a PluginDataObject or as the PluginError raised while reading
that id (for example DataNotFoundError or PluginPermissionError).
The result must contain only those requested ids. Extras should be
ignored if the remote API returns more.
Errors that affect the whole source (see SOURCE_ERRORS) must be
raised, not stored in the result.
Plugins whose source has no batch API can implement this in one line
with _read_each:
def get_data_objects_by_ids(self, object_ids): return self._read_each(object_ids, self._read_data_object)When the source can answer several ids in one request, no need to call
_read_each. Instead, call the remote API directly:
def get_data_objects_by_ids(self, object_ids): unique_ids = list(dict.fromkeys(object_ids)) try: remote = self.client.fetch_many(unique_ids) except ConnectionError as exc: raise UnreachableError(self.host) from exc
result = {} for object_id in unique_ids: item = remote.get(object_id) if item is None: result[object_id] = DataNotFoundError(object_id) else: result[object_id] = self._to_plugin_data_object(item) return resultParameters
Returns
dict[str, PluginDataObject | PluginError]- Mapping from object id to the object or a per-id plugin error.
Returns
{}whenobject_idsis empty.
get_metadata_for_data_objectsmethod#
def get_metadata_for_data_objects(object_ids: list[str]) -> dict[str, list[PluginMetadata] | PluginError]Return the metadata for each requested data object.
Returns
dict[str, list[PluginMetadata] | PluginError]- Mapping from object id to its metadata list, or to a per-id
PluginError. Returns{}whenobject_idsis empty. The default implementation returns an empty list for each distinct id, making this method optional for plugins that have no metadata. Only requested ids may appear in the result.
get_root_idmethodabstractstaticmethod#
def get_root_id() -> strReturn the unique identifier of the root data object.
Returns
str- Unique identifier of the root data object.
get_root_namemethodabstractstaticmethod#
def get_root_name() -> strReturn the name of the root data object.
Returns
str- Name of the root data object.
Inherited from plugins_api.base.ConnectionPlugin
closemethod#
plugins_api.base.ConnectionPluginView sourcedef close() -> NoneRelease connection resources (optional).
Plugins that hold long-lived connections (e.g. SSH, persistent HTTP
sessions) may override this method to close the underlying connection
and release file descriptors. Callers (e.g. core) should call
close() when done with the plugin instance.
The base implementation does nothing. Plugins that do not need to release resources need not override it.
get_connection_fieldsmethodabstractclassmethod#
plugins_api.base.ConnectionPluginView sourcedef get_connection_fields() -> tuple[list[ConnectionField], list[ConnectionField]]Retrieve the connection fields for the connection plugin.
Returns
tuple[list[ConnectionField], list[ConnectionField]](shared_fields, private_fields). Shared fields are instance-level (URL, host). Private fields are per-user secrets (token, password).
get_instance_namemethodabstract#
plugins_api.base.ConnectionPluginView sourcedef get_instance_name() -> strReturn the instance name of the connection plugin.
Returns
str- Instance name.
get_shared_parametersmethodabstract#
plugins_api.base.ConnectionPluginView sourcedef get_shared_parameters() -> dict[str, Any]Return all shared parameters in a cleaned form.
Returns
dict- Shared parameters cleaned.
is_sandboxmethodclassmethod#
plugins_api.base.ConnectionPluginView sourcedef is_sandbox(shared_parameters: dict[str, Any]) -> boolReturn whether the shared parameters are for a sandbox connection or not.
Parameters
| Name | Type | Description |
|---|---|---|
shared_parameters | dict | The shared parameters to check. |
Returns
bool- Plugin is sandbox.
on_before_store_credentialsmethodclassmethod#
plugins_api.base.ConnectionPluginView sourcedef on_before_store_credentials(shared_params: dict, private_params: dict) -> dictTransform the private parameters just before they are written to the Vault.
Allows the plugin to transform or enrich the private_params. By default, returns the params unchanged.
on_delete_credentialsmethodclassmethod#
plugins_api.base.ConnectionPluginView sourcedef on_delete_credentials(shared_params: dict, vault_secret: dict) -> NoneClean up just before the credentials are deleted from the Vault.
Allows the plugin to perform cleanup on the remote server side. By default, does nothing.
Inherited from plugins_api.base.MadbotPlugin
get_authormethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_author() -> strReturn plugin author.
Returns
str- Plugin author.
get_colormethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_color() -> strReturn the color of the plugin.
Typically a hex value such as "#3B82F6".
Returns
str- Plugin color.
get_descriptionmethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_description() -> strReturn plugin description.
Returns
str- Plugin description.
get_idmethodclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_id() -> strReturn the unique identifier of the plugin.
Returns
str- Unique identifier in the format ‘module.class’.
get_logomethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_logo() -> strReturn the static path of the plugin logo.
Path is relative to the plugin package static files.
Example: "images/ena.png".
Returns
str- Plugin logo static path.
get_namemethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_name() -> strReturn the plugin name.
Returns
str- Plugin name.
get_slugmethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_slug() -> strReturn the plugin slug.
Lowercase, no spaces (e.g. "ena", "omero").
Used as MetadataFieldID.source and as the schema source at reload.
Returns
str- Plugin slug.
get_statusmethodabstractclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_status() -> dict[str, PluginStatus]Return the maturity status of the plugin, keyed by capability type.
A plugin implementing several capability types at once (e.g. a class
inheriting from both SubmissionPlugin and
MetadataFieldDefinitionPlugin) must return one entry per type it
implements, using the same type strings as get_types().
Returns
dict[str, PluginStatus]- Mapping of plugin type (as returned by
get_types()) to its maturity status.
Examples
A plugin implementing a single type:
>>> print(self.get_status()){"data_plugin": PluginStatus.PREVIEW}A plugin implementing several types at once:
>>> print(self.get_status()){ "submission_plugin": PluginStatus.STABLE, "metadata_field_definition_plugin": PluginStatus.STABLE,}get_typesmethodclassmethod#
plugins_api.base.MadbotPluginView sourcedef get_types() -> list[str]Return the list of the plugin types in snake_case.
Returns
list- List of plugin types.

