Skip to content
Reference of plugins_api 3.0.0.

plugins_api.data_plugins

Data plugins.

This module contains plugin classes for data management operations.

Attributes

NameDescription
DEFAULT_BATCH_READ_SIZEDefault maximum number of ids madbot sends per batch-read call.

Classes

NameDescription
DataPluginAbstract class defining the minimum API for a data plugin.

DEFAULT_BATCH_READ_SIZEattributemodule attribute#

DEFAULT_BATCH_READ_SIZE = 100

Default maximum number of ids madbot sends per batch-read call.

Plugins can override DataPlugin.get_batch_read_size when their source has a different limit, or return None to disable chunking.

DataPluginclass#

class DataPlugin(connection_parameters: dict[str, Any])

Bases: ConnectionPlugin

Abstract class defining the minimum API for a data plugin.

Data plugins connect to external data sources and provide access to files and directories. They are used to browse, list, and download data from external systems.

browse_data_objectsmethodabstract#

def browse_data_objects(parent_id: str | None) -> list[PluginDataObject]

Return the direct children of a parent data object.

Parameters

NameTypeDescription
parent_idstr or NoneID of the parent object. When None, browse from the root of the source (same as passing get_root_id()).

Returns

List[PluginDataObject]
Direct children of parent_id, or of the root when parent_id is None.

download_datamethodabstract#

def download_data(external_dataobject_id: str, destination_path: str) -> str

Download a data object to local storage and returns an download report.

It is strongly recommended to implement data integrity verification in this method. This may include, but is not limited to:

  • Checksum validation (e.g., MD5, SHA256) if available.
  • File size comparison with the source.
  • Any other consistency checks relevant to the data source.

Implementing integrity checks helps prevent data corruption or incomplete downloads, ensuring reliable processing of the imported data.

Parameters

NameTypeDescription
external_dataobject_idstrUnique identifier of the data object in the external system.
destination_pathstrLocal path where the file should be saved.

Returns

str
Path to the local file.

Raises

NotImplementedError
This method must be implemented in a subclass.

get_batch_read_sizemethodclassmethod#

def get_batch_read_size() -> int | None

Return how many ids madbot should send per batch-read call.

Madbot uses this when calling get_data_objects_by_ids, get_metadata_for_data_objects and get_data_associations_for_data_objects for a long list of ids on one connection.

Returns

int or None
A positive integer to split the id list into chunks of that size, or None to pass the full list in a single call. The default is :data:DEFAULT_BATCH_READ_SIZE (100).

get_data_associations_for_data_objectsmethod#

def get_data_associations_for_data_objects(object_ids: list[str]) -> dict[str, list[PluginDataAssociation] | PluginError]

Return data associations for each requested data object.

Each association represents a group of data objects that should be associated together. A data object can participate in multiple associations, each returned as a separate PluginDataAssociation.

Associations should be returned bidirectionally - if data objects A, B, and C are associated together, then when querying A, B, or C, they should all return the same association group containing all three.

Parameters

NameTypeDescription
object_idslist[str]IDs of the data objects.

Returns

dict[str, list[PluginDataAssociation] | PluginError]
Mapping from object id to its association list, or to a per-id PluginError. Returns {} when object_ids is empty. The default implementation returns an empty list for each distinct id, making this method optional for plugins that have no associations. Only requested ids may appear in the result.

Examples

If data object A is associated with B and C in one group:

>>> {"A": [PluginDataAssociation(associated_external_ids=["A", "B", "C"])]}

If data object A has two separate associations (one with B, one with C):

>>> {
... "A": [
... PluginDataAssociation(associated_external_ids=["A", "B"]),
... PluginDataAssociation(associated_external_ids=["A", "C"]),
... ]
... }

With metadata:

>>> {
... "A": [
... PluginDataAssociation(
... associated_external_ids=["A", "B", "C"],
... metadata=[
... PluginMetadata(
... field=MetadataFieldID(
... slug="association_type",
... source="referential",
... name="Association Type",
... ),
... value=MetadataValue(value="paired"),
... )
... ],
... )
... ]
... }

get_data_objects_by_idsmethodabstract#

def get_data_objects_by_ids(object_ids: list[str]) -> dict[str, PluginDataObject | PluginError]

Return detailed information for each requested data object.

Every distinct id from object_ids must appear in the result, either as a PluginDataObject or as the PluginError raised while reading that id (for example DataNotFoundError or PluginPermissionError). The result must contain only those requested ids. Extras should be ignored if the remote API returns more.

Errors that affect the whole source (see SOURCE_ERRORS) must be raised, not stored in the result.

Plugins whose source has no batch API can implement this in one line with _read_each:

def get_data_objects_by_ids(self, object_ids):
return self._read_each(object_ids, self._read_data_object)

When the source can answer several ids in one request, no need to call _read_each. Instead, call the remote API directly:

def get_data_objects_by_ids(self, object_ids):
unique_ids = list(dict.fromkeys(object_ids))
try:
remote = self.client.fetch_many(unique_ids)
except ConnectionError as exc:
raise UnreachableError(self.host) from exc
result = {}
for object_id in unique_ids:
item = remote.get(object_id)
if item is None:
result[object_id] = DataNotFoundError(object_id)
else:
result[object_id] = self._to_plugin_data_object(item)
return result

Parameters

NameTypeDescription
object_idslist[str]IDs of the data objects to read. May be empty or contain duplicates.

Returns

dict[str, PluginDataObject | PluginError]
Mapping from object id to the object or a per-id plugin error. Returns {} when object_ids is empty.

get_metadata_for_data_objectsmethod#

def get_metadata_for_data_objects(object_ids: list[str]) -> dict[str, list[PluginMetadata] | PluginError]

Return the metadata for each requested data object.

Parameters

NameTypeDescription
object_idslist[str]IDs of the data objects.

Returns

dict[str, list[PluginMetadata] | PluginError]
Mapping from object id to its metadata list, or to a per-id PluginError. Returns {} when object_ids is empty. The default implementation returns an empty list for each distinct id, making this method optional for plugins that have no metadata. Only requested ids may appear in the result.

get_root_idmethodabstractstaticmethod#

def get_root_id() -> str

Return the unique identifier of the root data object.

Returns

str
Unique identifier of the root data object.

get_root_namemethodabstractstaticmethod#

def get_root_name() -> str

Return the name of the root data object.

Returns

str
Name of the root data object.
Inherited from plugins_api.base.ConnectionPlugin

closemethod#

def close() -> None

Release connection resources (optional).

Plugins that hold long-lived connections (e.g. SSH, persistent HTTP sessions) may override this method to close the underlying connection and release file descriptors. Callers (e.g. core) should call close() when done with the plugin instance.

The base implementation does nothing. Plugins that do not need to release resources need not override it.

get_connection_fieldsmethodabstractclassmethod#

def get_connection_fields() -> tuple[list[ConnectionField], list[ConnectionField]]

Retrieve the connection fields for the connection plugin.

Returns

tuple[list[ConnectionField], list[ConnectionField]]
(shared_fields, private_fields). Shared fields are instance-level (URL, host). Private fields are per-user secrets (token, password).

get_instance_namemethodabstract#

def get_instance_name() -> str

Return the instance name of the connection plugin.

Returns

str
Instance name.

get_shared_parametersmethodabstract#

def get_shared_parameters() -> dict[str, Any]

Return all shared parameters in a cleaned form.

Returns

dict
Shared parameters cleaned.

is_sandboxmethodclassmethod#

def is_sandbox(shared_parameters: dict[str, Any]) -> bool

Return whether the shared parameters are for a sandbox connection or not.

Parameters

NameTypeDescription
shared_parametersdictThe shared parameters to check.

Returns

bool
Plugin is sandbox.

on_before_store_credentialsmethodclassmethod#

def on_before_store_credentials(shared_params: dict, private_params: dict) -> dict

Transform the private parameters just before they are written to the Vault.

Allows the plugin to transform or enrich the private_params. By default, returns the params unchanged.

on_delete_credentialsmethodclassmethod#

def on_delete_credentials(shared_params: dict, vault_secret: dict) -> None

Clean up just before the credentials are deleted from the Vault.

Allows the plugin to perform cleanup on the remote server side. By default, does nothing.

Inherited from plugins_api.base.MadbotPlugin

get_authormethodabstractclassmethod#

def get_author() -> str

Return plugin author.

Returns

str
Plugin author.

get_colormethodabstractclassmethod#

def get_color() -> str

Return the color of the plugin.

Typically a hex value such as "#3B82F6".

Returns

str
Plugin color.

get_descriptionmethodabstractclassmethod#

def get_description() -> str

Return plugin description.

Returns

str
Plugin description.

get_idmethodclassmethod#

def get_id() -> str

Return the unique identifier of the plugin.

Returns

str
Unique identifier in the format ‘module.class’.
def get_logo() -> str

Return the static path of the plugin logo.

Path is relative to the plugin package static files. Example: "images/ena.png".

Returns

str
Plugin logo static path.

get_namemethodabstractclassmethod#

def get_name() -> str

Return the plugin name.

Returns

str
Plugin name.

get_slugmethodabstractclassmethod#

def get_slug() -> str

Return the plugin slug.

Lowercase, no spaces (e.g. "ena", "omero"). Used as MetadataFieldID.source and as the schema source at reload.

Returns

str
Plugin slug.

get_statusmethodabstractclassmethod#

def get_status() -> dict[str, PluginStatus]

Return the maturity status of the plugin, keyed by capability type.

A plugin implementing several capability types at once (e.g. a class inheriting from both SubmissionPlugin and MetadataFieldDefinitionPlugin) must return one entry per type it implements, using the same type strings as get_types().

Returns

dict[str, PluginStatus]
Mapping of plugin type (as returned by get_types()) to its maturity status.

Examples

A plugin implementing a single type:

>>> print(self.get_status())
{"data_plugin": PluginStatus.PREVIEW}

A plugin implementing several types at once:

>>> print(self.get_status())
{
"submission_plugin": PluginStatus.STABLE,
"metadata_field_definition_plugin": PluginStatus.STABLE,
}

get_typesmethodclassmethod#

def get_types() -> list[str]

Return the list of the plugin types in snake_case.

Returns

list
List of plugin types.