core-plugins.md

Core Plugins
============

## Motivation

The `service.properties` file of openBIS Application Server (AS) and
Data Store Server (DSS) can be quite big because of all the
configuration data for maintenance tasks, drop-boxes, reporting and
processing plugins, etc. Making this configuration more modular will
improve the structure. It would also allow to have core plugins shipped
with distribution and customized plugins separately. This makes
maintenance of these plugins more independent. For example, a new
maintenance task plugin can be added in an update without any need for
an admin to put the configuration data manually into the
`service.properties` file.

## Core Plugins Folder Structure

All plugins whether they are a part of the distribution or added and
maintained are stored in the folder usually called `core-plugins`.
Standard (i.e. core) plugins are part of the distribution. During
installation the folder `core-plugins` is unpacked as a sibling folder
of `openBIS-server` and` datastore_server`.

The folder structure is organized as follows:

-   The file `core-plugins.properties` containing the following
    properties:  
    -   `enabled-modules`: comma-separated list of regular expressions
        for all enabled modules.
    -   `disabled-core-plugins`: comma-separated list of disabled
        plugins. All plugins are disabled for which the beginning of
        full plugin ID matches one of the terms of this list. To disable
        initialization of master data of a module - disable it's core
        plugin "initialize-master-data"
-   The children of `core-plugins` are folders denoting modules like the
    standard technologies, `proteomics` and `screening`. For
    customization, any module can be added.
-   Each module folder has children which are numbered folders. The
    number denotes the version of the plugins of that module. The
    version with the largest number will be used. Different modules can
    have different largest version numbers.
-   Every version folder has the subfolder `as` and/or` dss `which have
    subfolders for the various types of plugins. The types are different
    for AS and DSS:  
    -   AS:  
        -   `maintenance-tasks`: Maintenance tasks triggered by some
            time schedule.`                            `Property `class`
            denotes fully-qualified class name of a class implementing
            `ch.systemsx.cisd.common.maintenance.IMaintenanceTask`.  
            For more details see [Maintenance
            Tasks](/display/openBISDoc2010/Maintenance+Tasks).
        -   `dss-data-sources`: Definition of data sources with
            corresponding data source definitions for DSS. For more
            details see [Installation and Administrator Guide of the
            openBIS
            Server](/display/openBISDoc2010/Installation+and+Administrator+Guide+of+the+openBIS+Server).
        -   `query-databases`: Databases for SQL queries. For more
            details see [Custom Database
            Queries](/display/openBISDoc2010/Custom+Database+Queries).
        -   `custom-imports`: Custom file imports to DSS via Web
            interface. For more details see [Custom
            Import](/display/openBISDoc2010/Custom+Import).
        -   `services`: Custom services. For more details see [Custom
            Application Server
            Services](/display/openBISDoc2010/Custom+Application+Server+Services).
        -   `webapps`: HTML5 applications that use the openBIS API. For
            more details see [openBIS
            webapps](/display/openBISDoc2010/openBIS+webapps).
        -   `miscellaneous`: Any additional properties.
    -   `DSS:`  
        -   `drop-boxes`: ETL server threads for registration of data
            sets.`                            `
        -   `reporting-plugins`: Reports visible in openBIS.  
            Property `class` denotes fully-qualified class name of a
            class implementing
            `ch.systemsx.cisd.openbis.dss.generic.server.plugins.tasks.IReportingPluginTask`.  
            For more details see [Reporting
            Plugins](/display/openBISDoc2010/Reporting+Plugins).
        -   `processing-plugins`: Processing tasks triggered by
            users.`                            `Property `class` denotes
            fully-qualified class name of a class implementing
            `ch.systemsx.cisd.openbis.dss.generic.server.plugins.tasks.IProcessingPluginTask`.`                            `For
            more details see [Processing
            Plugins](/display/openBISDoc2010/Processing+Plugins).`                            `
        -   `maintenance-tasks`: Maintenance tasks triggered by some
            time schedule.`              `Property `class` denotes
            fully-qualified class name of a class implementing
            `ch.systemsx.cisd.common.maintenance.IMaintenanceTask`.`                            `For
            more details see [Maintenance
            Tasks](/display/openBISDoc2010/Maintenance+Tasks).
        -   `search-domain-services`: Services for variaous search
            domains (e.g. search on sequence databases using BLAST).  
            Property `class` denotes fully-qualified class name of a
            class implementing
            `ch.systemsx.cisd.openbis.dss.generic.shared.api.internal.v2.ISearchDomainService`.
        -   `data-sources`: Internal or external database sources.
        -   `services`: Services based on servlets.  
            Property `class` denotes fully-qualified class name of a
            class implementing `javax.servlet.Servlet`.
        -   `imaging-overview-plugins`: Data set type specific provider
            of the overview image of a data set.  
            Property `class` denotes fully-qualified class name of a
            class implementing
            `ch.systemsx.cisd.openbis.dss.generic.server.IDatasetImageOverviewPlugin`.
        -   `file-system-plugins`: Provider of a custom DSS file system
            (FTP/SFTP) view hierarchy.  
            Property `class` denotes fully-qualified class name of a
            class
            implementing `ch.systemsx.cisd.openbis.dss.generic.server.fs.IResolverPlugin`  
            Property code denotes the name of the top-level directory
            under which the custom hierarchy will be visible
        -   `miscellaneous`: Any additional
            properties.`                            `
-   Folders of each of these types can have an arbitrary number of
    subfolders. But if the type folder is present it should have at
    least one subfolder. Each defining one plugin. The name of these
    subfolders define the plugin ID. It has to be unique over all
    plugins independent of module and plugin type. It should not contain
    the characters space ' ', comma '`,`', and equal sign '`=`'.
-   Each plugin folder should contain at least the file
    `plugin.properties`. There could be additional files (referred in
    `plugin.properties`) but no subfolders.

Here is an example of a typical structure of a core plugins folder:

    core-plugins
      core-plugins.properties
      proteomics
        1
          as
            initialize-master-data.py
          dss
            drop-boxes
              ms-injection
                plugin.properties
            maintenance-tasks
              data-set-clean-up
                plugin.properties
      screening
        1
          core-plugin.properties
          as
            initialize-master-data.py
            maintenance-tasks
              material-reporting
                mapping.txt
                plugin.properties
            custom-imports
              myCustomImport
                plugin.properties
          dss
            drop-boxes
              hcs-dropbox
                lib
                  custom-lib.jar
                hcs-dropbox.py
                plugin.properties

You might noticed the file `initialize-master-data.py` in AS core
plugins sections  in this example. It is a script to register master
data in the openBIS core database. For more details see [Installation
and Administrator Guide of the openBIS
Server](/display/openBISDoc2010/Installation+and+Administrator+Guide+of+the+openBIS+Server).

Each plugin can refer to any number of files. These files are part of
the plugin folder. In `plugin.properties` they are referred relative to
the plugin folder, that is by file name. Example:

**plugin.properties**

    incoming-dir = ${incoming-root-dir}/incoming-hcs
    incoming-data-completeness-condition = auto-detection
    top-level-data-set-handler = ch.systemsx.cisd.openbis.dss.etl.jython.JythonPlateDataSetHandler
    script-path = hcs-dropbox.py
    storage-processor = ch.systemsx.cisd.openbis.dss.etl.PlateStorageProcessor
    storage-processor.data-source = imaging-db
    storage-processor.define-channels-per-experiment = false

## Merging Configuration Data

At start up of AS and DSS merges  the content of  `service.properties`
with the content of all `plugin.properties` of the latest version per
enabled module. Plugin properties can be deleted by adding
`<plugin ID>.<plugin property key> = __DELETED__` to service.properties.
Example:

    simple-dropbox.incoming-data-completeness-condition = __DELETED__

This leads to a deletion of the property
`incoming-data-completeness-condition` specified in `plugins.properties`
of the plugin `simple-dropbox`.

Merging is done by injection the properties of `plugin.properties` into
`service.properties `by adding the plugin ID as a prefix to the property
key (not for `miscellaneous). `For example, the property `script-path`
of plugin `hcs-dropbox` becomes `hcs-dropbox.script-path`. References to
files inside the plugin are replaced by a path relative to the working
directory. For the various plugin types (except `miscellaneous`) the
plugin ID is appended to the related property in  `service.properties`
for this plugin type. For example, plugins of type `drop-boxes` are
added to the property `inputs`.

## Enabling Modules and Disabling Plugins

There are three methods to control which plugins are available and witch
not:

-   enabling by property `enabled-modules` in` core-plugins.properties`:
    This enables all plugins of certain modules.
-   disabling by property `disabled-core-plugins` in
    `       core-plugins.properties` : This allows to disable on a fine
    grade level specific plugins.
-   disabling by marker file: Plugin developers should use this method
    when developing new plugins.

### Enabling Modules

The property `enabled-modules` in `core-plugins.properties` is a
comma-separated list of regular expressions denoting modules. All
plugins in a module folder of `core-plugins` folder are enabled if the
module name matches one of these regular expressions. If this list is
empty or the property hasn't been specified no core-plugin will be used.
Note, that this property is manipulated by openBIS Installer for
Standard Technologies. Example:

**service.properties**

    enabled-modules = screening, proteomics, dev-module-.*

### Disabling Core Plugins by Property

The property `disabled-core-plugins` in `core-plugins.properties` allows
to disable plugins selectively either by module name, module combined
with plugin type or full plugin ID. Example:

**service.properties**

    disabled-core-plugins = screening, proteomics:reporting-plugins, proteomics:maintenance-tasks:data-set-clean-up

### Disabling Core Plugins by Marker File

The empty marker file `disabled` in a certain plugin folder disables the
particular plugin.

## Core Plugin Dependency

A core plugin can depend on another core plugin. The dependency is
specified in `<module>/<version>/core-plugin.properties`. It has a
property named `required-plugins`. Its value is a comma-separated list
of core-plugins on which it depends. The dependency can be specified
selectively either by module name, module combined with plugin type or
full plugin ID. Example:

**core-plugin.properties**

    required-plugins = module-a, module-b:initialize-master-data, module-b:reporting-plugins, module-a:drop-boxes:generic

## Rules for Plugin Writers

As a consequence of the way plugins are merged with 
`service.properties` writers of plugins have to obey the following
rules:

-   Plugin IDs have to be unique among all plugins whether they are
    defined in `service.properties` or as core plugins. The only
    exceptions are plugins of type `miscellaneous`.
-   In `plugin.properties` other properties can be referred by the usual
    `${<property key>`} notation. The referred property can be in
    `service.properties` or in any `plugin.properties`.
-   As convention use `${incoming-root-dir`} when defining the incoming
    folder for a drop box.
-   Refer files in `plugin.properties` only by names and add them as
    siblings of `plugin.properties` to the plugin folder. Note, that
    different plugins can refer files with the same name. There will be
    no ambiguity which file is meant.
-   In order to be completely independent from updates of the core
    plugins which are part of the distribution create your own module,
    like `my-plugins`, and put all your plugins there. Do not forget to
    add your module to the property `enabled-modules` in
    `core-plugins.properties`.

## Using Java libraries in Core Plugins

OpenBIS allows you to include Java libraries in core plugin folders. The
\*.jar files have to be stored in "&lt;code plugin folder&gt;/lib"
folder. For instance, in order to use "my-lib.jar" in "my-dropbox" a
following file structure is needed:

**service.properties**

    my-technology
        1
          dss
            drop-boxes
              my-dropbox
                lib
                  my-lib.jar
                dropbox.py
                plugin.properties

Having this structure, Java classes from "my-lib.jar" can be imported
and used in "dropbox.py" script.

NOTICE: Currently this feature is only supported for DSS core
plugins. Under the hood, a symbolic link to a jar file is created in
"datastore\_server/lib" folder during DSS startup.