jython-based-reporting-and-processing-plugins.md

Jython-based Reporting and Processing Plugins

Overview

It is possible to implement logic of reporting and processing plugins as Jython scripts instead of providing new classes implementing IReportingPluginTask and IProcessingPluginTask Java interfaces.

The scripts require only a single method to be implemented and have access to a few services that simplify tasks such as data retrieval, creation of tables or sending emails:

• access to files of a data set is done via IHierarchicalContent interface (through IDataSet.getContent())

• access to openBIS AS via ISearchService (through searchService and searchServiceUnfiltered variables),

• access to data sources specified in DSS via IDataSourceQueryService (through queryService variable),

• creation of tables in reporting script via ISimpleTableModelBuilderAdaptor (provided as a function argument),

• sending emails via IMailService (through mailServicevariable)

  • it is easy to send a file or a text as an attachment to the user (subject and text body can be provided optionally)
  • it is also possible to use a reporting script as a processing plugin sending the report as an attachment to the user

• checking user access privileges via the IAuthorizationService

   (available from the authorizationService variable).

All jython plugins use Jython version configured by the service.properties property jython-version which can be either 2.5 or 2.7.

Configuration

Jython-based plugins are configured in exactly the same way as other reporting and processing plugins. Apart from standard mandatory plugin properties one needs to specify a path to the script - script-path.

Additional third-party JAR files have to be added to the core plugin in a sub-folder lib/.

Here are some configuration examples for core plugins of the type reporting-plugins and processing-plugins, respectively:

Jython-based Reporting Plugin

plugin.properties

label = Jython Reporting
dataset-types = HCS_IMAGE
class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.jython.JythonBasedReportingPlugin
script-path = data-set-reporting.py

Jython Aggregation Service 

plugin.properties

label = Jython Aggregation Reporting
class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.jython.JythonAggregationService
script-path = aggregating.py

Jython Ingestion Service

plugin.properties

label = Jython Aggregation Reporting
class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.jython.JythonIngestionService
script-path = processing.py

Note, that property dataset-types is not needed and will be ignored.

Jython-based Processing Plugin

plugin.properties

label = Jython Processisng
dataset-types = HCS_IMAGE
class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.jython.JythonBasedProcessingPlugin
script-path = data-set-processing.py

Processing Plugin based on Reporting Script

One can also configure a special processing plugin (ch.systemsx.cisd.openbis.dss.generic.server.plugins.jython.ReportingBasedProcessingPlugin) which uses a reporting script instead of processing script. The reporting script's task is as usual - to describe contents of a table. The processing plugin will then convert the generated table to a text form and send it in an email as an attachment. This feature facilitates code reuse - one can write one script and use it for both reporting and processing plugins.

Configuration of the plugin is as simple as the one for basic jython-based plugins with a few additional properties for specifying email content:

service.properties

...

# --------------------------------------------------------------------------------------------------
# Jython-based Processing Plugin based on Reporting Script
# --------------------------------------------------------------------------------------------------
jython-processing-with-report.label = Jython Processing based on Reporting Script
jython-processing-with-report.dataset-types = HCS_IMAGE
jython-processing-with-report.class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.jython.ReportingBasedProcessingPlugin
jython-processing-with-report.script-path = /resource/examples/data-set-reporting.py
# Optional properties:
# - subject of the email with generated report;
#   defaults to empty subject
#jython-processing-with-report.email-subject = Report
# - body of the email with generated report -
#   defaults to empty subject
#jython-processing-with-report.email-body = The report was successfuly generated and is attached to this email.
# - name of the attachment with generated report;
#   defaults to 'report.txt'
#jython-processing-with-report.attachment-name = report-attachment.txt
# - whether there should be a single report for all processed data sets generated and send in an email to the user,
#   or rather should it be done for processed every data set separately (with one report & email per data set);
#   defaults to false
#jython-processing-with-report.single-report = true

...

Sending an email directly in a processing script is more flexible in defining content of the email than what is described above. It might be preferable if one wants to decide on the email's subject, body or attachment name dynamically, based e.g. on metadata.

Script interfaces and environment

Reporting script

The script file (e.g. "data-set-reporting.py") needs to implement one method:

describe(dataSets, tableBuilder)

which takes a list of data sets (implementing IDataSet interface) and a table builder (ISimpleTableModelBuilderAdaptor) that will be used to generate the table shown in openBIS AS or sent in an email. The method shouldn't return anything. Instead one should call methods of the table builder and a the table model will be created outside of the script using the builder.

Aggregation Service script

The script file (e.g. "aggregating.py") needs to implement one method:

aggregate(parameters, tableBuilder)

which takes some parameters (a java.util.Map with String keys and generic Object values) and a table builder (ISimpleTableModelBuilderAdaptor) that will be used to generate the table shown in openBIS AS or sent in an email. The method shouldn't return anything. Instead one should call methods of the table builder and a the table model will be created outside of the script using the builder.

Ingestion Service script

The script file (e.g. "processing.py") needs to implement one method:

process(transaction, parameters, tableBuilder)

which takes a transaction, some parameters (a java.util.Map with String keys and generic Object values) and a table builder (ISimpleTableModelBuilderAdaptor) that will be used to generate the table shown in openBIS AS or sent in an email. The method shouldn't return anything. Instead one should call methods of the table builder and a the table model will be created outside of the script using the builder. The transaction interface is the same as what is provided to a dropbox. See Dropboxes for a description of what can be done with a transaction.

Processing script

The script file (e.g. "data-set-processing.py") needs to implement one method:

process(dataSet)

which takes a single data set (implementing IDataSet interface). The method shouldn't return anything.

Environment

Both Processing and Reporting script functions are invoked in an environment with following services available as global variables:

Variable Name	Service Interface	Description
searchService	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/shared/api/internal/v2/ISearchService.html","ISearchService")	Methods for listing entities from openBIS AS filtered to those visible to the user.
searchServiceUnfiltered	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/shared/api/internal/v2/ISearchService.html","ISearchService")	As above, but unfiltered
queryService	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/shared/api/internal/IDataSourceQueryService.html","IDataSourceQueryService")	Methods for querying external databases specified in DSS
mailService	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/server/plugins/jython/api/IMailService.html","IMailService")	Methods for sending emails with an attachment to the user who triggered reporting/processing
contentProvider	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/shared/api/internal/v2/IDataSetContentProvider.html","IDataSetContentProvider")	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-openbis-common/ch/systemsx/cisd/openbis/common/io/hierarchical_content/api/IHierarchicalContent.html","Has a method for retrieving a IHierarchicalContent for a specified data set code, filtered to data visible by user.")
contentProviderUnfiltered	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/shared/api/internal/v1/IDataSetContentProvider.html","IDataSetContentProvider")	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-openbis-common/ch/systemsx/cisd/openbis/common/io/hierarchical_content/api/IHierarchicalContent.html","Has a method for retrieving a IHierarchicalContent for a specified data set code, unfiltered.")
authorizationService	=HYPERLINK("https://openbis.ch/javadoc/20.10.x/javadoc-datastore-server/ch/systemsx/cisd/openbis/dss/generic/shared/api/internal/v2/authorization/IAuthorizationService.html","IAuthorizationService")	Methods for determining user access privileges and filter entities to those visible to the user.
userId	String	The id of the user who requested this report (supposed to be used for authorization)

From method arguments one can:

• retrieve basic metadata information about data set and access its files through object implementing IHierarchicalContent (received by calling IDataSet.getContent()),
• (reporting script) build contents of a table using ISimpleTableModelBuilderAdaptor,

Example scripts

Simple reporting plugin

CODE = "Code"
TYPE = "Type"
SIZE = "Size"
LOCATION = "Location"
SPEED_HINT = "Speed Hint"
MAIN_PATTERN = "Main Data Set Pattern"
MAIN_PATH = "Main Data Set Path"
INSTANCE = "Instance"
SPACE = "Space"
PROJECT = "Project"
EXPERIMENT_CODE = "Experiment Code"
EXPERIMENT_IDENTIFIER = "Experiment Identifier"
EXPERIMENT_TYPE = "Experiment Type"
SAMPLE_CODE = "Sample Code"
SAMPLE_IDENTIFIER = "Sample Identifier"
SAMPLE_TYPE = "Sample Type"

def describe(dataSets, tableBuilder):

    tableBuilder.addHeader(CODE)
    tableBuilder.addHeader(TYPE)
    tableBuilder.addHeader(SIZE)
    tableBuilder.addHeader(LOCATION)
    tableBuilder.addHeader(SPEED_HINT)
    tableBuilder.addHeader(MAIN_PATTERN)
    tableBuilder.addHeader(MAIN_PATH)
    tableBuilder.addHeader(INSTANCE)
    tableBuilder.addHeader(SPACE)
    tableBuilder.addHeader(PROJECT)
    tableBuilder.addHeader(EXPERIMENT_CODE)
    tableBuilder.addHeader(EXPERIMENT_IDENTIFIER)
    tableBuilder.addHeader(EXPERIMENT_TYPE)
    tableBuilder.addHeader(SAMPLE_CODE)
    tableBuilder.addHeader(SAMPLE_IDENTIFIER)
    tableBuilder.addHeader(SAMPLE_TYPE)

    for dataSet in dataSets:
        print "script reporting " + dataSet.getDataSetCode()

        row = tableBuilder.addRow()
        row.setCell(CODE, dataSet.getDataSetCode())
        row.setCell(TYPE, dataSet.getDataSetTypeCode())
        row.setCell(SIZE, dataSet.getDataSetSize())
        row.setCell(LOCATION, dataSet.getDataSetLocation())
        row.setCell(SPEED_HINT, dataSet.getSpeedHint())
        row.setCell(MAIN_PATTERN, dataSet.getMainDataSetPattern())
        row.setCell(MAIN_PATH, dataSet.getMainDataSetPath())
        row.setCell(INSTANCE, dataSet.getInstanceCode())
        row.setCell(SPACE, dataSet.getSpaceCode())
        row.setCell(PROJECT, dataSet.getProjectCode())
        row.setCell(EXPERIMENT_CODE, dataSet.getExperimentCode())
        row.setCell(EXPERIMENT_IDENTIFIER, dataSet.getExperimentIdentifier())
        row.setCell(EXPERIMENT_TYPE, dataSet.getExperimentTypeCode())
        row.setCell(SAMPLE_CODE, dataSet.getSampleCode())
        row.setCell(SAMPLE_IDENTIFIER, dataSet.getSampleIdentifier())
        row.setCell(SAMPLE_TYPE, dataSet.getSampleTypeCode())

Reporting plugin accessing openBIS AS

CODE = "Data Set Code"
EXPERIMENT_IDENTIFIER = "Experiment Identifier"
EXPERIMENT_TYPE = "Experiment Type"
EXPERIMENT_DESCRIPTION = "Description"

def describe(dataSets, tableBuilder):

    tableBuilder.addHeader(CODE)
    tableBuilder.addHeader(EXPERIMENT_IDENTIFIER)
    tableBuilder.addHeader(EXPERIMENT_TYPE)
    tableBuilder.addHeader(EXPERIMENT_DESCRIPTION)

    for dataSet in dataSets:
        projectIdentifier = "/" + dataSet.getSpaceCode() + "/" + dataSet.getProjectCode()
        print "script reporting " + dataSet.getDataSetCode() + " from " + projectIdentifier
        experiments = searchService.listExperiments(projectIdentifier)

        for experiment in experiments:
            row = tableBuilder.addRow()
            row.setCell(CODE, dataSet.getDataSetCode())
            row.setCell(EXPERIMENT_IDENTIFIER, experiment.getExperimentIdentifier())
            row.setCell(EXPERIMENT_TYPE, experiment.getExperimentType())
            row.setCell(EXPERIMENT_DESCRIPTION, experiment.getPropertyValue("DESCRIPTION"))

Reporting plugin accessing external DB

Lets assume that a Path Info Database was configured as a data source named "path-info-db".

One shouldn't assume anything about Path Info DB schema. Code below should serve just as an example of accessing an external DB in a jython reporting/processing script.

DATA_SOURCE = "path-info-db"
QUERY = """
    SELECT ds.code as "data_set_code", dsf.*
    FROM data_sets ds, data_set_files dsf
    WHERE ds.code = ?{1} AND dsf.dase_id = ds.id
"""

"""reporting table column names"""
DATA_SET_CODE = "Data Set"
RELATIVE_PATH = "Relative Path"
FILE_NAME = "File Name"
SIZE_IN_BYTES = "Size"
IS_DIRECTORY = "Is Directory?"
LAST_MODIFIED = "Last Modified"

def describe(dataSets, tableBuilder):

    tableBuilder.addHeader(DATA_SET_CODE)
    tableBuilder.addHeader(RELATIVE_PATH)
    tableBuilder.addHeader(FILE_NAME)
    tableBuilder.addHeader(SIZE_IN_BYTES)
    tableBuilder.addHeader(IS_DIRECTORY)
    tableBuilder.addHeader(LAST_MODIFIED)

    for dataSet in dataSets:
        results = queryService.select(DATA_SOURCE, QUERY, [dataSet.getDataSetCode()])
        print "Found " + str(len(results)) + " results for data set '" + dataSet.getDataSetCode() + "':"
        for r in results:
            print r # debugging
            row = tableBuilder.addRow()
            row.setCell(DATA_SET_CODE, r.get("DATA_SET_CODE".lower()))
            row.setCell(RELATIVE_PATH, r.get("RELATIVE_PATH".lower()))
            row.setCell(FILE_NAME, r.get("FILE_NAME".lower()))
            row.setCell(SIZE_IN_BYTES, r.get("SIZE_IN_BYTES".lower()))
            row.setCell(IS_DIRECTORY, r.get("IS_DIRECTORY".lower()))
            row.setCell(LAST_MODIFIED, r.get("LAST_MODIFIED".lower()))
        results.close()