maintenance-tasks.md

Maintenance Tasks

Maintenance Task Classification

Category
Feature
Consistency and other Reports
Consistency Repair and Manual Migrations

Relevancy
Default
Relevant
Rare
Deprecated

Introduction

A maintenance task is a process which runs once or in regular time intervals. It is defined by a core plugin of type maintenance-tasks. Usually a maintenance task can only run on AS or DSS but not in both environments.

The following properties are common for all maintenance tasks:

Property Key	Description
class	The fully-qualified Java class name of the maintenance task. The class has to implement IMaintenanceTask.
execute-only-once	A flag which has to be set to true if the task should be executed only once. Default value: false
interval	A time interval (in seconds) which defines the pace of execution of the maintenance task. Can be specified with one of the following time units: ms, msec, s, sec, m, min, h, hours, d, days. Default time unit is sec. Default value: one day.
start	A time at which the task should be executed the first time. Format: HH:mm. where HH is a two-digit hour (in 24h notation) and mm is a two-digit minute. By default the task is execute at server startup.
run-schedule	Scheduling plan for task execution. Properties execute-only-once, interval, and start will be ignored if specified. Crontab syntax: cron: <second> <minute> <hour> <day> <month> <weekday> Examples: cron: 0 0 * * * *: the top of every hour of every day. cron: */10 * * * * *: every ten seconds. cron: 0 0 8-10 * * *: 8, 9 and 10 o'clock of every day. cron: 0 0 6,19 * * *: 6:00 AM and 7:00 PM every day. cron: 0 0/30 8-10 * * *: 8:00, 8:30, 9:00, 9:30, 10:00 and 10:30 every day. cron: 0 0 9-17 * * MON-FRI: on the hour nine-to-five weekdays. cron: 0 0 0 25 12 ?: every Christmas Day at midnight. Non-crontab syntax: Comma-separated list of definitions with following syntax: [[<counter>.]<week day>] [<month day>[.<month>]] <hour>[:<minute>] where <counter> counts the specified week day of the month. <week day> is MO, MON, TU, TUE, WE, WED, TH, THU, FR, FRI, SA, SAT, SU, or SUN (ignoring case). <month> is either the month number (followed by an optionl '.') or JAN, FEB, MAR, APR, MAY, JUN, JUL, AUG, SEP, OCT, NOV, or DEC (ignoring case). Examples: 6, 18: every day at 6 AM and 6 PM. 3.FR 22:15: every third friday of a month at 22:15. 1. 15:50: every first day of a month at 3:50 PM. SAT 1:30: every saturday at 1:30 AM. 1.Jan 5:15, 1.4. 5:15, 1.7 5:15, 1. OCT 5:15: every first day of a quarter at 5:15 AM.
run-schedule-file	File where the timestamp for next execution is stored. It is used if run-schedule is specified. Default: <installation folder>/<plugin name>_<class name>
retry-intervals-after-failure	Optional comma-separated list of time intervals (format as for interval) after which a failed execution will be retried. Note, that a maintenance task will be execute always when the next scheduled timepoint occurs. This feature allows to execute a task much earlier in case of temporary errors (e.g. temporary unavailibity of another server).

Feature

ArchivingByRequestTask

Environment: AS

Relevancy: Relevant

Description: Triggers archiving for data sets where the 'requested archiving' flag is set. Waits with archiving until enough data sets for a group come together. This is necessary for taped-base archiving where the files to be stored have to be larger than a minimum size.

Configuration:

Property Key	Description
keep-in-store	If true the archived data set will not be removed from the store. That is, only a backup will be created. Default: false
minimum-container-size-in-bytes	Minimum size of an archive container which has one or more data set. This is important for Multi Data Set Archiving. Default: 10 GB
maximum-container-size-in-bytes	Maximum size of an archive container which has one or more data set. This is important for Multi Data Set Archiving. Default: 80 GB
configuration-file-path	Path to the configuration file as used by User Group Management. Here only the group keys are needed. They define a set of groups. If there is no configuration file at the specified path this set is empty.A data set requested for archiving belongs the a specified group if its space starts with the group key followed by an underscore character '_'. Otherwise it belongs to no group. This maintenance task triggers archiving an archive container with one or more data set from the same group if the container fits the specified minimum and maximum size. Note, that data sets which do not belong to a group are handled as a group too. If a data set is larger than the maximum container size it will be archived even though the container is to large. The group key (in lower case) is provided to the archiver. The Multi Data Set Archiver will use this for storing the archive container in a sub folder of the same name. Default: etc/user-management-maintenance-config.json

Example:

plugin.properties

class = ch.systemsx.cisd.openbis.generic.server.task.ArchivingByRequestTask
interval = 1 d
minimum-container-size-in-bytes =  20000000000
maximum-container-size-in-bytes = 200000000000
configuration-file-path = ../../../data/groups.json

Notes:  In practice every instance using multi dataset archiving feature and also the ELN-LIMS should have this enabled.

AutoArchiverTask 

Environment: DSS

Relevancy: Rare

Description: Triggers archiving of data sets that have not been archived yet.

Configuration:

Property Key	Description
remove-datasets-from-store	If true the archived data set will be removed from the store. Default: false
data-set-type	Data set type of the data sets to be archived. If undefined all data set of all types might be archived.
older-than	Minimum number of days a data set to be archived hasn't been accessed. Default: 30
archive-candidate-discoverer.class	Discoverer of candidates to be archived: ch.systemsx.cisd.etlserver.plugins.AgeArchiveCandidateDiscoverer: All data sets with an access time stamp older than specified by property older-than are candidates. This is the default discoverer. ch.systemsx.cisd.etlserver.plugins.TagArchiveCandidateDiscoverer: All data sets which are marked by one of the tags specified by the property archive-candidate-discoverer.tags are candidates.
policy.class	A policy specifies which data set candidates should be archived. If undefined all candidates will be archived. Has to be a fully-qualified name of a Java class implementing ch.systemsx.cisd.etlserver.IAutoArchiverPolicy.
policy.*	Properties specific for the policy specified by policy.class. More about policies can be found here.

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.AutoArchiverTask
interval = 10 days
archive-candidate-discoverer.class = ch.systemsx.cisd.etlserver.plugins.TagArchiveCandidateDiscoverer
archive-candidate-discoverer.tags = /admin-user/archive
policy.class = ch.systemsx.cisd.etlserver.plugins.GroupingPolicy
policy.minimal-archive-size = 1500000
policy.maximal-archive-size = 3000000
policy.grouping-keys = Space#DataSetType, Space#Experiment:merge

BlastDatabaseCreationMaintenanceTask 

Environment: DSS

Relevancy: Default (ELN-LIMS)

Description: Creates BLAST databases from FASTA and FASTQ files of data sets and/or properties of experiments, samples, and data sets.

The title of all entries of the FASTA and FASTQ files will be extended by the string [Data set: <data set code>, File: <path>]. Sequences provide by an entity property will have identifiers of the form <entity kind>+<perm id>+<property type>+<time stamp>. This allows to determine where the matching sequences are stored in openBIS. A sequence can be a nucleic acid sequence or an amino acid sequence.

For each data set a BLAST nucl and prot databases will be created (if not empty) by the tool makeblastdb. For all entities of a specified kind and type one BLAST database (one for nucleic sequences and one for amino acid sequences) will be created from the plain sequences stored in the specified property (white spaces will be removed). In addition an index is created by the tool makembindex if the sequence file of the database (file type .nsq) is larger than 1MB. The name of the databases are <data set code>-nucl/prot and <entity kind>+<entity type code>+<property type code>+<time stamp>-nucl/prot. These databases are referred in the virtual database all-nucl (file: all-nucl.nal) and all-prot (file: all-prot.pal).

If a data set is deleted the corresponding BLAST nucl and prot databases will be automatically removed the next time this maintenance task runs. If an entity of specified type has been modified the BLAST databases will be recalculated the next time this maintenance task runs.

Works only if BLAST+ tool suite has been installed. BLAST+ can be downloaded from ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/

Notes:  It comes pre-configured with the ELN-LIMS but if additional properties need to scanned they should be added to the plugin.properties

Configuration:

Property Key	Description
dataset-types	Comma-separated list of regular expressions of data set types. All FASTA and FASTQ files from those data sets are handled. All data sets of types not matching at least one of the regular expression are not handled.
entity-sequence-properties	Comma-separated list of descriptions of entity properties with sequences. A description is of the form <entity kind>+<entity type code>+<property type code> where <entity kind> is either EXPERIMENT, SAMPLE or DATA_SET (Materials are not supported).
file-types	Space separated list of file types. Data set files of those file types have to be FASTA or FASTQ files. Default: .fasta .fa .fsa .fastq
blast-tools-directory	Path in the file system where all BLAST tools are located. If it is not specified or empty the tools directory has to be in the PATH environment variable.
blast-databases-folder	Path to the folder where all BLAST databases are stored. Default: <data store root>/blast-databases
blast-temp-folder	Path to the folder where temporary FASTA files are stored. Default: <blast-databases-folder>/tmp
last-seen-data-set-file	Path to the file which stores the id of the last seen data set. Default: <data store root>/last-seen-data-set-for-BLAST-database-creation

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.BlastDatabaseCreationMaintenanceTask
interval = 1 h
dataset-types = BLAST-.+
entity-sequence-properties = SAMPLE+OLIGO+SEQUENCE, EXPERIMENT+YEAST+PLASMID_SEQUENCE
blast-tools-directory = /usr/local/ncbi/blast/bin

DeleteDataSetsAlreadyDeletedInApplicationServerMaintenanceTask 

Environment: DSS

Relevancy: Default

Description: Deletes data sets which have been deleted on AS.

If this task isn't configured neither in service.properties nor as a core plugin it will be established automatically by using default configuration and running every 5 minutes.

Configuration:

Property Key	Description
last-seen-data-set-file	Path to a file which will store the code of the last data set handled. Default: deleteDatasetsAlreadyDeletedFromApplicationServerTaskLastSeen
timing-parameters.max-retries	Maximum number of retries in case of currently not available filesystem of the share containing the data set. Default:11
timing-parameters.failure-interval	Waiting time (in seconds) between retries. Default: 10
chunk-size	Number of data sets deleted together. The task is split into deletion tasks with maximum number of data sets. Default: No chunk size. That is, all data sets to be deleted are deleted in one go.

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.DeleteDataSetsAlreadyDeletedInApplicationServerMaintenanceTask
interval = 60
last-seen-data-set-file = lastSeenDataSetForDeletion.txt

DeleteFromArchiveMaintenanceTask 

Environment: DSS

Relevancy: Rare

Description: Deletes archived data sets which have been deleted on AS. This tasks needs the archive plugin to be configured in service.properties. This task only works with non multi data set archivers.

Configuration:

Property Key	Description
status-filename	Path to a file which will store the technical ID of the last data set deletion event on AS.
chunk-size	Maximum number of entries deleted in one maintenance task run. Default: Unlimited

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.DeleteFromArchiveMaintenanceTask
interval = 3600
status-filename = ../archive-cleanup-status.txt

DeleteFromExternalDBMaintenanceTask 

Environment: DSS

Relevancy: Rare

Description: Deletes database entries which are related to data sets deleted in AS. The database is can be any relational database accessible by DSS.

Configuration:

Property Key	Description
data-source	Key of a data source configured in service.properties or in a core plugin of type 'data-sources'. A data source defines the credentials to access the database.
synchronization-table	Name of the table which stores the technical ID of the last data set deletion event on AS. This is ID is used to ask AS for all new data set deletion events. Default value: EVENTS
last-seen-event-id-column	Name of the column in the database table defined by property synchronization-table which stores the ID of the last data set deletion event. Default value: LAST_SEEN_DELETION_EVENT_ID
data-set-table-name	Comma-separated list of table names which contain stuff related to data sets to be deleted. In case of cascading deletion only the tables at the beginning of the cascade should be mentioned. Default value: image_data_sets, analysis_data_sets.
data-set-perm-id	Name of the column in all tables defined by data-set-table-name which stores the data set code. Default value: PERM_ID
chunk-size	Maximum number of entries deleted in one maintenance task run. Default: Unlimited

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.DeleteFromExternalDBMaintenanceTask
interval = 300
data-source = proteomics-db
data-set-table-name = data_sets

EventsSearchMaintenanceTask

Environment: AS

Relevancy: Default

Description: Populates EVENTS_SEARCH database table basing on entries from EVENTS database table. EVENTS_SEARCH table contains the same information as EVENTS table but in a more search friendly format (e.g. a single entry in EVENTS table may represent a deletion of multiple objects deleted at the same time, in EVENT_SEARCH table such entry is split into separate entries - one for each deleted object.). This is set up automatically.

Configuration:

There are no specific configuration parameters for this task.

Example:

plugin.properties

class = ch.systemsx.cisd.openbis.generic.server.task.events_search.EventsSearchMaintenanceTask
interval = 1 day

ExperimentBasedArchivingTask 

Environment: DSS

Relevancy: rare, used when no MultiDataSetArchiver is used and AutoArchiverTask is too complex.

Description: Archives all data sets of experiments which fulfill some criteria. This tasks needs the archive plugin to be configured in service.properties.

Configuration:

Property Key	Description
excluded-data-set-types	Comma-separated list of data set types. Data sets of such types are not archived. Default: No data set type is excluded.
estimated-data-set-size-in-KB.	Specifies for the data set type the average size in KB. If is DEFAULT it will be used for all data set types with unspecified estimated size.
free-space-provider.class	Fully qualified class name of the free space provider (implementing ch.systemsx.cisd.common.filesystem.IFreeSpaceProvider). Depending on the free space provider additional properties, all starting with prefix free-space-provider.,  might be needed. Default: ch.systemsx.cisd.common.filesystem.SimpleFreeSpaceProvider
monitored-dir	Path to the directory to be monitored by the free space provider.
minimum-free-space-in-MB	Minimum free space in MB. If the free space is below this limit the task archives data sets. Default: 1 GB

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.ExperimentBasedArchivingTask
interval = 86400
minimum-free-space-in-MB = 2048
monitored-dir = /my-data/
estimated-data-set-size-in-KB.RAW_DATA = 12000
estimated-data-set-size-in-KB.DEFAULT = 35000

If there is not enough free space the task archives all data sets experiment by experiment until free space is above the specified limit. The oldest experiments are archived first. The age of an experiment is determined by the youngest modification/registration time stamp of all its data sets which are not excluded by data set type or archiving status.

The free space is only calculated once when the task starts to figure out whether archiving is necessary or not. This value is than used together with estimated data set sizes to get an estimated free space which is used for the stopping criteria. Why not calculating the free space again with the free space provider after the data sets of an experiment have been archived? The reason is that providing the free space might be an expensive operation. This is the case when archiving means removing data from a database which have been fed by data from data sets of certain type. In this case archiving (i.e. deleting) those data in the database do not automatically frees disk space because freeing disk space is for databases often an expensive operation.

The DSS admin will be informed by an e-mail about which experiments have been archived.

HierarchicalStorageUpdater

Environment: DSS

Description: Creates/updates a mirrot of the data store. Data set are organized hierachical in accordance to their experiment and samples

Relevancy: Deprecated

Configuration:

Property Key	Description
storeroot-dir-link-path	Path to the root directory of the store as to be used for creating symbolic links. This should be used if the path to the store as seen by clients is different than seen by DSS.
storeroot-dir	Path to the root directory of the store. Used if storeroot-dir-link-path is not specified.
hierarchy-root-dir	Path to the root directory of mirrored store.
link-naming-strategy.class	Fully qualified class name of the strategy to generate the hierarchy (implementing ch.systemsx.cisd.etlserver.plugins.IHierarchicalStorageLinkNamingStrategy). Depending on the actual strategy additional properties, all starting with prefix link-naming-strategy.,  mighty be needed. Default: ch.systemsx.cisd.etlserver.plugins.TemplateBasedLinkNamingStrategy
link-source-subpath.	Link source subpath for the specified data set type. Only files and folder in this relative path inside a data set will be mirrored. Default: The complete data set folder will be mirroed.
link-from-first-child.	Flag which specifies whether only the first child of or the complete folder (either the data set or the one specified by link-source-subpath.). Default: False
with-meta-data	Flag, which specifies whether directories with meta-data.tsv and a link should be created or only links. The default behavior is to create links-only. Default: false
link-naming-strategy.template	The exact form of link paths produced by TemplateBasedLinkNamingStrategy is defined by this template. The variables dataSet, dataSetType, sample, experiment, project and space will be recognized and replaced in the actual link path. Default: ${space}/${project}/${experiment}/${dataSetType}+${sample}+${dataSet}
link-naming-strategy.component-template	If defined, specifies the form of link paths for component datasets. If undefined, component datasets links are formatted with link-naming-strategy.template. Works as link-naming-strategy.template, but has these additional variables: containerDataSetType, containerDataSet, `containerSample. Default: Undefined.

Example:

plugin.properties

class = ch.systemsx.cisd.etlserver.plugins.HierarchicalStorageUpdater
storeroot-dir = ${root-dir}
hierarchy-root-dir = ../../mirror
link-naming-strategy.template = ${space}/${project}/${experiment}/${sample}/${dataSetType}-${dataSet}
link-naming-strategy.component-template = ${space}/${project}/${experiment}/${containerSample}/${containerDataSetType}-${containerDataSet}/${dataSetType}-${dataSet} 

MultiDataSetDeletionMaintenanceTask 

Environment: DSS

Relevancy: Relevant

Description: Deletes data sets which are already deleted on AS also from multi-data-set archives. This maintenance task works only if the Multi Data Set Archiver  is configured. It does the following:

1. Extracts the not-deleted data sets of a TAR container with deleted data sets into the store.
2. Marks them as not present in archive.
3. Deletes the TAR containers with deleted data sets.
4. Requests archiving of the non-deleted data sets.

The last step requires that the maintenance task ArchivingByRequestTask is configured.

Configuration:

Property Key	Description
last-seen-event-id-file	File which contains the last seen event id.
mapping-file	Optional file which maps data sets to share ids and archiving folders (for details see Mapping File for Share Ids and Archiving Folders). If not specified the first share which has enough free space and which isn't a unarchiving scratch share will be used for extracting the not-deleted data sets.

Example:

plugin.properties

class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.standard.archiver.MultiDataSetDeletionMaintenanceTask
interval = 1 d
last-seen-event-id-file = ${storeroot-dir}/MultiDataSetDeletionMaintenanceTask-last-seen-event-id.txt
mapping-file = etc/mapping.tsv 

NOTE: Should be configured on any instance using the multi dataset archiver when the archive data should be deletable.

MultiDataSetUnarchivingMaintenanceTask

Environment: DSS

Relevancy: Relevant

Description: Triggers unarchiving of multi data set archives. Is only needed if the configuration property delay-unarchiving of the Multi Data Set Archiver is set true.

This maintenance task allows to reduce the stress of the tape system by otherwise random unarchiving events triggered by the users.

Configuration: No specific properties.

Example:

plugin.properties

class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.standard.archiver.MultiDataSetUnarchivingMaintenanceTask
interval = 1 d
start = 01:00  

MultiDataSetArchiveSanityCheckMaintenanceTask

Environment: DSS

Relevancy: Default

Description: Task that verifies checksums of data sets archived within a specific time window. It reads archives from the final destination and checks if they are consistent with path info database entries.

The task assumes MultiDataSetArchiver task is configured (the
task uses some of the multi data set archiver configuration properties
e.g. final destination location).

Configuration:

Property Key	Description
status-file	Path to a JSON file that keeps a list of already checked archive containers
notify-emails	List of emails to notify about problematic archive containers
interval	Interval in seconds
check-to-date	"To date" of the time window to be checked. Date in format yyyy-MM-dd HH:mm
check-from-date	"From date" of the time window to be checked. Date in format yyyy-MM-dd HH:mm

Example: