README.rst

Welcome to pyBIS!
=================

pyBIS is a Python module for interacting with openBIS, designed to be
used in Jupyter. It offers a sort of IDE for openBIS, supporting TAB
completition and input checks, making the life of a researcher hopefully
easier.

SYNOPSIS
========

connecting to OpenBIS
---------------------

::

    from pybis import Openbis
    o = Openbis('https://example.com:8443', verify_certificates=False)
    o.login('username', 'password', save_token=True)   # saves the session token in ~/.pybis/example.com.token
    o.token
    o.is_session_active()
    o.get_datastores()
    o.logout()

Masterdata
----------

::

    o.get_experiment_types()
    o.get_sample_types()
    o.get_sample_type('YEAST')
    o.get_material_types()
    o.get_dataset_types()
    o.get_dataset_types()[0]
    o.get_dataset_type('RAW_DATA')
    o.get_terms()
    o.get_terms('MATING_TYPE')
    o.get_tags()

Users, Groups and RoleAssignments
---------------------------------

::

    o.get_groups()
    group = o.new_group(code='group_name', description='...')
    group = o.get_group('group_name')
    group.save()
    group.assign_role(role='ADMIN', space='DEFAULT')
    group.get_roles() 
    group.revoke_role(role='ADMIN', space='DEFAULT')

    group.add_persons(['admin'])
    group.get_persons()
    group.del_persons(['admin'])
    group.delete()

    o.get_persons()
    person = o.new_person(userId='username')
    person.space = 'USER_SPACE'
    person.save()

    person.assign_role(role='ADMIN', space='MY_SPACE')
    person.assign_role(role='OBSERVER')
    person.get_roles()
    person.revoke_role(role='ADMIN', space='MY_SPACE')
    person.revoke_role(role='OBSERVER')

    o.get_role_assignments()
    o.get_role_assignments(space='MY_SPACE')
    o.get_role_assignments(group='MY_GROUP')
    ra = o.get_role_assignment(techId)
    ra.delete()

Spaces
------

::

    space = o.new_space(code='space_name', description='')
    space.save()
    space.delete('reason for deletion')
    o.get_spaces()
    o.get_space('MY_SPACE')

Projects
--------

::

    project = o.new_project(
        space=space, 
        code='project_name',
        description='some project description'
    )
    project = space.new_project( code='project_code', description='project description')
    project.save()

    o.get_projects()
    o.get_projects(space='MY_SPACE')
    space.get_projects()

    project.get_experiments()
    project.get_attachments()
    p.add_attachment(fileName='testfile', description= 'another file', title= 'one more attachment')
    project.download_attachments()

Samples
-------

Samples are nowadays called **Objects** in openBIS. pyBIS is not yet
thoroughly supporting this term in all methods where «sample» occurs.

::

    sample = o.new_sample(
        type='YEAST', 
        space='MY_SPACE', 
        parents=[parent_sample, '/MY_SPACE/YEA66'], 
        children=[child_sample]
    )
    sample = space.new_sample( type='YEAST' )
    sample.save()

    sample = o.get_sample('/MY_SPACE/MY_SAMPLE_CODE')
    sample = o.get_sample('20170518112808649-52')

    sample.space
    sample.code
    sample.permId
    sample.identifier
    sample.type  # once the sample type is defined, you cannot modify it
    sample.space
    sample.space = 'MY_OTHER_SPACE'
    sample.experiment    # a sample can belong to one experiment only
    sample.experiment = 'MY_SPACE/MY_PROJECT/MY_EXPERIMENT'
    sample.tags
    sample.tags = ['guten_tag', 'zahl_tag' ]
    sample.get_parents()
    sample.parents = ['/MY_SPACE/PARENT_SAMPLE_NAME']
    sample.add_parents('/MY_SPACE/PARENT_SAMPLE_NAME')
    sample.del_parents('/MY_SPACE/PARENT_SAMPLE_NAME')

    sample.get_childeren()
    sample.children = ['/MY_SPACE/CHILD_SAMPLE_NAME']
    sample.add_children('/MY_SPACE/CHILD_SAMPLE_NAME')
    sample.del_children('/MY_SPACE/CHILD_SAMPLE_NAME')
    sample.get_childeren()

    sample.props
    sample.p                              # same thing as .props
    sample.p.my_property = "some value"   # set the value of a property (value is checked)
    sample.p + TAB                        # in IPython or Jupyter: show list of available properties
    sample.p.my_property_ + TAB           # in IPython or Jupyter: show datatype or controlled vocabulary

    sample.get_attachments()
    sample.download_attachments()
    sample.add_attachment('testfile.xls')

    samples = o.get_samples(
        space='MY_SPACE',
        type='YEAST',
        tags=['*'],                          # tags must be present
        NAME = 'some name',                  # properties are always uppercase to distinguish them from attributes
        **{ "SOME.WEIRD:PROPERTY": "value"}  # in case your property name contains a dot or a colon which cannot be passed as an argument name 
        props=['NAME', 'MATING_TYPE','SHOW_IN_PROJECT_OVERVIEW'] # show these properties in the results
    )
    samples.df                            # returns a pandas dataframe object
    samples.get_datasets(type='ANALYZED_DATA')

Note: Project samples are not implemented yet.

Experiments
-----------

::

    o.new_experiment
        type='DEFAULT_EXPERIMENT',
        space='MY_SPACE',
        project='YEASTS'
    )
    o.get_experiments(
        project='YEASTS',
        space='MY_SPACE', 
        type='DEFAULT_EXPERIMENT',
        tags='*', 
        finished_flag=False,
        props=['name', 'finished_flag']
    )
    exp = o.get_experiment('/MY_SPACE/MY_PROJECT/MY_EXPERIMENT')

    exp.props
    exp.p                              # same thing as .props
    exp.p.finished_flag=True
    exp.p.my_property = "some value"   # set the value of a property (value is checked)
    exp.p + TAB                        # in IPython or Jupyter: show list of available properties
    exp.p.my_property_ + TAB           # in IPython or Jupyter: show datatype or controlled vocabulary

    exp.attrs
    exp.a     # same as exp.attrs
    exp.attrs.tags = ['some', 'extra', 'tags']
    exp.tags = ['some', 'extra', 'tags']          # same thing
    exp.save()

Datasets
--------

::

    sample.get_datasets()
    ds = o.get_dataset('20160719143426517-259')
    ds.get_parents()
    ds.get_children()
    sample = ds.sample
    experiment = ds.experiment
    ds.physicalData
    ds.status        # AVAILABLE LOCKED ARCHIVED UNARCHIVE_PENDING ARCHIVE_PENDING BACKUP_PENDING
    ds.archive()
    ds.unarchive()
    ds.get_files(start_folder="/")
    ds.file_list
    ds.add_attachment()
    ds.get_attachments()
    ds.download_attachments()
    ds.download(destination='/tmp', wait_until_finished=False)

    ds_new = o.new_dataset(
        type='ANALYZED_DATA', 
        experiment=exp, 
        sample= samp,
        files = ['my_analyzed_data.dat'], 
        props={'name': 'we give this dataset a name', 'notes': 'and we might need some notes, too'})
    )
    ds_new.save()

    ds.props
    ds.p                              # same thing as .props
    ds.p.my_property = "some value"   # set the value of a property (value is checked)
    ds.p + TAB                        # in IPython or Jupyter: show list of available properties
    ds.p.my_property_ + TAB           # in IPython or Jupyter: show datatype or controlled vocabulary

    # complex query with chaining. props adds a "name" column. To filter for some property, the name of the property must be in UPPERCASE
    datasets = o.get_experiments(project='YEASTS').get_samples(type='FLY').get_datasets(type='ANALYZED_DATA', props=['MY_PROPERTY'],MY_PROPERTY='some analyzed data')

    # another example
    datasets = o.get_experiment('/MY_NEW_SPACE/VERMEUL_PROJECT/MY_EXPERIMENT4').get_samples(type='UNKNOWN').get_parents().get_datasets(type='RAW_DATA')

    # get a pandas dataFrame object
    datasets.df

    # use it in a for-loop:
    for dataset in datasets:
        print(ds.permID)

Semantic Annotations
--------------------

::

    # create semantic annotation for sample type (predicate and descriptor values omitted for brevity)
    sa = o.new_semantic_annotation(entityType = 'UNKNOWN')
    sa.save()

    # create semantic annotation for property type (predicate and descriptor values omitted for brevity)
    sa = o.new_semantic_annotation(propertyType = 'DESCRIPTION')
    sa.save()

    # create semantic annotation for sample property assignment (predicate and descriptor values omitted for brevity)
    sa = o.new_semantic_annotation(entityType = 'UNKNOWN', propertyType = 'DESCRIPTION')
    sa.save()

    # create semantic annotation with additional fields
    sa = o.new_semantic_annotation(entityType = 'UNKNOWN',
                          predicateOntologyId = 'po_id',
                          predicateOntologyVersion = 'po_version',
                          predicateAccessionId = 'pa_id',
                          descriptorOntologyId = 'do_id',
                          descriptorOntologyVersion = 'do_version',
                          descriptorAccessionId = 'da_id')
    sa.save()

    # get all semantic annotations
    o.get_semantic_annotations()

    # get semantic annotation by perm id
    sa = o.get_semantic_annotation("20171015135637955-30")

    # update semantic annotation
    sa.predicateOntologyId = 'new_po_id'
    sa.descriptorOntologyId = 'new_do_id'
    sa.save()

    # delete semantic annotation
    sa.delete('reason')

Tags
----

::

    new_tag = o.new_tag('my_tag', description='some descriptive text')
    new_tag.save()
    o.get_tags()
    tag = o.get_tag('/username/TAG_Name')
    tag.description = 'some new description'
    tag.save()
    tag.get_experiments()
    tag.get_samples()
    tag.delete()

Vocabualry and VocabularyTerms
------------------------------

An entity such as Sample (Object), Experiment (Collection), Material or
DataSet can be of a specific type:

-  Sample Type
-  Experiment Type
-  DataSet Type
-  Material Type

Every type defines which Properties may be defined. Properties are like
Attributes, but they are Type specific. Properties can contain all sorts
of information, such as free text, XML, Hyperlink, Boolean and also
*Controlled Vocabulary*. Such a Controlled Vocabulary consists of many
VocabularyTerms. They are used to check the terms entered in a Property
field.

So for example, you want to add a property called **Animal** to a Sample
and you want to control which terms are entered in this Property field.
For this you need to do a couple of steps:

1. create a new vocabulary *AnimalVocabulary*
2. add terms to that vocabulary: *Cat, Dog, Mouse*
3. create a new PropertyType (e.g. *Animal*) of DataType
   *CONTROLLEDVOCABULARY* and assign the *AnimalVocabulary* to it
4. create a new SampleType (e.g. *Pet*) and *assign* the created
   PropertyType to that Sample type.
5. If you now create a new Sample of type *Pet* you will be able to add
   a property *Animal* to it which only accepts the terms *Cat, Dog* or
   *Mouse*.

**create new Vocabulary with three VocabularyTerms**

::

    voc = o.new_vocabulary(
        code = 'BBB',
        description = 'description of vocabulary aaa',
        urlTemplate = 'https://ethz.ch',
        terms = [
            { "code": 'term_code1', "label": "term_label1", "description": "term_description1"},
            { "code": 'term_code2', "label": "term_label2", "description": "term_description2"},
            { "code": 'term_code3', "label": "term_label3", "description": "term_description3"}
        ]   
    )
    voc.save()

**create additional VocabularyTerms**

::

    term = o.new_term(
        code='TERM_CODE_XXX', 
        vocabularyCode='BBB', 
        label='here comes a label',
        description='here is a meandingful description'
    )
    term.save()

**fetching Vocabulary and VocabularyTerms**

Requirements and organization
=============================

Dependencies and Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  pyBIS relies the openBIS API v3; openBIS version 16.05.2 or newer
-  pyBIS uses Python 3.3 and pandas
-  pyBIS needs the jupyter-api to be installed, in order to register new
   datasets

Installation
~~~~~~~~~~~~

-  locate the ``jupyter-api`` folder found in ``pybis/src/coreplugins``
-  copy this folder to ``openbis/servers/core-plugins`` in your openBIS
   installation
-  register the plugin by editing
   ``openbis/servers/core-plugins/core-plugins.properties`` :
-  ``enabled-modules = jupyter-api`` (separate multiple plugins with
   comma)
-  restart your DSS to activate the plugin

Project Organization
~~~~~~~~~~~~~~~~~~~~

This project is devided in several parts:

-  src/python/\ **PyBis** Python module which holds all the method to
   interact with OpenBIS
-  src/python/\ **OBis** a command-line tool to register large datasets
   in OpenBIS without actually copying the data. Uses git annex for
   version control and OpenBIS linkedDataSet objects to register the
   metadata.
-  src/python/\ **JupyterBis** a JupyterHub authenticator module which
   uses pyBIS for authenticating against openBIS, validating and storing
   the session token
-  src/core-plugins/\ **jupyter-api**, an ingestion plug-in for openBIS,
   allowing people to upload new datasets
-  src/vagrant/\ **jupyter-bis/Vagrantfile** to set up JupyterHub on a
   virtual machine (CentOS 7), which uses the JupyterBis authenticator
   module
-  src/vagrant/\ **obis/Vagrantfile** to set up a complete OpenBIS
   instance on a virtual machine (CentOS 7)
-