Skip to content
Snippets Groups Projects

Welcome to pyBIS!

pyBIS is a Python module for interacting with openBIS, designed to be used in Jupyter. It offers a sort of IDE for openBIS, supporting TAB completition and input checks, making the life of a researcher hopefully easier.

SYNOPSIS

connecting to OpenBIS

from pybis import Openbis
o = Openbis('https://example.com:8443', verify_certificates=False)
o.login('username', 'password', save_token=True)   # saves the session token in ~/.pybis/example.com.token
o.token
o.is_session_active()
o.get_datastores()
o.logout()

Masterdata

o.get_experiment_types()
o.get_sample_types()
o.get_sample_type('YEAST')
o.get_material_types()
o.get_dataset_types()
o.get_dataset_types()[0]
o.get_dataset_type('RAW_DATA')
o.get_terms()
o.get_terms('MATING_TYPE')
o.get_tags()

Users, Groups and RoleAssignments

o.get_groups()
group = o.new_group(code='group_name', description='...')
group = o.get_group('group_name')
group.save()
group.assign_role(role='ADMIN', space='DEFAULT')
group.get_roles() 
group.revoke_role(role='ADMIN', space='DEFAULT')

group.add_persons(['admin'])
group.get_persons()
group.del_persons(['admin'])
group.delete()

o.get_persons()
person = o.new_person(userId='username')
person.space = 'USER_SPACE'
person.save()

person.assign_role(role='ADMIN', space='MY_SPACE')
person.assign_role(role='OBSERVER')
person.get_roles()
person.revoke_role(role='ADMIN', space='MY_SPACE')
person.revoke_role(role='OBSERVER')

o.get_role_assignments()
o.get_role_assignments(space='MY_SPACE')
o.get_role_assignments(group='MY_GROUP')
ra = o.get_role_assignment(techId)
ra.delete()

Spaces

space = o.new_space(code='space_name', description='')
space.save()
space.delete('reason for deletion')
o.get_spaces()
o.get_space('MY_SPACE')

Projects

project = o.new_project(
    space=space, 
    code='project_name',
    description='some project description'
)
project = space.new_project( code='project_code', description='project description')
project.save()

o.get_projects()
o.get_projects(space='MY_SPACE')
space.get_projects()

project.get_experiments()
project.get_attachments()
p.add_attachment(fileName='testfile', description= 'another file', title= 'one more attachment')
project.download_attachments()

Samples

Samples are nowadays called Objects in openBIS. pyBIS is not yet thoroughly supporting this term in all methods where «sample» occurs.

sample = o.new_sample(
    type='YEAST', 
    space='MY_SPACE', 
    parents=[parent_sample, '/MY_SPACE/YEA66'], 
    children=[child_sample]
)
sample = space.new_sample( type='YEAST' )
sample.save()

sample = o.get_sample('/MY_SPACE/MY_SAMPLE_CODE')
sample = o.get_sample('20170518112808649-52')

sample.space
sample.code
sample.permId
sample.identifier
sample.type  # once the sample type is defined, you cannot modify it
sample.space
sample.space = 'MY_OTHER_SPACE'
sample.experiment    # a sample can belong to one experiment only
sample.experiment = 'MY_SPACE/MY_PROJECT/MY_EXPERIMENT'
sample.tags
sample.tags = ['guten_tag', 'zahl_tag' ]
sample.get_parents()
sample.parents = ['/MY_SPACE/PARENT_SAMPLE_NAME']
sample.add_parents('/MY_SPACE/PARENT_SAMPLE_NAME')
sample.del_parents('/MY_SPACE/PARENT_SAMPLE_NAME')

sample.get_childeren()
sample.children = ['/MY_SPACE/CHILD_SAMPLE_NAME']
sample.add_children('/MY_SPACE/CHILD_SAMPLE_NAME')
sample.del_children('/MY_SPACE/CHILD_SAMPLE_NAME')
sample.get_childeren()

sample.props
sample.p                              # same thing as .props
sample.p.my_property = "some value"   # set the value of a property (value is checked)
sample.p + TAB                        # in IPython or Jupyter: show list of available properties
sample.p.my_property_ + TAB           # in IPython or Jupyter: show datatype or controlled vocabulary

sample.get_attachments()
sample.download_attachments()
sample.add_attachment('testfile.xls')

samples = o.get_samples(
    space='MY_SPACE',
    type='YEAST',
    tags=['*'],                          # tags must be present
    NAME = 'some name',                  # properties are always uppercase to distinguish them from attributes
    **{ "SOME.WEIRD:PROPERTY": "value"}  # in case your property name contains a dot or a colon which cannot be passed as an argument name 
    props=['NAME', 'MATING_TYPE','SHOW_IN_PROJECT_OVERVIEW'] # show these properties in the results
)
samples.df                            # returns a pandas dataframe object
samples.get_datasets(type='ANALYZED_DATA')

Note: Project samples are not implemented yet.

Experiments

o.new_experiment
    type='DEFAULT_EXPERIMENT',
    space='MY_SPACE',
    project='YEASTS'
)
o.get_experiments(
    project='YEASTS',
    space='MY_SPACE', 
    type='DEFAULT_EXPERIMENT',
    tags='*', 
    finished_flag=False,
    props=['name', 'finished_flag']
)
exp = o.get_experiment('/MY_SPACE/MY_PROJECT/MY_EXPERIMENT')

exp.props
exp.p                              # same thing as .props
exp.p.finished_flag=True
exp.p.my_property = "some value"   # set the value of a property (value is checked)
exp.p + TAB                        # in IPython or Jupyter: show list of available properties
exp.p.my_property_ + TAB           # in IPython or Jupyter: show datatype or controlled vocabulary

exp.attrs
exp.a     # same as exp.attrs
exp.attrs.tags = ['some', 'extra', 'tags']
exp.tags = ['some', 'extra', 'tags']          # same thing
exp.save()

Datasets

sample.get_datasets()
ds = o.get_dataset('20160719143426517-259')
ds.get_parents()
ds.get_children()
sample = ds.sample
experiment = ds.experiment
ds.physicalData
ds.status        # AVAILABLE LOCKED ARCHIVED UNARCHIVE_PENDING ARCHIVE_PENDING BACKUP_PENDING
ds.archive()
ds.unarchive()
ds.get_files(start_folder="/")
ds.file_list
ds.add_attachment()
ds.get_attachments()
ds.download_attachments()
ds.download(destination='/tmp', wait_until_finished=False)

ds_new = o.new_dataset(
    type='ANALYZED_DATA', 
    experiment=exp, 
    sample= samp,
    files = ['my_analyzed_data.dat'], 
    props={'name': 'we give this dataset a name', 'notes': 'and we might need some notes, too'})
)
ds_new.save()

ds.props
ds.p                              # same thing as .props
ds.p.my_property = "some value"   # set the value of a property (value is checked)
ds.p + TAB                        # in IPython or Jupyter: show list of available properties
ds.p.my_property_ + TAB           # in IPython or Jupyter: show datatype or controlled vocabulary

# complex query with chaining. props adds a "name" column. To filter for some property, the name of the property must be in UPPERCASE
datasets = o.get_experiments(project='YEASTS').get_samples(type='FLY').get_datasets(type='ANALYZED_DATA', props=['MY_PROPERTY'],MY_PROPERTY='some analyzed data')

# another example
datasets = o.get_experiment('/MY_NEW_SPACE/VERMEUL_PROJECT/MY_EXPERIMENT4').get_samples(type='UNKNOWN').get_parents().get_datasets(type='RAW_DATA')

# get a pandas dataFrame object
datasets.df

# use it in a for-loop:
for dataset in datasets:
    print(ds.permID)

Semantic Annotations

# create semantic annotation for sample type 'UNKNOWN'
sa = o.new_semantic_annotation(entityType = 'UNKNOWN',
                      predicateOntologyId = 'po_id',
                      predicateOntologyVersion = 'po_version',
                      predicateAccessionId = 'pa_id',
                      descriptorOntologyId = 'do_id',
                      descriptorOntologyVersion = 'do_version',
                      descriptorAccessionId = 'da_id')
sa.save()

# create semantic annotation for property type (predicate and descriptor values omitted for brevity)
sa = o.new_semantic_annotation(propertyType = 'DESCRIPTION', ...)
sa.save()

# create semantic annotation for sample property assignment (predicate and descriptor values omitted for brevity)
sa = o.new_semantic_annotation(entityType = 'UNKNOWN', propertyType = 'DESCRIPTION', ...)
sa.save()

# create a semantic annotation directly from a sample type
# will also create sample property assignment annotations when propertyType is given
st = o.get_sample_type("ORDER")
st.new_semantic_annotation(...)

# get all semantic annotations
o.get_semantic_annotations()

# get semantic annotation by perm id
sa = o.get_semantic_annotation("20171015135637955-30")

# update semantic annotation
sa.predicateOntologyId = 'new_po_id'
sa.descriptorOntologyId = 'new_do_id'
sa.save()

# delete semantic annotation
sa.delete('reason')

Tags

new_tag = o.new_tag('my_tag', description='some descriptive text')
new_tag.save()
o.get_tags()
tag = o.get_tag('/username/TAG_Name')
tag.description = 'some new description'
tag.save()
tag.get_experiments()
tag.get_samples()
tag.delete()

Vocabualry and VocabularyTerms

An entity such as Sample (Object), Experiment (Collection), Material or DataSet can be of a specific type:

  • Sample Type
  • Experiment Type
  • DataSet Type
  • Material Type

Every type defines which Properties may be defined. Properties are like Attributes, but they are Type specific. Properties can contain all sorts of information, such as free text, XML, Hyperlink, Boolean and also Controlled Vocabulary. Such a Controlled Vocabulary consists of many VocabularyTerms. They are used to check the terms entered in a Property field.

So for example, you want to add a property called Animal to a Sample and you want to control which terms are entered in this Property field. For this you need to do a couple of steps:

  1. create a new vocabulary AnimalVocabulary
  2. add terms to that vocabulary: Cat, Dog, Mouse
  3. create a new PropertyType (e.g. Animal) of DataType CONTROLLEDVOCABULARY and assign the AnimalVocabulary to it
  4. create a new SampleType (e.g. Pet) and assign the created PropertyType to that Sample type.
  5. If you now create a new Sample of type Pet you will be able to add a property Animal to it which only accepts the terms Cat, Dog or Mouse.

create new Vocabulary with three VocabularyTerms

voc = o.new_vocabulary(
    code = 'BBB',
    description = 'description of vocabulary aaa',
    urlTemplate = 'https://ethz.ch',
    terms = [
        { "code": 'term_code1', "label": "term_label1", "description": "term_description1"},
        { "code": 'term_code2', "label": "term_label2", "description": "term_description2"},
        { "code": 'term_code3', "label": "term_label3", "description": "term_description3"}
    ]   
)
voc.save()

create additional VocabularyTerms

term = o.new_term(
	code='TERM_CODE_XXX', 
	vocabularyCode='BBB', 
	label='here comes a label',
	description='here is a meandingful description'
)
term.save()

fetching Vocabulary and VocabularyTerms

Requirements and organization

Dependencies and Requirements

  • pyBIS relies the openBIS API v3; openBIS version 16.05.2 or newer
  • pyBIS uses Python 3.3 and pandas
  • pyBIS needs the jupyter-api to be installed, in order to register new datasets

Installation

  • locate the jupyter-api folder found in pybis/src/coreplugins
  • copy this folder to openbis/servers/core-plugins in your openBIS installation
  • register the plugin by editing openbis/servers/core-plugins/core-plugins.properties :
  • enabled-modules = jupyter-api (separate multiple plugins with comma)
  • restart your DSS to activate the plugin

Project Organization

This project is devided in several parts:

  • src/python/PyBis Python module which holds all the method to interact with OpenBIS
  • src/python/OBis a command-line tool to register large datasets in OpenBIS without actually copying the data. Uses git annex for version control and OpenBIS linkedDataSet objects to register the metadata.
  • src/python/JupyterBis a JupyterHub authenticator module which uses pyBIS for authenticating against openBIS, validating and storing the session token
  • src/core-plugins/jupyter-api, an ingestion plug-in for openBIS, allowing people to upload new datasets
  • src/vagrant/jupyter-bis/Vagrantfile to set up JupyterHub on a virtual machine (CentOS 7), which uses the JupyterBis authenticator module
  • src/vagrant/obis/Vagrantfile to set up a complete OpenBIS instance on a virtual machine (CentOS 7)