diff --git a/pybis/src/python/CHANGELOG.md b/pybis/src/python/CHANGELOG.md
index ddc9cefd6540d09e80d603c682be4513a9200b1a..e6651980ce2cedd3490b534541b29ac1b6d061d7 100644
--- a/pybis/src/python/CHANGELOG.md
+++ b/pybis/src/python/CHANGELOG.md
@@ -1,3 +1,8 @@
+## Changes with pybis-1.12.0
+
+- added possibility to get any additional attributes in the get_samples() method
+- added possibility to get any additional attributes in the get_dataSets() method
+
## Changes with pybis-1.11.1
- added automatically accepting host key, otherwise mount() will hang the first time
diff --git a/pybis/src/python/README.md b/pybis/src/python/README.md
index 61b7e0777841c6d59276def3ff1cf9824abf2136..b19055202cece87044a5d2776850e1f8ee5a24ab 100644
--- a/pybis/src/python/README.md
+++ b/pybis/src/python/README.md
@@ -1,33 +1,62 @@
# Welcome to pyBIS!
-pyBIS is a Python module for interacting with openBIS, designed to be used in Jupyter. It offers some sort of IDE for openBIS, supporting TAB completition and input checks, making the life of a researcher hopefully easier.
+
+pyBIS is a Python module for interacting with openBIS. pyBIS is designed to be most useful in a [Jupyter Notebook](https://jupyter.org) or IPython environment, especially if you are developing Python scripts for automatisation. Jupyter Notebooks offer some sort of IDE for openBIS, supporting TAB completition and input checks, making the life of a researcher hopefully easier.
## Dependencies and Requirements
- pyBIS relies the openBIS API v3
- openBIS version 16.05.2 or newer is required
- 18.06.2 or later is recommended
-- pyBIS uses Python 3.3 and pandas
+- pyBIS uses Python 3.3 and Pandas
## Installation
```
-pip install pybis
+pip install --upgrade pybis
```
-That command will download install pybis and all its dependencies.
-
-If you haven't done yet, install Jupyter Notebook:
+That command will download install pyBIS and all its dependencies. If pyBIS is already installed, it will be upgraded to the latest version.
+If you haven't done yet, install Jupyter and/or Jupyter Lab (the next Generation of Jupyter):
```
pip install jupyter
+pip install jupyterlab
```
# General Usage
-## Tab completition and other hints
-Used in a Jupyter Notebook environment, pybis helps you to enter the commands. After every dot `.` you might hit the `TAB` key in order to look at the available commands.
+### TAB completition and other hints in Jupyter / IPython
+
+* in a Jupyter Notebook or IPython environment, pybis helps you to enter the commands
+* After every dot `.` you might hit the `TAB` key in order to look at the available commands.
+* if you are unsure what parameters to add to a , add a question mark right after the method and hit `SHIFT+ENTER`
+* Jupyter will then look up the signature of the method and show some helpful docstring
+
-If you are unsure what parameters to add to a , add a question mark right after the method and hit `SHIFT+ENTER`. Jupyter will then look up the signature of the method and show some helpful docstring.
+### Checking input
-When working with properties of entities, they might use a **controlled vocabulary** or are of a specific **property type**. Add an underscore `_` character right after the property and hit `SHIFT+ENTER` to show the valid values. When a property only acceps a controlled vocabulary, you will be shown the valid terms in a nicely formatted table.
+* When working with properties of entities, they might use a **controlled vocabulary** or are of a specific **property type**.
+* Add an underscore `_` character right after the property and hit `SHIFT+ENTER` to show the valid values
+* When a property only acceps a controlled vocabulary, you will be shown the valid terms in a nicely formatted table
+* if you try to assign an **invalid value** to a property, you'll receive an error immediately
+
+### Glossary
+
+* **spaces:** used for authorisation eg. to separate two working groups. If you have permissions in a space, you can see everything which in that space, but not necessarily in another space (unless you have the permission).
+* **projects:** a space consists of many projects.
+* **experiments / collections:** a projects contain many experiments. Experiments can have *properties*
+* **samples / objects:** an experiment contains many samples. Samples can have *properties*
+* **dataSet:** a dataSet which contains the actual *data files*, either pyhiscal (stored in openBIS dataStore) or linked
+* **attributes:** every entity above contains a number of attributes. They are the same accross all instances of openBIS and independent of their type.
+* **properties:** Additional specific key-value pairs, available for these entities:
+ * experiments
+ * samples
+ * dataSets
+
+ every single instance of an entity must be of a specific **entity type** (see below). The type defines the set of properties.
+* **experiment type / collection type:** a type for experiments which specifies its properties
+* **sample type / object type:** a type for samples / objects which specifies its properties
+* **dataSet type:** a type for dataSets which specifies its properties
+* **property type:** a single property, as defined in the entity types above. It can be of a classic data type (e.g. INTEGER, VARCHAR, BOOLEAN) or its values can be controlled (CONTROLLEDVOCABULARY).
+* **plugin:** a script written in [Jython](https://www.jython.org) which allows to check property values in a even more detailed fashion
# connect to OpenBIS
@@ -55,7 +84,7 @@ o.login(os.environ['OPENBIS_USERNAME'], os.environ['OPENBIS_PASSWORD'])
```
-Check whether the session token is still valid and log out:
+Check whether the **session token** is still valid and log out:
```
o.token
@@ -67,7 +96,7 @@ o.logout()
### Prerequisites: FUSE / SSHFS
-Mounting an openBIS dataStore server requires FUSE / SSHFS to be installed (requires root privileges):
+Mounting an openBIS dataStore server requires FUSE / SSHFS to be installed (requires root privileges). The mounting itself requires no root privileges.
**Mac OS X**
@@ -128,10 +157,10 @@ st.get_validationPlugin() # returns a plugin object
st.get_property_assignments() # show the list of properties
# for that sample type
-
o.get_material_types()
o.get_dataset_types()
o.get_experiment_types()
+o.get_collection_types()
o.get_property_types()
pt = o.get_property_type('BARCODE_COMPLEXITY_CHECKER')
@@ -191,37 +220,39 @@ The `dataType` attribute can contain any of these values:
When choosing `CONTROLLEDVOCABULARY`, you must specify a `vocabulary` attribute (see example). Likewise, when choosing `MATERIAL`, a `materialType` attribute must be provided. PropertyTypes that start with a $ belong by definition to the `internalNameSpace` and therefore this attribute must be set to True.
-## create sample types
+## create sample types / object types
+
+The new name for `sample_type` is `object_type`. You can use boths names interchangeably.
```
sample_type = o.new_sample_type(
- code = 'my_own_sample_type', # mandatory
- generatedCodePrefix = 'S', # mandatory
- description = '',
- autoGeneratedCode = True,
- subcodeUnique = False,
- listable = True,
- showContainer = False,
- showParents = True,
- showParentMetadata = False,
- validationPlugin = 'Has_Parents' # see plugins below
+ code = 'my_own_sample_type', # mandatory
+ generatedCodePrefix = 'S', # mandatory
+ description = '',
+ autoGeneratedCode = True,
+ subcodeUnique = False,
+ listable = True,
+ showContainer = False,
+ showParents = True,
+ showParentMetadata = False,
+ validationPlugin = 'Has_Parents' # see plugins below
)
sample_type.save()
```
-## assign properties to sample type
+## assign properties to sample type / object type
A sample type needs to be saved before properties can be assigned to. This assignment procedure applies to all entity types (dataset type, experiment type, material type).
```
sample_type.assign_property(
- prop = 'diff_time', # mandatory
- section = '',
- ordinal = 5,
- mandatory = True,
+ prop = 'diff_time', # mandatory
+ section = '',
+ ordinal = 5,
+ mandatory = True,
initialValueForExistingEntities = 'initial value'
- showInEditView = True,
- showRawValueInForms = True
+ showInEditView = True,
+ showRawValueInForms = True
)
sample_type.revoke_property('diff_time')
sample_type.get_property_assignments()
@@ -231,12 +262,12 @@ sample_type.get_property_assignments()
```
dataset_type = o.new_dataset_type(
- code = 'my_dataset_type', # mandatory
- description=None,
- mainDataSetPattern=None,
- mainDataSetPath=None,
- disallowDeletion=False,
- validationPlugin=None,
+ code = 'my_dataset_type', # mandatory
+ description = None,
+ mainDataSetPattern = None,
+ mainDataSetPath = None,
+ disallowDeletion = False,
+ validationPlugin = None,
)
dataset_type.save()
dataset_type.assign_property('property_name')
@@ -244,13 +275,15 @@ dataset_type.revoke_property('property_name')
dataset_type.get_property_assignments()
```
-## create experiment types
+## create experiment types / collection types
+
+The new name for `experiment_type` is `collection_type`. You can use boths names interchangeably.
```
experiment_type = o.new_experiment_type(
code,
- description=None,
- validationPlugin=None,
+ description = None,
+ validationPlugin = None,
)
experiment_type.save()
experiment_type.assign_property('property_name')
@@ -280,9 +313,9 @@ Plugins are Jython scripts that can accomplish more complex data-checks than ord
```
pl = o.new_plugin(
name ='my_new_entry_validation_plugin',
- pluginType ='ENTITY_VALIDATION', # or 'DYNAMIC_PROPERTY' or 'MANAGED_PROPERTY',
- entityKind = None, # or 'SAMPLE', 'MATERIAL', 'EXPERIMENT', 'DATA_SET'
- script = 'def calculate(): pass' # a JYTHON script
+ pluginType ='ENTITY_VALIDATION', # or 'DYNAMIC_PROPERTY' or 'MANAGED_PROPERTY',
+ entityKind = None, # or 'SAMPLE', 'MATERIAL', 'EXPERIMENT', 'DATA_SET'
+ script = 'def calculate(): pass' # a JYTHON script
)
pl.save()
```
@@ -328,8 +361,8 @@ space = o.new_space(code='space_name', description='')
space.save()
space.delete('reason for deletion')
o.get_spaces(
- start_with = 1, # start_with and count
- count = 7, # enable paging
+ start_with = 0, # start_with and count
+ count = 10, # enable paging
)
space = o.get_space('MY_SPACE')
space.code
@@ -344,20 +377,20 @@ space.attrs.all() # returns a dict containing all attributes
## Projects
```
project = o.new_project(
- space=space,
- code='project_name',
- description='some project description'
+ space = space,
+ code = 'project_name',
+ description = 'some project description'
)
project = space.new_project(
- code='project_code',
- description='project description'
+ code = 'project_code',
+ description = 'project description'
)
project.save()
o.get_projects(
- space = 'MY_SPACE', # show only projects in MY_SPACE
- start_with = 1, # start_with and count
- count = 7, # enable paging
+ space = 'MY_SPACE', # show only projects in MY_SPACE
+ start_with = 0, # start_with and count
+ count = 10, # enable paging
)
o.get_projects(space='MY_SPACE')
space.get_projects()
@@ -369,7 +402,7 @@ project.download_attachments()
project.code
project.description
-# ... and many more
+# ... any other attribute
project.attrs.all() # returns a dict containing all attributes
project.freeze = True
@@ -378,19 +411,24 @@ project.freezeForSamples = True
```
-## Samples
-Samples are nowadays called **Objects** in openBIS. pyBIS is not yet thoroughly supporting this term in all methods where «sample» occurs.
+## Samples / Objects
+
+The new name for `sample` is `object`. You can use boths names interchangeably:
+
+* `get_sample()` = `get_object()`
+* `new_sample()` = `new_object()`
+* `get_samples()` = `get_objects()`
-NOTE: In openBIS, `samples` entities have recently been renamed to `objects`. All methods have synonyms using the term `object`, e.g. `get_object`, `new_object`, `get_object_types`.
+etc.
```
sample = o.new_sample(
- type = 'YEAST',
- space = 'MY_SPACE',
+ type = 'YEAST',
+ space = 'MY_SPACE',
experiment = '/MY_SPACE/MY_PROJECT/EXPERIMENT_1',
- parents = [parent_sample, '/MY_SPACE/YEA66'],
- children = [child_sample],
- props = {"name": "some name", "description": "something interesting"}
+ parents = [parent_sample, '/MY_SPACE/YEA66'],
+ children = [child_sample],
+ props = {"name": "some name", "description": "something interesting"}
)
sample = space.new_sample( type='YEAST' )
sample.save()
@@ -478,22 +516,30 @@ sample.p.my_property = "some value" # set the value of a property
sample.save() # update the sample in openBIS
```
-### querying samples
+### search for samples / objects
+
+The result of a search is always list, even when no items are found. The `.df` attribute returns
+the Pandas dataFrame of the results.
```
samples = o.get_samples(
- space ='MY_SPACE',
- type ='YEAST',
- tags =['*'], # only sample with existing tags
- start_with = 1, # start_with and count
- count = 7, # enable paging
- NAME = 'some name', # properties are always uppercase
+ space ='MY_SPACE',
+ type ='YEAST',
+ tags =['*'], # only sample with existing tags
+ start_with = 0, # start_with and count
+ count = 10, # enable paging
+ NAME = 'some name', # properties are always uppercase
# to distinguish them from attributes
**{ "SOME.WEIRD:PROP": "value"} # property name contains a dot or a
# colon: cannot be passed as an argument
- props=['NAME', 'MATING_TYPE'] # show these properties in the result
+ attrs=[ # show these attributes in the dataFrame
+ 'sample.code',
+ 'registrator.email',
+ 'type.generatedCodePrefix'
+ ],
+ props=['$NAME', 'MATING_TYPE'] # show these properties in the result
)
-samples.df # returns a pandas DataFrame object
+samples.df # returns a Pandas DataFrame object
samples.get_datasets(type='ANALYZED_DATA')
```
@@ -507,9 +553,13 @@ sample.freezeForParents = True
sample.freezeForDataSets = True
```
-## Experiments
+## Experiments / Collections
-NOTE: In openBIS, `experiment` entities have recently been renamed to `collection`. All methods have synonyms using the term `collection`, e.g. `get_collections`, `new_collection`, `get_collection_types`.
+The new name for `experiment` is `collection`. You can use boths names interchangeably:
+
+* `get_experiment()` = `get_collection()`
+* `new_experiment()` = `new_collection()`
+* `get_experiments()` = `get_collections()`
```
exp = o.new_experiment
@@ -586,7 +636,7 @@ ds.download_attachments()
### download dataSets
```
-ds.get_files(start_folder="/") # get file list as pandas table
+ds.get_files(start_folder="/") # get file list as Pandas table
ds.file_list # get file list as array
ds.download() # simply download all files to hostname/permId/
@@ -613,10 +663,34 @@ ds.attrs.all() # returns all attributes as a dict
ds.props.all() # returns all properties as a dict
```
-### querying dataSets
+### search for dataSets
+
+* The result of a search is always list, even when no items are found
+* The `.df` attribute returns the Pandas dataFrame of the results
+* properties must be in UPPERCASE to distinguish them from attributes
+
+```
+dataSets = o.get_dataSets(
+ type ='MY_DATASET_TYPE',
+ NAME = 'some name', # properties are always uppercase
+ # to distinguish them from attributes
+ **{ "SOME.WEIRD:PROP": "value"}, # property name contains a dot or a
+ # colon: cannot be passed as an argument
+ start_with = 0, # start_with and count
+ count = 10, # enable paging
+ attrs=[ # show these attributes in the dataFrame
+ 'sample.code',
+ 'registrator.email',
+ 'type.generatedCodePrefix'
+ ],
+ props=['$NAME', 'MATING_TYPE'] # show these properties in the result
+)
+
+df = dataSets.df # returns the Pandas dataFrame object
+```
-* examples of a complex queries with methods chaining.
-* NOTE: properties must be in UPPERCASE to distinguish them from attributes
+In some cases, you might want to retrieve precisely certain datasets. This can be achieved by
+methods chaining (but be aware, it might not be very performant):
```
datasets = o.get_experiments(project='YEASTS')\
@@ -627,7 +701,7 @@ datasets = o.get_experiments(project='YEASTS')\
MY_PROPERTY='some analyzed data'
)
```
-
+* another example:
```
datasets = o.get_experiment('/MY_NEW_SPACE/MY_PROJECT/MY_EXPERIMENT4')\
.get_samples(type='UNKNOWN')\
@@ -635,16 +709,6 @@ datasets = o.get_experiment('/MY_NEW_SPACE/MY_PROJECT/MY_EXPERIMENT4')\
.get_datasets(type='RAW_DATA')
```
-### deal with dataSets query results
-
-```
-datasets.df # get a pandas dataFrame object
-
-# use it in a for-loop:
-for dataset in datasets:
- print(dataset.permID)
- dataset.delete('give me a reason')
-```
### freeze dataSets
* once a dataSet has been frozen, it cannot be changed by anyone anymore
* so be careful!
@@ -673,8 +737,9 @@ ds_new.save()
### create dataSet with zipfile
-```
-# DataSet containing one zipfile which will be unzipped in openBIS
+DataSet containing one zipfile which will be unzipped in openBIS:
+
+```python
ds_new = o.new_dataset(
type = 'RAW_DATA',
sample = '/SPACE/SAMP1',
@@ -693,15 +758,17 @@ ds_new.save()
* `my_file.txt` --> `/my_file.txt`
* `../somwhere/else/my_other_file.txt` --> `/my_other_file.txt`
* `some/folder/file.txt` --> `/file.txt`
+* useful if DataSet contains files and folders
+* the content of the folder will be zipped (on-the-fly) and uploaded to openBIS
+* openBIS will keep the folder structure intact
+* relative path will be shortened to its basename. For example:
+
+| local | openBIS |
+|----------------------------|------------|
+| `../../myData/` | `myData/` |
+| `some/experiment/results/` | `results/` |
```
-# Dataset containing files and folders
-# the content of the folder will be zipped (on-the-fly) and uploaded to openBIS.
-# openBIS will keep the folder structure intact.
-# relative path will be shortened to its basename. For example:
-# local openBIS
-# ../../myData/ myData/
-# some/experiment/results/ results/
ds_new = o.new_dataset(
type = 'RAW_DATA',
sample = '/SPACE/SAMP1',
@@ -712,8 +779,9 @@ ds_new.save()
### create dataSet container
+A DataSet of kind=CONTAINER contains other DataSets, but no files:
+
```
-# DataSet CONTAINER (contains other DataSets, but no files)
ds_new = o.new_dataset(
type = 'ANALYZED_DATA',
experiment = '/SPACE/PROJECT/EXP1',
@@ -766,8 +834,11 @@ dataset.del_components(['20170115220259155-412'])
```
## Semantic Annotations
+
+create semantic annotation for sample type 'UNKNOWN':
+
```
-# create semantic annotation for sample type 'UNKNOWN'
+
sa = o.new_semantic_annotation(
entityType = 'UNKNOWN',
predicateOntologyId = 'po_id',
@@ -778,33 +849,56 @@ sa = o.new_semantic_annotation(
descriptorAccessionId = 'da_id'
)
sa.save()
+```
+
+Create semantic annotation for property type (predicate and descriptor values omitted for brevity)
-# create semantic annotation for property type
-# (predicate and descriptor values omitted for brevity)
+```
sa = o.new_semantic_annotation(propertyType = 'DESCRIPTION', ...)
sa.save()
+```
-# create semantic annotation for sample property assignment (predicate and descriptor values omitted for brevity)
-sa = o.new_semantic_annotation(entityType = 'UNKNOWN', propertyType = 'DESCRIPTION', ...)
+**Create** semantic annotation for sample property assignment (predicate and descriptor values omitted for brevity)
+
+```
+sa = o.new_semantic_annotation(
+ entityType = 'UNKNOWN',
+ propertyType = 'DESCRIPTION',
+ ...
+)
sa.save()
+```
-# create a semantic annotation directly from a sample type
-# will also create sample property assignment annotations when propertyType is given
+**Create** a semantic annotation directly from a sample type. Will also create sample property assignment annotations when propertyType is given:
+
+```
st = o.get_sample_type("ORDER")
st.new_semantic_annotation(...)
+```
+
+**Get all** semantic annotations
-# get all semantic annotations
+```
o.get_semantic_annotations()
+```
-# get semantic annotation by perm id
+**Get** semantic annotation by perm id
+
+```
sa = o.get_semantic_annotation("20171015135637955-30")
+```
+
+**Update** semantic annotation
-# update semantic annotation
+```
sa.predicateOntologyId = 'new_po_id'
sa.descriptorOntologyId = 'new_do_id'
sa.save()
+```
-# delete semantic annotation
+**Delete** semantic annotation
+
+```
sa.delete('reason')
```
@@ -830,8 +924,8 @@ tag.delete('why?')
An entity such as Sample (Object), Experiment (Collection), Material or DataSet can be of a specific *entity type*:
-* Sample Type
-* Experiment Type
+* Sample Type (Object Type)
+* Experiment Type (Collection Type)
* DataSet Type
* Material Type
diff --git a/pybis/src/python/pybis/__init__.py b/pybis/src/python/pybis/__init__.py
index b0e578dcb1db65cd510e7b661f80be5f42689c29..f4acf1fd1cd6ab42c3d18823bd19b17e74d58d78 100644
--- a/pybis/src/python/pybis/__init__.py
+++ b/pybis/src/python/pybis/__init__.py
@@ -1,7 +1,7 @@
name = 'pybis'
__author__ = 'Swen Vermeul'
__email__ = 'swen@ethz.ch'
-__version__ = '1.11.1'
+__version__ = '1.12.0'
from . import pybis
from .pybis import Openbis
diff --git a/pybis/src/python/pybis/pybis.py b/pybis/src/python/pybis/pybis.py
index 4729f9e0d3c8beed22bcb9468c4baa0483e2204f..4496e54d94d7db1acea79771615b4bdb05ec4f58 100644
--- a/pybis/src/python/pybis/pybis.py
+++ b/pybis/src/python/pybis/pybis.py
@@ -1727,9 +1727,36 @@ class Openbis:
self, identifier=None, code=None, permId=None,
space=None, project=None, experiment=None, collection=None, type=None,
start_with=None, count=None,
- withParents=None, withChildren=None, tags=None, props=None, **properties
+ withParents=None, withChildren=None, tags=None, attrs=None, props=None, **properties
):
- """ Get a list of all samples for a given space/project/experiment (or any combination)
+ """Returns a DataFrame of all samples for a given space/project/experiment (or any combination)
+ Filters:
+ --------
+ space -- a space code or a space object
+ project -- a project code or a project object
+ experiment -- an experiment code or an experiment object
+ collection -- same as experiment
+ tags -- only return samples with the specified tags
+ type -- a sampleType code
+
+ Paging:
+ -------
+ start_with -- default=None
+ count -- number of samples that should be fetched. default=None.
+
+ Include:
+ --------
+ withParents -- the list of parent's permIds in a column 'parents'
+ withChildren -- the list of children's permIds in a column 'children'
+ attrs -- list of all desired attributes. Examples:
+ space, project, experiment: just return their identifier
+ space.code, project.code, experiment.code
+ registrator.email, registrator.firstName
+ type.generatedCodePrefix
+ props -- list of all desired properties. Returns an empty string if
+ a) property is not present
+ b) property is not defined for this sampleType
+
"""
if collection is not None:
@@ -1743,6 +1770,7 @@ class Openbis:
if space:
sub_criteria.append(_subcriteria_for(space, 'space'))
+
if project:
sub_criteria.append(_subcriteria_for(project, 'project'))
if experiment:
@@ -1771,35 +1799,54 @@ class Openbis:
"operator": "AND"
}
+ options = self._get_fetchopts_for_attrs(attrs)
+
# build the various fetch options
fetchopts = fetch_option['sample']
fetchopts['from'] = start_with
fetchopts['count'] = count
- for option in ['tags', 'properties', 'registrator', 'modifier', 'experiment']:
+ for option in ['tags', 'properties', 'registrator', 'modifier']+options:
fetchopts[option] = fetch_option[option]
request = {
"method": "searchSamples",
- "params": [self.token,
- criteria,
- fetchopts,
- ],
+ "params": [
+ self.token,
+ criteria,
+ fetchopts,
+ ],
}
resp = self._post_request(self.as_v3, request)
return self._sample_list_for_response(
response=resp['objects'],
+ attrs=attrs,
props=props,
start_with=start_with,
count=count,
totalCount=resp['totalCount'],
)
-
get_objects = get_samples # Alias
+ def _get_fetchopts_for_attrs(self, attrs=None):
+ if attrs is None:
+ return []
+
+ fetchopts = []
+ for attr in attrs:
+ if attr.startswith('space'): fetchopts.append('space')
+ if attr.startswith('project'): fetchopts.append('project')
+ if attr.startswith('experiment'): fetchopts.append('experiment')
+ if attr.startswith('sample'): fetchopts.append('sample')
+ if attr.startswith('registrator'): fetchopts.append('registrator')
+ if attr.startswith('modifier'): fetchopts.append('modifier')
+
+ return fetchopts
+
+
def get_experiments(
self, code=None, permId=None, type=None, space=None, project=None,
start_with=None, count=None,
@@ -1896,8 +1943,36 @@ class Openbis:
self, code=None, type=None, withParents=None, withChildren=None,
start_with=None, count=None, kind=None,
status=None, sample=None, experiment=None, collection=None, project=None,
- tags=None, props=None, **properties
+ tags=None, attrs=None, props=None, **properties
):
+ """Returns a DataFrame of all dataSets for a given project/experiment/sample (or any combination)
+ Filters:
+ --------
+ project -- a project code or a project object
+ experiment -- an experiment code or an experiment object
+ sample -- a sample code/permId or a sample/object
+ collection -- same as experiment
+ tags -- only return dataSets with the specified tags
+ type -- a dataSetType code
+
+ Paging:
+ -------
+ start_with -- default=None
+ count -- number of dataSets that should be fetched. default=None.
+
+ Include:
+ --------
+ withParents -- the list of parent's permIds in a column 'parents'
+ withChildren -- the list of children's permIds in a column 'children'
+ attrs -- list of all desired attributes. Examples:
+ project, experiment, sample: just return their identifier
+ space.code, project.code, experiment.code
+ registrator.email, registrator.firstName
+ type.generatedCodePrefix
+ props -- list of all desired properties. Returns an empty string if
+ a) property is not present
+ b) property is not defined for this dataSetType
+ """
if 'object' in properties:
sample = properties['object']
@@ -1945,6 +2020,7 @@ class Openbis:
}
fetchopts['from'] = start_with
fetchopts['count'] = count
+
if kind:
kind = kind.upper()
if kind not in ['PHYSICAL_DATA', 'CONTAINER', 'LINK']:
@@ -1952,7 +2028,8 @@ class Openbis:
fetchopts['kind'] = kind
raise NotImplementedError('you cannot search for dataSet kinds yet')
- for option in ['tags', 'properties', 'sample', 'experiment', 'physicalData']:
+ options = self._get_fetchopts_for_attrs(attrs)
+ for option in ['tags', 'properties', 'physicalData']+options:
fetchopts[option] = fetch_option[option]
request = {
@@ -1966,6 +2043,7 @@ class Openbis:
return self._dataset_list_for_response(
response=resp['objects'],
+ attrs=attrs,
props=props,
start_with=start_with,
count=count,
@@ -3320,22 +3398,40 @@ class Openbis:
def _dataset_list_for_response(
- self, response, props=None,
+ self, response, attrs=None, props=None,
start_with=None, count=None, totalCount=0
):
"""returns a Things object, containing a DataFrame plus some additional information
"""
+ def extract_attribute(attribute_to_extract):
+ def return_attribute(obj):
+ if obj is None: return ''
+ return obj.get(attribute_to_extract,'')
+ return return_attribute
parse_jackson(response)
- attrs = ['permId', 'type', 'experiment', 'sample',
+
+ if attrs is None: attrs=[]
+ default_attrs = ['permId', 'type', 'experiment', 'sample',
'registrationDate', 'modificationDate',
'location', 'status', 'presentInArchive', 'size',
'properties'
]
+ display_attrs = default_attrs + attrs
+
if len(response) == 0:
- datasets = DataFrame(columns=attrs)
+ datasets = DataFrame(columns=display_attrs)
else:
datasets = DataFrame(response)
+ for attr in attrs:
+ if '.' in attr:
+ entity, attribute_to_extract = attr.split('.')
+ datasets[attr] = datasets[entity].map(extract_attribute(attribute_to_extract))
+ for attr in attrs:
+ # if no dot supplied, just display the code of the space, project or experiment
+ if attr in ['space', 'project', 'experiment', 'sample']:
+ datasets[attr] = datasets[attr].map(extract_nested_identifier)
+
datasets['registrationDate'] = datasets['registrationDate'].map(format_timestamp)
datasets['modificationDate'] = datasets['modificationDate'].map(format_timestamp)
datasets['experiment'] = datasets['experiment'].map(extract_nested_identifier)
@@ -3351,13 +3447,18 @@ class Openbis:
if isinstance(props, str):
props = [props]
for prop in props:
- datasets[prop.upper()] = datasets['properties'].map(lambda x: x.get(prop.upper(), ''))
- attrs.append(prop.upper())
+ if datasets.get('properties') is not None:
+ datasets[prop.upper()] = datasets['properties'].map(
+ lambda x: x.get(prop.upper(), '')
+ )
+ else:
+ datasets[prop.upper()] = ''
+ display_attrs.append(prop.upper())
return Things(
openbis_obj = self,
entity = 'dataset',
- df = datasets[attrs],
+ df = datasets[display_attrs],
identifier_name = 'permId',
start_with=start_with,
count=count,
@@ -3430,40 +3531,63 @@ class Openbis:
props=props,
)
+
def _sample_list_for_response(
- self, response, props=None,
+ self, response, attrs=None, props=None,
start_with=None, count=None, totalCount=0
):
- """returns a Things object, containing a DataFrame plus some additional information
+ """returns a Things object, containing a DataFrame plus additional information
"""
+ def extract_attribute(attribute_to_extract):
+ def return_attribute(obj):
+ if obj is None: return ''
+ return obj.get(attribute_to_extract,'')
+ return return_attribute
parse_jackson(response)
- attrs = ['identifier', 'permId', 'experiment', 'type',
+
+ if attrs is None: attrs = []
+ default_attrs = ['identifier', 'permId', 'type',
'registrator', 'registrationDate', 'modifier', 'modificationDate']
+ display_attrs = default_attrs + attrs
+
if len(response) == 0:
- samples = DataFrame(columns=attrs)
+ samples = DataFrame(columns=display_attrs)
else:
samples = DataFrame(response)
+ for attr in attrs:
+ if '.' in attr:
+ entity, attribute_to_extract = attr.split('.')
+ samples[attr] = samples[entity].map(extract_attribute(attribute_to_extract))
+ for attr in attrs:
+ # if no dot supplied, just display the code of the space, project or experiment
+ if attr in ['space', 'project', 'experiment']:
+ samples[attr] = samples[attr].map(extract_nested_identifier)
+
samples['registrationDate'] = samples['registrationDate'].map(format_timestamp)
samples['modificationDate'] = samples['modificationDate'].map(format_timestamp)
samples['registrator'] = samples['registrator'].map(extract_person)
samples['modifier'] = samples['modifier'].map(extract_person)
samples['identifier'] = samples['identifier'].map(extract_identifier)
samples['permId'] = samples['permId'].map(extract_permid)
- samples['experiment'] = samples['experiment'].map(extract_nested_identifier)
samples['type'] = samples['type'].map(extract_nested_permid)
if props is not None:
if isinstance(props, str):
props = [props]
for prop in props:
- samples[prop.upper()] = samples['properties'].map(lambda x: x.get(prop.upper(), ''))
- attrs.append(prop.upper())
+ if samples.get('properties') is not None:
+ samples[prop.upper()] = samples['properties'].map(
+ lambda x: x.get(prop.upper(), '')
+ )
+ else:
+ samples[prop.upper()] = ''
+ display_attrs.append(prop.upper())
return Things(
openbis_obj = self,
entity = 'sample',
- df = samples[attrs],
+ df = samples[display_attrs],
identifier_name = 'identifier',
start_with=start_with,
count=count,
diff --git a/pybis/src/python/setup.py b/pybis/src/python/setup.py
index 08a6e22b1766a31aa4cf8219855a2d1023e9b372..db38bf27d7f69d51f0c738a33c5256a775f71836 100644
--- a/pybis/src/python/setup.py
+++ b/pybis/src/python/setup.py
@@ -11,7 +11,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
setup(
name='PyBIS',
- version= '1.11.1',
+ version= '1.12.0',
author='Swen Vermeul • ID SIS • ETH Zürich',
author_email='swen@ethz.ch',
description='openBIS connection and interaction, optimized for using with Jupyter',