Commit bd3656c9 authored by vermeul's avatar vermeul
Browse files

Merge branch 'release/0.2.0'

parents edc18fb8 1885b110
## jupyter-openbis-server 0.2.0
- added full API description
- added mount/unmount support
- added logout support
## jupyter-openbis-server 0.1.3
- enforce compatibility with pyBIS 1.14.3
......
# Jupyter openBIS Server
The server part of the `jupyter-openbis-extension` and `jupyterlab-openbis` notebook extension. Uses the `pyBIS` module internally to communicate with openBIS. Communicates with the notebook extensions via the tornado webserver.
This server is an extension to the Jupyter notebook server and is part of the `jupyter-openbis-extension` and `jupyterlab-openbis` notebook extensions. It uses the `pyBIS` module internally to communicate with openBIS and ommunicates with the notebook extensions via the built-in tornado webserver.
## Install the server extension
......@@ -46,3 +45,312 @@ Unfortunately, `pip` doesn't automatically clean up the Jupyter configuration wh
$ jupyter serverextension disable --py jupyter-openbis-server
$ pip uninstall jupyter-openbis-server
```
## Server extension API documentation
### XSRF Token in `POST`, `PUT` and `DELETE` requests
XSRF (or CSRF) stands for Cross-Site-Request-Forgery.
For all **POST**, **PUT** and **DELETE** requests, the following **http headers** must be submitted as http headers:
```
"X-XSRFToken": xsrf_token,
"credentials": "same-origin"
```
The value of the `xsrf_token` is the value of the `_xsrf` cookie which is stored in the users' browser. Without this http header information, the request will fail. All **GET** requests can be established without a special header.
The underlying Tornado-Webserver which handles all requests to the Jupyter serverextension will throw an error if the X-XSRF Token is not present.
### Errors
Errors caused by a `POST`, `PUT` and `DELETE` request will result in a HTTP Status > 300 and an error message:
```
{
"reason": "Incorrect username or password for openBIS instance"
}
```
### get openBIS connections
**GET `/openbis/conns`**
Returns an array of JSON objects:
```
{
"status": 200,
"connections": [
{
"name": "openBIS instance",
"url": "https://openbis.instance.ch",
"status": "connected",
"username": "user_name",
"password": "******",
"isMounted": false,
"mountpoint": ""
}
],
"notebook_dir": "/home/user_name/project_dir"
}
```
* the **`name`** is the name of the connection being used when downloading or uploading dataSets (see below)
* the **`url`** of the openBIS instance
* the values of `status` can be either **connected** or **not connected**
* the **`username`** being used in openBIS
* the **`password`** really only consists of a number of asteriks **\***. If they are passed as such to re-connect to openBIS, the server tries to use the internally saved password instead. The password only lives in memory of the singleuser notebook-server and is not saved persistently.
* **`isMounted`** is either **true** or **false**, depending whether there is a current FUSE/SSHFS mountpoint available which connects to the openBIS dataStore
* `mountpoint` is the path to the mounted openBIS dataStore. It defaults to `$HOME/<openbis hostname>`
### login to an openBIS connection
An openBIS connection that has to be established or has timed out: a new login has to take place.
**PUT `/openbis/conn`**
Body:
```
{
"username": username,
"password": password,
"action": "login",
}
```
The `action` attribute defaults to `login`. Returns:
```
{
"status": 200,
"connection": {
"name": "openBIS instance",
"url": "https://openbis.instance.ch",
"status": "connected",
"username": "some_username",
"password": "******",
"isMounted": false,
"mountpoint": ""
}
}
```
### logout
Logs out from an openBIS instance, i.e. the token is invalidated. The mount might still persist, as it is a separate connection. The status changes from **connected** to **not connected**
**PUT `/openbis/conn`**
Body:
```
{
"action": "logout",
}
```
Returns:
```
{
"status": 200,
"connection": {
"name": "openBIS instance",
"url": "https://openbis.instance.ch",
"status": "not connected",
"username": "some_username",
"password": "******",
"isMounted": true,
"mountpoint": "/Users/some_username/openbis.instance.ch"
}
}
```
### Mount to an openBIS dataStore
#### Prerequisites
On the Jupyter Server, FUSE/SSHFS must be installed beforehand (requires root privileges). For the actual mount to the openBIS dataStore, no special privileges are required.
For **Mac OS X**, follow the installation instructions on [https://osxfuse.github.io](https://osxfuse.github.io)
For **Unix Cent OS 7**, do the following:
```
$ sudo yum install epel-release
$ sudo yum --enablerepo=epel -y install fuse-sshfs
$ user="$(whoami)"
$ usermod -a -G fuse "$user"
```
**Windows** is currently not supported, sorry!
By default, the mountpoint is the same as the hostname of the instance and it is located inside the home of the user. FUSE/SSHFS needs an empty directory to do this, so it will automatically be created.
**PUT `/openbis/conn`**
Body:
```
{
"username": username,
"password": password,
"action" : "mount"
}
```
Returns:
```
{
"status": 200,
"connection": {
"name": "openBIS instance",
"url": "https://openbis.instance.ch",
"status": "connected",
"username": "some_username",
"password": "******",
"isMounted": true,
"mountpoint": "/Users/some_username/openbis.instance.ch"
}
}
```
### Unmount from openBIS dataStore
**PUT `/openbis/conn`**
Body:
```
{
"action" : "mount"
}
```
Returns:
```
{
"status": 200,
"connection": {
"name": "openBIS instance",
"url": "https://openbis.instance.ch",
"status": "connected",
"username": "some_username",
"password": "******",
"isMounted": false,
"mountpoint": ""
}
}
```
### Register a new openBIS connection
For the lifetime (runtime) of the Jupyter server, this will create a connection to openBIS.
**POST `/openbis/conns`**
Body:
```
{
"name": connection_name,
"url": connection_url,
"username": username,
"password": password
}
```
### Upload a dataSet
**POST `/openbis/dataset/<connection_name>/<permId>/<downloadPath>`**
### Download a dataSet
**GET `/openbis/dataset/<connection_name>/<permId>/<downloadPath>`**
* the `connection_name` is the name of the connection given in the connections dialog.
* the `permId` is the identifer of the dataSet that needs to be downloaded.
* the `downloadPath` is the absolute path on the host system where the dataSet files should be downloaded to. The `downloadPath` must be URL-encoded to not to be confused with the URL itself.
In case of a **successful download**, the API returns a JSON like this
```
{
'url' : conn.url,
'permId' : dataset.permId,
'path' : path,
'dataStore' : dataset.dataStore,
'location' : dataset.physicalData.location,
'size' : dataset.physicalData.size,
'files' : dataset.file_list,
'statusText': 'Data for DataSet {} was successfully downloaded to: {}'.format(dataset.permId, path)
}
```
In case of an **error**, the API returns one of these errors (HTTP Status > 200):
**general connection error**
```
HTTP-Status: 500
{
"reason": 'connection to {} could not be established: {}'.format(conn.name, exc)
}
```
**dataSet not found error**
```
HTTP-Status: 404
{
"reason": 'No such dataSet found: {}'.format(permId)
}
```
**dataSet download error**
```
HTTP-Status: 500
{
"reason": 'Data for DataSet {} could not be downloaded: {}'.format(permId, exc)
}
```
### Save `requirements.txt` and `runtime.txt` file
Note: The requirements list and the runtime must be evaluated by executing actual Python or R code from wtihin a notebook cell. The Python used by the Jupyter server might differ from the Python used by the kernel. The usual `pip freeze` doesn't work, as we cannot access the pip CLI from within Python.
For the Python `requirements.txt` we use this script:
```
import pkg_resources
print(
"\n".join(
["{}=={}".format(i.key, i.version) for i in pkg_resources.working_set]
)
)
```
For the Python `runtime.txt`:
```
import sys
print('python-' + str(sys.version_info[0]) + '.' + str(sys.version_info[1]))
```
Once submitted to the server, the server will join the relative `notebook_path` (from the UI) with the server-side `notebook_dir`. These files will be stored in the same location on the filesystem as the notebook itself.
**POST `/openbis/requirements`**
Body:
```
{
"notebook_path": notebook_path,
"requirements_list": state.requirements_list,
"requirements_filename": state.requirements_filename,
"runtime": state.runtime,
"runtime_filename": state.runtime_filename
}
```
\ No newline at end of file
name = 'jupyter-openbis-server.server'
__author__ = 'Swen Vermeul'
__email__ = 'swen@ethz.ch'
__version__ = '0.1.4'
__version__ = '0.2.0'
def _jupyter_server_extension_paths():
return [{
......
......@@ -3,6 +3,7 @@ from pybis import Openbis
from notebook.base.handlers import IPythonHandler
openbis_connections = {}
fake_password = "******"
def register_connection(connection_info):
......@@ -51,17 +52,48 @@ class OpenBISConnection:
def login(self, username=None, password=None):
if username is None:
username=self.username
if password is None:
if password is None or password == fake_password:
password=self.password
self.openbis.login(
username = username,
password = password
password = password,
)
# store username and password in memory
self.username = username
self.password = password
self.status = 'connected'
def logout(self):
self.openbis.logout()
self.status = "not connected"
def mount(self, username=None, password=None, hostname=None):
if username is None:
username=self.username
if password is None or password == fake_password:
password=self.password
if hostname is None:
hostname=self.openbis.hostname
self.openbis.mount(
username = username,
password = password,
hostname = hostname,
)
self.mount_status = 'mounted'
def unmount(self):
mountpoint = getattr(self.openbis, 'mountpoint', None)
if mountpoint is None:
try:
mountpoint = self.openbis.get_mountpoint(search_mountpoint=True)
except Exception:
pass
self.openbis.unmount(mountpoint=mountpoint)
def get_info(self):
is_mounted = self.openbis.is_mounted()
mountpoint = ''
......@@ -73,7 +105,7 @@ class OpenBISConnection:
'url' : self.url,
'status' : self.status,
'username' : self.username,
'password' : "******",
'password' : fake_password,
'isMounted' : is_mounted,
'mountpoint': mountpoint,
}
......@@ -132,6 +164,31 @@ class OpenBISConnectionHandler(IPythonHandler):
notebook_dir = self.config.NotebookApp.notebook_dir
return notebook_dir
def delete(self, connection_name):
"""logout and unmount a connection, remove from connection list
"""
try:
conn = openbis_connections[connection_name]
except KeyError:
self.set_status(404)
self.write({
"reason" : 'No such connection: {}'.format(data)
})
return
try:
conn.unmount()
except Exception:
pass
try:
conn.logout()
except Exception:
pass
del openbis_connections[connection_name]
def put(self, connection_name):
"""reconnect to a current connection
:return: an updated connection object
......@@ -147,6 +204,53 @@ class OpenBISConnectionHandler(IPythonHandler):
})
return
if data.get('action','') == 'mount':
try:
conn.mount(
username=data.get('username'),
password=data.get('password'),
)
self.write({
'status' : 200,
'connection' : conn.get_info(),
})
except Exception as err:
self.set_status(500)
self.write({
"reason": "Mounting failed: {}".format(err)
})
return
if data.get('action','') == 'unmount':
try:
conn.unmount()
self.write({
'status' : 200,
'connection' : conn.get_info(),
})
except Exception as err:
self.set_status(500)
self.write({
"reason": "Unmounting failed: {}".format(err)
})
return
if data.get('action','') == 'logout':
try:
conn.logout()
self.write({
'status' : 200,
'connection' : conn.get_info(),
})
except Exception as err:
self.set_status(500)
self.write({
"reason": "logout failed: {}".format(err)
})
return
# no action given, try to connect instead
try:
conn.login(data.get('username'), data.get('password'))
except ConnectionError:
......@@ -168,9 +272,9 @@ class OpenBISConnectionHandler(IPythonHandler):
})
self.write({
'status' : 200,
'connection' : conn.get_info(),
'' : self._notebook_dir()
'status' : 200,
'connection' : conn.get_info(),
'notebook_dir' : self._notebook_dir()
})
def get(self, connection_name):
......@@ -194,5 +298,3 @@ class OpenBISConnectionHandler(IPythonHandler):
'noteboook_dir' : self._notebook_dir()
})
return
......@@ -11,7 +11,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
setup(
name='jupyter-openbis-server',
version= '0.1.4',
version= '0.2.0',
author='Swen Vermeul | ID SIS | ETH Zürich',
author_email='swen@ethz.ch',
description='Server Extension for Jupyter notebooks to connect to openBIS and download/upload datasets, inluding the notebook itself',
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment