Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
JupyterHub for openBIS
======================
## Overview
SIS provides a Docker image for the installation of a JupyterHub server,
available at <https://hub.docker.com/r/openbis/>.
This guide explains how to modify JupyterHub docker images and save
them. It is aimed at users who are not familiar with Docker, but it
should not be considered a substitute of the official Docker
documentation.
** **We
advise non expert users, to first test the instructions provided in this
guide on their local machine, to familiarise themselves with the
process, before making changes on the JupyterHub server.
Docker images are stateless, which means that after rebooting all
changes made will not be saved. This guarantees a stable environment,
which is particularly desirable to run services.
If a user wants to introduce changes, the docker image needs to be
updated. There are two possibilities for this:
- **For testing**: Login into the Docker container, modify it and save
the modified container as a new image. This method is not
recommended for production because it is not compatible with
official JupyterHub Docker image upgrades released by SIS.
- **For correct maintenance**: Extend the current image using a Docker
recipe that includes only your changes. This method is recommended
for production, because when a new official JupyterHub Docker image
is released by SIS, it will be possible to quickly apply the changes
to this image from the Docker recipe.
### Nomenclature
**Docker**: A computer program that performs operating-system-level
virtualisation also known as containerisation. The official website can
be found here <https://www.docker.com/>.
**Docker image**: Docker images describe the environment to virtualise.
Docker images are stateless.
**Docker container**: Docker containers provide the environment to
execute the images.
### Prerequisites for testing in a local environment
1. **Docker environment**. All examples shown below require a working
docker environment. Please visit <https://www.docker.com> to
download the Docker Community Edition for your OS.
2. **JupyterHub Docker image**. The jupyterhub-openbis images can be
found at <https://hub.docker.com/r/openbis/>. They can be installed
locally like any other Docker Hub image.
3. **openBIS installation** (optional).
How to run the official JupyterHub for openBIS image in your local machine
--------------------------------------------------------------------------
1\. After downloading the jupyterhub-openbis, find the id of your image.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
openbis/jupyterhub-openbis-sis-20180405 latest 585a9adf333b 23 hours ago 4.75GB
2\. Run the image with one of the two following commands:
a\. if you want to connect to your productive openBIS instance (e.g.
https://openbis-elnlims.ch), use the following command:
docker run -e OPENBIS_URL=https://openbis-elnlims.ch -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p 8002:8002 585a9adf333b ./vagrant/initialize/start_jupyterhub.sh
b\. if you have a local openBIS installation for testing, you can run
the following command:
docker run -v /Users/juanf/jupyterhub-local/home:/home -v /Users/juanf/jupyterhub-local/config/certificates:/vagrant/config/certificates -e OPENBIS_URL=https://129.132.228.42:8443 -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p 8002:8002 585a9adf333b ./vagrant/initialize/start_jupyterhub.sh
> :warning: **Please note the following configuration options:**
> 1. -v /Users/juanf/jupyterhub-local/home:/home -
> This option is only required if you want to store the changes you are making. You need to have a home directory for this. It is not necessary for testing, as the image will provide a default one. This directory should contain a "vagrant" sub directory.
> 2. -v /Users/juanf/jupyterhub-local/config/certificates:/vagrant/config/certificates -
> This option is only required in production environments where you need valid certificates. It is not necssary for testing, as the image will provide a default one.
> 3. OPENBIS_URL= https://129.132.228.42:8443 - By defaut docker is in bridge mode, which means that your docker container accesses your local machine network directly through it. If you have a local openBIS installation please use your IP address; if you use a server installation use the typical address you use to access it.
To stop a running docker container, run "**docker kill container\_ID". **
The container\_ID can be found by running the command **"docker ps".**
-------------------------------------------------------
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
Modify a currently running container - From UI (for users)
----------------------------------------------------------
Please note that libraries installed in this way are NOT permanently
saved. After upgrade of the image, the libraries need to be
re-installed.
### Check Available Python 2 Libraries
help("modules")
### Add Python 2 Library
It can probably be done but we are currently not supporting it.
### Check Available Octave Libraries
pkg list
### Add Octave Library
It can probably be done but we are currently not supporting it.
### Check Available Python 3 Libraries
pip freeze
### Add Python 3 Library
1\. Use pip install as you would normally do. The Python 3 kernel often
doesn't need to be restarted to pick up new libraries, but is
recommended to do so.
### Check Available R Libraries
my_packages <- library()$results
head(my_packages, 1000000)
### Add R Library
1\. Use the install command as you would normally do. The R kernel needs
to be restarted to pick up new libraries.
Modify a currently running container - From Console (for admins)
----------------------------------------------------------------
1\. Find the container id of the image currently running.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2b76d1dd204 jupyterhub-openbis-sis-20180405 "./vagrant/initial..." 4 seconds ago Up 2 seconds 0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8081->8081/tcp nervous_leakey
2\. Log into the container.
$ docker exec -it a2b76d1dd204 bash
### Add Python Library
Add a new library to Python 3
# First we should move to the environment used by JupyterHub
[root@a2b76d1dd204 /]# export PATH=/vagrant_installation/miniconda3/bin:$PATH
[root@a2b76d1dd204 /]# export LC_ALL=en_US.utf8
[root@a2b76d1dd204 /]# export LANG=en_US.utf8
# Install a new python lib using pip
[root@a2b76d1dd204 /]# python --version
Python 3.6.4 :: Anaconda, Inc.
[root@a2b76d1dd204 /]# pip install prettytable
This type of changes can be validated straightaway in JupyterHub, by
just starting a Python 3 notebook. Other changes could require to reboot
JupyterHub.
Please note that this approach should only be used for testing. To
preserve the changes, the running container should be saved as a new
image, otherwise when the container is shutdown these changes will be
lost.
### Add R Library
Add a new library to R
# First we should move to the environment used by JupyterHub
[root@a2b76d1dd204 /]# export PATH=/vagrant_installation/miniconda3/bin:$PATH
[root@a2b76d1dd204 /]# export LC_ALL=en_US.utf8
[root@a2b76d1dd204 /]# export LANG=en_US.utf8
# Install a new r lib using conda
[root@a2b76d1dd204 /]# sudo conda list r-
[root@a2b76d1dd204 /]# sudo conda install -c r -y r-base64enc
This type of changes can be validated straightaway in JupyterHub, by
just starting a R notebook. Other changes could require to reboot
JupyterHub.
### Save the state of a running container as a new image
If you know that you have made significant changes that you want to keep
until you build a new docker recipe, you have the option to save the
running container as a new image.
bs-mbpr28:jupyterhub_reference_installation juanf$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2b76d1dd204 jupyterhub-openbis-sis-20180405 "./vagrant/initial..." 37 minutes ago Up 37 minutes 0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8081->8081/tcp lucid_stonebraker
$ docker commit a2b76d1dd204 jupyterhub-openbis-sis-juanextensions-20180406
sha256:5dd0036664c75a21d6a62b80bf5780e70fcad345bb12a7ad248d01e29a3caa99
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyterhub-openbis-sis-juanextensions-20180406 latest 5dd0036664c7 4 seconds ago 4.75GB
jupyterhub-openbis-sis-20180405 latest 585a9adf333b 23 hours ago 4.75GB
Extend a docker image using a docker recipe (for maintenance)
-------------------------------------------------------------
The recommended approach for maintenance purposes is to extend the
latest official docker image distributed by SIS.
Using our last example, let's create a file called "Dockerfile" and with
the content shown below.
# vim:set ft=dockerfile:
FROM openbis/jupyterhub-openbis-sis-20180405
## Adding Python 3 library
RUN export PATH=/vagrant_installation/miniconda3/bin:$PATH && \
export LC_ALL=en_US.utf8 && \
export LANG=en_US.utf8 && \
pip install prettytable
Please change the name of the image in the file to the one you are
using.
Now we can create a new image using as a starting point the latest from
the official repository.
**** It
is best practice to include both the name of the user and the creation
date in the image name. This will help when dealing with many versions
created by different users at different times.
$ docker build -t jupyterhub-openbis-sis-juanextensions-recipe-20180406 .
Sending build context to Docker daemon 4.957GB
Step 1/2 : FROM openbis/jupyterhub-openbis-sis-20180405
....
Step 2/2 : RUN export PATH=/vagrant_installation/miniconda3/bin:$PATH && export LC_ALL=en_US.utf8 && export LANG=en_US.utf8 && pip install prettytable
....
Successfully tagged jupyterhub-openbis-sis-juanextensions-recipe-20180406:latest
The new image is now available and can be started as described above.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyterhub-openbis-sis-juanextensions-recipe-20180406 latest a0106501b223 3 minutes ago 4.75GB
openbis/jupyterhub-openbis-sis-20180405 latest 585a9adf333b 23 hours ago 4.75GB
How to start a jupyterhub-openbis docker image on a productive JupyterHub server
--------------------------------------------------------------------------------
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
**** You
can only have **ONE** jupyterhub-openbis image running on a server at
one given time, since JupyterHub makes use of certain ports on the
machine that are also configured in openBIS.
1. Find the jupyterhub-openbis-start.sh file in your server (please ask
your admin).
2\. Find the container id of the image that is currently running.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2b76d1dd204 jupyterhub-openbis-sis-20180405 "./vagrant/initial..." 4 seconds ago Up 2 seconds 0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8081->8081/tcp nervous_leakey
3\. Stop the current container.
$ docker kill a2b76d1dd204
a2b76d1dd204
4\. Edit the jupyterhub-openbis-start.sh file in your server and update
the name of the image it runs to the one of your choice
docker run -v /Users/juanf/Documents/programming/git/jupyter-openbis-integration/jupyterhub_reference_installation/home:/home -v /Users/juanf/Documents/programming/git/jupyter-openbis-integration/jupyterhub_reference_installation/vagrant/config/certificates:/vagrant/config/certificates -e OPENBIS_URL=https://129.132.229.37:8443 -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p 8002:8002 jupyterhub-openbis-sis-20180405 ./vagrant/initialize/start_jupyterhub.sh
5\. Start the new image.
$ ./jupyterhub-openbis-start.sh
Other useful Docker commands
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
### Save an image as a tar file to share it
**** It
is best practice to include both the name of the user and the creation
date in the image name. This will help when dealing with many versions
created by different users at different times.
$ docker save jupyterhub-openbis-sis-20180405 > jupyterhub-openbis-sis-20180405.tar
$ ls -lah
total 9681080
-rw-r--r-- 1 juanf 1029 4.6G Apr 5 15:38 jupyterhub-openbis-sis-20180405.tar
### Load an image from a tar file
$ docker load < jupyterhub-openbis-sis-20180405.tar
8feeda13d3ce: Loading layer [==================================================>] 27.65kB/27.65kB
622cd2c170f3: Loading layer [==================================================>] 152MB/152MB
633fa40a6caa: Loading layer [==================================================>] 2.048kB/2.048kB
7219a9159e4f: Loading layer [==================================================>] 223.9MB/223.9MB
678b55e862c7: Loading layer [==================================================>] 4.377GB/4.377GB
Loaded image: jupyterhub-openbis-sis-20180405:latest
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyterhub-openbis-sis-20180405 latest 585a9adf333b 24 hours ago 4.75GB
### Remove an image
$ docker rmi jupyterhub-openbis-sis-juanextensions-recipe-20180406
### Remove all stopped containers
$ docker rm $(docker ps -aq)
openBIS ELN Integration Configuration
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
On the openBIS end, what needs to be done is to append the following
lines into your ELN instance profile:
servers/core-plugins/eln-lims/1/as/webapps/eln-lims/html/etc/InstanceProfile.js
# Ansible yml syntax, replace the variables in the double curly braces by the appropriate values:
this.jupyterIntegrationServerEndpoint = "https://{{ openbis_jupyterhub_hostname }}:{{ openbis_jupyterhub_communication_port }}";
this.jupyterEndpoint = "https://{{ openbis_jupyterhub_hostname }}/";
# Example:
this.jupyterIntegrationServerEndpoint = "https://jupyterhub-demo.labnotebook.ch:80";
this.jupyterEndpoint = "https://jupyterhub-demo.labnotebook.ch/";
On the jupyterhub end, the docker command would then look as follows:
docker run -e OPENBIS_URL=https://{{ openbis_public_hostname }} -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p {{ openbis_jupyterhub_communication_port }}:8002 585a9adf333b ./vagrant/initialize/start_jupyterhub.sh
# Example:
openbis_public_hostname: openbis-test.ethz.ch
openbis_jupyterhub_hostname: jupyterhub-test.ethz.ch
openbis_jupyterhub_communication_port: 80
The only port you need to open on your jupyterhub instance is the one
matching {{ openbis\_jupyterhub\_communication\_port }}. Using
firewall-cmd this would look as follows:
firewall-cmd --permanent --zone=public --add-rich-rule='rule family="ipv4" source address="{{ openbis_jupyterhub_openbis_hostname }}" port protocol="tcp" port="{{ openbis_jupyterhub_communication_port }}" accept'
Troubleshooting Connectivity to openBIS
---------------------------------------
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
Currently only connecting to the openBIS server used to validate your
log in is supported.
### Session is no longer valid. Please log in again error
This error can be show due to two reasons:
- The openBIS server has a self-signed certificates for whatever
reason, typically this is true for test servers.
- The session has timeout.
For each one of them a different fix needs to be applied.
#### Session is no longer valid. The openBIS server has a self-signed certificate
To fix this issue please allow self signed certificates when connecting
as on the example shown below using the verify\_certificates modifier.
#### Session is no longer valid. The session has timeout
The session token obtained during the login is stored by the Jupiter
server during its startup. This session token has most likely timeout
after a couple of days without use and needs to be refreshed. If you
just log in to JupyterHub there is a new session available that needs to
be handed over to the Jupyter server. For that just stop and start it
again.
Step 1 : Go to your control panel clicking on the button of the top
right corner.
Step 2: Press Stop My Server
Step 3 : Press Start My Server
Step 4: Press Launch My Server
Step 5: Wait for the server to startup, after the startup finishes go
back to your notebook, it should connect now.