Merge branch 'tempdoc20108-2' into 'master'

Tempdoc20108 2

See merge request !43

docs/user-documentation/advance-features/openbis-data-modelling.md

@@ -11,7 +11,7 @@ openBIS has the following data structure:
 4.  **Object**: entity with *user-defined properties* 
 5.  **Dataset**: folder where data files are stored. A dataset has *user-defined properties*      
       
-![image info](img/openbis-data-model.png)
+![image info](img/openbis-data-model-v3.png)
 
 
 *Space* is the top level. Below *Spaces* there are *Projects* and below *Projects* there are *Experiments/Collections*. 
@@ -32,7 +32,7 @@ Access to openBIS is controlled at the *Space* level, *Project* level or openBIS
 
 In the openBIS ELN-LIMS a simplified data model is used, as shown below.  
 
-![image info](img/ELN-data-model.png). 
+![image info](img/ELN-data-model-v2.png). 
 
 In this case, *Objects* can only belong to *Experiments/Collections*.
 

docs/user-documentation/general-admin-users/admins-documentation/import-openbis-exports.md

 # Imports of openBIS exports
 
-It is possible to export metadata and data from one openBIS instance to another.
+It is possible to export metadata and data from one openBIS instance and import them to another openBIS instance.
 The export process is described [here](../../general-users/data-export.md).
 
-The exported metadata (and related masterdata) can be imported in another openBIS instance by an instance admin via the admin UI, as described in [mastedata import and export](./masterdata-exports-and-imports.md).
 
-Exported datasets need to be imported via the [eln-lims default dropbox](../../general-users/data-upload.md#data-upload-via-dropbox).
+## Metadata import
 
-When importing both metadata and data in a different openBIS instance, first the metadata need to be imported and afterwards the data.
+The exported metadata (and related masterdata) can be imported in another openBIS instance by an instance admin via the admin UI, as described in [mastedata import and export](./masterdata-exports-and-imports.md). 
+
+Exported metadata (and masterdata) are contained in a **xlsx** folder, as shown below.
+
+![image info](img/xlxs-folder.png)
+
+
+Metadata and masterdata are contained in the **metadata.xlsx** file. If some of the exported types have validation plugins or dynamic property plugins, these are exported to a **scripts** folder inside the **xlsx** folder. If some exported entities contain either large text fields that exceed the length of an Excel cell or metadata in spreadheets, these are exported to a **data** folder inside the **xlsx** folder.
+
+If a **data** folder and/or a **scripts** folder are present in the exported **xlsx** folder, the **xlsx** folder needs to be zipped and the **xlsx.zip** file can be imported via admin UI.
+If only the **metadata.xlsx** file is contained in the **xlsx** folder, the metadata.xlsx file can be directly uploaded via admin UI.
+
+
+
+## Datasets import
+
+Exported datasets are contained in a **data** folder in a format ready to be imported via [eln-lims default dropbox](../../general-users/data-upload.md#data-upload-via-dropbox).
+
+![image info](img/import-data-folder.png)
+
+The folders contained in the **data** folder need to be placed in the **eln-lims incoming directory** and from here will be uploaded to the corresponsing openBIS entities. The metadata of the datasets is read from the **metadata.json** file contained inside each dataset folder.
+
+When importing both metadata and data in a different openBIS instance, first the metadata need to be imported and afterwards the data. 
 
-If the exported metadata consists of an xlsx folder that contains a metadata.xlsx file and a script folder (see below), the xlsx folder needs to be zipped before import via the admin UI (xlsx.zip).
 
-![image info](img/xls-folder.png)
 

docs/user-documentation/general-users/additional-functionalities.md

@@ -3,11 +3,13 @@ Additional Functionalities
  
 ## Print PDF 
 
-For every entity in openBIS it is possible to generate a pdf using the **Print PDF** option from the **More..** dropdown menu.
+For every entity in openBIS it is possible to generate a pdf using the **Print PDF** option from the **More..** dropdown menu. 
+
+The generated pdf file can be printed or downloaded from the browser. 
 
 An example for a Space is shown in the picture below.
 
-![image info](img/print-pdf.png)
+![image info](img/print-pdf-2.png)
 
 
 ## Visualise Relationships

docs/user-documentation/general-users/data-export.md

-Data Export
+Export
 ====
  
 ## Export to File
@@ -14,26 +14,26 @@ the **Export** option in the **More..** drop down, as shown below.
 
 **Space**
 
-![image info](img/new-export-space.png)
+![image info](img/new-export-space-2.png)
 
 **Project**
 
-![image info](img/new-export-project.png)
+![image info](img/new-export-project-2.png)
 
 
 **Experiment/Collection**
 
-![image info](img/new-export-experiment.png)
+![image info](img/new-export-experiment-2.png)
 
 
 **Object**
 
-![image info](img/new-export-object.png)
+![image info](img/new-export-object-2.png)
 
 
 **Dataset**
 
-![image info](img/new-export-dataset.png)
+![image info](img/new-export-dataset-2.png)
 
 
 In each case, the following export options are available: 
@@ -41,23 +41,119 @@ In each case, the following export options are available:
 ![image info](img/new-export-options.png)
 
  
-- **Make import compatible**. If selected, the exported data are in a format ready to be uploaded in openBIS using the default eln-lims dropbox, and the metadata XLSX file is in a format ready to be uploaded via the admin UI.
-- **Export metadata as pdf**. Metadata is exported in pdf files. One pdf for each exported entity is created.
-- **Export metadata as XLSX**. Metadata is exported in one xlsx file. One file containing the metadata of all exported entities and the corresponding masterdata. If **Make import compatible** is selected, this format is suitable for re-import in openBIS.
-- **Export data**. Default size limit is 10GB. This can be changed in .. If **Make import compatible** is selected, exported data is in a format ready to be uploaded in openBIS using the default eln-lims dropbox. If not, the datasets will be in a folder structure that matches the ELN hierarchy.
-- **Include levels below from same space**. If selected, all hirerachy levels below the selected entity and belonging to the same Space are exported.
-- **Include Object and Dataset parents from same space**. If selected, Object parents and Dataset parents from the same Space are exported. Example: I export Object A, in Experiment A, in Space 1. Object A has Object B as parent, in Experiment B in Space 1. When this option is selected, Object B is also exported, otherwise it is not.
-- **Include Objects and Datasets parents and children from different spaces**. This exports Object and Dataset parents and children that belong to a different Space than the Space from where Objects and Datasets are being exported. Example: I export Object A in Space 1, which has parents in Space 2. If this option is selected, the parents in Space 2 are also exported.
-- **Wait for download to complete in browser**. This is suitable when exporting only metadata or small datasets. When the dowload is ready, the zip file will be available to download from the browser.
+- **Make import compatible**. If selected, datasets are exported in a **data** folder and are in a format ready to be uploaded in openBIS using the default eln-lims dropbox; the metadata are exported in a **xlsx** folder which contains information in a format ready to be uploaded via the openBIS admin UI.
+
+- **Export metadata as PDF**. Metadata are exported in a **pdf** folder that keeps the folder structure of the ELN. At each level, one pdf file for each exported entity is generated.
+
+- **Export metadata as XLSX**. Metadata are exported in one **xlsx** folder. The folder contains the metadata of all exported entities and the corresponding masterdata in a **metadata.xlsx** file. If **Make import compatible** is selected, this file is suitable for re-import in openBIS. If not, the file contains some fields which are not compatible with re-imports. These fields are: PermId of entities, registrator, registration date, modifier, modification date. In addition to the metadata.xlsx file, the **xlsx** folder might contain a **scripts** folder, which contains scripts associated with types in the metadata.xlsx file, if these are present, and a **data** folder which holds the content of spreadsheet fields and large text fields that exceed the size of an Excel cell.
+
+- **Export data**. The default maximum size of all datasets to be exported is 10GB. This can be configured by a system admin in the [AS service.properties file](../../system-documentation/configuration/optional-application-server-configuration.md). We recommend to use [sftp](../general-users/lab-notebook.md#data-access) to download large datasets. 
+If **Make import compatible** is selected, datasets are exported in a **data** folder in a format ready to be uploaded in openBIS using the default eln-lims dropbox. If not, the datasets are exported in a **pdf** folder that matches the ELN hierarchy.
+
+- **Include levels below from same space**. If selected, all hierachy levels below the selected entity and belonging to the same Space are exported.
+
+- **Include Object and Dataset parents from same space**. If selected, Object parents and Dataset parents from the same Space are exported. Example: I export Object A, in Experiment A, in Space 1. Object B in Experiment B also in Space 1 is parent of Object A. When this option is selected, Object B is also exported, otherwise it is not.
+
+- **Include Objects and Datasets parents and children from different spaces**. This allows to export Object and Dataset parents and children that belong to a different Space than the Space from where Objects and Datasets are being exported. Example: I export Object A in Space 1, which has parents in Space 2. If this option is selected, the parents in Space 2 are also exported, otherwise they are not.
+
+- **Wait for download to complete in browser**. This is suitable when exporting only metadata or small datasets. When the dowload is ready, a zip file will be available to download from the browser.
 
 _Note: ensure that pop-ups are not disabled in your browser_.
-- **Receive results by email**. If this option is selected, when the export is ready, you will receive an email notification with a download link.  Email
-notification needs to be configured on *system level* during or after
-installation, as explained in [Configure Data Store
-Server](../../system-documentation/standalone/optional-datastore-server-configuration.md)
+- **Receive results by email**. If this option is selected, when the export is ready, you will receive an email notification with a download link.  Email notification needs to be configured on *system level* during or after installation, as explained in [Configure Data Store
+Server](../../system-documentation/configuration/optional-datastore-server-configuration.md)
 
 
- 
+> ![image info](img/warning.png) **Known limitation of openBIS 20.10.8**: images embedded in text fields are not currently exported. This limitation will be fixed in the upcoming 20.10.8.1 release. 
+
+
+
+
+We provide below a couple of examples of the export, to clarify how it works.
+
+### **1. Import-compatible export of a Space selecting all options**
+
+We select all options from the export widget, as shown below.
+
+ ![image info](img/space-export-all-options-import-compatible.png)
+
+We export a Space called CATERINA in the Lab Notebook with all its sublevels (see below). 
+
+![image info](img/export_example-lab-notebook.png)
+
+One Object in this Space has a parent in a Space called METHODS (see below). 
+
+![image info](img/export_example-inventory.png)
+
+The exported zip file contains 3 folders:
+
+**A.** **data** folder
+
+This contains the datasets in the correct format to be uploaded via eln-lims dropbox, as shown below.
+
+
+![image info](img/IC-data-folder-1.png)
+
+![image info](img/IC-data-folder-2.png)
+
+**B.** **pdf** folder
+
+This contains folders that match the openBIS hierarchy (Space/Project/Experiment/Object).
+
+In this case 2 Space folders are present:
+
+1. **CATERINA**: is the exported space.
+2. **METHODS**: contains an Object which is parent of an Object in the space CATERINA. This was exported because the option **Include Objects and Datasets parents and children from different spaces** was selected for export.
+
+![image info](img/IC-pdf-folder-1.png)
+
+![image info](img/IC-pdf-folder-2.png)
+
+
+Inside each folder, there is a pdf of the corresponding entity. Example: 
+
+- in the Space folder **CATERINA** there is a **CATERINA.pdf** file that contains the metadata of the Space;
+- in the Project folder **PROJECT_1** there is a **PROJECT_1.pdf** file that contains the metadata of the Project;
+- in the Experiment folder **My second experiment (PROJECT_1_EXP_1)** there is a **My second experiment (PROJECT_1_EXP_1).pdf** file with the metadata of the Experiment;
+- in the Object folder **Step A (EXP4)**  there is a **Step A(EXP4).pdf** file with the metadata of the Object and a **20240523100301019-55.pdf** file that contains the metadata of the dataset that belongs to this Object.
+
+
+**C.** **xlsx** folder. 
+
+This contains:
+
+- a **metadata.xlsx** file which has the metadata of the exported entities and the corresponding masterdata (types and properties) in the correct format to be re-imported in another openBIS instance;
+- a **scripts** folder that contains evaluation plugins associated to two types defined in the metadata.xlsx file;
+- a **data** folder that contains the information stored in the spreadsheet field of one of the Objects in this Space.
+
+![image info](img/IC-xlsx-folder-data.png)
+
+![image info](img/IC-xlsx-folder-scripts.png)
+
+
+
+### **2. Non import-compatible export of a Space selecting all options**
+
+We export the same Space as described in Example 1, with all options selected, but the export this time is not import-compatible, as shown below.
+
+ ![image info](img/space-export-all-options-non-import-compatible.png)
+
+ 
+In this case the exported zip file contains only 2 folders: **pdf** and **xlsx**. Data are exported inside the pdf folder, instead of being in a separate **data** folder.
+
+**A.** **pdf** folder
+
+This contains the same folder structure as described above. In addition, in this case, inside the Object **Step A (EXP4)** folder there is a **data** folder that contains the dataset belonging to this Object, as shown below. The metadata of the dataset is provided as a metadata.json file inside the data folder and as pdf file inside the Object folder (**Step A (EXP4)**).
+
+
+![image info](img/NIC-pdf-folder.png)
+
+**B.** **xlsx** folder 
+
+This contains the same files and folders as described in Example 1 (see below). The only difference in this case is that the metadata.xlsx is not import-compatible. It contains some fields which are not compatible with openBIS re-import, as explained above.
+
+![image info](img/NIC-xlsx-folder-data.png)
+
+![image info](img/NIC-xlsx-folder-scripts.png)
 
  
 ## Export to Zenodo