Merge branch 'temp-master-20109' into 'master'

Temp master 20109

See merge request !46

docs/user-documentation/advance-features/command-line-tool.md

@@ -619,7 +619,7 @@ PAT generation is explained in depth in `token` command section.
 The Big Data Link Services can be used to download files which are contained in an obis repository.
 The services are included in the installation folder of openBIS,
 under `servers/big_data_link_services`. For how to configure and run them, consult
-the [README.md](https://sissource.ethz.ch/sispub/openbis/blob/master/big_data_link_server/README.md)
+the [README.md](https://sissource.ethz.ch/sispub/openbis/-/blob/master/server-external-data-store/README.md)
 file.
 
 ## 8. Rationale for obis

docs/user-documentation/general-users/data-export.md

@@ -43,12 +43,15 @@ In each case, the following export options are available:
  
 - **Make import compatible**. If selected, datasets are exported in a **data** folder and are in a format ready to be uploaded in openBIS using the default eln-lims dropbox; the metadata are exported in a **xlsx** folder which contains information in a format ready to be uploaded via the openBIS admin UI.
 
-- **Export metadata as PDF**. Metadata are exported in a **pdf** folder that keeps the folder structure of the ELN. At each level, one pdf file for each exported entity is generated.
+- **Export metadata as PDF**. Metadata are exported in a **hierarchy** folder that keeps the folder structure of the ELN. At each level, one pdf file for each exported entity is generated.
 
-- **Export metadata as XLSX**. Metadata are exported in one **xlsx** folder. The folder contains the metadata of all exported entities and the corresponding masterdata in a **metadata.xlsx** file. If **Make import compatible** is selected, this file is suitable for re-import in openBIS. If not, the file contains some fields which are not compatible with re-imports. These fields are: PermId of entities, registrator, registration date, modifier, modification date. In addition to the metadata.xlsx file, the **xlsx** folder might contain a **scripts** folder, which contains scripts associated with types in the metadata.xlsx file, if these are present, and a **data** folder which holds the content of spreadsheet fields and large text fields that exceed the size of an Excel cell.
+- **Export metadata as XLSX**. Metadata are exported in one **xlsx** folder. The folder contains the metadata of all exported entities and the corresponding masterdata in a **metadata.xlsx** file. If **Make import compatible** is selected, this file is suitable for re-import in openBIS. If not, the file contains some fields which are not compatible with re-imports. These fields are: PermId of entities, registrator, registration date, modifier, modification date. In addition to the metadata.xlsx file, the **xlsx** folder might contain additional folders:
+    - a **scripts** folder, which contains scripts associated with types in the metadata.xlsx file, if these are present;
+    - a **data** folder which holds the content of spreadsheet fields and large text fields that exceed the size of an Excel cell;
+    - a **miscellaneous** folder which contain images embedded in text of exported entries, if present.
 
 - **Export data**. The default maximum size of all datasets to be exported is 10GB. This can be configured by a system admin in the [AS service.properties file](../../system-documentation/configuration/optional-application-server-configuration.md). We recommend to use [sftp](../general-users/lab-notebook.md#data-access) to download large datasets. 
-If **Make import compatible** is selected, datasets are exported in a **data** folder in a format ready to be uploaded in openBIS using the default eln-lims dropbox. If not, the datasets are exported in a **pdf** folder that matches the ELN hierarchy.
+If **Make import compatible** is selected, datasets are exported in a **data** folder in a format ready to be uploaded in openBIS using the default eln-lims dropbox. If not, the datasets are exported in a **hiearchy** folder that matches the ELN hierarchy.
 
 - **Include levels below from same space**. If selected, all hierachy levels below the selected entity and belonging to the same Space are exported.
 
@@ -63,7 +66,6 @@ _Note: ensure that pop-ups are not disabled in your browser_.
 Server](../../system-documentation/configuration/optional-datastore-server-configuration.md)
 
 
-> ![image info](img/warning.png) **Known limitation of openBIS 20.10.8**: images embedded in text fields are not currently exported. This limitation will be fixed in the release after 20.10.8. 
 
 
 
@@ -91,22 +93,21 @@ The exported zip file contains 3 folders:
 This contains the datasets in the correct format to be uploaded via eln-lims dropbox, as shown below.
 
 
-![image info](img/IC-data-folder-1.png)
+![image info](img/IC-20109-export-data-folder-v2.png)
 
-![image info](img/IC-data-folder-2.png)
 
-**B.** **pdf** folder
+**B.** **hiearchy** folder
 
 This contains folders that match the openBIS hierarchy (Space/Project/Experiment/Object).
 
 In this case 2 Space folders are present:
 
 1. **CATERINA**: is the exported space.
-2. **METHODS**: contains an Object which is parent of an Object in the space CATERINA. This was exported because the option **Include Objects and Datasets parents and children from different spaces** was selected for export.
+2. **METHODS**: contains an Object which is parent of an Object in the Space CATERINA. This was exported because the option **Include Objects and Datasets parents and children from different spaces** was selected for export.
 
-![image info](img/IC-pdf-folder-1.png)
+![image info](img/IC-20109-export-hierarchy-1.png)
 
-![image info](img/IC-pdf-folder-2.png)
+![image info](img/IC-20109-export-hiearchy-2.png)
 
 
 Inside each folder, there is a pdf of the corresponding entity. Example: 
@@ -114,7 +115,7 @@ Inside each folder, there is a pdf of the corresponding entity. Example:
 - in the Space folder **CATERINA** there is a **CATERINA.pdf** file that contains the metadata of the Space;
 - in the Project folder **PROJECT_1** there is a **PROJECT_1.pdf** file that contains the metadata of the Project;
 - in the Experiment folder **My second experiment (PROJECT_1_EXP_1)** there is a **My second experiment (PROJECT_1_EXP_1).pdf** file with the metadata of the Experiment;
-- in the Object folder **Step A (EXP4)**  there is a **Step A(EXP4).pdf** file with the metadata of the Object and a **20240523100301019-55.pdf** file that contains the metadata of the dataset that belongs to this Object.
+- in the Object folder **Step A (EXP4)**  there is a **Step A(EXP4).pdf** file with the metadata of the Object and a **20240726094631217-68.pdf** file that contains the metadata of the dataset that belongs to this Object.
 
 
 **C.** **xlsx** folder. 
@@ -122,14 +123,17 @@ Inside each folder, there is a pdf of the corresponding entity. Example:
 This contains:
 
 - a **metadata.xlsx** file which has the metadata of the exported entities and the corresponding masterdata (types and properties) in the correct format to be re-imported in another openBIS instance;
-- a **scripts** folder that contains evaluation plugins associated to two types defined in the metadata.xlsx file;
-- a **data** folder that contains the information stored in the spreadsheet field of one of the Objects in this Space.
+- a **scripts** folder that contains evaluation plugins associated to two types defined in the metadata.xlsx file. This folder is present only if the exported types have plugins associated with them.
+- a **data** folder that contains the information stored in the spreadsheet field of one of the Objects in this Space. This folder is present only if the exported entities contain information in spreadsheet or if there are text fields with more than 32,767 characters (this is the limit of the Excel cells).
+- a **miscellaneous** folder that contains images that are embedded in text fields of the exported entities. This folder is present only if exported entities contain images embedded in text.
 
-![image info](img/IC-xlsx-folder-data.png)
+![image info](img/IC-20109-export-xlsx-scripts.png)
 
-![image info](img/IC-xlsx-folder-scripts.png)
+![image info](img/IC-20109-export-xlsx-data.png)
 
 
+![image info](img/IC-20109-export-miscellaneous.png)
+
 
 ### **2. Non import-compatible export of a Space selecting all options**
 
@@ -138,23 +142,24 @@ We export the same Space as described in Example 1, with all options selected, b
  ![image info](img/space-export-all-options-non-import-compatible.png)
 
  
-In this case the exported zip file contains only 2 folders: **pdf** and **xlsx**. Data are exported inside the pdf folder, instead of being in a separate **data** folder.
+In this case the exported zip file contains only 2 folders: **hierarchy** and **xlsx**. Data are exported inside the **hierachy** folder, instead of being in a separate **data** folder.
 
-**A.** **pdf** folder
+**A.** **hierarchy** folder
 
 This contains the same folder structure as described above. In addition, in this case, inside the Object **Step A (EXP4)** folder there is a **data** folder that contains the dataset belonging to this Object, as shown below. The metadata of the dataset is provided as a metadata.json file inside the data folder and as pdf file inside the Object folder (**Step A (EXP4)**).
 
 
-![image info](img/NIC-pdf-folder.png)
+![image info](img/NIC-20109-hiearchy-folder.png)
 
 **B.** **xlsx** folder 
 
 This contains the same files and folders as described in Example 1 (see below). The only difference in this case is that the metadata.xlsx is not import-compatible. It contains some fields which are not compatible with openBIS re-import, as explained above.
 
-![image info](img/NIC-xlsx-folder-data.png)
+![image info](img/NIC-20109-xlsx-scripts.png)
 
-![image info](img/NIC-xlsx-folder-scripts.png)
+![image info](img/NIC-20109-xlsx-data.png)
 
+![image info](img/NIC-20109-miscellaneous.png)
  
 ## Export to Zenodo