CERN runs CERNBox, a synchronisation and file-sharing platform integrated with the laboratory’s computing infrastructure and specialised applications, such as JupyterLab, HTC (HTCondor), and viewers/editors for ROOT and markdown formats.
As part of ScienceMesh, CERNBox enables seamless data sharing and collaboration across diverse platforms without requiring a common authentication infrastructure.
CERN has significantly contributed to ScienceMesh’s technology and standards, including Open Cloud Mesh (OCM) and Cloud Storage Synchronisation and Sharing (CS3) APIs, and led the CS3Mesh4EOSC project.
By integrating ScienceMesh with EOSC Data Commons services, CERNBox will allow users to discover and directly import datasets via the Matchmaker, facilitated by the Data Player.
Processing can be done on integrated HPC resources, JupyterLab, and HTC (HTCondor). Users can easily share preliminary analyses, collaborate via Jupyter notebooks, and finalise results, which are then deposited and discoverable through CERNBox or project-specific repositories.
Use Case Status Before Joining EOSC Data Commons
Processing happens through integration with other systems and current examples include JupyterLab, HTC (HTCondor batch), and various viewers and editors for particular fi letypes, e.g. markdown and ROOT.
Objectives in the Project
- Integration of HPC resources into ScienceMesh via CERNBox support for CephFS; making data directly available to processing resources (JupyterLab and HPC).
- ScienceMesh integration into EOSC Data Commons services:
- Integrated with discovery mechanisms.
- Ingestion and processing of search results.
- Publishing data to digital repositories.
- Integration with the EOSC EU Node. The EOSC EU Node and the upcoming EOSC Federation are providing the same storage federation technology, based on Open Cloud Mesh (OCM). OCM v1.2 support is implemented in CERNBox, and its Jupyter integration is extended to fulfil the Data Commons workflows.
Integration with EOSC Data Commons Services and Components
- Integration of ScienceMesh into the EOSC Matchmaker through a “Run on ScienceMesh” button, which will enable the processing analysis and datasets on ScienceMesh.
- Integration of ScienceMesh into the EOSC Data Player will happen by the development of a ScienceMeshVRE class in the Dispatcher, which will dispatch packages for processing to a ScienceMesh node through OCM shares embedding a Package for Processing Datasets. ScienceMesh will have some interaction with the Package for Processing Datasets through format specification and feedback.
Technical Integration Plan
Local HPC integration finished, enabling complete web and dav access to CERNBox/CephFS. A proof of concept workflow support in the Dispatcher and presence as a tool in the EOSC Data player and specification of the package for processing datasets to deploy and process, along with a demonstration of data federation with the EU EOSC Node.
Full workflow from the EOSC Matchmaker to the EOSC Data Player to the ScienceMesh instance finished. Sync functionality on both desktop and mobile clients. Support for authenticated and public shares on HPC data. Demo of processing data on HPC via a console (e.g. “slurm submit”).
Extended workflow support finished, along with application sharing on HPC data through full Jupyter integration including share management, CephFS, and Slurm integration. Support for publishing data and analysis to digital repositories should also be done by this release and possible directions depending on results and feedback.