The full lifecycle management of scientific data at Hefei Light Source soft X-ray microscopy station

1.   Introduction

The National Synchrotron Radiation Laboratory (NSRL) is China’s first national laboratory, which operates the synchrotron radiation facility Hefei Light Source (HLS). HLS is a vacuum ultraviolet and soft X-ray synchrotron radiation light source that enables cutting-edge scientific experiments in various fields. It plays a crucial role in generating scientific breakthroughs in fundamental and applied research through interdisciplinary studies^[1–5]. The soft X-ray microscopy experimental station at HLS leverages the high brightness and strong penetrability of synchrotron radiation, combined with the unique properties of soft X-rays, to perform nanoresolution three-dimensional imaging of intact water-thick samples. This capability significantly advances research in many fields, including life sciences, medicine, materials science, and environmental sciences^[6–9].

However, the rapid development of experimental technology has led to a significant increase in the amount of scientific data and metadata, which hinders research efficiency without a centralized management platform^[8–15]. To establish a centralized scientific data management platform, HLS is in the process of developing a test system that covers the entire lifecycle of scientific data, including data generation, acquisition, storage, processing, and destruction.

In the soft X-ray microscopy experimental station, the ZEISS Xradia 825 synchrotron serves as the core experimental instrument, with its data acquisition and processing relying on commercial proprietary software. However, this software system is closed-source. Consequently, as a prerequisite for centralized scientific data management, the automation of data acquisition presents significant challenges.

This paper aims to present the scientific data acquisition, storage, and processing of their lifecycle on the basis of the soft X-ray microscopy experimental station.

2.   System architecture

The core of a scientific data management platform is metadata catalogs. The scientific data management platform currently being developed by HLS utilizes SciCat as the metadata catalog^{[16, 17]}. SciCat is based on MongoDB and NodeJS, which support high concurrent metadata reading and writing and exhibit a flexible data structure and strong scalability.

The HLS scientific data management platform consists of six components: a data acquisition (DAQ) system, a data storage system, a data processing system, a metadata database, a website back-end, and a website front-end. The DAQ system acquires the raw data and related metadata from experiments conducted at the experimental station. The storage system employs data redundancy and data integrity validation to ensure the safety and integrity of the stored data. The data analysis platform enables users to perform online processing of their scientific data via custom methods. The metadata database, website front-end, and back-end are built on the SciCat framework, which is responsible for storing metadata and providing user interfaces. The entire data management system is designed as shown in Fig. 1.

Figure  1.  Architecture diagram of the HLS scientific data management platform.

DownLoad: Full-Size Img PowerPoint

To acquire and store scientific data from the soft X-ray microscopy experimental station, we developed a semi-automatic DAQ system that bypasses the limitations of the station’s commercial proprietary software. We also built an online data processing platform to assist users in analyzing their data.

3.   Scientific data acquisition procedure

Owing to the variety of instruments used across different experimental stations, the implementation of the DAQ system needs to be specific to each experimental station. The DAQ system consists of two main parts: metadata acquisition and scientific data acquisition. Metadata acquisition involves two types of metadata: administrative metadata from the user management system of HLS and experimental metadata from the experimental station. Scientific data were also acquired from the experimental station.

The formats of scientific data generated at different experimental stations are often inconsistent. To unify these formats, we encapsulated the data in HDF5 (Hierarchical Data Format version 5). HDF is a file format used for storing and organizing large amounts of data^[18]. Originally developed by the National Supercomputing Center in the United States, the HDF is now technically supported by the HDF Group, a nonprofit organization. HDF is supported by a variety of commercial and noncommercial software platforms, including MATLAB, Java, Python, R, and Julia. HDF5 is the latest version of HDF, which has been widely used in the storage of scientific data. HDF5 has no restrictions on data structures but requires customization when used.

3.1   Data acquisition at the soft X-ray microscopy experimental station

The soft X-ray microscopy experimental station employs a ZEISS Xradia 825 Synchrotron experimental instrument to acquire microscopy images. The operation of the instrument relies on its commercial proprietary software. During the experimental procedure, the user generates the data file via the program, while the relevant metadata are recorded manually. After the experiment is completed, the user obtains the data through a mobile storage medium. This procedure is not only inefficient but also prone to errors when the metadata are recorded manually. Additionally, there is no automatic backup of data. To improve the experimental procedures, we developed a semi-automatic DAQ system specifically for the soft X-ray microscopy experimental station.

3.2   Data acquisition procedure

The DAQ program for the soft X-ray microscopy experimental station was developed via the PyQt5 framework^[19]. The user interface is composed of several components, including a user identification page, guide page, user information page, file uploading page, metadata configuration, and system configuration. To use the DAQ program, users must first identify themselves by tapping their user ID cards on the proximity reader. This allows the program to obtain the user’s information, as well as information related to the experiment from the user management system of HLS. The guide page includes a software introduction and frequently asked questions. The user information page displays information related to the user and the experiment, including user accounts, research topics, basic information about the experiment, and experimental machine hours. The file uploading page is the main function of the DAQ program and enables users to upload data files from their experiments. The metadata configuration page acquires additional metadata, such as the sample name, experimental energy, keywords, comments, and other relevant information. Finally, the system configuration allows users to set various parameters used internally.

3.3   Experimental procedure

The process for conducting experiments at HLS involves several steps that ensure efficient and accurate data acquisition and management. Users first need to apply for an account and submit their experimental application to the Chinese Academy of Sciences Large Research Infrastructures User Service Platform^[20]. The platform then sends this information to the user management system of HLS, which associates the user’s ID card with the system.

Before the experiment, the user’s ID card is verified to ensure that the experiment-related metadata are accurately recorded in the DAQ program and to eliminate potential errors. After the identification step, the DAQ program provides a guide page to help first-time users understand the software. During the experiment, the user manually operates the instrument and generates data files that are the same as before. These data files are Xradia OLE (object linking and embedding) files (.txrm) and contain built-in experimental metadata, including image information and coordinate information. After the experiment, the user selects the data file directory to be uploaded, and the DAQ program automatically analyzes the files to extract experimental metadata and scientific data. Any additional metadata can be added manually on the metadata configuration page if needed, and the user can review all the data before they are uploaded. To standardize the user’s metadata, experiment-related metadata, and scientific data, the DAQ program encapsulates them into HDF5 files and calculates the HDF5 file’s checksum. Then, the HDF5 file is uploaded to the storage server, and the metadata are encapsulated in JSON format and sent to the SciCat back-end. When uploading is completed, the DAQ program triggers the file checksum checking procedure on the file server to ensure data integrity. The process ensures efficient data acquisition and management, accurate metadata recording, and easy access to data for further analysis and collaboration. The detailed process is illustrated in Fig. 2.

Figure  2.  The experimental procedure of the soft X-ray microscopy experimental station.

DownLoad: Full-Size Img PowerPoint

Notably, the metadata in an HDF5 file and a JSON file are identical at the time of initial upload. This duplication ensures that the metadata catalog always reflects the most current information, whereas the metadata embedded within the raw data files remain static and are never updated. The metadata within the catalog are subject to updates as new insights are gained and can be augmented to include additional details that enhance searchability or serve other functions, such as supporting statistical analysis within the data catalog.

3.4   HDF5 file format

HDF5 is a file format that allows for a flexible and efficient way of storing complex data relationships, including metadata. The structure of an HDF5 file needs to be customized. NeXus is a widely adopted data format specifically designed for neutron, X-ray, and muon sciences, facilitating the management and sharing of data within these research communities. NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files in addition to a dictionary of well-defined domain-specific field names. For the soft X-ray station, we have adopted the NeXus format because of its extensive support for domain-specific data structures and metadata definitions. The structure is as follows (names in parentheses are NeXus class types):

/entry(NXentry)

/user(NXuser)

/sample(NXsample)

/instrument(NXinstrument)

/data(NXdata)

The top level of any NeXus file contains entry groups. These contain all the data that are required to describe an experimental scan. The user group contains information from users registered in the user management system of HLS. The sample group contains detailed information pertaining to the sample, including its chemical composition, mass, and environmental variables, among other characteristics. The instrument group is designed to encapsulate all the instrumental information that might be relevant to a measurement, including the instrumental status and configuration of Xradia 825. Finally, the data group contains the raw data from the txrm files generated by Xradia 825 software. The raw data include built-in experimental metadata such as image width and height, x/y positions, and x/y shifts, among other parameters. The DAQ program extracts this metadata and uploads it to the metadata catalog.

4.   Centralized storage of scientific data

Previously, user data files were stored on the computers of the experimental station. The beam station is capable of generating a daily data volume ranging from 1 to 4 GB. To centralize scientific data storage, the DAQ program now uploads the data files and related metadata to a storage server located in the server room. To ensure data integrity, the storage server uses a distributed storage system that provides backup and redundancy for scientific data files. Users can access their data from anywhere by logging into the SciCat website via the Internet. The data are transferred via the Internet using the transport layer security (TLS) protocol to ensure security. By means of a distributed storage system and secure data transmission protocols, such as TLS, user data are protected from potential data loss or security breaches, ensuring that the data are available and secure for further analysis and collaboration.

The DAQ program sends metadata to an Apache Kafka message queue first. The message queue serves as an asynchronous processing system to ensure reliable transmission of metadata and prevent data loss for reasons such as high load and network failure. The message queue receives the metadata and then transmits it to the Catamel back-end API server for SciCat. Catamel is a NodeJS application based on the loopback framework and provides RESTful (representational state transfer) APIs. The RESTful architecture separates the front- and back-ends of the website, improving flexibility, security, and development efficiency. Finally, Catamel writes the metadata to the MongoDB database. MongoDB is a document-oriented nonrelational database that uses a JSON-like document architecture to store metadata and data related to the SciCat system. Compared with traditional relational databases, nonrelational databases are efficient at storing massive amounts of data and are capable of high concurrent reading and writing. The relevant process is illustrated in Fig. 3.

Figure  3.  Data acquisition and retrieval procedure.

DownLoad: Full-Size Img PowerPoint

After an experiment is completed, users can access the data management system and obtain experimental data via the Internet. To improve the user experience and avoid multiple accounts and passwords, it is better to integrate the data management system with the Chinese Academy of Sciences Large Research Infrastructures User Service Platform. However, since SciCat does not support the user authentication protocol of this sharing service platform, we developed an authentication module for SciCat to achieve this goal. Additionally, we developed a storage website back-end to provide data downloading services. The downloading service is authenticated via JSON Web Token, and data are transmitted through TLS for security. Furthermore, the service allows users to download multiple files in a compressed zip format.

Currently, centralized storage retains scientific data for a duration of 3 years. Upon reaching the 3-year retention period, data are evaluated for deletion on the basis of a combination of regulatory requirements, user needs, and storage capacity constraints. We plan to expand the storage capacity after collaborating with other experimental stations, which will enable us to accommodate more data.

5.   Online data processing platform

After conducting experiments at the soft X-ray microscopy experimental station, users may need to perform computed tomography (CT) reconstruction on the raw data, which requires significant computational resources. To facilitate users in analyzing their data, we built an online data processing platform that consists of JupyterLab^[21] and OpenWhisk^[22].

JupyterHub is an online interactive development environment that supports multiple users and can run JupyterLab notebooks, text editors, terminals, etc. It also supports over 40 programming languages, including Python and R. We deploy a docker-based JupyterHub container for image reconstruction^[17]. For user authentication, we have developed a JupyterHub authenticator that can be integrated with SciCat. Users from SciCat do not need to log in to JupyterHub separately. A user’s scientific data storage is mounted through the high-speed storage area network, allowing scientific data to be accessed quickly within JupyterLab. Additionally, we provide graphics processing unit acceleration for JupyterLab to reduce the computational time for image reconstruction.

In addition to image reconstruction, users may also want to visualize the raw data by taking a frame from it and viewing it as a picture. However, since the raw data are not in image format, they need to be preprocessed before visualization. OpenWhisk is an open-source distributed function as a service (FaaS) computing platform that can provide users with an event-driven, stateless computing model. All the components of OpenWhisk run in a docker container, which has the characteristics of high performance, scalability, easy integration, and deployment. This is particularly suitable for data preprocessing scenarios that do not require much time (i.e., less than 5 min) for data reduction, verification, or visualization.

As shown in Fig. 4, the specific process of using OpenWhisk to achieve visualization requirements is as follows: (i) The data acquisition service acquires scientific data and metadata from the soft X-ray microscopy experimental station and uploads them to the file storage system and metadata ingestion service, respectively. (ii) The metadata ingestion service detects the metadata from the soft X-ray microscopy experimental station and submits a request for data visualization to the message queue Kafka. (iii) The message queue Kafka triggers OpenWhisk’s predefined Python script for data visualization. (iv) The script generates data images on the basis of the scientific data files obtained from the file storage system and then writes them into SciCat through Kafka. (v) The image is now available on the web page. Additionally, users can select scripts with different functions from the data processing interface of the website front-end to call OpenWhisk for data processing.

Figure  4.  The data preprocessing procedure of OpenWhisk.

DownLoad: Full-Size Img PowerPoint

6.   Conclusions

To achieve comprehensive lifecycle management of experimental data at soft X-ray microscopy stations, we developed a semi-automatic data acquisition program and built an online data analysis platform. Currently, the platform has been successfully deployed and is under testing at the HLS. In the future, we plan to collaborate with other experimental stations to integrate more data into the platform for scientific data lifecycle management and provide services to more users. This will facilitate the sharing and reuse of scientific data, promote collaboration among researchers, and accelerate scientific discoveries.