System Software team

Professor Osamu Tatebe

• Research on parallel and distributed system software for HPC, big data and AI

We conduct research on system software for very large-scale big data analysis, data-intensive computing and high performance computing.

Handling ever larger data requires an I/O mechanism that scales with both the size of the data and the computing speed of the machine. To that end we study parallel and distributed system software that scales out, including distributed file systems, parallel I/O and application frameworks.

Our research uses large-scale PC clusters for HPC, Pegasus (the University of Tsukuba’s supercomputer), Miyabi (operated jointly with the University of Tokyo) and other systems.

• Research on ad hoc file systems ( CHFS , FINCHFS )

On supercomputers, ad hoc file systems — which build a temporary parallel file system out of the local storage of the compute nodes — are seen as a promising way to close the gap between the performance of the parallel file system and that of the computation itself.

We are developing CHFS and FINCHFS as ad hoc file systems. CHFS ranked 21st in the 10-node research category of the June 2023 edition of IO500, the world ranking of I/O performance.

• Research on caching file systems (CHFS/Cache)

An ad hoc file system is a temporary parallel file system built on node-local storage, and data must therefore be moved between it and the underlying parallel file system. A caching file system is a mechanism that performs this data movement automatically.

We are developing CHFS/Cache, a version of CHFS that works as a caching file system. Conventional caching file systems suffer from poor performance when accessing small files; in CHFS/Cache we address this problem by relaxing the consistency requirements between the cache and the parallel file system.

There was also a problem of degraded performance when flushing modified data back to the parallel file system, which we address by proposing an I/O-aware flushing mechanism.

• Research on parallel I/O libraries

Ad hoc file systems and similar software are implemented at user level, so some effort is needed before applications can use them. So that applications can be used without modification, we are extending libraries such as MPI-IO, HDF5, Apache Arrow and TensorStore, so that any application built on those libraries works unchanged.

We are also devising a variety of techniques to further improve the I/O performance of applications.

• Study of the storage system for the next flagship machine

We are studying the storage system for the flagship machine that will succeed Fugaku. The following is an interim report.

• The Gfarm file system

We research and develop the Gfarm file system as a wide-area distributed file system. Gfarm is a storage system that can be accessed securely over the Internet, can distribute storage across wide areas, scales out in both performance and capacity, has no single point of failure, guarantees data integrity, and can cope with silent data corruption.

Gfarm is in production use in systems such as the HPCI shared storage. The HPCI shared storage is a wide-area storage system used across the domestic HPC infrastructure centred on Fugaku, which is promoted by MEXT; it can be mounted and used from anywhere, including supercomputer centres throughout Japan. Files are automatically replicated to the eastern site (Kashiwa) and the western site (Kobe), so they remain accessible even when a failure occurs.

Everyday life in the team

• Meetings are held in person. Team meetings are weekly, and the whole-laboratory meeting is roughly monthly.
• We also hold a reading group within the team. This year we are reading Architecture and Design of the Linux Storage Stack .
• There are no core hours.
• On weekday afternoons there are typically two or three people in the laboratory.
• The laboratory is equipped with a microwave oven, a refrigerator, an electric kettle, a coffee maker, sofas and so on, all of which students are free to use. Snacks and drinks are also on sale on site.
• The goal for fourth-year undergraduates is to give an oral presentation at the HPC research meeting in March. Of course, if progress is fast, they can present at an international conference.
  • Last year one fourth-year student presented in the poster session of SC24, a top-tier conference.
• There are a number of enjoyable events, such as the implementation camp and the alumni gathering.
  • The photograph below is from last year’s alumni gathering.

Computing resources and operational services of the laboratory

• We have about seven nodes equipped with Intel Optane persistent memory.
• We have close to forty nodes with InfiniBand, two of which reach 400 Gbps.
• Including the nodes used to run services, the team has about seventy of its own nodes, and root is available on all of them.
• The cluster at the Center for Computational Sciences and the laboratory are connected by a 10G link.

Because we have many experimental nodes on which root privileges are available and even the kernel can be replaced, a wide range of research is possible. When root privileges are not required, the University of Tsukuba’s Cygnus and Pegasus supercomputers are also available, and after discussion with the faculty we sometimes use supercomputers at other universities as well.

The shared file system, DNS and other services run by the team are operated by volunteers within the team, mainly master’s students. Web services and DNS operation for the whole laboratory are handled by the admin group. Neither is compulsory, so you are free either to concentrate on research and produce results, or to gain operational experience alongside your research.