Flexible and Efficient Parallel I/O for Large-Scale Multi-component Simulations
X. Ma and J. Jiao and M. Campbell and M. Winslett
In Proceedings of The 4th Workshop on Parallel and Distributed
Scientific and Engineering Computing with Applications,
in conjunction with the 2003 International Parallel and Distributed
Processing Symposium.
Available format:
PDF
Abstract:
In this paper, we discuss our experience of providing high performance
parallel I/O for a large-scale, on-going, multi-disciplinary
simulation project for solid propellant rockets. We describe the
performance and data management issues observed in this project and
present our solutions, including (1) support for relatively fine-grained
distribution of irregular datasets in parallel I/O, (2) a flexible
data management facility for inter-module communication, and (3) two
schemes to overlap computation with I/O. Performance results obtained
from the rocket simulation's development and production platforms show that our
I/O optimizations can dramatically reduce the simulation's visible I/O
cost, as well as the number of disk files, and significantly improve
the overall performance. Meanwhile, our data management facility helps
to provide simulation developers with simple user interfaces for
parallel I/O.