Active Buffering Plus Compressed Migration: An Integrated Solution to Parallel Simulations' Data Transport Needs

J. Lee and X. Ma and M. Winslett and S. Yu

In Proceedings of the International Conference on Supercomputing, 2002.

Available format: postscript

Abstract:

Scientific simulations running on parallel platforms output intermediate data periodically, typically moving the output to a remote machine for visualization. Due to the large data size and slow network, improvements in output and migration performance can significantly reduce simulation turnaround time. In this paper, we propose a novel simulation execution environment that integrates active buffering and compressed migration to hide and reduce the cost of output and migration. Our implementation removes previous shortcomings of active buffering and of compressed migration, and our experiments with real world data sets show that integrating the two techniques brings comparable application-visible I/O performance and much more flexibility compared to compressed migration without active buffering. The performance advantage will increase with slower-to-write file formats such as HDF.