PGREM3D (Parallel Groundwater Transport and Remediation Codes) is a suite of massively parallel codes intended for high resolution numerical simulation of groundwater transport and remediation experiments (Mahinthakumar, 1997). The main feature of the code is that it can solve very large problems (in the order of 10's of millions of cells) efficiently on todays parallel supercomputers. Scalability has been demonstrated for upto 1024 processors of the Intel Paragon XPS/150. As an example, transport calculations involving about 160 million degrees of freedom took less than 2 seconds per time step (each time step involving a full matrix solve of 40 million unknowns) to execute on 1024 parallel processors of the Intel Paragon XPS/150 supercomputer. More recent investigations on the IBM SP supercomputer has involved transport problems of up to 320 Million degrees of freedom involving 6 components.
In its current version, flow module solves single phase saturated systems and the transport module solves multicomponent transport with sorption, bioremediation, and reversible kinetic reactions. The codes are based on the Galerkin finite-element method using hexahedral elements. A logically rectangular nodal structure is assumed for computational efficiency but irregular geometries are supported with distorted hexahedral elements. Parallelization is based on two-dimensional domain decomposition with explicit message passing using MPI. These codes are primarily written in Fortran (some Fortran 90 features and some C). The code uses fast solvers based on multigrid (used in the flow module for perfectly rectangular problems) and Krylov subspace methods (used in both flow and transport).
Multiple horizontal and vertical wells can be handled in heterogeneous porous media. A flexible input interface is available to handle a variety of boundary conditions (including time dependent and cyclic) and scenarios. The codes have been tested on a variety of parallel supercomputer architectures including the Intel Paragon, IBM SP, SGI Origin 2400, Convex Exemplar SPP/1200, Cray T3E, and Compaq/Alpha SC system. These codes are being used in many research applications including groundwater remediation analyses, subsurface heterogeneity (Painter and Mahinthakumar 1999), feasibility of chemical waveforms for subsurface detection, inverse modeling for subsurface characterization (Mahinthakumar et al. 1999, Sayeed and Mahinthakumar 2002), investigation of sparse matrix solvers (Mahinthakumar and Saied 1996, Saied and Mahinthakumar 1998, Mahinthakumar et al. 1997), multi-platform performance analyses (Mahinthakumar and Sayeed 1999, 2002), parallel I/O research (Mackay et al. 1998), and computational steering (Mahinthakumar et al. 1997). These codes have also been used in support of a field remediation experiment at the Portsmouth gaseous diffusion plant (Mahinthakumar and West, 1997).
Mahinthakumar, G., (1998). PGREM3D: Parallel groundwater transport and remediation codes, Users Guide, April 1998, updated 1999.
Mahinthakumar, G., and F. Saied, (1999). Implementation and Performance Analysis of a Parallel Multicomponent Groundwater Transport Code, CD-ROM proccedings of the 1999 SIAM Parallel Processing Meeting, March 22-25, San Antonio, TX.
Saied, F. S., and G. Mahinthakumar, (1998). Efficient Parallel Multigrid Based Solvers for Large Scale Groundwater Flow Simulations,, Computers Math. Applic., Vol 35, No. 7, p.45-54, 1998.
Mahinthakumar, G., F. Saied, and A. J. Valocchi, (1997a). Comparison of some parallel krylov solvers for large scale contaminant transport simulations, High Performance Computing 1997 (Editor: A. M. Tentner), p. 134-139, 1997.
Mackay, D., E.F. D'Azevedo, and G. Mahinthakumar, (1998). A study oof I/O in a parallel finite-element groundwater transport code, Int. J. of High Performance Computing Applications, 12(3), Fall 1998, p. 307-319.
Mahinthakumar, G. and F. Saied, (1996). Distributed Memory Implementation of Multigrid Methods for Groundwater Flow problems with Rough Coefficients, High Performance Computing 1996 (Editor: A. M. Tentner), p. 51-57, 1996.
Mahinthakumar, G. and O. R. West, (1997). High resolution numerical experiments in support of the ISCOR experiment at Portsmouth, Unpublished Report submitted to PORTS, Nov 1997.
G. Mahinthakumar, R.Sharpley, A. Kaulgud, and S. Johnson, (1997b). Groundwater remediation experiments using interactive computational steering on the Intel Paragon, Intel Supercomputer User's Group Conference, Alberquerque, NM, Jun 18-20, 1997.
Painter, S. and G. Mahinthakumar, (1999). Prediction Uncertainty for Tracer Migration in Random Heterogeneities with Multifractal Character, Advances in Water Resources, 23, p. 49-57, 1999.
Mahinthakumar G., Gwo J.P., Moline G.R., Webb O. F., (1999). Subsurface biological activity zone detection using genetic search algorithms, ASCE Journal of Environmental Engineering, vol. 125, no. 12, p 1103-1112, December 1999.