Application:
GYRO
Simulation:
B2-cy
Performance
Tool: TAU
Number of
Processors: 16
Platform:
Teragrid at NCSA (Mercury: Itanium Cluster)
OUTPUT
FROM PPROF:
Reading Profile files in
profile.*
NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 9.134E+05
6.729E+08 1 2808
672917690 .TAU application
89.5
8.334E+06 6.023E+08 1000 17008
602330 DO_FULLADVANCE
85.5
2.149E+08 5.753E+08 1000 28000
575298 TIMESTEP_SSP_322
31.8
1.993E+08 2.142E+08 3000 6000 71385 GET_KINETIC_ADVANCE
14.8
9.98E+07 9.98E+07 24000 0 4158 MPI_Alltoall()
6.9
4.161E+07 4.645E+07 1001 2002 46401 GET_FIELD_EXPLICIT
4.4
2.851E+07 2.982E+07 1 770
29816758 MAKE_MAXWELL_MATRIX
3.6
2.444E+07 2.444E+07 10979 0 2226 MPI_Allreduce()
2.6
1.705E+07 1.737E+07 10 700
1737015 WRITE_BIG
2.4
1.617E+07 1.617E+07 1 0
16172266 MAKE_LAMBDA_GRID
2.1
1.381E+07 1.42E+07 1 130
14196821 MAKE_POISSON_MATRIX
0.3
6.581E+05 2.073E+06 1000 1000 2073 CATCH_HALT_SIGNAL
0.3
1.859E+06 1.859E+06 15555 0 119 MPI_Recv()
0.2
1.659E+06 1.659E+06 1 8
1659225 MPI_Init()
0.2
1.427E+05 1.603E+06 1000 1000 1603 GET_ERROR
0.2
1.601E+06 1.601E+06 1122 0 1427 MPI_Bcast()
0.1
6.877E+05 6.877E+05 1 0
687724 MAKE_THETA_OPERATORS
0.1
1.582E+05 5.59E+05 1 257
559006 MAKE_BLEND_ARRAYS
0.1
4.696E+05 4.696E+05 1 0
469577 MAKE_PROFILES
0.1
3.879E+05 3.879E+05 1 0
387932 MAKE_RADIAL_OPERATORS
0.1
3.365E+05 3.672E+05 1 6
367178 MAKE_IMPLICIT_ADVECT
0.0
3.096E+04 3.494E+04 1 17 34943 WRITE_INPUT
0.0
1.461E+04 1.461E+04 2 0 7303 MPI_Comm_split()
0.0
1.338E+04 1.35E+04 1 4 13495 MPI_Finalize()
0.0
5756 5756 12 0 480 MPI_Reduce()
0.0
113 113 4 0 28 MPI_Keyval_free()
0.0
103 103 4 0 26 MPI_Attr_put()
0.0
101 101 4 0 25 MPI_Attr_get()
0.0 72 72 4 0 18 MPI_Comm_rank()
0.0
21 21 2 0 10 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 0,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
1.556E+04
8192 8 111.5
761.7 Message size received from
all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 1;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.205E+05
6.585E+08 1 2808
658504246 .TAU application
89.6
7.071E+06 5.898E+08 1000 2008
589818 DO_FULLADVANCE
88.2
2.173E+08 5.807E+08 1000 28000
580747 TIMESTEP_SSP_322
32.8
2.005E+08 2.159E+08 3000 6000 71973 GET_KINETIC_ADVANCE
15.2
1.003E+08 1.003E+08 24000 0 4179 MPI_Alltoall()
7.2
4.221E+07 4.728E+07 1001 2002 47228 GET_FIELD_EXPLICIT
4.5
2.855E+07 2.982E+07 1 770
29821568 MAKE_MAXWELL_MATRIX
3.8
2.53E+07 2.53E+07 10979 0 2304 MPI_Allreduce()
2.5
1.621E+07 1.621E+07 1 0
16206357 MAKE_LAMBDA_GRID
2.2
1.385E+07 1.427E+07 1 130
14273814 MAKE_POISSON_MATRIX
0.6
4.193E+06 4.193E+06 1122 0 3737 MPI_Bcast()
0.2
1.41E+05 1.547E+06 1000 1000 1547 GET_ERROR
0.2
7.565E+04 1.529E+06 1000 1000 1529 CATCH_HALT_SIGNAL
0.1
7.025E+05 7.025E+05 1 0
702483 MAKE_THETA_OPERATORS
0.1
1.591E+05 5.933E+05 1 257
593308 MAKE_BLEND_ARRAYS
0.1
2.944E+05 5.667E+05 10 160 56666 WRITE_BIG
0.1
3.909E+05 3.909E+05 1 0
390862 MAKE_RADIAL_OPERATORS
0.1
3.441E+05 3.759E+05 1 6
375861 MAKE_IMPLICIT_ADVECT
0.0
6.816E+04 6.842E+04 1 8 68416 MPI_Init()
0.0
1.349E+04 1.349E+04 2 0 6745 MPI_Comm_split()
0.0
406 7372 1 2 7372 WRITE_INPUT
0.0
5402 5519 1 4 5519 MPI_Finalize()
0.0
2627 2627 12 0 219 MPI_Reduce()
0.0
1746 1746 1 0 1746 MAKE_PROFILES
0.0
128 128 4
0 32 MPI_Attr_put()
0.0
123 123 4 0 31 MPI_Attr_get()
0.0
117 117 4 0 29 MPI_Keyval_free()
0.0
82 82 4 0
20 MPI_Comm_rank()
0.0
23 23 2 0 12 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 1,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 2;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.06E+05
6.502E+08 1 2808
650222258 .TAU application
89.6
7.058E+06 5.826E+08 1000 2008
582636 DO_FULLADVANCE
88.2
2.136E+08 5.734E+08 1000 28000
573409 TIMESTEP_SSP_322
32.9
1.993E+08 2.142E+08 3000 6000 71402 GET_KINETIC_ADVANCE
15.2
9.887E+07 9.887E+07 24000 0 4120 MPI_Alltoall()
7.2
4.18E+07 4.673E+07 1001 2002 46685 GET_FIELD_EXPLICIT
4.6
2.851E+07 2.98E+07 1 770
29798700 MAKE_MAXWELL_MATRIX
3.8
2.477E+07 2.477E+07 10979 0 2256 MPI_Allreduce()
2.5
1.617E+07 1.617E+07 1 0
16173149 MAKE_LAMBDA_GRID
2.2
1.383E+07 1.418E+07 1 130
14182907 MAKE_POISSON_MATRIX
0.5
3.27E+06 3.27E+06 1122 0 2914 MPI_Bcast()
0.3
1.43E+05 1.727E+06 1000 1000 1727 GET_ERROR
0.2
7.211E+04 1.536E+06 1000 1000 1536 CATCH_HALT_SIGNAL
0.1
6.943E+05 6.943E+05 1 0
694294 MAKE_THETA_OPERATORS
0.1
1.565E+05 5.986E+05 1 257
598572 MAKE_BLEND_ARRAYS
0.1
2.827E+05 5.561E+05 10 160 55610 WRITE_BIG
0.1 3.882E+05
3.882E+05 1 0
388200 MAKE_RADIAL_OPERATORS
0.1
3.396E+05 3.706E+05 1 6
370578 MAKE_IMPLICIT_ADVECT
0.0
6.802E+04 6.823E+04 1 8 68234 MPI_Init()
0.0 3.139E+04
3.139E+04 2 0 15695 MPI_Comm_split()
0.0
5203 5311 1 4 5311 MPI_Finalize()
0.0
380 4187 1 2 4187 WRITE_INPUT
0.0
3194 3194 12 0 266 MPI_Reduce()
0.0
1672 1672 1 0 1672 MAKE_PROFILES
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
108 108 4 0 27 MPI_Keyval_free()
0.0
104 104 4 0 26 MPI_Attr_get()
0.0
63 63 4 0 16 MPI_Comm_rank()
0.0
27 27 2
0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 2,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0
0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 3;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.013E+05
6.661E+08 1 2808
666084533 .TAU application
89.7
7.064E+06 5.973E+08 1000 2008
597290 DO_FULLADVANCE
88.1
2.229E+08 5.87E+08 1000 28000
586993 TIMESTEP_SSP_322
32.5
2.007E+08 2.163E+08 3000 6000 72098 GET_KINETIC_ADVANCE
15.1
1.006E+08 1.006E+08 24000 0 4191 MPI_Alltoall()
7.1
4.222E+07 4.728E+07 1001 2002 47235 GET_FIELD_EXPLICIT
4.5
2.857E+07 2.984E+07 1 770
29841822 MAKE_MAXWELL_MATRIX
4.0
2.68E+07 2.68E+07 10979 0 2441 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16200438 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.427E+07 1 130
14272815 MAKE_POISSON_MATRIX
0.6
4.252E+06 4.252E+06 1122 0 3790 MPI_Bcast()
0.3
1.396E+05 1.689E+06 1000 1000 1689 GET_ERROR
0.2
2.905E+05 1.662E+06 10 160
166234 WRITE_BIG
0.2
6.887E+04 1.504E+06 1000 1000 1504 CATCH_HALT_SIGNAL
0.1
7.008E+05 7.008E+05 1 0
700830 MAKE_THETA_OPERATORS
0.1
1.6E+05 6.319E+05 1 257
631945 MAKE_BLEND_ARRAYS
0.1
3.914E+05 3.914E+05 1 0
391401 MAKE_RADIAL_OPERATORS
0.1
3.394E+05 3.709E+05 1 6
370889 MAKE_IMPLICIT_ADVECT
0.0
7.054E+04 7.075E+04 1 8 70746 MPI_Init()
0.0
4.182E+04 4.182E+04 2 0 20908 MPI_Comm_split()
0.0
4992 5095 1 4 5095 MPI_Finalize()
0.0
402 4391 1 2 4391 WRITE_INPUT
0.0
2669 2669 12 0 222 MPI_Reduce()
0.0
1707 1707 1 0 1707 MAKE_PROFILES
0.0
108 108 4 0 27 MPI_Attr_put()
0.0
103 103 4 0 26 MPI_Keyval_free()
0.0 100 100 4 0 25 MPI_Attr_get()
0.0
62 62 4 0 16 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 3,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 4;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.061E+05
6.589E+08 1 2808
658865988 .TAU application
89.7
7.07E+06 5.911E+08 1000 2008
591142 DO_FULLADVANCE
88.3
2.224E+08 5.82E+08 1000 28000
582019 TIMESTEP_SSP_322
32.5
1.997E+08 2.142E+08 3000 6000 71407 GET_KINETIC_ADVANCE
15.0
9.902E+07 9.902E+07 24000 0 4126 MPI_Alltoall()
7.0
4.164E+07 4.64E+07 1001 2002 46349 GET_FIELD_EXPLICIT
4.5
2.851E+07 2.98E+07 1 770
29796103 MAKE_MAXWELL_MATRIX
3.7
2.413E+07 2.413E+07 10979 0 2198 MPI_Allreduce()
2.5
1.617E+07 1.617E+07 1 0
16172614 MAKE_LAMBDA_GRID
2.2
1.382E+07 1.419E+07 1 130
14188506 MAKE_POISSON_MATRIX
0.5
3.26E+06 3.26E+06 1122 0 2905 MPI_Bcast()
0.2 1.397E+05
1.63E+06 1000 1000 1630 GET_ERROR
0.2
7.65E+04 1.452E+06 1000 1000 1452 CATCH_HALT_SIGNAL
0.1
6.929E+05 6.929E+05 1 0
692904 MAKE_THETA_OPERATORS
0.1
1.576E+05 5.997E+05 1 257
599683 MAKE_BLEND_ARRAYS
0.1
2.875E+05 5.364E+05 10 160 53640 WRITE_BIG
0.1
4.408E+05 4.716E+05 1 6
471577 MAKE_IMPLICIT_ADVECT
0.1
4.232E+05 4.232E+05 1 0
423227 MAKE_RADIAL_OPERATORS
0.0
7.065E+04 7.086E+04 1 8 70860 MPI_Init()
0.0
3.096E+04 3.096E+04 2 0 15478 MPI_Comm_split()
0.0
4872 4982
1 4 4982 MPI_Finalize()
0.0
4506 4506 12 0 376 MPI_Reduce()
0.0
395 4131 1 2 4131 WRITE_INPUT
0.0
1729 1729 1
0 1729 MAKE_PROFILES
0.0
110 110 4 0 28 MPI_Attr_put()
0.0
110 110 4 0 28 MPI_Keyval_free()
0.0
103 103 4
0 26 MPI_Attr_get()
0.0
61 61 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 4,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0
0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 5;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.062E+05
6.637E+08 1 2808
663710572 .TAU application
89.6
7.067E+06 5.949E+08 1000 2008
594873 DO_FULLADVANCE
88.3
2.228E+08 5.858E+08 1000 28000
585765 TIMESTEP_SSP_322
32.4
2.01E+08 2.154E+08 3000 6000 71789 GET_KINETIC_ADVANCE
15.2
1.007E+08 1.007E+08 24000 0 4198 MPI_Alltoall()
7.1
4.206E+07 4.694E+07 1001 2002 46892 GET_FIELD_EXPLICIT
4.5
2.855E+07 2.983E+07 1 770
29830203 MAKE_MAXWELL_MATRIX
3.6
2.413E+07 2.413E+07 10979 0 2198 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16200503 MAKE_LAMBDA_GRID
2.1
1.384E+07 1.423E+07 1 130
14228102 MAKE_POISSON_MATRIX
0.6
4.218E+06 4.218E+06 1122 0 3759 MPI_Bcast()
0.2
1.38E+05 1.603E+06 1000 1000 1603 GET_ERROR
0.2
7.681E+04 1.495E+06 1000 1000 1495 CATCH_HALT_SIGNAL
0.1
7.014E+05 7.014E+05 1 0
701393 MAKE_THETA_OPERATORS
0.1
1.581E+05 6.262E+05 1 257
626185 MAKE_BLEND_ARRAYS
0.1
2.976E+05 5.572E+05 10 160 55722 WRITE_BIG
0.1
4.408E+05 4.719E+05 1 6
471923 MAKE_IMPLICIT_ADVECT
0.1
4.268E+05 4.268E+05 1 0
426792 MAKE_RADIAL_OPERATORS
0.0
6.787E+04 6.807E+04 1 8 68073 MPI_Init()
0.0
4.009E+04 4.009E+04 2 0 20045 MPI_Comm_split()
0.0
5158 5270 1 4 5270 MPI_Finalize()
0.0
376 4350 1 2 4350 WRITE_INPUT
0.0
2566 2566 12 0 214 MPI_Reduce()
0.0
1691 1691 1 0 1691 MAKE_PROFILES
0.0
112 112 4 0 28 MPI_Keyval_free()
0.0
106 106 4 0 26 MPI_Attr_put()
0.0
98 98 4 0 24 MPI_Attr_get()
0.0
63 63 4
0 16 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 5,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 6;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.068E+05
6.674E+08 1
2808 667410026 .TAU
application
89.7
7.073E+06 5.985E+08 1000 2008
598528 DO_FULLADVANCE
88.3
2.232E+08 5.895E+08 1000 28000
589487 TIMESTEP_SSP_322
32.4
2.01E+08 2.161E+08 3000
6000 72045 GET_KINETIC_ADVANCE
15.4
1.031E+08 1.031E+08 24000 0 4295 MPI_Alltoall()
7.1
4.209E+07 4.715E+07 1001 2002 47103 GET_FIELD_EXPLICIT
4.5
2.855E+07 2.986E+07 1
770 29859682 MAKE_MAXWELL_MATRIX
3.7
2.497E+07 2.497E+07 10979 0 2274 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16201668 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.421E+07 1 130
14209130 MAKE_POISSON_MATRIX
0.6
4.229E+06 4.229E+06 1122 0 3769 MPI_Bcast()
0.2
1.391E+05 1.523E+06 1000 1000 1523 GET_ERROR
0.2
7.703E+04 1.496E+06 1000 1000 1496 CATCH_HALT_SIGNAL
0.1
7.014E+05 7.014E+05 1 0
701410 MAKE_THETA_OPERATORS
0.1
1.584E+05 6.255E+05 1 257
625474 MAKE_BLEND_ARRAYS
0.1
2.975E+05 5.643E+05 10 160 56431 WRITE_BIG
0.1
4.475E+05 4.787E+05 1 6
478666 MAKE_IMPLICIT_ADVECT
0.1
4.271E+05 4.271E+05 1 0
427096 MAKE_RADIAL_OPERATORS
0.0
6.794E+04 6.814E+04 1 8 68143 MPI_Init()
0.0
4.037E+04 4.037E+04 2 0 20184 MPI_Comm_split()
0.0
5109 5212 1 4 5212 MPI_Finalize()
0.0
403 4488 1 2 4488 WRITE_INPUT
0.0
3371 3371 12 0 281 MPI_Reduce()
0.0
1682 1682 1 0 1682 MAKE_PROFILES
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
103 103 4 0 26 MPI_Keyval_free()
0.0
98 98 4 0 24 MPI_Attr_get()
0.0
62 62 4 0 16 MPI_Comm_rank()
0.0 29 29 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 6,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 7;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.056E+05
6.617E+08 1 2808
661675282 .TAU application
89.7
7.066E+06 5.938E+08 1000 2008
593772 DO_FULLADVANCE
88.4
2.225E+08 5.846E+08 1000 28000
584646 TIMESTEP_SSP_322
32.5
1.996E+08 2.15E+08 3000 6000 71655 GET_KINETIC_ADVANCE
15.2
1.005E+08 1.005E+08 24000 0 4189 MPI_Alltoall()
7.1
4.166E+07 4.673E+07 1001 2002 46686 GET_FIELD_EXPLICIT
4.5
2.853E+07 2.991E+07 1 770
29906524 MAKE_MAXWELL_MATRIX
3.8
2.545E+07 2.545E+07 10979 0 2318 MPI_Allreduce()
2.4
1.617E+07 1.617E+07 1 0
16173416 MAKE_LAMBDA_GRID
2.2
1.383E+07 1.425E+07 1 130
14248644 MAKE_POISSON_MATRIX
0.5
3.258E+06 3.258E+06 1122 0 2903 MPI_Bcast()
0.2
1.407E+05 1.635E+06 1000 1000 1635 GET_ERROR
0.2
7.555E+04 1.462E+06 1000 1000 1462 CATCH_HALT_SIGNAL
0.1
6.938E+05 6.938E+05 1 0
693754 MAKE_THETA_OPERATORS
0.1
1.581E+05 6.024E+05 1 257
602375 MAKE_BLEND_ARRAYS
0.1
2.894E+05 5.368E+05 10 160 53684 WRITE_BIG
0.1
4.404E+05 4.71E+05 1 6
471017 MAKE_IMPLICIT_ADVECT
0.1
4.24E+05 4.24E+05 1 0
423953 MAKE_RADIAL_OPERATORS
0.0
7.075E+04 7.096E+04 1 8 70965 MPI_Init()
0.0
3.029E+04 3.029E+04 2 0 15144 MPI_Comm_split()
0.0
4847 4950 1 4 4950 MPI_Finalize()
0.0
383 4233 1 2 4233 WRITE_INPUT
0.0
2555 2555 12
0 213 MPI_Reduce()
0.0
1758 1758 1 0 1758 MAKE_PROFILES
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
103 103 4 0
26 MPI_Attr_get()
0.0
103 103 4 0 26 MPI_Keyval_free()
0.0
61 61 4 0 15 MPI_Comm_rank()
0.0
25 25 2 0 12 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 7,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0
0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 8;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.096E+05
6.514E+08 1 2808
651438422 .TAU application
89.6
7.064E+06 5.838E+08 1000 2008
583796 DO_FULLADVANCE
88.2
2.14E+08 5.747E+08 1000 28000
574687 TIMESTEP_SSP_322
32.8
1.995E+08 2.138E+08 3000 6000 71280 GET_KINETIC_ADVANCE
15.4
1.004E+08 1.004E+08 24000 0 4184 MPI_Alltoall()
7.1
4.181E+07 4.65E+07 1001 2002 46457 GET_FIELD_EXPLICIT
4.6
2.851E+07 2.985E+07 1 770
29851824 MAKE_MAXWELL_MATRIX
3.7
2.393E+07 2.393E+07 10979 0 2180 MPI_Allreduce()
2.5
1.617E+07 1.617E+07 1 0
16172549 MAKE_LAMBDA_GRID
2.2
1.382E+07 1.42E+07 1 130
14202728 MAKE_POISSON_MATRIX
0.5
3.228E+06 3.228E+06 1122 0 2877 MPI_Bcast()
0.2
1.421E+05 1.604E+06 1000 1000 1604 GET_ERROR
0.2
7.083E+04 1.455E+06 1000 1000 1455 CATCH_HALT_SIGNAL
0.1
6.929E+05 6.929E+05 1 0
692860 MAKE_THETA_OPERATORS
0.1
1.585E+05 6.077E+05 1 257
607738 MAKE_BLEND_ARRAYS
0.1
2.831E+05 5.553E+05 10 160 55530 WRITE_BIG
0.1
3.882E+05 3.882E+05 1 0
388243 MAKE_RADIAL_OPERATORS
0.1
3.472E+05 3.781E+05 1 6
378134 MAKE_IMPLICIT_ADVECT
0.0
6.844E+04 6.866E+04 1 8 68655 MPI_Init()
0.0
3.011E+04 3.011E+04 2 0 15054 MPI_Comm_split()
0.0 6545 6545 12 0 545 MPI_Reduce()
0.0
5109 5225 1 4 5225 MPI_Finalize()
0.0
396 4041 1 2 4041 WRITE_INPUT
0.0
1753 1753 1 0 1753 MAKE_PROFILES
0.0
116 116 4 0 29 MPI_Keyval_free()
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
104 104 4 0 26 MPI_Attr_get()
0.0
60 60 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 8,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0
0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 9;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.098E+05
6.619E+08 1 2808
661891024 .TAU application
89.6
7.071E+06 5.931E+08 1000 2008
593069 DO_FULLADVANCE
88.2
2.188E+08 5.839E+08 1000 28000
583944 TIMESTEP_SSP_322
32.7
2.008E+08 2.166E+08 3000 6000 72185 GET_KINETIC_ADVANCE
15.3
1.014E+08 1.014E+08 24000 0 4226 MPI_Alltoall()
7.1
4.221E+07 4.724E+07 1001
2002 47195 GET_FIELD_EXPLICIT
4.5
2.856E+07 2.99E+07 1 770
29896056 MAKE_MAXWELL_MATRIX
3.9
2.571E+07 2.571E+07 10979 0 2342 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1
0 16199637 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.422E+07 1 130
14223379 MAKE_POISSON_MATRIX
0.6
4.234E+06 4.234E+06 1122 0 3774 MPI_Bcast()
0.2
1.437E+05 1.613E+06 1000 1000 1613 GET_ERROR
0.2
7.473E+04 1.511E+06 1000 1000 1511 CATCH_HALT_SIGNAL
0.1
7.021E+05 7.021E+05 1 0
702077 MAKE_THETA_OPERATORS
0.1
1.587E+05 6.366E+05 1 257
636627 MAKE_BLEND_ARRAYS
0.1
2.932E+05 5.588E+05 10 160 55878 WRITE_BIG
0.1
3.914E+05 3.914E+05 1 0
391418 MAKE_RADIAL_OPERATORS
0.1
3.47E+05 3.789E+05 1 6
378915 MAKE_IMPLICIT_ADVECT
0.0
7.02E+04 7.041E+04 1 8 70410 MPI_Init()
0.0
4.039E+04 4.039E+04 2 0 20196 MPI_Comm_split()
0.0
4912 5012 1 4 5012 MPI_Finalize()
0.0
385 4438 1 2 4438 WRITE_INPUT
0.0
2660 2660 12 0 222 MPI_Reduce()
0.0
1703 1703 1 0 1703 MAKE_PROFILES
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
104 104 4 0 26 MPI_Attr_get()
0.0
100 100 4 0 25 MPI_Keyval_free()
0.0
61 61 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 9,
CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 10;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.077E+05
6.631E+08 1 2808
663108041 .TAU application
89.5
7.066E+06 5.937E+08 1000 2008
593706 DO_FULLADVANCE
88.2
2.187E+08 5.846E+08 1000 28000
584573 TIMESTEP_SSP_322
32.8
2.004E+08 2.174E+08 3000 6000 72457 GET_KINETIC_ADVANCE
15.2
1.007E+08 1.007E+08 24000 0 4195 MPI_Alltoall()
7.2
4.219E+07 4.784E+07 1001 2002 47796 GET_FIELD_EXPLICIT
4.5
2.858E+07 2.992E+07 1 770
29921344 MAKE_MAXWELL_MATRIX
4.2
2.754E+07 2.754E+07 10979 0 2508 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16199928 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.421E+07 1 130
14208194 MAKE_POISSON_MATRIX
0.7
4.818E+06 4.818E+06 1122 0 4294 MPI_Bcast()
0.3
7.392E+04 2.098E+06 1000 1000 2098 CATCH_HALT_SIGNAL
0.2 1.435E+05
1.624E+06 1000 1000 1624 GET_ERROR
0.1
6.999E+05 6.999E+05 1 0
699926 MAKE_THETA_OPERATORS
0.1
1.576E+05 6.374E+05 1 257
637410 MAKE_BLEND_ARRAYS
0.1
2.904E+05 5.614E+05 10 160 56143 WRITE_BIG
0.1
3.913E+05 3.913E+05 1 0
391346 MAKE_RADIAL_OPERATORS
0.1
3.419E+05 3.737E+05 1 6
373682 MAKE_IMPLICIT_ADVECT
0.0
7.153E+04 7.174E+04 1 8 71740 MPI_Init()
0.0
4.005E+04 4.005E+04 2 0 20024 MPI_Comm_split()
0.0
4806 4930 1 4 4930 MPI_Finalize()
0.0
392 4488 1 2 4488 WRITE_INPUT
0.0
3698 3698 12 0 308 MPI_Reduce()
0.0
1701 1701 1 0 1701 MAKE_PROFILES
0.0
124 124 4
0 31 MPI_Keyval_free()
0.0
108 108 4 0 27 MPI_Attr_put()
0.0
98 98 4 0 24 MPI_Attr_get()
0.0
65 65 4 0
16 MPI_Comm_rank()
0.0
26 26 2 0 13 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
10, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 11;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.034E+05
6.605E+08 1 2808
660484560 .TAU application
89.8
7.07E+06 5.928E+08 1000 2008
592841 DO_FULLADVANCE
88.4
2.225E+08 5.837E+08 1000 28000
583674 TIMESTEP_SSP_322
32.4
1.993E+08 2.142E+08 3000 6000 71410 GET_KINETIC_ADVANCE
15.2
1.002E+08 1.002E+08 24000 0 4174 MPI_Alltoall()
7.1
4.182E+07 4.677E+07 1001 2002 46722 GET_FIELD_EXPLICIT
4.5
2.853E+07 2.978E+07 1 770
29783053 MAKE_MAXWELL_MATRIX
3.7
2.477E+07 2.477E+07 10979 0 2256 MPI_Allreduce()
2.4
1.617E+07 1.617E+07 1 0
16173864 MAKE_LAMBDA_GRID
2.2
1.382E+07 1.423E+07 1 130
14227240 MAKE_POISSON_MATRIX
0.5
3.295E+06 3.295E+06 1122 0 2936 MPI_Bcast()
0.3
1.423E+05 1.675E+06 1000 1000 1675 GET_ERROR
0.2
6.839E+04 1.463E+06 1000 1000 1463 CATCH_HALT_SIGNAL
0.1
6.925E+05 6.925E+05 1 0
692499 MAKE_THETA_OPERATORS
0.1
1.567E+05 6.066E+05 1 257
606591 MAKE_BLEND_ARRAYS
0.1
2.875E+05 5.356E+05 10 160 53562 WRITE_BIG
0.1
3.887E+05 3.887E+05 1 0
388707 MAKE_RADIAL_OPERATORS
0.1
3.409E+05 3.716E+05 1 6
371568 MAKE_IMPLICIT_ADVECT
0.0
6.854E+04 6.876E+04 1 8 68755 MPI_Init()
0.0
3.07E+04 3.07E+04 2 0 15348 MPI_Comm_split()
0.0
5101 5197 1 4 5197 MPI_Finalize()
0.0
398 4066 1 2 4066 WRITE_INPUT
0.0
2681 2681 12 0 223 MPI_Reduce()
0.0
1720 1720 1 0 1720 MAKE_PROFILES
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
103 103
4 0 26 MPI_Attr_get()
0.0
96 96 4 0 24 MPI_Keyval_free()
0.0
60 60 4 0 15 MPI_Comm_rank()
0.0
26 26 2 0 13 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
11, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0
0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 12;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.002E+05
6.636E+08 1 2808
663590455 .TAU application
89.6
7.068E+06 5.945E+08 1000 2008
594467 DO_FULLADVANCE
88.2
2.23E+08 5.853E+08 1000 28000
585311 TIMESTEP_SSP_322
32.4
1.994E+08 2.151E+08 3000 6000 71716 GET_KINETIC_ADVANCE
15.1
1.004E+08 1.004E+08 24000 0 4185 MPI_Alltoall()
7.1
4.169E+07 4.68E+07 1001 2002 46756 GET_FIELD_EXPLICIT
4.5
2.85E+07 2.987E+07 1 770
29871704 MAKE_MAXWELL_MATRIX
4.1
2.71E+07 2.71E+07 10979 0 2468 MPI_Allreduce()
2.4
1.617E+07 1.617E+07 1 0
16171651 MAKE_LAMBDA_GRID
2.1
1.382E+07 1.42E+07 1 130
14200228 MAKE_POISSON_MATRIX
0.5
3.291E+06 3.291E+06 1122 0 2933 MPI_Bcast()
0.2
1.414E+05 1.657E+06 1000 1000 1657 GET_ERROR
0.2
7.361E+04 1.458E+06 1000 1000 1458 CATCH_HALT_SIGNAL
0.1
6.944E+05 6.944E+05 1 0
694412 MAKE_THETA_OPERATORS
0.1
1.564E+05 6.084E+05 1 257
608435 MAKE_BLEND_ARRAYS
0.1
2.868E+05 5.459E+05 10 160 54587 WRITE_BIG
0.1
4.36E+05 4.667E+05 1 6
466654 MAKE_IMPLICIT_ADVECT
0.1
4.236E+05 4.236E+05 1 0
423629 MAKE_RADIAL_OPERATORS
0.0
7.09E+04 7.111E+04 1 8 71112 MPI_Init()
0.0
3.04E+04 3.04E+04 2 0 15200 MPI_Comm_split()
0.0
7760 7760 12 0 647 MPI_Reduce()
0.0
4825 4944 1 4 4944 MPI_Finalize()
0.0
362 4065 1 2 4065 WRITE_INPUT
0.0
1740 1740 1
0 1740 MAKE_PROFILES
0.0
119 119 4 0 30 MPI_Keyval_free()
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
102 102 4 0 26 MPI_Attr_get()
0.0
60 60 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
12, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 13;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.068E+05
6.648E+08 1 2808
664804205 .TAU application
89.6
7.07E+06 5.959E+08 1000 2008
595864 DO_FULLADVANCE
88.3
2.232E+08 5.867E+08 1000 28000
586732 TIMESTEP_SSP_322
32.5
2.006E+08 2.158E+08 3000 6000 71938 GET_KINETIC_ADVANCE
15.1
1.006E+08 1.006E+08 24000 0 4192 MPI_Alltoall()
7.1
4.204E+07 4.714E+07 1001 2002 47094 GET_FIELD_EXPLICIT
4.5
2.855E+07 2.988E+07 1 770
29877760 MAKE_MAXWELL_MATRIX
3.8
2.528E+07 2.528E+07 10979 0 2303 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16201956 MAKE_LAMBDA_GRID
2.1
1.384E+07 1.423E+07 1 130
14226087 MAKE_POISSON_MATRIX
0.6
4.208E+06 4.208E+06 1122 0 3750 MPI_Bcast()
0.2 1.4E+05
1.623E+06 1000 1000 1623 GET_ERROR
0.2
7.412E+04 1.49E+06 1000 1000 1490 CATCH_HALT_SIGNAL
0.1
7.018E+05 7.018E+05 1 0
701750 MAKE_THETA_OPERATORS
0.1
1.582E+05 6.292E+05 1 257
629237 MAKE_BLEND_ARRAYS
0.1
2.954E+05 5.575E+05 10 160 55746 WRITE_BIG
0.1
4.488E+05 4.801E+05 1 6
480122 MAKE_IMPLICIT_ADVECT
0.1
4.275E+05 4.275E+05 1 0
427490 MAKE_RADIAL_OPERATORS
0.0
6.838E+04 6.859E+04 1 8 68589 MPI_Init()
0.0
3.983E+04 3.983E+04 2 0 19916 MPI_Comm_split()
0.0
5193 5293
1 4 5293 MPI_Finalize()
0.0
404 4492 1 2 4492 WRITE_INPUT
0.0
2605 2605 12 0 217 MPI_Reduce()
0.0
1720 1720 1
0 1720 MAKE_PROFILES
0.0
108 108 4 0 27 MPI_Attr_put()
0.0
100 100 4 0 25 MPI_Keyval_free()
0.0
98 98 4
0 24 MPI_Attr_get()
0.0
66 66 4 0 16 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
13, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0
0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 14;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.065E+05
6.62E+08 1 2808
661956940 .TAU application
89.8
7.068E+06 5.941E+08 1000 2008
594114 DO_FULLADVANCE
88.2
2.231E+08 5.839E+08 1000 28000
583907 TIMESTEP_SSP_322
32.5
1.996E+08 2.149E+08 3000 6000 71621 GET_KINETIC_ADVANCE
15.0
9.935E+07 9.935E+07 24000 0 4140 MPI_Alltoall()
7.1
4.165E+07 4.667E+07 1001 2002 46624 GET_FIELD_EXPLICIT
4.5
2.853E+07 2.983E+07 1 770
29831277 MAKE_MAXWELL_MATRIX
4.0
2.622E+07 2.622E+07 10979 0 2388 MPI_Allreduce()
2.4
1.617E+07 1.617E+07 1 0
16173722 MAKE_LAMBDA_GRID
2.1
1.383E+07 1.419E+07 1 130
14188874 MAKE_POISSON_MATRIX
0.5
3.315E+06 3.315E+06 1122 0 2954 MPI_Bcast()
0.2
2.887E+05 1.638E+06 10 160
163792 WRITE_BIG
0.2
1.407E+05 1.615E+06 1000 1000 1615 GET_ERROR
0.2
7.499E+04 1.47E+06 1000 1000 1470 CATCH_HALT_SIGNAL
0.1
6.923E+05 6.923E+05 1 0
692265 MAKE_THETA_OPERATORS
0.1
1.592E+05 6.066E+05 1
257 606611
MAKE_BLEND_ARRAYS
0.1
4.419E+05 4.723E+05 1 6
472338 MAKE_IMPLICIT_ADVECT
0.1
4.242E+05 4.242E+05 1 0
424226 MAKE_RADIAL_OPERATORS
0.0
6.893E+04 6.914E+04 1 8 69141 MPI_Init()
0.0
3.047E+04 3.047E+04 2 0 15236 MPI_Comm_split()
0.0
5428 5530 1 4 5530 MPI_Finalize()
0.0
377 4170 1 2 4170 WRITE_INPUT
0.0
3388 3388 12 0 282 MPI_Reduce()
0.0
1709 1709 1 0 1709 MAKE_PROFILES
0.0
112 112 4 0 28 MPI_Attr_put()
0.0
102 102 4 0 26 MPI_Keyval_free()
0.0
100 100 4 0 25 MPI_Attr_get()
0.0
62 62 4
0 16 MPI_Comm_rank()
0.0
26 26 2 0 13 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
14, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 15;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.079E+05
6.66E+08 1 2808
665989266 .TAU application
89.7
7.066E+06 5.972E+08 1000 2008
597154 DO_FULLADVANCE
88.1
2.229E+08 5.866E+08 1000 28000
586572 TIMESTEP_SSP_322
32.5
2.008E+08 2.163E+08 3000 6000 72097 GET_KINETIC_ADVANCE
15.1
1.003E+08 1.003E+08 24000 0 4180 MPI_Alltoall()
7.1
4.207E+07 4.712E+07 1001 2002 47075 GET_FIELD_EXPLICIT
4.5
2.855E+07 2.985E+07 1 770
29849707 MAKE_MAXWELL_MATRIX
4.0
2.683E+07 2.683E+07 10979 0 2444 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16202264 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.421E+07 1 130
14207535 MAKE_POISSON_MATRIX
0.6 4.214E+06
4.214E+06 1122 0 3756 MPI_Bcast()
0.3
2.965E+05 1.967E+06 10 160
196737 WRITE_BIG
0.3
1.4E+05 1.667E+06 1000 1000 1667 GET_ERROR
0.2
7.324E+04 1.506E+06 1000 1000 1506 CATCH_HALT_SIGNAL
0.1
7.004E+05 7.004E+05 1 0
700445 MAKE_THETA_OPERATORS
0.1
1.592E+05 6.268E+05 1 257
626836 MAKE_BLEND_ARRAYS
0.1
4.394E+05 4.707E+05 1 6
470680 MAKE_IMPLICIT_ADVECT
0.1
4.278E+05 4.278E+05 1 0
427814 MAKE_RADIAL_OPERATORS
0.0
7.049E+04 7.07E+04 1 8 70700 MPI_Init()
0.0
3.973E+04 3.973E+04 2 0 19863 MPI_Comm_split()
0.0
4776 4883 1 4 4883 MPI_Finalize()
0.0
385 4361 1 2 4361 WRITE_INPUT
0.0
2471 2471 12 0 206 MPI_Reduce()
0.0
1732 1732 1 0 1732 MAKE_PROFILES
0.0
109 109 4 0 27 MPI_Attr_put()
0.0
107 107 4 0 27 MPI_Keyval_free()
0.0
101 101 4 0 25 MPI_Attr_get()
0.0
64 64 4 0 16 MPI_Comm_rank()
0.0
22 22 2 0 11 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
15, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096
0 Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 16;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.14E+05
6.55E+08 1 2808
654977686 .TAU application
89.5
7.075E+06 5.86E+08 1000 2008
585983 DO_FULLADVANCE
88.1
2.185E+08 5.768E+08 1000 28000
576828 TIMESTEP_SSP_322
32.7
1.995E+08 2.139E+08 3000 6000 71302 GET_KINETIC_ADVANCE
14.9
9.776E+07 9.776E+07 24000 0 4073 MPI_Alltoall()
7.1
4.187E+07 4.674E+07 1001 2002 46695 GET_FIELD_EXPLICIT
4.6
2.855E+07 2.982E+07 1 770
29824996 MAKE_MAXWELL_MATRIX
3.9
2.546E+07 2.546E+07 10979 0 2319 MPI_Allreduce()
2.5
1.617E+07 1.617E+07 1 0
16172384 MAKE_LAMBDA_GRID
2.2
1.384E+07 1.424E+07 1 130
14237753 MAKE_POISSON_MATRIX
0.5
3.291E+06 3.291E+06 1122 0 2933 MPI_Bcast()
0.3
1.426E+05 1.654E+06 1000 1000 1654 GET_ERROR
0.2
7.478E+04 1.474E+06 1000 1000 1474 CATCH_HALT_SIGNAL
0.1
6.926E+05 6.926E+05 1 0
692607 MAKE_THETA_OPERATORS
0.1
1.587E+05 6.075E+05 1 257
607479 MAKE_BLEND_ARRAYS
0.1
2.857E+05 5.386E+05 10 160 53862 WRITE_BIG
0.1
3.879E+05 3.879E+05 1 0
387944 MAKE_RADIAL_OPERATORS
0.1
3.428E+05 3.736E+05 1 6
373572 MAKE_IMPLICIT_ADVECT
0.0
6.818E+04 6.84E+04 1 8 68398 MPI_Init()
0.0
3.076E+04 3.076E+04 2 0 15382 MPI_Comm_split()
0.0
5698 5698 12 0 475 MPI_Reduce()
0.0
5132 5246 1 4 5246 MPI_Finalize()
0.0
397 4278 1 2 4278 WRITE_INPUT
0.0
1644 1644 1 0 1644 MAKE_PROFILES
0.0
114 114 4 0 28 MPI_Keyval_free()
0.0
113 113 4
0 28 MPI_Attr_put()
0.0
103 103 4 0 26 MPI_Attr_get()
0.0
61 61 4 0 15 MPI_Comm_rank()
0.0
32 32 2 0 16 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
16, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122
8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 17;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.028E+05
6.666E+08 1 2808
666617149 .TAU application
89.4
7.07E+06 5.959E+08 1000 2008
595887 DO_FULLADVANCE
88.0
2.23E+08 5.867E+08 1000 28000
586712 TIMESTEP_SSP_322
32.3
2.005E+08 2.153E+08 3000 6000 71766 GET_KINETIC_ADVANCE
15.2
1.013E+08 1.013E+08 24000 0 4220 MPI_Alltoall()
7.1
4.221E+07 4.719E+07 1001 2002 47141 GET_FIELD_EXPLICIT
4.5
2.857E+07 2.984E+07 1 770
29842761 MAKE_MAXWELL_MATRIX
3.7
2.478E+07 2.478E+07 10979 0 2257 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16199612 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.422E+07 1 130
14224838 MAKE_POISSON_MATRIX
0.9
6.109E+06 6.109E+06 1122 0 5445 MPI_Bcast()
0.5 6.969E+04 3.395E+06 1000 1000 3395 CATCH_HALT_SIGNAL
0.2
1.426E+05 1.629E+06 1000 1000 1629 GET_ERROR
0.1
7.004E+05 7.004E+05 1 0
700351 MAKE_THETA_OPERATORS
0.1
1.58E+05 6.332E+05 1 257
633221 MAKE_BLEND_ARRAYS
0.1
2.947E+05 5.956E+05 10 160 59555 WRITE_BIG
0.1
3.915E+05 3.915E+05 1 0
391547 MAKE_RADIAL_OPERATORS
0.1
3.394E+05 3.713E+05 1 6
371272 MAKE_IMPLICIT_ADVECT
0.0
7.016E+04 7.037E+04 1 8 70369 MPI_Init()
0.0
4.053E+04 4.053E+04 2 0 20266 MPI_Comm_split()
0.0
4743 4854 1 4 4854 MPI_Finalize()
0.0
416 4311 1 2 4311 WRITE_INPUT
0.0
2947 2947 12 0 246 MPI_Reduce()
0.0
1729 1729 1 0 1729 MAKE_PROFILES
0.0
111 111 4 0 28 MPI_Keyval_free()
0.0
108 108 4 0 27 MPI_Attr_put()
0.0
99 99
4 0 25 MPI_Attr_get()
0.0
65 65 4 0 16 MPI_Comm_rank()
0.0
25 25 2 0 12 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
17, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 18;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.101E+05
6.581E+08 1 2808
658087759 .TAU application
89.7
7.074E+06 5.903E+08 1000 2008
590268 DO_FULLADVANCE
88.3
2.187E+08 5.811E+08 1000 28000
581064 TIMESTEP_SSP_322
32.7 1.995E+08
2.155E+08 3000 6000 71835 GET_KINETIC_ADVANCE
15.2
9.974E+07 9.974E+07 24000 0 4156 MPI_Alltoall()
7.2
4.179E+07 4.716E+07 1001 2002 47112 GET_FIELD_EXPLICIT
4.5 2.854E+07
2.987E+07 1 770
29873188 MAKE_MAXWELL_MATRIX
4.0
2.639E+07 2.639E+07 10979 0 2404 MPI_Allreduce()
2.5
1.617E+07 1.617E+07 1 0
16171852 MAKE_LAMBDA_GRID
2.2 1.383E+07 1.425E+07 1 130
14254846 MAKE_POISSON_MATRIX
0.5
3.289E+06 3.289E+06 1122 0 2932 MPI_Bcast()
0.3
1.409E+05 1.702E+06 1000 1000 1702 GET_ERROR
0.2
7.522E+04 1.524E+06 1000 1000 1524 CATCH_HALT_SIGNAL
0.1
6.924E+05 6.924E+05 1 0
692395 MAKE_THETA_OPERATORS
0.1
1.587E+05 6.103E+05 1 257
610281 MAKE_BLEND_ARRAYS
0.1
2.895E+05 5.392E+05 10 160 53916 WRITE_BIG
0.1
3.887E+05 3.887E+05 1 0
388710 MAKE_RADIAL_OPERATORS
0.1
3.438E+05 3.748E+05 1 6
374750 MAKE_IMPLICIT_ADVECT
0.0
6.775E+04 6.796E+04 1 8 67965 MPI_Init()
0.0
3.014E+04 3.014E+04 2 0 15070 MPI_Comm_split()
0.0
5087 5189 1 4 5189 MPI_Finalize()
0.0
381 4069 1 2 4069 WRITE_INPUT
0.0
3540 3540 12 0 295 MPI_Reduce()
0.0
1742 1742 1 0 1742 MAKE_PROFILES
0.0
111 111 4 0
28 MPI_Attr_put()
0.0
104 104 4 0 26 MPI_Attr_get()
0.0
102 102 4 0 26 MPI_Keyval_free()
0.0
67 67 4 0
17 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
18, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 19;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.056E+05
6.614E+08 1 2808
661397145 .TAU application
89.6
7.072E+06 5.924E+08 1000 2008
592414 DO_FULLADVANCE
88.2
2.208E+08 5.832E+08 1000 28000
583239 TIMESTEP_SSP_322
32.7
2.005E+08 2.16E+08 3000 6000 71993 GET_KINETIC_ADVANCE
15.0
9.897E+07 9.897E+07 24000 0 4124 MPI_Alltoall()
7.2
4.225E+07 4.755E+07 1001 2002 47498 GET_FIELD_EXPLICIT
4.5
2.857E+07 2.993E+07 1 770
29934953 MAKE_MAXWELL_MATRIX
3.9
2.589E+07 2.589E+07 10979 0 2358 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16200181 MAKE_LAMBDA_GRID
2.2
1.385E+07 1.425E+07 1 130
14245960 MAKE_POISSON_MATRIX
0.6
4.259E+06 4.259E+06 1122 0 3796 MPI_Bcast()
0.3
1.415E+05 1.67E+06 1000 1000 1670 GET_ERROR
0.2
7.032E+04 1.553E+06 1000 1000 1553 CATCH_HALT_SIGNAL
0.1
7.014E+05 7.014E+05 1 0
701449 MAKE_THETA_OPERATORS
0.1
1.589E+05 6.254E+05 1 257
625394 MAKE_BLEND_ARRAYS
0.1
2.868E+05 5.501E+05 10 160 55006 WRITE_BIG
0.1
3.929E+05 3.929E+05 1 0
392916 MAKE_RADIAL_OPERATORS
0.1
3.502E+05 3.816E+05 1 6
381633 MAKE_IMPLICIT_ADVECT
0.0
7.055E+04 7.076E+04 1 8 70755 MPI_Init()
0.0
4.01E+04 4.01E+04 2 0 20048 MPI_Comm_split()
0.0
4837 4952 1 4 4952 MPI_Finalize()
0.0
410 4402 1 2 4402 WRITE_INPUT
0.0
2659 2659 12 0 222 MPI_Reduce()
0.0
1708 1708 1 0 1708 MAKE_PROFILES
0.0
115 115 4 0 29 MPI_Keyval_free()
0.0
110 110 4
0 28 MPI_Attr_put()
0.0
97 97 4 0 24 MPI_Attr_get()
0.0
64 64 4 0 16 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
19, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122
8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 20;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.042E+05
6.66E+08 1 2808
665990061 .TAU application
89.6
7.072E+06 5.97E+08 1000 2008
596962 DO_FULLADVANCE
88.3
2.236E+08 5.878E+08 1000 28000
587768 TIMESTEP_SSP_322
32.4
2.006E+08 2.156E+08 3000 6000 71869 GET_KINETIC_ADVANCE
15.2
1.015E+08 1.015E+08 24000 0 4229 MPI_Alltoall()
7.1
4.208E+07 4.71E+07 1001 2002 47049 GET_FIELD_EXPLICIT
4.5
2.852E+07 2.981E+07 1 770
29812346 MAKE_MAXWELL_MATRIX
3.8
2.508E+07 2.508E+07 10979 0 2284 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16200798 MAKE_LAMBDA_GRID
2.1
1.384E+07 1.425E+07 1 130
14253571 MAKE_POISSON_MATRIX
0.6
4.311E+06 4.311E+06 1122 0 3843 MPI_Bcast()
0.3 1.399E+05 1.668E+06 1000 1000 1668 GET_ERROR
0.2
7.336E+04 1.583E+06 1000 1000 1583 CATCH_HALT_SIGNAL
0.1
7.011E+05 7.011E+05 1 0
701085 MAKE_THETA_OPERATORS
0.1
1.593E+05 6.326E+05 1 257
632630 MAKE_BLEND_ARRAYS
0.1
2.951E+05 5.769E+05 10 160 57687 WRITE_BIG
0.1
4.34E+05 4.651E+05 1 6
465082 MAKE_IMPLICIT_ADVECT
0.1
4.28E+05 4.28E+05 1 0
428015 MAKE_RADIAL_OPERATORS
0.0
6.824E+04 6.845E+04 1 8 68447 MPI_Init()
0.0
4.002E+04 4.002E+04 2 0 20008 MPI_Comm_split()
0.0
2.126E+04 2.126E+04 12 0 1772 MPI_Reduce()
0.0
5057 5158 1 4 5158 MPI_Finalize()
0.0
389 4602 1 2 4602 WRITE_INPUT
0.0
1753 1753 1 0 1753 MAKE_PROFILES
0.0
106 106 4 0 26 MPI_Attr_put()
0.0
102 102 4 0 26 MPI_Attr_get()
0.0
101 101 4 0 25 MPI_Keyval_free()
0.0
63 63 4 0 16 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
20, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 21;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.113E+05
6.607E+08 1 2808
660709838 .TAU application
89.7
7.073E+06 5.929E+08 1000 2008
592870 DO_FULLADVANCE
88.3
2.23E+08 5.837E+08 1000 28000
583718 TIMESTEP_SSP_322
32.4 1.994E+08
2.138E+08 3000 6000 71281 GET_KINETIC_ADVANCE
15.2
1.004E+08 1.004E+08 24000 0 4183 MPI_Alltoall()
7.0
4.164E+07 4.657E+07 1001 2002 46519 GET_FIELD_EXPLICIT
4.5 2.852E+07
2.981E+07 1 770
29807542 MAKE_MAXWELL_MATRIX
3.7
2.429E+07 2.429E+07 10979 0 2212 MPI_Allreduce()
2.4
1.617E+07 1.617E+07 1 0
16172632 MAKE_LAMBDA_GRID
2.2 1.382E+07 1.422E+07 1 130
14220638 MAKE_POISSON_MATRIX
0.5
3.312E+06 3.312E+06 1122 0 2952 MPI_Bcast()
0.3
1.411E+05 1.658E+06 1000 1000 1658 GET_ERROR
0.2
7.651E+04 1.509E+06 1000 1000 1509 CATCH_HALT_SIGNAL
0.1
6.94E+05 6.94E+05 1 0
694039 MAKE_THETA_OPERATORS
0.1
1.572E+05 6.044E+05 1 257
604398 MAKE_BLEND_ARRAYS
0.1
2.867E+05 5.32E+05 10 160 53204 WRITE_BIG
0.1
4.434E+05 4.737E+05 1 6
473696 MAKE_IMPLICIT_ADVECT
0.1
4.241E+05 4.241E+05 1 0
424101 MAKE_RADIAL_OPERATORS
0.0
7.06E+04 7.081E+04 1 8 70806 MPI_Init()
0.0
2.982E+04 2.982E+04 2 0 14909 MPI_Comm_split()
0.0
4779 4887 1 4 4887 MPI_Finalize()
0.0
382 4096 1 2 4096 WRITE_INPUT
0.0
2462 2462 12 0 205 MPI_Reduce()
0.0
1729 1729 1 0 1729 MAKE_PROFILES
0.0
109 109 4 0
27 MPI_Attr_put()
0.0
108 108 4 0 27 MPI_Keyval_free()
0.0
100 100 4 0 25 MPI_Attr_get()
0.0
67 67 4 0
17 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
21, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 22;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.064E+05
6.667E+08 1 2808
666697171 .TAU application
89.7
7.07E+06 5.977E+08 1000 2008
597695 DO_FULLADVANCE
88.3
2.231E+08 5.886E+08 1000 28000
588601 TIMESTEP_SSP_322
32.4
2.008E+08 2.163E+08 3000 6000 72090 GET_KINETIC_ADVANCE
15.3
1.019E+08 1.019E+08 24000 0 4245 MPI_Alltoall()
7.1
4.208E+07 4.738E+07 1001 2002 47334 GET_FIELD_EXPLICIT
4.5
2.854E+07 2.986E+07 1 770
29855394 MAKE_MAXWELL_MATRIX
3.9
2.578E+07 2.578E+07 10979 0 2348 MPI_Allreduce()
2.4
1.62E+07 1.62E+07 1 0
16200886 MAKE_LAMBDA_GRID
2.1
1.385E+07 1.426E+07 1 130
14260930 MAKE_POISSON_MATRIX
0.6
4.244E+06 4.244E+06 1122 0 3783 MPI_Bcast()
0.2
1.412E+05 1.581E+06 1000 1000 1581 GET_ERROR
0.2
7.285E+04 1.545E+06 1000 1000 1545 CATCH_HALT_SIGNAL
0.1
6.991E+05 6.991E+05 1 0
699061 MAKE_THETA_OPERATORS
0.1
1.583E+05 6.307E+05 1 257
630688 MAKE_BLEND_ARRAYS
0.1
2.935E+05 5.626E+05 10 160 56264 WRITE_BIG
0.1
4.381E+05 4.69E+05 1 6
469047 MAKE_IMPLICIT_ADVECT
0.1
4.28E+05 4.28E+05 1 0
427962 MAKE_RADIAL_OPERATORS
0.0
6.807E+04 6.828E+04 1 8 68280 MPI_Init()
0.0
3.966E+04 3.966E+04 2 0 19832 MPI_Comm_split()
0.0
5057 5175 1 4 5175 MPI_Finalize()
0.0
4558 4558 12 0 380 MPI_Reduce()
0.0
370 4358 1 2 4358 WRITE_INPUT
0.0
1663 1663 1 0 1663 MAKE_PROFILES
0.0
118 118 4 0 30 MPI_Keyval_free()
0.0
109 109 4
0 27 MPI_Attr_put()
0.0
100 100 4 0 25 MPI_Attr_get()
0.0
62 62 4 0 16 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE
22, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue
MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04
4096 4096 4096 0
Message size for all-to-all
1122
8 4
4.232 0.9345 Message size for broadcast
1.098E+04
4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0
Message size received from all nodes
0 0 0 0 0
Message size sent to all nodes
---------------------------------------------------------------------------------------
NODE 23;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 8.089E+05
6.6E+08 1 2808
659993587 .TAU application
89.7
7.074E+06 5.921E+08 1000 2008
592115 DO_FULLADVANCE
88.3
2.23E+08 5.83E+08 1000 28000
583002 TIMESTEP_SSP_322
32.4
1.996E+08 2.139E+08 3000 6000 71305 GET_KINETIC_ADVANCE
15.1
9.968E+07 9.968E+07 24000 0 4153 MPI_Alltoall()