Application:
GYRO
Simulation:
B2-cy
Performance
Tool: TAU
Number of
Processors: 16
Platform:
Teragrid at NCSA (Mercury: Itanium Cluster)
User event
ledger
Function
Ledger and User event ledger:
CALL PATH RELATIONS
INDIVIDUAL
FUNCTIONS:
Detailed
Histogram for TIMESTEP_SSP_322:
OUTPUT
FROM PPROF:
Reading
Profile files in profile.*
NODE 0;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.875E+06
1.131E+10 1 2808 11311369160 .TAU application
98.7
1.67E+08 1.117E+10 1000 17008
11166548 DO_FULLADVANCE
97.0
8.821E+09 1.097E+10 1000 28000
10973161 TIMESTEP_SSP_322
12.0
1.357E+09 1.361E+09 3000 6000
453572 GET_KINETIC_ADVANCE
3.6
4.058E+08 4.058E+08 24000 0 16910 MPI_Alltoall()
3.4
3.845E+08 3.856E+08 1001 2002
385250 GET_FIELD_EXPLICIT
0.5
5.534E+07 5.552E+07 1 770
55522977 MAKE_MAXWELL_MATRIX
0.4
4.036E+07 4.042E+07 1 130
40420920 MAKE_POISSON_MATRIX
0.2
2.592E+07 2.614E+07 10 700
2614036 WRITE_BIG
0.1
1.617E+07 1.617E+07 1 0
16173551 MAKE_LAMBDA_GRID
0.1
1.184E+07 1.184E+07 1 0
11837231 MAKE_RADIAL_OPERATORS
0.1
1.053E+07 1.054E+07 1 6
10538419 MAKE_IMPLICIT_ADVECT
0.1
7.355E+06 7.355E+06 10979 0 670 MPI_Allreduce()
0.0
1.689E+05 2.299E+06 1000 1000 2299 GET_ERROR
0.0
1.632E+06 1.632E+06 15555 0 105 MPI_Recv()
0.0
1.462E+06 1.518E+06 1 257
1518309 MAKE_BLEND_ARRAYS
0.0
6.709E+05 1.425E+06 1000 1000 1425 CATCH_HALT_SIGNAL
0.0
8.418E+05 8.418E+05 1122 0 750 MPI_Bcast()
0.0 6.942E+05
6.942E+05 1 0
694191 MAKE_THETA_OPERATORS
0.0
4.801E+05 4.801E+05 1 0
480103 MAKE_PROFILES
0.0
7.555E+04 7.573E+04 1 8 75731 MPI_Init()
0.0
3.119E+04 3.397E+04 1 17 33969 WRITE_INPUT
0.0
4681 4779 1 4 4779 MPI_Finalize()
0.0
3796 3796 12 0 316 MPI_Reduce()
0.0
2976 2976 2 0 1488 MPI_Comm_split()
0.0
98 98 4 0 24 MPI_Keyval_free()
0.0
92 92 4 0 23 MPI_Attr_get()
0.0
92 92 4
0 23 MPI_Attr_put()
0.0
61 61 4 0 15 MPI_Comm_rank()
0.0
29 29 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
1.556E+04
8192 8 111.5
761.7 Message
size received from all nodes
0 0 0 0 0 Message size sent to
all nodes
---------------------------------------------------------------------------------------
NODE 1;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.77E+06
1.122E+10 1 2808 11216229318 .TAU application
99.1
1.66E+08 1.111E+10 1000 3008
11110809 DO_FULLADVANCE
97.5
8.818E+09 1.094E+10 1000 28000
10938059 TIMESTEP_SSP_322
12.0 1.339E+09 1.342E+09 3000 6000
447495 GET_KINETIC_ADVANCE
3.6
3.983E+08 3.983E+08 24000 0 16596 MPI_Alltoall()
3.4
3.783E+08 3.795E+08 1001 2002
379094 GET_FIELD_EXPLICIT
0.3
3.123E+07 3.141E+07 1 770
31411127 MAKE_MAXWELL_MATRIX
0.2
2.776E+07 2.782E+07 1 130
27819369 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16201288 MAKE_LAMBDA_GRID
0.1 1.228E+07 1.228E+07 1 0
12281285 MAKE_RADIAL_OPERATORS
0.1
1.077E+07 1.078E+07 1 6
10782957 MAKE_IMPLICIT_ADVECT
0.1
6.75E+06 6.75E+06 10979 0 615 MPI_Allreduce()
0.1
6.233E+06 6.435E+06 10 196
643486 WRITE_BIG
0.0
1.7E+05 1.552E+06 1000 1000 1552 GET_ERROR
0.0
1.465E+06 1.52E+06 1 257
1519951 MAKE_BLEND_ARRAYS
0.0
8.749E+05 8.749E+05 1122 0 780 MPI_Bcast()
0.0
6.489E+04 8.436E+05 1000 1000 844 CATCH_HALT_SIGNAL
0.0
7.032E+05 7.032E+05 1 0
703249 MAKE_THETA_OPERATORS
0.0
1.328E+05 1.328E+05 1037 0 128 MPI_Send()
0.0
621 8.851E+04 1 3 88508 WRITE_INPUT
0.0
7.386E+04 7.409E+04 1 8 74088 MPI_Init()
0.0
4287 4394 1 4 4394 MPI_Finalize()
0.0
3011 3011 2 0 1506 MPI_Comm_split()
0.0
2650 2650 12 0 221 MPI_Reduce()
0.0
1860 1860 1 0 1860 MAKE_PROFILES
0.0
119 119 4 0 30 MPI_Attr_put()
0.0
109 109 4 0 27 MPI_Attr_get()
0.0
107 107 4 0 27 MPI_Keyval_free()
0.0
83 83 4 0 21 MPI_Comm_rank()
0.0
42 42 2 0 21 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 2;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.796E+06
1.122E+10 1 2808 11218442695 .TAU application
99.1
1.66E+08 1.111E+10 1000 3008
11113105 DO_FULLADVANCE
97.5
8.81E+09 1.094E+10 1000 28000
10940778 TIMESTEP_SSP_322
12.0
1.342E+09 1.346E+09 3000 6000
448651 GET_KINETIC_ADVANCE
3.6
4.042E+08 4.042E+08 24000 0 16841 MPI_Alltoall()
3.4
3.794E+08 3.806E+08 1001 2002
380199 GET_FIELD_EXPLICIT
0.3
3.12E+07 3.138E+07 1 770
31383072 MAKE_MAXWELL_MATRIX
0.2
2.776E+07 2.781E+07 1 130
27812674 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16200620 MAKE_LAMBDA_GRID
0.1
1.228E+07 1.228E+07 1 0
12279874 MAKE_RADIAL_OPERATORS
0.1
1.075E+07 1.076E+07 1 6
10759097 MAKE_IMPLICIT_ADVECT
0.1
6.325E+06 6.529E+06 10 196
652903 WRITE_BIG
0.1
6.205E+06 6.205E+06 10979 0 565 MPI_Allreduce()
0.0
1.468E+06 1.524E+06 1 257
1523799 MAKE_BLEND_ARRAYS
0.0
1.658E+05 1.024E+06 1000 1000 1024 GET_ERROR
0.0
8.618E+05 8.618E+05 1122 0 768 MPI_Bcast()
0.0
6.854E+04 8.398E+05 1000 1000 840 CATCH_HALT_SIGNAL
0.0
7.026E+05 7.026E+05 1 0
702551 MAKE_THETA_OPERATORS
0.0
1.396E+05 1.396E+05 1037 0 135 MPI_Send()
0.0
6.152E+04 6.174E+04 1 8 61741 MPI_Init()
0.0
652 5.792E+04 1 3 57923 WRITE_INPUT
0.0
4433 4526 1
4 4526 MPI_Finalize()
0.0
3604 3604 12 0 300 MPI_Reduce()
0.0
3058 3058 2 0 1529 MPI_Comm_split()
0.0
1839 1839 1 0
1839 MAKE_PROFILES
0.0
117 117 4 0 29 MPI_Attr_put()
0.0
106 106 4 0 26 MPI_Attr_get()
0.0
93 93 4 0
23 MPI_Keyval_free()
0.0
59 59 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 3;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.753E+06
1.12E+10 1 2808 11201926024 .TAU application
99.1
1.66E+08 1.11E+10 1000 3008
11098058 DO_FULLADVANCE
97.5
8.807E+09 1.093E+10 1000 28000
10925448 TIMESTEP_SSP_322
12.0
1.34E+09 1.343E+09 3000 6000
447671 GET_KINETIC_ADVANCE
3.5
3.961E+08 3.961E+08 24000 0 16503 MPI_Alltoall()
3.4
3.785E+08 3.796E+08 1001 2002
379262 GET_FIELD_EXPLICIT
0.3
3.063E+07 3.081E+07 1 770
30808590 MAKE_MAXWELL_MATRIX
0.2
2.722E+07 2.728E+07 1 130
27278152 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1 0
16172738 MAKE_LAMBDA_GRID
0.1
1.217E+07 1.217E+07 1 0
12170285 MAKE_RADIAL_OPERATORS
0.1
1.066E+07 1.067E+07 1 6
10669851 MAKE_IMPLICIT_ADVECT
0.1
6.613E+06 6.613E+06 10979 0 602 MPI_Allreduce()
0.1
6.151E+06 6.331E+06 10 196
633061 WRITE_BIG
0.0
1.461E+06 1.517E+06 1 257
1517106 MAKE_BLEND_ARRAYS
0.0
1.705E+05 1.472E+06 1000 1000 1472 GET_ERROR
0.0
8.387E+05 8.387E+05 1122 0 747 MPI_Bcast()
0.0
6.767E+04 8.105E+05 1000 1000 810 CATCH_HALT_SIGNAL
0.0
6.922E+05 6.922E+05 1 0
692245 MAKE_THETA_OPERATORS
0.0
1.414E+05 1.414E+05 1037 0 136 MPI_Send()
0.0
6.445E+04 6.468E+04 1 8 64685 MPI_Init()
0.0
640 4.422E+04 1
3 44217 WRITE_INPUT
0.0
4302 4408 1 4 4408 MPI_Finalize()
0.0
3025 3025 2 0 1512 MPI_Comm_split()
0.0
2783 2783 12 0
232 MPI_Reduce()
0.0
1937 1937 1 0 1937 MAKE_PROFILES
0.0
125 125 4 0 31 MPI_Attr_put()
0.0
107 107 4 0 27 MPI_Attr_get()
0.0
106 106 4 0 26 MPI_Keyval_free()
0.0
58 58 4 0 14 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 4;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.75E+06
1.12E+10 1 2808 11196780243 .TAU application
99.1
1.66E+08 1.109E+10 1000 3008
11093035 DO_FULLADVANCE
97.5
8.804E+09 1.092E+10 1000 28000
10920493 TIMESTEP_SSP_322
12.0
1.339E+09 1.343E+09 3000 6000
447607 GET_KINETIC_ADVANCE
3.5
3.947E+08 3.947E+08 24000 0 16447 MPI_Alltoall()
3.4
3.784E+08 3.796E+08 1001 2002
379225 GET_FIELD_EXPLICIT
0.3
3.06E+07 3.078E+07 1 770
30781653 MAKE_MAXWELL_MATRIX
0.2
2.717E+07 2.723E+07 1 130
27225204 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1 0
16171888 MAKE_LAMBDA_GRID
0.1
1.217E+07 1.217E+07 1 0
12169137 MAKE_RADIAL_OPERATORS
0.1
1.068E+07 1.069E+07 1 6
10694571 MAKE_IMPLICIT_ADVECT
0.1
6.49E+06 6.49E+06 10979 0 591 MPI_Allreduce()
0.1
6.159E+06 6.339E+06 10 196
633908 WRITE_BIG
0.0
1.456E+06 1.511E+06 1 257
1511018 MAKE_BLEND_ARRAYS
0.0
1.701E+05 1.386E+06 1000 1000 1386 GET_ERROR
0.0
7.932E+05 7.932E+05 1122 0 707 MPI_Bcast()
0.0
6.572E+04 7.644E+05 1000 1000 764 CATCH_HALT_SIGNAL
0.0
6.911E+05 6.911E+05 1 0
691109 MAKE_THETA_OPERATORS
0.0
1.464E+05 1.464E+05 1037 0 141 MPI_Send()
0.0
6.187E+04 6.21E+04 1 8 62095 MPI_Init()
0.0
595 3.659E+04 1 3 36589 WRITE_INPUT
0.0
1.043E+04 1.043E+04 12 0 869 MPI_Reduce()
0.0
4478 4587 1 4 4587 MPI_Finalize()
0.0
2739 2739 2 0 1370 MPI_Comm_split()
0.0
1804 1804 1 0 1804 MAKE_PROFILES
0.0
124 124 4 0 31 MPI_Attr_put()
0.0
109 109 4 0 27 MPI_Keyval_free()
0.0
102 102 4 0 26 MPI_Attr_get()
0.0
77 77 4 0 19 MPI_Comm_rank()
0.0
27 27
2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 4, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0
0
0
0
0 Message size received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 5;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.785E+06
1.125E+10 1 2808 11246624453 .TAU application
99.1
1.66E+08 1.114E+10 1000 3008
11141320 DO_FULLADVANCE
97.5
8.838E+09 1.097E+10 1000 28000
10968576 TIMESTEP_SSP_322
12.0
1.343E+09 1.346E+09 3000 6000
448738 GET_KINETIC_ADVANCE
3.6
4.046E+08 4.046E+08 24000 0 16857 MPI_Alltoall()
3.4
3.794E+08 3.806E+08 1001 2002
380233 GET_FIELD_EXPLICIT
0.3
3.123E+07 3.141E+07 1 770
31405655 MAKE_MAXWELL_MATRIX
0.2
2.777E+07 2.783E+07 1 130
27825572 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16201920 MAKE_LAMBDA_GRID
0.1
1.228E+07 1.228E+07 1 0
12282987 MAKE_RADIAL_OPERATORS
0.1
1.071E+07 1.072E+07 1 6
10724790 MAKE_IMPLICIT_ADVECT
0.1
6.369E+06 6.571E+06 10 196
657140 WRITE_BIG
0.1
6.567E+06 6.567E+06 10979 0 598 MPI_Allreduce()
0.0
1.467E+06 1.523E+06 1 257
1523162 MAKE_BLEND_ARRAYS
0.0 1.677E+05
1.385E+06 1000 1000 1385 GET_ERROR
0.0
8.271E+05 8.271E+05 1122 0 737 MPI_Bcast()
0.0
6.916E+04 8.1E+05 1000 1000 810 CATCH_HALT_SIGNAL
0.0
7.02E+05 7.02E+05 1 0
701973 MAKE_THETA_OPERATORS
0.0
1.493E+05 1.493E+05 1037 0 144 MPI_Send()
0.0
6.413E+04 6.435E+04 1 8 64350 MPI_Init()
0.0
605 6.016E+04 1 3 60162 WRITE_INPUT
0.0
4273 4384 1 4 4384 MPI_Finalize()
0.0
3081 3081 2 0 1540 MPI_Comm_split()
0.0
3001 3001 12
0 250 MPI_Reduce()
0.0
1876 1876 1 0 1876 MAKE_PROFILES
0.0
116 116 4 0 29 MPI_Attr_put()
0.0
111 111 4 0
28 MPI_Keyval_free()
0.0
107 107 4 0 27 MPI_Attr_get()
0.0
59 59 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 5, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 6;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.785E+06
1.124E+10 1 2808 11239684154 .TAU application
99.1
1.66E+08 1.114E+10 1000 3008
11135672 DO_FULLADVANCE
97.5
8.838E+09 1.096E+10 1000 28000
10963215 TIMESTEP_SSP_322
12.0
1.343E+09 1.346E+09 3000 6000
448691 GET_KINETIC_ADVANCE
3.6
3.992E+08 3.992E+08 24000 0 16634 MPI_Alltoall()
3.4
3.794E+08 3.806E+08 1001 2002
380257 GET_FIELD_EXPLICIT
0.3
3.065E+07 3.083E+07 1 770
30829763 MAKE_MAXWELL_MATRIX
0.2
2.723E+07 2.728E+07 1 130
27283623 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1 0
16172226 MAKE_LAMBDA_GRID
0.1
1.217E+07 1.217E+07 1 0
12173969 MAKE_RADIAL_OPERATORS
0.1
1.07E+07 1.072E+07 1 6
10716450 MAKE_IMPLICIT_ADVECT
0.1
6.312E+06 6.517E+06 10 196
651656 WRITE_BIG
0.1
6.353E+06 6.353E+06 10979 0 579 MPI_Allreduce()
0.0 1.462E+06 1.518E+06 1 257
1517615 MAKE_BLEND_ARRAYS
0.0
1.669E+05 1.161E+06 1000 1000 1161 GET_ERROR
0.0
8.312E+05 8.312E+05 1122 0 741 MPI_Bcast()
0.0
6.972E+04 8.083E+05 1000 1000 808 CATCH_HALT_SIGNAL
0.0
6.943E+05 6.943E+05 1 0
694317 MAKE_THETA_OPERATORS
0.0
1.422E+05 1.422E+05 1037 0 137 MPI_Send()
0.0
6.247E+04 6.268E+04 1 8 62685 MPI_Init()
0.0
600 3.795E+04 1 3 37951 WRITE_INPUT
0.0
1.011E+04 1.011E+04 12 0 842 MPI_Reduce()
0.0
4429 4531 1 4
4531 MPI_Finalize()
0.0
2999 2999 2 0 1500 MPI_Comm_split()
0.0
2113 2113 1 0 2113 MAKE_PROFILES
0.0
116 116 4 0
29 MPI_Attr_put()
0.0
102 102 4 0 26 MPI_Keyval_free()
0.0
99 99 4 0 25 MPI_Attr_get()
0.0
69 69 4 0 17 MPI_Comm_rank()
0.0
30 30 2 0 15 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 6, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 7;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.759E+06
1.12E+10 1 2808 11203093503 .TAU application
99.1
1.66E+08 1.11E+10 1000 3008
11097707 DO_FULLADVANCE
97.5
8.812E+09 1.093E+10 1000 28000
10925195 TIMESTEP_SSP_322
12.0
1.339E+09 1.342E+09 3000 6000
447435 GET_KINETIC_ADVANCE
3.5
3.92E+08 3.92E+08 24000 0 16334 MPI_Alltoall()
3.4
3.783E+08 3.794E+08 1001 2002
379053 GET_FIELD_EXPLICIT
0.3
3.13E+07 3.148E+07 1 770
31478392 MAKE_MAXWELL_MATRIX
0.2 2.781E+07
2.786E+07 1 130
27862106 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16200941 MAKE_LAMBDA_GRID
0.1
1.228E+07 1.228E+07 1 0
12283225 MAKE_RADIAL_OPERATORS
0.1
1.079E+07 1.08E+07 1 6
10803034 MAKE_IMPLICIT_ADVECT
0.1
6.502E+06 6.502E+06 10979 0 592 MPI_Allreduce()
0.1
6.166E+06 6.344E+06 10 196
634366 WRITE_BIG
0.0
1.464E+06 1.52E+06 1 257
1519859 MAKE_BLEND_ARRAYS
0.0
1.693E+05 1.353E+06 1000 1000 1353 GET_ERROR
0.0
7.921E+05 7.921E+05 1122 0 706 MPI_Bcast()
0.0
6.43E+04 7.7E+05 1000 1000 770 CATCH_HALT_SIGNAL
0.0
7.024E+05 7.024E+05 1 0
702442 MAKE_THETA_OPERATORS
0.0
1.505E+05 1.505E+05 1037 0 145 MPI_Send()
0.0
6.465E+04 6.487E+04 1 8 64870 MPI_Init()
0.0
615 5.978E+04 1 3 59783 WRITE_INPUT
0.0
4316 4423 1 4 4423 MPI_Finalize()
0.0
2927 2927 2 0 1464 MPI_Comm_split()
0.0
2764 2764 12 0 230 MPI_Reduce()
0.0
1872 1872 1 0 1872 MAKE_PROFILES
0.0
115 115 4 0 29 MPI_Attr_put()
0.0
107 107 4 0 27 MPI_Keyval_free()
0.0
105 105 4 0 26 MPI_Attr_get()
0.0
68 68 4
0 17 MPI_Comm_rank()
0.0
24 24 2 0 12 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 7, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 8;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.791E+06
1.121E+10 1
2808 11213558761 .TAU application
99.1
1.66E+08 1.111E+10 1000 3008
11109351 DO_FULLADVANCE
97.5
8.808E+09 1.094E+10 1000 28000
10936590 TIMESTEP_SSP_322
12.0
1.343E+09 1.346E+09 3000
6000 448828 GET_KINETIC_ADVANCE
3.6
4.013E+08 4.013E+08 24000 0 16720 MPI_Alltoall()
3.4
3.795E+08 3.807E+08 1001 2002
380326 GET_FIELD_EXPLICIT
0.3
3.071E+07 3.089E+07 1
770 30893601 MAKE_MAXWELL_MATRIX
0.2
2.725E+07 2.731E+07 1 130
27309715 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16200215 MAKE_LAMBDA_GRID
0.1
1.228E+07 1.228E+07 1
0 12284686 MAKE_RADIAL_OPERATORS
0.1
1.065E+07 1.067E+07 1 6
10667053 MAKE_IMPLICIT_ADVECT
0.1
6.595E+06 6.595E+06 10979 0 601 MPI_Allreduce()
0.1
6.367E+06 6.57E+06 10 196
657048 WRITE_BIG
0.0
1.467E+06 1.523E+06 1 257
1522789 MAKE_BLEND_ARRAYS
0.0
1.685E+05 1.408E+06 1000 1000 1408 GET_ERROR
0.0
8.305E+05 8.305E+05 1122 0
740 MPI_Bcast()
0.0
6.562E+04 8.099E+05 1000 1000 810 CATCH_HALT_SIGNAL
0.0
7.011E+05 7.011E+05 1 0
701059 MAKE_THETA_OPERATORS
0.0
1.499E+05 1.499E+05 1037 0 145 MPI_Send()
0.0
6.229E+04 6.253E+04 1 8 62534 MPI_Init()
0.0
592 3.803E+04 1 3 38029 WRITE_INPUT
0.0
1.128E+04 1.128E+04 12 0 940 MPI_Reduce()
0.0
4482 4578 1 4 4578 MPI_Finalize()
0.0
3074 3074 2 0 1537 MPI_Comm_split()
0.0
1997 1997 1 0 1997 MAKE_PROFILES
0.0
126 126 4 0 32 MPI_Attr_get()
0.0
119 119 4 0 30 MPI_Attr_put()
0.0
96 96 4 0 24 MPI_Keyval_free()
0.0
58 58 4 0 14 MPI_Comm_rank()
0.0
26 26 2 0 13 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 8, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 9;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.76E+06
1.119E+10 1 2808 11185440434 .TAU application
99.1
1.659E+08 1.108E+10 1000 3008
11080256 DO_FULLADVANCE
97.5
8.801E+09 1.091E+10 1000 28000
10907884 TIMESTEP_SSP_322
12.0
1.339E+09 1.343E+09 3000 6000
447509 GET_KINETIC_ADVANCE
3.4
3.848E+08 3.848E+08 24000 0 16035 MPI_Alltoall()
3.4
3.784E+08 3.795E+08 1001 2002
379152 GET_FIELD_EXPLICIT
0.3
3.13E+07 3.147E+07 1 770
31474261 MAKE_MAXWELL_MATRIX
0.2
2.78E+07 2.786E+07 1 130
27857743 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1 0
16172086 MAKE_LAMBDA_GRID
0.1
1.218E+07 1.218E+07 1 0
12177013 MAKE_RADIAL_OPERATORS
0.1
1.076E+07 1.077E+07 1 6
10771211 MAKE_IMPLICIT_ADVECT
0.1
6.405E+06 6.405E+06 10979 0 583 MPI_Allreduce()
0.1
6.108E+06 6.287E+06 10 196
628681 WRITE_BIG
0.0
1.457E+06 1.512E+06 1 257
1512226 MAKE_BLEND_ARRAYS
0.0
1.694E+05 1.284E+06 1000 1000 1284 GET_ERROR
0.0
7.927E+05 7.927E+05 1122 0 706 MPI_Bcast()
0.0
6.215E+04 7.739E+05 1000 1000 774 CATCH_HALT_SIGNAL
0.0
6.945E+05 6.945E+05 1 0
694465 MAKE_THETA_OPERATORS
0.0
1.388E+05 1.388E+05 1037 0 134 MPI_Send()
0.0
6.388E+04 6.41E+04 1 8 64095 MPI_Init()
0.0
605 6.009E+04 1 3 60093 WRITE_INPUT
0.0
4132 4236 1
4 4236 MPI_Finalize()
0.0
3072 3072 2 0 1536 MPI_Comm_split()
0.0
2793 2793 12 0 233 MPI_Reduce()
0.0
1920 1920 1
0 1920 MAKE_PROFILES
0.0
119 119 4 0 30 MPI_Attr_put()
0.0
104 104 4 0 26 MPI_Keyval_free()
0.0
97 97 4 0 24 MPI_Attr_get()
0.0
59 59 4 0 15 MPI_Comm_rank()
0.0
31 31 2 0 16 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 9, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 10;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.756E+06
1.12E+10 1 2808 11200611329 .TAU application
99.1
1.659E+08 1.11E+10 1000 3008
11096387 DO_FULLADVANCE
97.5
8.814E+09 1.092E+10 1000 28000
10923925 TIMESTEP_SSP_322
12.0
1.339E+09 1.343E+09 3000 6000
447547 GET_KINETIC_ADVANCE
3.5
3.885E+08 3.885E+08 24000 0 16189 MPI_Alltoall()
3.4
3.784E+08 3.795E+08 1001 2002
379165 GET_FIELD_EXPLICIT
0.3
3.073E+07 3.091E+07 1 770
30912337 MAKE_MAXWELL_MATRIX
0.2
2.728E+07 2.733E+07 1 130
27334209 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16200763 MAKE_LAMBDA_GRID
0.1
1.229E+07 1.229E+07 1 0
12285457 MAKE_RADIAL_OPERATORS
0.1
1.075E+07 1.076E+07 1 6
10761575 MAKE_IMPLICIT_ADVECT
0.1
6.485E+06 6.485E+06 10979 0 591 MPI_Allreduce()
0.1 6.112E+06
6.292E+06 10 196
629157 WRITE_BIG
0.0
1.462E+06 1.518E+06 1 257
1517971 MAKE_BLEND_ARRAYS
0.0
1.662E+05 1.359E+06 1000 1000 1359 GET_ERROR
0.0
7.994E+05 7.994E+05 1122 0 712 MPI_Bcast()
0.0
6.117E+04 7.746E+05 1000 1000 775 CATCH_HALT_SIGNAL
0.0
7.021E+05 7.021E+05 1 0
702069 MAKE_THETA_OPERATORS
0.0
1.457E+05 1.457E+05 1037 0 140 MPI_Send()
0.0
6.177E+04 6.201E+04 1 8 62013 MPI_Init()
0.0
615 4.102E+04 1 3 41024 WRITE_INPUT
0.0
7911 7911 12 0 659 MPI_Reduce()
0.0
4584 4680 1 4 4680 MPI_Finalize()
0.0
3109 3109 2 0 1554 MPI_Comm_split()
0.0
1991 1991 1
0 1991 MAKE_PROFILES
0.0
124 124 4 0 31 MPI_Attr_put()
0.0
119 119 4 0 30 MPI_Attr_get()
0.0
96 96 4
0 24 MPI_Keyval_free()
0.0
58 58 4 0 14 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 10, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 11;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.784E+06
1.121E+10 1 2808 11212346728 .TAU application
99.1
1.66E+08 1.111E+10 1000 3008
11107048 DO_FULLADVANCE
97.5
8.816E+09 1.093E+10 1000 28000
10934550 TIMESTEP_SSP_322
12.0
1.343E+09 1.346E+09 3000 6000
448777 GET_KINETIC_ADVANCE
3.5
3.915E+08 3.915E+08 24000 0 16314 MPI_Alltoall()
3.4
3.795E+08 3.807E+08 1001
2002 380305 GET_FIELD_EXPLICIT
0.3
3.13E+07 3.148E+07 1 770
31478067 MAKE_MAXWELL_MATRIX
0.2
2.779E+07 2.785E+07 1 130
27850372 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1
0 16172956 MAKE_LAMBDA_GRID
0.1
1.218E+07 1.218E+07 1 0
12178596 MAKE_RADIAL_OPERATORS
0.1
1.077E+07 1.078E+07 1 6
10778563 MAKE_IMPLICIT_ADVECT
0.1
6.376E+06 6.577E+06 10
196 657713 WRITE_BIG
0.1
6.372E+06 6.372E+06 10979 0 580 MPI_Allreduce()
0.0
1.458E+06 1.514E+06 1 257
1514132 MAKE_BLEND_ARRAYS
0.0
1.702E+05 1.15E+06 1000
1000 1150 GET_ERROR
0.0
8.185E+05 8.185E+05 1122 0 730 MPI_Bcast()
0.0
6.384E+04 8.011E+05 1000 1000 801 CATCH_HALT_SIGNAL
0.0
6.919E+05 6.919E+05 1 0
691947 MAKE_THETA_OPERATORS
0.0
1.46E+05 1.46E+05 1037 0 141 MPI_Send()
0.0
6.46E+04 6.484E+04 1 8 64835 MPI_Init()
0.0
609 6.26E+04 1 3 62604 WRITE_INPUT
0.0
4225 4319 1 4 4319 MPI_Finalize()
0.0
3141 3141 2 0 1570 MPI_Comm_split()
0.0
2869 2869 12 0 239 MPI_Reduce()
0.0
1927 1927 1 0 1927 MAKE_PROFILES
0.0
140 140 4 0 35 MPI_Attr_put()
0.0
100 100 4 0 25 MPI_Attr_get()
0.0
94 94 4 0 24 MPI_Keyval_free()
0.0
61 61 4 0 15 MPI_Comm_rank()
0.0
27 27 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 11, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size received
from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 12;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.785E+06
1.12E+10 1 2808 11200565320 .TAU application
99.1
1.66E+08 1.11E+10 1000 3008
11096283 DO_FULLADVANCE
97.5
8.81E+09 1.092E+10 1000 28000
10923759 TIMESTEP_SSP_322
12.0
1.341E+09 1.344E+09 3000 6000
448109 GET_KINETIC_ADVANCE
3.5
3.893E+08 3.893E+08 24000 0 16220 MPI_Alltoall()
3.4
3.789E+08 3.801E+08 1001 2002
379670 GET_FIELD_EXPLICIT
0.3
3.076E+07 3.094E+07 1 770
30936043 MAKE_MAXWELL_MATRIX
0.2
2.728E+07 2.734E+07 1 130
27341029 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16201857 MAKE_LAMBDA_GRID
0.1
1.229E+07 1.229E+07 1 0
12286825 MAKE_RADIAL_OPERATORS
0.1
1.07E+07 1.071E+07 1 6
10708534 MAKE_IMPLICIT_ADVECT
0.1
6.256E+06 6.449E+06 10 196
644878 WRITE_BIG
0.1
6.425E+06 6.425E+06 10979 0 585 MPI_Allreduce()
0.0
1.465E+06 1.52E+06 1 257
1520412 MAKE_BLEND_ARRAYS
0.0
1.698E+05 1.285E+06 1000 1000 1285 GET_ERROR
0.0
7.97E+05 7.97E+05 1122 0 710 MPI_Bcast()
0.0
6.081E+04 7.808E+05 1000 1000 781 CATCH_HALT_SIGNAL
0.0
7.017E+05 7.017E+05 1 0
701709 MAKE_THETA_OPERATORS
0.0
1.635E+05 1.635E+05 1037 0 158 MPI_Send()
0.0
6.452E+04 6.474E+04 1 8 64740 MPI_Init()
0.0
600 4.065E+04 1 3 40646 WRITE_INPUT
0.0
9138 9138 12 0 762 MPI_Reduce()
0.0
4223 4332 1 4 4332 MPI_Finalize()
0.0
3201 3201 2
0 1600 MPI_Comm_split()
0.0
2069 2069 1 0 2069 MAKE_PROFILES
0.0
117 117 4 0 29 MPI_Attr_put()
0.0
109 109 4 0 27 MPI_Keyval_free()
0.0
103 103 4 0 26 MPI_Attr_get()
0.0
57 57 4 0 14 MPI_Comm_rank()
0.0
35 35 2 0 18 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 12, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122
8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 13;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.76E+06
1.121E+10 1 2808 11211966858 .TAU application
99.1
1.659E+08 1.111E+10 1000 3008
11106764 DO_FULLADVANCE
97.5
8.818E+09 1.093E+10 1000 28000
10934503 TIMESTEP_SSP_322
12.0
1.341E+09 1.345E+09 3000 6000
448222 GET_KINETIC_ADVANCE
3.5
3.917E+08 3.917E+08 24000 0 16320 MPI_Alltoall()
3.4
3.79E+08 3.802E+08 1001
2002 379790
GET_FIELD_EXPLICIT
0.3
3.129E+07 3.147E+07 1 770
31467926 MAKE_MAXWELL_MATRIX
0.2
2.781E+07 2.786E+07 1 130
27864572 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1 0
16173150 MAKE_LAMBDA_GRID
0.1
1.218E+07 1.218E+07 1 0
12182237 MAKE_RADIAL_OPERATORS
0.1
1.075E+07 1.076E+07 1 6
10761703 MAKE_IMPLICIT_ADVECT
0.1
6.205E+06 6.395E+06 10 196
639468 WRITE_BIG
0.1
6.195E+06 6.195E+06 10979 0 564 MPI_Allreduce()
0.0
1.465E+06 1.521E+06 1 257
1520947 MAKE_BLEND_ARRAYS
0.0
1.67E+05 1.033E+06 1000 1000 1033 GET_ERROR
0.0
8.096E+05 8.096E+05 1122 0 722 MPI_Bcast()
0.0
6.299E+04 7.968E+05 1000 1000 797 CATCH_HALT_SIGNAL
0.0
6.951E+05 6.951E+05 1 0
695106 MAKE_THETA_OPERATORS
0.0
1.553E+05 1.553E+05 1037 0 150 MPI_Send()
0.0
592 6.324E+04 1 3 63236 WRITE_INPUT
0.0
6.276E+04 6.299E+04 1 8 62986 MPI_Init()
0.0
4381 4483 1 4 4483 MPI_Finalize()
0.0
3373 3373 2 0 1686 MPI_Comm_split()
0.0
2834 2834 12
0 236 MPI_Reduce()
0.0
1947 1947 1 0 1947 MAKE_PROFILES
0.0
120 120 4 0 30 MPI_Attr_put()
0.0
104 104 4 0
26 MPI_Attr_get()
0.0
102 102 4 0 26 MPI_Keyval_free()
0.0
67 67 4 0 17 MPI_Comm_rank()
0.0
23 23 2 0
12 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 13, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 14;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.772E+06
1.119E+10 1 2808 11187968331 .TAU application
99.1
1.66E+08 1.108E+10 1000 3008
11082746 DO_FULLADVANCE
97.5
8.813E+09 1.091E+10 1000 28000
10910416 TIMESTEP_SSP_322
12.0
1.339E+09 1.342E+09 3000 6000
447431 GET_KINETIC_ADVANCE
3.4
3.783E+08 3.795E+08 1001 2002
379081 GET_FIELD_EXPLICIT
3.4
3.759E+08 3.759E+08 24000 0 15662 MPI_Alltoall()
0.3
3.129E+07 3.147E+07 1 770
31469981 MAKE_MAXWELL_MATRIX
0.2
2.78E+07 2.786E+07 1 130
27859506 MAKE_POISSON_MATRIX
0.1
1.617E+07 1.617E+07 1 0
16171947 MAKE_LAMBDA_GRID
0.1
1.218E+07 1.218E+07 1 0
12180255 MAKE_RADIAL_OPERATORS
0.1
1.078E+07 1.079E+07 1 6
10787750 MAKE_IMPLICIT_ADVECT
0.1 6.32E+06
6.32E+06
10979 0 576 MPI_Allreduce()
0.1
6.107E+06 6.287E+06 10 196
628657 WRITE_BIG
0.0
1.458E+06 1.513E+06 1 257
1512858 MAKE_BLEND_ARRAYS
0.0
1.684E+05 1.221E+06 1000 1000 1221 GET_ERROR
0.0
7.908E+05 7.908E+05 1122 0 705 MPI_Bcast()
0.0
6.727E+04 7.815E+05 1000 1000 781 CATCH_HALT_SIGNAL
0.0
6.942E+05 6.942E+05 1 0
694152 MAKE_THETA_OPERATORS
0.0
1.513E+05 1.513E+05 1037 0 146 MPI_Send()
0.0
656 6.578E+04 1 3 65783 WRITE_INPUT
0.0
6.242E+04 6.265E+04 1
8 62650 MPI_Init()
0.0
4487 4601 1 4 4601 MPI_Finalize()
0.0
3658 3658 12 0 305 MPI_Reduce()
0.0
3223 3223 2 0 1612 MPI_Comm_split()
0.0
1874 1874 1 0 1874 MAKE_PROFILES
0.0
126 126 4 0 32 MPI_Attr_put()
0.0 114
114
4 0 28 MPI_Keyval_free()
0.0
99 99 4 0 25 MPI_Attr_get()
0.0
73 73 4 0 18 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 14, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
NODE 15;CONTEXT
0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.785E+06
1.12E+10 1 2808 11202957837 .TAU application
99.1
1.659E+08 1.11E+10 1000 3008
11098616 DO_FULLADVANCE
97.5
8.81E+09 1.093E+10 1000 28000
10925973 TIMESTEP_SSP_322
12.0
1.343E+09 1.346E+09 3000 6000
448737 GET_KINETIC_ADVANCE
3.5
3.893E+08 3.893E+08 24000 0 16222 MPI_Alltoall()
3.4
3.795E+08 3.806E+08 1001 2002
380254 GET_FIELD_EXPLICIT
0.3
3.077E+07 3.095E+07 1 770
30947246 MAKE_MAXWELL_MATRIX
0.2
2.729E+07 2.735E+07 1 130
27345430 MAKE_POISSON_MATRIX
0.1
1.62E+07 1.62E+07 1 0
16200622 MAKE_LAMBDA_GRID
0.1
1.229E+07 1.229E+07 1 0
12287503 MAKE_RADIAL_OPERATORS
0.1
1.07E+07 1.072E+07 1 6
10717305 MAKE_IMPLICIT_ADVECT
0.1
6.392E+06 6.592E+06 10 196
659152 WRITE_BIG
0.1
6.422E+06 6.422E+06 10979 0 585 MPI_Allreduce()
0.0
1.464E+06 1.519E+06 1 257
1519055 MAKE_BLEND_ARRAYS
0.0
1.708E+05 1.275E+06 1000 1000 1275 GET_ERROR
0.0
8.285E+05 8.285E+05 1122 0 738 MPI_Bcast()
0.0
6.641E+04 8.194E+05 1000 1000 819 CATCH_HALT_SIGNAL
0.0
7.021E+05 7.021E+05 1 0
702109 MAKE_THETA_OPERATORS
0.0
1.569E+05 1.569E+05 1037 0 151 MPI_Send()
0.0
6.36E+04 6.383E+04 1 8 63831 MPI_Init()
0.0
617 4.346E+04 1 3 43464 WRITE_INPUT
0.0
4286 4383 1 4 4383 MPI_Finalize()
0.0
2938 2938 2 0 1469 MPI_Comm_split()
0.0
2769 2769 12 0 231 MPI_Reduce()
0.0
1800 1800 1 0 1800 MAKE_PROFILES
0.0
123 123 4 0 31 MPI_Attr_put()
0.0
107 107 4 0 27 MPI_Attr_get()
0.0
97 97 4 0 24 MPI_Keyval_free()
0.0
72 72 4 0 18 MPI_Comm_rank()
0.0
28 28 2 0 14 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 15, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev.
Event Name
---------------------------------------------------------------------------------------
2.4E+04 1.311E+05 1.311E+05 1.311E+05 0
Message size for all-to-all
1122 8 4
4.232 0.9345 Message size for broadcast
1.098E+04 4.915E+04 8 9349 5122
Message size for gather
12 16 8
14.67 2.981 Message size for reduce
0 0 0 0 0 Message size
received from all nodes
1037
8192 8 111.5
761.7 Message
size sent to all nodes
---------------------------------------------------------------------------------------
FUNCTION SUMMARY (total):
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts total counts
---------------------------------------------------------------------------------------
100.0 2.847E+07
1.794E+11 16 44928 11215597822 .TAU application
99.0
2.656E+09 1.777E+11 16000 62128
11108357 DO_FULLADVANCE
97.5
1.41E+11 1.75E+11 16000 448000
10934533 TIMESTEP_SSP_322
12.0
2.147E+10 2.153E+10 48000 96000
448439 GET_KINETIC_ADVANCE
3.5
6.307E+09 6.307E+09 384000 0 16425 MPI_Alltoall()
3.4
6.068E+09 6.086E+09 16016 32032
380020 GET_FIELD_EXPLICIT
0.3
5.203E+08 5.232E+08 16 12320
32700043 MAKE_MAXWELL_MATRIX
0.3
4.534E+08 4.543E+08 16 2080
28393137 MAKE_POISSON_MATRIX
0.1
2.59E+08 2.59E+08 16 0
16186798 MAKE_LAMBDA_GRID
0.1
1.953E+08 1.953E+08 16 0
12208785 MAKE_RADIAL_OPERATORS
0.1
1.714E+08 1.716E+08 16 96
10727679 MAKE_IMPLICIT_ADVECT
0.1
1.196E+08 1.227E+08 160 3640
766582 WRITE_BIG
0.1
1.041E+08 1.041E+08 175664 0 592 MPI_Allreduce()
0.0
2.34E+07 2.429E+07 16 4112
1518201 MAKE_BLEND_ARRAYS
0.0
2.699E+06 2.165E+07 16000 16000 1353 GET_ERROR
0.0
1.651E+06 1.341E+07 16000 16000 838 CATCH_HALT_SIGNAL
0.0 1.313E+07 1.313E+07 17952 0 731 MPI_Bcast()
0.0
1.116E+07 1.116E+07 16 0
697793 MAKE_THETA_OPERATORS
0.0
2.21E+06 2.21E+06 15555 0 142 MPI_Send()
0.0
1.632E+06 1.632E+06 15555 0 105 MPI_Recv()
0.0
1.034E+06 1.038E+06 16 128 64871 MPI_Init()
0.0
4.04E+04 8.34E+05 16 62 52124 WRITE_INPUT
0.0
5.089E+05 5.089E+05 16 0 31808 MAKE_PROFILES
0.0
8.239E+04 8.239E+04 192 0 429 MPI_Reduce()
0.0
7E+04 7.164E+04 16 64 4478 MPI_Finalize()
0.0
4.895E+04 4.895E+04 32 0 1530 MPI_Comm_split()
0.0
1912 1912 64 0 30 MPI_Attr_put()
0.0
1682 1682 64 0 26 MPI_Attr_get()
0.0
1645 1645 64 0 26 MPI_Keyval_free()
0.0
1039 1039 64 0 16 MPI_Comm_rank()
0.0
459 459 32 0 14 MPI_Comm_size()
FUNCTION SUMMARY (mean):
---------------------------------------------------------------------------------------
%Time Exclusive
Inclusive #Call #Subrs
Count/Call Name
counts
total counts
---------------------------------------------------------------------------------------
100.0 1.779E+06
1.122E+10 1 2808 11215597822 .TAU application
99.0
1.66E+08 1.111E+10 1000 3883
11108357 DO_FULLADVANCE
97.5
8.815E+09 1.093E+10 1000 28000
10934533 TIMESTEP_SSP_322
12.0
1.342E+09 1.345E+09 3000 6000
448439 GET_KINETIC_ADVANCE
3.5
3.942E+08 3.942E+08 24000 0 16425 MPI_Alltoall()
3.4
3.792E+08 3.804E+08 1001 2002
380020 GET_FIELD_EXPLICIT
0.3
3.252E+07 3.27E+07 1 770
32700043 MAKE_MAXWELL_MATRIX
0.3
2.834E+07 2.839E+07 1 130
28393137 MAKE_POISSON_MATRIX
0.1
1.619E+07 1.619E+07 1 0
16186798 MAKE_LAMBDA_GRID
0.1
1.221E+07 1.221E+07 1 0
12208785 MAKE_RADIAL_OPERATORS
0.1
1.072E+07 1.073E+07 1 6
10727679 MAKE_IMPLICIT_ADVECT
0.1
7.472E+06 7.666E+06 10 227.5
766582 WRITE_BIG
0.1
6.504E+06 6.504E+06 10979 0 592 MPI_Allreduce()
0.0
1.463E+06 1.518E+06 1 257
1518201 MAKE_BLEND_ARRAYS
0.0
1.687E+05 1.353E+06 1000 1000 1353 GET_ERROR
0.0
1.032E+05 8.381E+05 1000 1000 838 CATCH_HALT_SIGNAL
0.0
8.205E+05 8.205E+05 1122 0 731 MPI_Bcast()
0.0
6.978E+05 6.978E+05 1 0
697793 MAKE_THETA_OPERATORS
0.0
1.381E+05 1.381E+05 972.188 0 142 MPI_Send()
0.0
1.02E+05 1.02E+05 972.188 0 105 MPI_Recv()
0.0
6.465E+04 6.487E+04 1 8 64871 MPI_Init()
0.0
2525 5.212E+04 1 3.875 52124 WRITE_INPUT
0.0
3.181E+04 3.181E+04 1 0 31808 MAKE_PROFILES
0.0
5149 5149 12 0
429 MPI_Reduce()
0.0
4375 4478 1 4 4478 MPI_Finalize()
0.0
3059 3059 2 0 1530 MPI_Comm_split()
0.0
119.5 119.5 4 0
30 MPI_Attr_put()
0.0
105.1 105.1 4 0 26 MPI_Attr_get()
0.0
102.8 102.8 4 0 26 MPI_Keyval_free()
0.0
64.94 64.94 4 0 16 MPI_Comm_rank()
0.0
28.69 28.69 2 0 14 MPI_Comm_size()