% $Id$
%
% Earth System Modeling Framework
% Copyright (c) 2002-2024, University Corporation for Atmospheric Research,
% Massachusetts Institute of Technology, Geophysical Fluid Dynamics
% Laboratory, University of Michigan, National Centers for Environmental
% Prediction, Los Alamos National Laboratory, Argonne National Laboratory,
% NASA Goddard Space Flight Center.
% Licensed under the University of Illinois-NCSA License.

%\subsection{Use and Examples}

\subsubsection{Output a Timing Profile to Text}
\label{sec:BasicProfiling}

ESMF profiling is disabled by default. To profile an application,
set the {\tt ESMF\_RUNTIME\_PROFILE} variable to {\tt ON} prior
to executing the application.  You do not need to recompile
your code to enable profiling.

\begin{verbatim}
# csh shell
$ setenv ESMF_RUNTIME_PROFILE ON

# bash shell
$ export ESMF_RUNTIME_PROFILE=ON

# (from now on, only the csh shell version will be shown)
\end{verbatim}

Then execute the application in the usual way. At the end of
the run the profile information will be available at the end
of each PET log (if ESMF Logs are turned on) or in a set of
separate files, one per PET, with names {\em ESMF\_Profile.XXX}
where XXX is the PET number.  Below is an example timing
profile.  Some regions are left out for brevity.

\begin{verbatim}
Region                           Count  Total (s)   Self (s)    Mean (s)    Min (s)     Max (s)
  [esm] Init 1                   1      4.0878      0.0341      4.0878      4.0878      4.0878
    [OCN-TO-ATM] IPDv05p6b       1      2.6007      2.6007      2.6007      2.6007      2.6007
    [ATM-TO-OCN] IPDv05p6b       1      1.4333      1.4333      1.4333      1.4333      1.4333
    [ATM] IPDv00p2               1      0.0055      0.0055      0.0055      0.0055      0.0055
    [OCN] IPDv00p2               1      0.0023      0.0023      0.0023      0.0023      0.0023
    [ATM] IPDv00p1               1      0.0011      0.0011      0.0011      0.0011      0.0011
    [OCN] IPDv00p1               1      0.0009      0.0009      0.0009      0.0009      0.0009
    [ATM-TO-OCN] IPDv05p3        1      0.0008      0.0008      0.0008      0.0008      0.0008
    [ATM-TO-OCN] IPDv05p1        1      0.0008      0.0008      0.0008      0.0008      0.0008
    [ATM-TO-OCN] IPDv05p2b       1      0.0007      0.0007      0.0007      0.0007      0.0007
    [ATM-TO-OCN] IPDv05p4        1      0.0007      0.0007      0.0007      0.0007      0.0007
    [ATM-TO-OCN] IPDv05p2a       1      0.0007      0.0007      0.0007      0.0007      0.0007
    [ATM-TO-OCN] IPDv05p5        1      0.0007      0.0007      0.0007      0.0007      0.0007
    [OCN-TO-ATM] IPDv05p3        1      0.0006      0.0006      0.0006      0.0006      0.0006
    [OCN-TO-ATM] IPDv05p4        1      0.0006      0.0006      0.0006      0.0006      0.0006
    [OCN-TO-ATM] IPDv05p2b       1      0.0006      0.0006      0.0006      0.0006      0.0006
    [OCN-TO-ATM] IPDv05p2a       1      0.0006      0.0006      0.0006      0.0006      0.0006
    [OCN-TO-ATM] IPDv05p5        1      0.0006      0.0006      0.0006      0.0006      0.0006
    [OCN-TO-ATM] IPDv05p1        1      0.0005      0.0005      0.0005      0.0005      0.0005
  [esm] RunPhase1                1      2.7423      0.9432      2.7423      2.7423      2.7423
    [OCN-TO-ATM] RunPhase1       864    0.6094      0.6094      0.0007      0.0006      0.0179
    [ATM] RunPhase1              864    0.5296      0.2274      0.0006      0.0005      0.0011
      ATM:ModelAdvance           864    0.3022      0.3022      0.0003      0.0003      0.0005
    [ATM-TO-OCN] RunPhase1       864    0.3345      0.3345      0.0004      0.0002      0.0299
    [OCN] RunPhase1              864    0.3256      0.3256      0.0004      0.0003      0.0010
  [esm] FinalizePhase1           1      0.0029      0.0020      0.0029      0.0029      0.0029
    [OCN-TO-ATM] FinalizePhase1  1      0.0006      0.0006      0.0006      0.0006      0.0006
    [ATM-TO-OCN] FinalizePhase1  1      0.0002      0.0002      0.0002      0.0002      0.0002
    [OCN] FinalizePhase1         1      0.0001      0.0001      0.0001      0.0001      0.0001
    [ATM] FinalizePhase1         1      0.0000      0.0000      0.0000      0.0000      0.0000
\end{verbatim}

A timed region is either an ESMF component phase (e.g., initialize,
run, or finalize) or a user-defined region of code surrounded by calls to
{\tt ESMF\_TraceRegionEnter()} and {\tt ESMF\_TraceRegionExit()}. (See
section \ref{ex:TraceUserEx} for more information on instrumenting
user-defined regions.)
Regions are organized hierarchically with sub-regions nested.
For example, in the profile above,
the {\tt [OCN] RunPhase1} is a sub-region of {\tt [esm] RunPhase1} and is
entirely contained inside that region. Regions with the same name may appear
at multiple places in the hierarchy, and so would appear in multiple rows
in the table.  The statistics in that row apply to that region at that
location in the hierarchy. Component names appear in square brackets,
e.g., {\tt [ATM]},  {\tt [OCN]}, and {\tt [ATM-TO-OCN]}.
By default, timings are based on elapsed wall clock time and are collected
on a per-PET basis. Therefore, regions timings may differ across PETs. Regions
are sorted with the most expensive regions appearing at the top. The following
describes the meaning of the statistics in each column:

\begin{itemize}
\item [{\tt Count}] the number of times the region is executed
\item [{\tt Total}] the aggregate time spent in the region, inclusive of all sub-regions
\item [{\tt Self}]  the aggregate time spend in the region, exclusive of all sub-regions
\item [{\tt Mean}]  the average amount of time for one execution of the region
\item [{\tt Min}]   time of the fastest execution of the region
\item [{\tt Max}]   time of the slowest execution of the region
\end{itemize}


\subsubsection{Summarize Timings across Multiple PETs}
\label{sec:SummaryProfiling}

By default, separate timing profiles are generated for each PET
in the application.  The per-PET profiles can be aggregated together
and output to a single file, {\em ESMF\_Profile.summary}, by setting the
{\tt ESMF\_RUNTIME\_PROFILE\_OUTPUT} environment variable as follows:

\begin{verbatim}
$ setenv ESMF_RUNTIME_PROFILE ON              # turn on profiling
$ setenv ESMF_RUNTIME_PROFILE_OUTPUT SUMMARY  # specify summary output
\end{verbatim}

Note the {\tt ESMF\_RUNTIME\_PROFILE} environment variable must
also be set to {\tt ON} since this controls all profiling capabilities.
The {\em ESMF\_Profile.summary} file will contain a tree of
timed regions, but aggregated across all PETs. For example:

\begin{verbatim}
Region                           PETs   PEs    Count    Mean (s)    Min (s)     Min PET Max (s)     Max PET
  [esm] Init 1                   4      4      1        4.0880      4.0878      2       4.0883      1
    [OCN-TO-ATM] IPDv05p6b       4      4      1        2.6007      2.6007      2       2.6007      3
    [ATM-TO-OCN] IPDv05p6b       4      4      1        1.4335      1.4333      0       1.4337      3
    [ATM-TO-OCN] IPDv05p4        4      4      1        0.0037      0.0007      0       0.0060      1
    [ATM] IPDv00p2               4      4      1        0.0034      0.0020      1       0.0055      0
    [ATM-TO-OCN] IPDv05p1        4      4      1        0.0020      0.0007      2       0.0033      3
    [OCN] IPDv00p2               4      4      1        0.0019      0.0015      3       0.0024      2
    [ATM-TO-OCN] IPDv05p3        4      4      1        0.0010      0.0008      0       0.0013      1
    [ATM-TO-OCN] IPDv05p2a       4      4      1        0.0009      0.0007      0       0.0012      3
    [ATM] IPDv00p1               4      4      1        0.0009      0.0007      3       0.0011      0
    [ATM-TO-OCN] IPDv05p2b       4      4      1        0.0008      0.0007      0       0.0010      3
    [ATM-TO-OCN] IPDv05p5        4      4      1        0.0008      0.0007      0       0.0010      3
    [ATM-TO-OCN] IPDv05p6a       4      4      1        0.0008      0.0005      2       0.0012      3
    [OCN-TO-ATM] IPDv05p3        4      4      1        0.0008      0.0006      2       0.0010      3
    [OCN-TO-ATM] IPDv05p4        4      4      1        0.0008      0.0006      0       0.0009      3
    [OCN-TO-ATM] IPDv05p2b       4      4      1        0.0007      0.0006      2       0.0009      3
    [OCN] IPDv00p1               4      4      1        0.0007      0.0005      1       0.0009      2
    [OCN-TO-ATM] IPDv05p2a       4      4      1        0.0007      0.0006      2       0.0009      1
    [OCN-TO-ATM] IPDv05p5        4      4      1        0.0007      0.0006      0       0.0009      3
    [OCN-TO-ATM] IPDv05p1        4      4      1        0.0006      0.0005      0       0.0008      1
    [OCN-TO-ATM] IPDv05p6a       4      4      1        0.0006      0.0004      2       0.0007      1
  [esm] RunPhase1                4      4      1        2.7444      2.7423      0       2.7454      1
    [OCN-TO-ATM] RunPhase1       4      4      864      0.6123      0.6004      2       0.6244      1
    [ATM] RunPhase1              4      4      864      0.5386      0.5296      0       0.5530      1
      ATM:ModelAdvance           4      4      864      0.3038      0.3022      0       0.3065      1
    [OCN] RunPhase1              4      4      864      0.3471      0.3256      0       0.3824      1
    [ATM-TO-OCN] RunPhase1       4      4      864      0.2843      0.1956      1       0.3345      0
  [esm] FinalizePhase1           4      4      1        0.0029      0.0029      1       0.0030      2
    [OCN-TO-ATM] FinalizePhase1  4      4      1        0.0007      0.0006      0       0.0008      3
    [ATM-TO-OCN] FinalizePhase1  4      4      1        0.0002      0.0001      3       0.0002      1
    [OCN] FinalizePhase1         4      4      1        0.0001      0.0001      3       0.0001      0
    [ATM] FinalizePhase1         4      4      1        0.0001      0.0000      0       0.0001      2
\end{verbatim}

The meaning of the statistics in each column in as follows:
\begin{itemize}
\item [{\tt PETs}] the number of reporting PETs that executed the region
\item [{\tt PEs}] the number of PEs associated with the reporting PETs that executed the region
\item [{\tt Count}] the number of times each reporting PET executed the region
      or ``MULTIPLE'' if not all PETs executed the region the same number of times
\item [{\tt Mean}] the mean across all reporting PETs of the total time spent in the region
\item [{\tt Min}] the minimum across all reporting PETs of the total time spent in the region
\item [{\tt Min PET}] the PET that reported the minimum time
\item [{\tt Max}] the maximum across all reporting PETs of the total time spent in the region
\item [{\tt Max PET}] the PET that reported the maximum time
\end{itemize}

Note that setting the {\tt ESMF\_RUNTIME\_PROFILE\_PETLIST} environment variable
(described below) may reduce the number of reporting PETs. Only reporting PETs are
included in the summary profile. To output both the per-PET and summary timing profiles,
set the {\tt ESMF\_RUNTIME\_PROFILE\_OUTPUT} environment variable as follows:

\begin{verbatim}
$ setenv ESMF_RUNTIME_PROFILE_OUTPUT "TEXT SUMMARY"
\end{verbatim}


\subsubsection{Limit the Set of Profiled PETs}
\label{sec:LimitProfiling}

By default, all PETs in an application are profiled. It may be desirable
to only profile a subset of PETs to reduce the amount of output.
An explicit list of PETs can be specified by setting the
{\tt ESMF\_RUNTIME\_PROFILE\_PETLIST} environment variable.
The syntax of this environment variable is to list
PET numbers separated by spaces. PET ranges are also supported using
the ``X-Y'' syntax where X < Y.
For example:

\begin{verbatim}
# only profile PETs 0, 20, and 35 through 39
$ setenv ESMF_RUNTIME_PROFILE_PETLIST "0 20 35-39"
\end{verbatim}

When used in conjunction with the {\tt SUMMARY} option above, the summarized
profile will only aggregate over the specified set of PETs. The one exception is that
PET 0 is always profiled if {\tt ESMF\_RUNTIME\_PROFILE=ON}, regardless of the
{\tt ESMF\_RUNTIME\_TRACE\_PETLIST} setting.


\subsubsection{Include MPI Communication in the Profile}
\label{sec:MPIProfiling}

MPI functions can be included in the timing profile to indicate how much time
is spent inside communication calls.  This can also help to determine load imbalance
in the system, since large times spent inside MPI may indicate that communication
between PETs is not tightly synchronized.  This option includes {\em all} MPI calls in
the application, whether or not they originate from the ESMF library.  Here is a partial
example summary profile that contains MPI times:

\begin{verbatim}
Region                           PETs   Count    Mean (s)    Min (s)     Min PET Max (s)     Max PET
  [esm] RunPhase1                8      1        4.9307      4.6867      0       4.9656      1
    [OCN] RunPhase1              8      1824     0.8344      0.8164      0       0.8652      1
    [MED] RunPhase1              8      1824     0.8203      0.7900      5       0.8584      1
    [ATM] RunPhase1              8      1824     0.6387      0.6212      5       0.6610      1
    [ATM-TO-MED] RunPhase1       8      1824     0.5975      0.5317      0       0.6583      5
      MPI_Bcast                  8      1824     0.0443      0.0025      4       0.1231      5
      MPI_Wait                   8      MULTIPLE 0.0421      0.0032      0       0.0998      2
    [MED-TO-OCN] RunPhase1       8      1824     0.4879      0.4497      0       0.5362      4
      MPI_Wait                   8      MULTIPLE 0.0234      0.0030      0       0.0821      4
      MPI_Bcast                  8      1824     0.0111      0.0024      4       0.0273      5
    [OCN-TO-MED] RunPhase1       8      1824     0.4541      0.4075      0       0.4918      4
      MPI_Wait                   8      MULTIPLE 0.0339      0.0017      0       0.0824      4
      MPI_Bcast                  8      1824     0.0194      0.0026      4       0.0452      6
    [MED-TO-ATM] RunPhase1       8      1824     0.4487      0.4005      0       0.4911      5
      MPI_Bcast                  8      1824     0.0338      0.0026      4       0.0942      5
      MPI_Wait                   8      MULTIPLE 0.0241      0.0022      1       0.0817      2
  [esm] Init 1                   8      1        0.6287      0.6287      1       0.6287      4
    [ATM-TO-MED] IPDv05p6b       8      1        0.1501      0.1500      1       0.1501      2
      MPI_Barrier                8      242      0.0082      0.0006      3       0.0157      7
      MPI_Wait                   8      MULTIPLE 0.0034      0.0010      0       0.0053      7
      MPI_Allreduce              8      62       0.0030      0.0003      3       0.0063      7
      MPI_Alltoall               8      6        0.0015      0.0000      1       0.0022      5
      MPI_Allgather              8      21       0.0010      0.0002      1       0.0017      7
      MPI_Waitall                8      MULTIPLE 0.0006      0.0001      3       0.0015      7
      MPI_Send                   8      MULTIPLE 0.0004      0.0001      7       0.0008      6
      MPI_Allgatherv             8      6        0.0001      0.0001      4       0.0001      0
      MPI_Scatter                8      5        0.0000      0.0000      0       0.0000      7
      MPI_Reduce                 8      5        0.0000      0.0000      1       0.0000      0
      MPI_Recv                   8      MULTIPLE 0.0000      0.0000      0       0.0000      3
      MPI_Bcast                  8      1        0.0000      0.0000      0       0.0000      7
\end{verbatim}

The procedure for including MPI
functions in the timing profile depends on whether the application is
dynamically or statically linked. Most applications are dynamically linked,
however on some systems (such as Cray), static linking may be used.
Note that for either option, ESMF must be built with {\tt ESMF\_TRACE\_LIB\_BUILD=ON},
which is the default.

In {\em dynamically linked applications}, the {\tt LD\_PRELOAD} (Linux) or
{\tt DYLD\_INSERT\_LIBRARIES} (Darwin) environment variable must be used when
executing the MPI application.  This instructs the dynamic loader to interpose
certain MPI symbols so they can be captured by the ESMF profiler.  To simplify
this process, a script is provided at {\tt \$(ESMF\_INSTALL\_LIBDIR)/preload.sh}
that sets the appropriate variable.

For example, if you typically execute your application as as follows:

\begin{verbatim}
$ mpirun -np 8 ./myApp
\end{verbatim}

then you should add the {\em preload.sh} script in front of the
executable when starting the application as follows:

\begin{verbatim}
# replace $(ESMF_INSTALL_LIBDIR) with absolute path
# ... to the ESMF installation lib directory
$ mpirun -np 8 $(ESMF_INSTALL_LIBDIR)/preload.sh ./myApp
\end{verbatim}

An advantage of this approach is that your application does {\em not} need to
be recompiled. The MPI timing information will be included in the per-PET profiles and/or the summary
profile, depending on the setting of environment variable
{\tt ESMF\_RUNTIME\_PROFILE\_OUTPUT}.

Notice that an additional step is required for dynamically linked applications
on {\em Darwin} systems with System Integrity Protection (SIP) enabled! In
addition to using the {\tt \$(ESMF\_INSTALL\_LIBDIR)/preload.sh} script during
launching of the executable as shown above, the executable must {\em also be
linked} against the dynamic ESMF trace preload library. This must happen during
the link step of the executable. It is most easily accomplished by using
variable {\tt \$(ESMF\_F90ESMFPRELOADLINKLIBS)} instead of the typical
{\tt \$(ESMF\_F90ESMFLINKLIBS)} variable for the final link command. Both
variables are defined in the {\em esmf.mk} file that should be imported by
the application Makefile. For example:

\begin{verbatim}
# import esmf.mk
include $(ESMFMKFILE)

# other makefile targets here...

# example final link command, with $(ESMF_F90ESMFPRELOADLINKLIBS)
myApp: myApp.o driver.o model.o
        $(ESMF_F90LINKER) $(ESMF_F90LINKOPTS) $(ESMF_F90LINKPATHS) \
        $(ESMF_F90LINKRPATHS) -o $@ $^ $(ESMF_F90ESMFPRELOADLINKLIBS)
\end{verbatim}

In {\em statically linked applications}, the application must be re-linked
with specific options provided to the linker.  These options instruct the linker
to wrap the MPI symbols with the ESMF profiling functions. The linking flags that
must be provided are included in the {\em esmf.mk} Makefile fragment that
is part of the ESMF installation. These link flags should be imported into
your application Makefile, and included in the final link command. To do this,
first import the {\em esmf.mk} file into your application Makefile. The path
to this file is typically stored in the {\tt ESMFMKFILE} environment variable.
Then, pass the variables {\tt \$(ESMF\_TRACE\_STATICLINKOPTS)} and
{\tt \$(ESMF\_TRACE\_STATICLINKLIBS)} to the final linking command.  For example:

\begin{verbatim}
# import esmf.mk
include $(ESMFMKFILE)

# other makefile targets here...

# example final link command, with $(ESMF_TRACE_STATICLINKOPTS)
# ... and $(ESMF_TRACE_STATICLINKLIBS) added
myApp: myApp.o driver.o model.o
        $(ESMF_F90LINKER) $(ESMF_F90LINKOPTS) $(ESMF_F90LINKPATHS) \
        $(ESMF_F90LINKRPATHS) -o $@ $^ $(ESMF_F90ESMFLINKLIBS) \
        $(ESMF_TRACE_STATICLINKOPTS) $(ESMF_TRACE_STATICLINKLIBS)
\end{verbatim}

This option will statically wrap all of the MPI functions and include them
in the profile output.  Execute the application in the normal way
with the environment variable {\tt ESMF\_RUNTIME\_PROFILE} set to {\tt ON}.
You will see the MPI functions included in the timing profile.

\subsubsection{Output a Detailed Trace for Analysis}


ESMF tracing is disabled by default. To enable tracing, set the
{\tt ESMF\_RUNTIME\_TRACE} environment variable to {\tt ON}. You
do not need to recompile your code to enable tracing.

\begin{verbatim}
# csh shell
$ setenv ESMF_RUNTIME_TRACE ON

# bash shell
$ export ESMF_RUNTIME_TRACE=ON
\end{verbatim}

When enabled, the default behavior is to trace all PETs of the
ESMF application. Although the ESMF tracer is designed to write
events in a compact form, tracing can produce an extremely
large number of events depending on the total number of PETs and
the length of the run. To reduce output, it is possible to restrict
the PETs that produce trace output by setting the {\tt ESMF\_RUNTIME\_TRACE\_PETLIST}
environment variable. For example, this setting:

\begin{verbatim}
$ setenv ESMF_RUNTIME_TRACE_PETLIST "0 101 192-196"
\end{verbatim}

will instruct the tracer to only trace PETs 0, 101, and 192 through 196
(inclusive). The syntax of this environment variable is to list
PET numbers separated by spaces. PET ranges are also supported using
the ``X-Y'' syntax where X < Y. For PET counts greater than 100, it is
recommended to set this environment variable. The one exception is that
PET 0 is always traced, regardless of the {\tt ESMF\_RUNTIME\_TRACE\_PETLIST}
setting.

ESMF's profiling and tracing options can be used together.  A typical
use would be to set {\tt ESMF\_RUNTIME\_PROFILE=ON} for all PETs to
capture summary timings, and set {\tt ESMF\_RUNTIME\_TRACE=ON} and
{\tt ESMF\_RUNTIME\_TRACE\_PETLIST} to a subset of of PETs,
such as the root PET of each ESMF component. This helps to keep trace
sizes small while still providing timing summaries over all PETs.

When tracing is enabled, {\tt phase\_enter} and {\tt phase\_exit} events will
automatically be recorded for all initialize, run, and finalize phases of all
Components in the application. To trace {\em only} user-instrumented regions (via
the {\tt ESMF\_TraceRegionEnter()} and {\tt ESMF\_TraceRegionExit()} calls),
Component-level tracing can be turned off by setting:

\begin{verbatim}
$ setenv ESMF_RUNTIME_TRACE_COMPONENT OFF
\end{verbatim}

After running an ESMF application with tracing enabled, a directory
called {\em traceout} will be created in the run directory and it will
contain a {\em metadata} file and an event stream file {\em esmf\_stream\_XXXX}
for each PET with tracing enabled. Together these files form a valid
CTF trace which may be analyzed with any of the tools listed above.

Trace events are flushed to file at a regular interval. If the application
crashes, some of the most recent events may not be flushed to file. To
maximize the number of events appearing in the trace, an option is available
to flush events to file more frequently. Because this option may have
negative performance implications due to increased file I/O, it is not
recommended unless needed. To turn on eager flushing use:

\begin{verbatim}
$ setenv ESMF_RUNTIME_TRACE_FLUSH EAGER
\end{verbatim}

\subsubsection{Set the Clock used for Profiling/Tracing}
\label{sec:TracingClocks}

There are three options for the kind of clock to use to timestamp
events when profiling/tracing an application.
These options are controlled by setting the environment variable
{\tt ESMF\_RUNTIME\_TRACE\_CLOCK}.
\begin{itemize}
\item [{\tt REALTIME}] The {\tt REALTIME} clock timestamps events with the current time on
      the system.  This is the default clock if the above environment
      variable is not set.  This setting can be useful when tracing PETs that
      span multiple physical computing nodes assuming that the system clocks
      on each node are adequately synchronized.  On most HPC systems, system
      clocks are periodically updated to stay in sync.  A disadvantage of this
      clock is that periodic adjustments mean the clock is not monotonically
      increasing so some timings may be inaccurate if the system clock jumps
      forward or backward significantly. Testing has shown that this is not
      typically an issue on most systems.
\item [{\tt MONOTONIC}] The {\tt MONOTONIC} clock is guaranteed to be monotonically increasing
      and does not suffer from periodic adjustments.  The timestamps represent
      an amount of time since some arbitrary point in the past.  There is no
      guarantee that these timestamps will be synchronized across physical
      computing nodes, so this option should only be used for tracing a set of PETs
      running on a single physical machine.
\item [{\tt MONOTONIC\_SYNC}] The {\tt MONOTONIC\_SYNC} clock is similar to the {\tt MONOTONIC} clock
      in that it is guaranteed to be monotonically increasing. In addition, at
      application startup, all PET clocks are synchronized to a common time
      by determining a PET-local offset to be applied to timestamps. Therefore this option
      can be used to compare trace streams across physical nodes.
\end{itemize}
