|
You can use the Performance Analyzer to examine Message Passing Interface (MPI) applications to answer the following questions:
- Would tuning the MPI code produce significant performance improvement?
- Is the MPI performance characterized by synchronization or data transfer?
- Does the program contain load imbalances?
- How long is one iteration of program execution?
- How long does it take for program performance to equilibrate?
- What are the message-passing patterns in program execution?
- Which are most important: long or short messages?
- Do processes that send messages synchronize with processes that receive messages?
The preceding list is too broad to address in this single tutorial. The goal
of this tutorial is to guide you through some basic new features of the
Performance Analyzer including the following:
- The MPI Timeline tab, which graphically displays the MPI activity
that occurred during an application's run.
- The MPI Charts tab, which generates scatter plots and histograms
to visualize the performance data of MPI functions and MPI messages.
- The MPI data-zooming and data-filtering controls which you can use
to broaden or narrow your view of the data in the MPI Timeline and MPI Charts.
The MPI Timeline tab presents the data from a run of the test program as a
timeline. Initially, your view of the timeline encompasses the run from
beginning to end with all MPI functions and MPI messages represented
graphically in a condensed form. You'll learn how to expand this presentation
and move down from a complete view to a tightly focused view that can be as
granular as a single function. The MPI Timeline tab offers many different ways
to zoom, pan, and examine the data, together with MPI Charts tab. The MPI
Charts tab enables you to plot statistical data about the functions and
messages in graphical charts, to help you see what is happening in the run.
The tutorial is designed to be followed from beginning to end to show you
how to use the new MPI features, and covers the following topics:
Setting Up for the Tutorial
The Performance Analyzer works with the Sun
ClusterTools 7 and ClusterTools 8 software. HPC ClusterTools is an integrated
toolkit for creating and tuning MPI applications that run on high performance
clusters of Sun systems. This tutorial explains how to use the Performance
Analyzer on an example MPI application called ring_c, which is
included with the Sun HPC ClusterTools 8.1 software.
You must already have a cluster configured and functioning for this
tutorial.
Follow the steps below to get started.
- Download the ClusterTools 8.1 release at
http://www.sun.com/software/products/clustertools.
- Install the ClusterTools software as described in the
Quick Installation Guide, which is available in the
sun-hpc-ct-8.1-docs.tar.gz
documentation tar file on the Sun Download Center page.
- Add the
/SunStudio_installation_directory/bin directory and
the ClusterTools_installation_directory/bin directory to your
path.
- Copy the
/ClusterTools_installation_directory/examples
directory into a directory to which you have write access. This directory must
be visible from all the cluster nodes.
- Change directory to your newly copied
examples directory.
- Build the
ring_c example.
% make ring_c
mpicc -g -o ring_c ring_c.c
|
The program is compiled with the -g option,
which allows the Performance-Analyzer data collector to map MPI events to
source code.
The ring_c program simply passes a message from process to
process in a ring, then terminates.
Run the ring_c example with mpirun to make sure it
works correctly.
This example shows how to run the program on a two-node cluster;
each node handles up to 32 threads. The node names are specified in a
host file, along with the number of slots that are to be used on each node.
We have chosen to use 25 processes, and specify one slot on each host. You
should specify a number of processes and slots that is appropriate for your
system. See the mpirun(1) man page for more information about
specifying hosts and slots. You can also run this command on a standalone
host that isn't part of a cluster, but the results might be less educational.
The host file for this example is called clusterhosts and contains the following content:
hostA slots=1
hostB slots=1
|
You must have permission to use a remote shell (ssh/rsh) to each host
without logging into the hosts. By default, mpirun uses ssh.
 |
% mpirun -np 25 --hostfile clusterhosts ring_c
Process 0 sending 10 to 1, tag 201 (25 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting
Process 2 exiting
Process 3 exiting
Process 4 exiting
Process 5 exiting
Process 6 exiting
Process 7 exiting
Process 8 exiting
Process 9 exiting
Process 10 exiting
Process 11 exiting
Process 12 exiting
Process 13 exiting
Process 14 exiting
Process 15 exiting
Process 16 exiting
Process 17 exiting
Process 18 exiting
Process 19 exiting
Process 20 exiting
Process 21 exiting
Process 22 exiting
Process 23 exiting
Process 24 exiting
|
Run this command and if you get similar output, you are ready to collect
data on an example application as shown in the next section.
If you have problems with mpirun specifying ssh,
try using the option --mca plm_rsh_agent rsh to the
mpirun command to specify the rsh command:
% mpirun -np 25 --hostfile clusterhosts --mca plm_rsh_agent rsh -- ring_c
|
Collecting Data on the ring_c Example
- Change to the directory where your example binaries and source code
are located. This directory must be visible from all the cluster nodes.
- Run the following command:
% collect -M MPI_version -p off mpirun -np 25 --hostfile clusterhosts -- ring_c
|
The -M MPI_version option indicates that the collect is running
on an MPI program, and the -p off option turns off clock-based profiling to simplify the data collection. See the collect(1) man page for more information. The collect command might take a few moments to run and the output should be the same as the test run through the mpirun command.
The -np 25 option specifies 25 processes on the cluster, and
-hostfile clusterhosts indicates that the node names and the number
of slots that are to be used on each node are specified in a host file called
clusterhosts. We have chosen to use 25 processes on two hosts,
and specify one slot on each host. You should specify a number of processes
and slots that is appropriate for your system.
-
List the contents of the newly created
test.1.er directory and
make sure the date on the files reflects the latest execution. This means
you ran the command successfully and are ready to run the Performance Analyzer
on ring_c. The integer in test.1.er increments for each
collect command you run so the rest of this tutorial refers to
this name generically as test.*.er.
Opening the Experiment
- Change to the directory that contains the
ring_c.c
source file, the ring_c executable, and the test.*.er
directory.
Start the Performance Analyzer from the command line:
% /Sun_Studio_installation_directory/bin/analyzer
|
The Performance Analyzer opens a file browser for you to find and
open an experiment. If not, choose File > Open Experiment.
-
Find the test.*.er experiment that you just created and open it.
The Performance Analyzer window should look similar to that below.
The experiment opens on the MPI Timeline tab. The MPI Charts tab is next
to it. In the right panel you can see the MPI Chart Controls and MPI
Timeline Controls tabs.
The MPI Timeline shows a view of the data over time as the program was run
through the collector. The horizontal axis shows elapsed time. At the bottom,
the horizontal axis shows "relative time" with the origin at the left edge of
the display. At the top, the horizontal axis shows "absolute time" where the
origin is the start of the data. The vertical axis shows MPI process rank.
Therefore, for each MPI process you can look horizontally to see what the
process is doing as a function of elapsed time.
In this initial view of the timeline, you can answer the question "What is the time scale of program execution?" In the screen capture, you can see that it is about 5 seconds, but only from 3.90 to 4.05 is actual run time, the steady state of the application program. The collect tool uses MPI_Init and MPI_Finalize to set up and terminate data collection.
Navigating the MPI Timeline By Zooming and Panning
- Click the MPI Timeline tab if it's not already selected.
- Zoom in on the data by clicking and dragging from the left to the right
on any process row as shown by the directional arrow in the graphic below.
When you release the mouse button, the area inside the box automatically
expands to reveal a zoomed in view.
An alternative to clicking and dragging is to use the zooming slider
controls in the top left of the timeline.
Use the horizontal slider to change the time scale.
You'll see progressively smaller chunks of time, while still showing all
the processes, as you zoom.
Use the vertical slider to zoom in on the MPI processes.
Click the Zoom Undo button in the MPI Timeline Controls tab shown below to
go back to the previous level of zooming.
Click the Zoom Undo button a second time to return to the first zoom.
Pan across the data by sliding the scroll bars located at the bottom
and the right of the timeline.
Alternatively you can toggle between a pointer that zooms and a pointer
that pans by clicking the hand icon in the MPI Timeline Controls tab.
When the pointer is a hand, you can drag across the MPI Timeline to pan
horizontally.
Viewing Message Details
- Reset the view to the original, maximum, zoomed-out setting by clicking
the Zoom Reset button, which is located to the top left of the zoom sliders.
- Zoom in on the activity area by dragging on the area horizontally with
the mouse so it looks similar to what you see here.
In the zoomed in timeline, now you can see that the steady state portion of
the program execution appears to be from 3.93 seconds to 4.03 seconds.
You can also see that MPI functions are color coded. The black lines drawn
between events represent point-to-point messages exchanged by the MPI
processes.
With this view of the timeline, you can answer the question:
"How long is one iteration before the pattern repeats?" The answer is
roughly 10 milliseconds. Look at the relative time scale at the bottom to
see how often the loop seems to repeat.
Click one of the black message lines.
The line turns red and details about the message are displayed in the
right-hand panel MPI Timeline Controls tab.
In the MPI Timeline Controls tab, find the Messages slider, then click
and drag it to 0% as shown.
The Messages slider controls the number of message lines displayed on the
screen. At 0%, only functions are displayed in the MPI Timeline tab.
In this simple example, 100% of the messages can be displayed. However,
in complex applications, if all messages were displayed, the volume of
messages could be very high, overwhelming the tool with large data volume
and making the screen too cluttered to be usable. Select a lower percentage
of messages to reduce the volume of messages shown in the timeline. The tool
adjusts default levels for the message volume so the screen is readable and
the tool remains responsive. If fewer than 100% of the messages are shown,
the messages used are those messages that are most "costly" in terms of the
total time used in the message's send and receive functions.
- Set the Messages slider back to 100%.
Viewing Function Details and Application Source Code
Click on one of the MPI_Recv function events in the MPI Timeline
tab.
The function is highlighted in yellow, and details about the function are
displayed in the MPI Timeline Controls tab on the right.
- In the MPI Timeline Controls tab, click the button labeled
Show Call Stack if available.
After a few moments, the call stack for the highlighted state should be shown in the MPI Timeline Controls tab:
- When the Call Stack for Selected Event is displayed in the MPI
Timeline Controls tab, click on
main + 0x00000198, line 53 in
"ring_c.c
Click the Source tab in the main Performance Analyzer panel.
If you get a message such as "Object file (unknown) not readable",
make sure you selected the stack frame main + 0x00000198, line 53 in
"ring_c.c.
Note - Source is only visible when the source is in the same location
it was in when the program was run through the collector, or when it can be
found in the $expts path as set in .er.rc or in View >
Set Data Presentation > Search Path. Source also needs to be compiled with
-g.
If the source code is not visible, you may not have started the Analyzer
from the directory containing the ring_c binary and source code.
If this is the case, quit the Performance Analyzer and restart after you
cd to the directory containing ring_c.
When the source becomes visible, you should see the following.
The source should show where main() calls MPI_Recv().
As you can see, MPI_Recv() is called from line 53 in the source.
The green bar highlights metrics with high values. 274 receives are associated
with line 53. If you look further down, you can see 274 sends are associated
with MPI_Send on line 60.
Click the Functions tab in the main Performance Analyzer panel.
The Functions tab shows the same MPI Send and MPI Receive metrics in
columns on the left side of a table. You can sort the table by clicking
in the column headers.
Click the MPI Timeline tab to return to the MPI timeline.
Do not click the regular Timeline tab because it does not apply to MPI
programs.
Filtering Data in the MPI Tabs
The filtering facility lets you select different views of the
collected messaging data. You can undo and redo the filters using the
filtering controls in either the MPI Chart Controls tab or the MPI Timeline
Controls tab.
The first control filters the data by removing everything that is not
currently in view.
The second control is the Filter Undo button which provides an associated
drop down list for removing filters. Clicking this button removes the last
filter applied. Clicking the down arrow presents a list of the filters applied,
in the order they were applied, with the most recent at the top of the list.
When you select a filter in this list, the selected filter and all filters
above it on the list are removed.
The third control is the Filter Redo button, and it also has an associated
drop down list for reapplying filters. Clicking the button reapplies the last
filter that you removed. Clicking the down arrow opens a list of all the
filters that have been removed, in the order in which they were removed.
When you select a filter in this list, the selected filter and all filters
above it on the list are reapplied.
You can redo and undo the filters by using the arrows, similar to going
backward and forward in a web browser. You can even remove and apply more
than one filter in one click by using the down arrows next to the filter
buttons.
The following steps explain how to use a filter to focus on the steady s
tate portion of the program by filtering out the MPI_Init and
MPI_Finalize functions.
- Zoom in on the area of absolute time t=3.93 to 4.03 by dragging as show
below.
- Click the Filter button in the MPI Timeline Controls tab:
It may look like nothing happened because the filtering is not evident
until you change your view by zooming out or by looking at a chart.
-
Click the Zoom Undo button to go back out to the previous zoom.
The display now shows Uninstrumented in place of the
MPI_Init and MPI_Finalize functions. White areas
labeled as Uninstrumented indicate that there is no MPI data
collected for that area or the data has been filtered out.
Using the Filter Stack
-
Drag vertically until you have zoomed-in far enough to see a single
MPI_Send process. You may have to first drag horizontally to zoom in close
enough to see some MPI_Send processes.
-
Click the Filter button one time to filter out all data except
MPI_Send data.
-
Click the MPI Timeline tab again and click the Zoom Reset button.
The MPI Timeline might appear to show everything as Uninstrumented, but
there is a hidden MPI_Send function.
-
There is always at least one transition where the label of a function
starts, so zoom in on the beginning of the Uninstrumented states on the right
side of the timeline until you see the hidden MPI_Send state.
Now suppose you want to go back and undo some of the filtering you have
done.
-
Click the filter Undo drop down button to reveal a list of applied filters.
This list lets you choose which filters to remove. It works like a stack:
if you select No filters applied, everything on top of it will be
taken off, which means there will be no filters applied. You should see
something like the following in the list of filters.
Timeline(Time(range)3398546253123,350489981242),Process(0,24) |
No filters applied |
Select the top filter from the Filter Undo drop-down list (Timeline(Time(range)...)
The timeline should now look similar to the following.
-
Reset the zoom to confirm that your original filter is still in effect.
The timeline should look similar to the following.
Using the MPI Chart Tab
Now you can explore the MPI Chart features with your filtered data. There are two types of data that you can view in chart form: Functions and Messages. In the following chart, we'll get an overview of which functions took the most time.
Click the MPI Chart tab to see a chart similar to the following.
The MPI Chart tab opens with a chart that shows the sum of the durations
of the functions as they ran in all the processes. The vertical colored scale,
to the right of the chart, shows a scale of 0.01 seconds to 2.92 seconds.
The MPI_Send and Application functions took almost no time
whereas the MPI_Recv function took the full 2.92 seconds.
-
Click on the bar for the MPI_Recv function.
The exact value of the bar is displayed in the MPI Chart Controls tab.
2.926024072 seconds were spent in this function across all process ranks.
-
Click near the Application and MPI_Send chart bars to
see their values.
In this particular application, every process waits until the token has
passed to every other process. As a result 2.92 seconds were spent in
MPI_Recv and only 0.03 seconds in Application, a state that
represents time between MPI functions. All processes are waiting an equal
amount, but any delays in the delivery of the token affects the whole
application.
Varying the MPI Chart Controls
This section shows how to use the MPI Chart controls in different ways to visualize the data. Depending on the program you are analyzing, some forms of charting are more useful than others. In this particular program, ring_c, we are focusing on message latency.
The following are the chart attributes you can set.
Data Type
- Functions - Plot data about the MPI functions used by the program
- Messages - Plot data about the MPI messages sent between process ranks
Chart
- Y Histogram - One dimensional chart with the data plotted on the vertical axis as a
function of another metric on the horizontal axis. You must select the type
of data to plot on the Y axis and the metric for the X axis.
X Histogram - One dimensional chart with the data plotted on the horizontal axis as a
function of time on the vertical axis. You must select the type of data to
plot on the X axis, and the metric for the Y axis.
- 2-D chart - Two dimensional chart with data plotted on both X and Y axis, making a
2-D matrix or scatter plot. You must specify what to plot on the X and Y
axis, and the metric.
X-Axis - Select the type of data to plot on the horizontal axis, for
X Histogram or 2-D Chart.
Y-Axis - Select the type of data to plot on the vertical axis, for
Y Histogram or 2-D Chart.
Metric - Select what is shown as a function of X and/or Y. The
metric value is indicated through color in the charts.
X-Axis, Y-Axis, and Metric options when looking at Functions:
- Time (range) - The range of times from entry to exit of a function
-
Entry Time - The time when a function is called
- Exit Time - The time when a function returns to the caller
- Duration - The time difference between function entry and function exit
- Process - The MPI global ranks in numerical order. Each function call has
a unique process rank associated with it.
- Function - The MPI function called.
- Send Bytes - Number of bytes sent in an MPI function call
- Receive Bytes - Number of bytes received in an MPI function call
- 1 (only for Metric) - Specifying 1 as the metric simply specifies an
attribute whose value is always 1. This can be used to count data records or
signal the presence or absence of data. For example, to count the number of
function calls for each function, set Y Axis: Function, Metric: 1, Operation:
Sum. To detect whether any function calls were made, set Operation: Maximum.
X-Axis, Y-Axis, and Metric options when looking at Messages:
Time (range) - The range of time from send to receive of a message
- Send Time - The time that a message was sent
- Receive Time - The time that a message was received
- Duration - The time difference between send and receive of a message
- Send Process - The process that sent a message
- Receive Process - The process that received a message
- Communicator - An arbitrarily defined ID that uniquely labels the
communicator (set of processes) used to send and receive the message
- Tag - The MPI tag used to identify the message
- Send Function - The function that sent the message
- Receive Function - The function that received the message
- Bytes - Number of bytes in the message
- 1 (only for Metric) - Specifying 1 as the metric simply specifies an
attribute whose value is always 1. This can be used to count data records or
signal the presence or absence of data. For example, to count the number of
function calls for functions that send a message, set Y Axis: Send Function,
Metric: 1, Operation: Sum. To detect whether any function calls were made for
each function that sends a message, set Operation: Maximum.
Operator: How multiple metric values are combined in the chart
-
Sum - calculates the sum of the selected Metric, which must be one of: Time,
Duration, Send/Receive Bytes, or "1"
- Maximum - calculates the maximum value of the selected Metric, which must
be one of: Time, Duration, Send/Receive Bytes, or "1"
- Minimum - calculates the minimum value of the selected Metric, which must
be one of: Time, Duration, Send/Receive Bytes, or "1"
- Average - calculates the average value of the selected Metric, which must
be one of: Time, Duration, Send/Receive Bytes, or "1"
- Fair - The Fair operator operates on any type of metric. When many metric
values are all assigned to the same chart bin, the Fair operator picks a
single one of those values "fairly". For example, let's say that 90% of the
MPI messages use the communicator MPI_COMM_WORLD, but 10% of them use a
user-defined communicator mycomm. If we form a message chart using
"Communicator" as the metric and "Fair" as the operator, the chart would
report MPI_COMM_WORLD 90% of the time but mycomm 10% of the time.
Make a chart that shows where messages are being sent
Create a chart to look at messages by making the following selections in
the MPI Chart Controls tab:
| Data Type: |
Messages |
| Chart: |
2-D Chart |
| X Axis: |
Send Process |
| Y Axis: |
Receive Process |
| Metric: |
Duration |
| Operator: |
Maximum |
Click Redraw to draw a new chart:
This chart shows that Process 0 sends only to Process 1. Process 1 only sends to Process 2, and so on. The color of each box is set by the metric selected (Duration) and the operator (Maximum). Since this graph's Data Type is Messages, this will be the sum of duration of the messages, or the length of message lines in the time dimension.
Interestingly, the color key shows the range of message durations is from 0.3 msec to 9.7 msec. The messages that took the longest to arrive were sent from P14 to P15.
Click on the square at Send Process = P14 and Receive Process = P15.
The details in MPI Chart Controls tab show that 7.675218 msec was spent sending messages from P14 to P15.
Make a chart to show which ranks waited longest to receive a message
Make the following selections in the MPI Chart Controls tab:
| Data Type: |
Messages |
| Chart: |
Y Histogram |
| Y Axis: |
Receive Process |
| Metric: |
Duration |
| Operator: |
Maximum |
Click Redraw to draw a new chart.
The chart above shows that the P15 rank waited the longest to receive a message, at 7.67 msecs.
Other processes with lengthy waits are P13, P7, and P5.
To show when and where these large delays occurred, select a 2-D chart
with time range on the X Axis:
| Data Type: |
Messages |
| Chart: |
2-D Chart |
| X Axis: |
Receive Time |
| Y Axis: |
Receive Process |
| Metric: |
Duration |
| Operator: |
Maximum |
Click Redraw.
For the P15 rank, click the red line and check the details in the right panel. You can see that the delay
occurred at 3.981063709 seconds.
To show a histogram for when these long duration messages occurred:
| Data Type: |
Messages |
| Chart: |
X Histogram |
| X Axis: |
Receive Time |
| Metric: |
Duration |
| Operator: |
Maximum |
Click Redraw.
The slowest message was received at 3.981 seconds.
Look for an effect of the slow messages on time spent in MPI functions
To see the effect of the long duration messages, we will create a graph
that shows duration of functions versus time.
- Make the following selections in the MPI Chart Controls tab and click
Redraw:
| Data Type: |
Functions |
| Chart: |
2-D Chart |
| X Axis: |
Exit Time |
| Y Axis: |
Duration |
| Metric: |
Duration |
| Operator: |
Maximum |
The resulting graph shows clear regions of functions of long duration,
especially for some functions ending at around t=3.995, which
last 20.69 seconds.
Isolate these long duration functions by dragging a box around them to
zoom:
Click the filter button.
The resulting image shows two dots near 3.995 and duration 20.7 msec.
-
Click the MPI Timeline tab.
You can now identify the high duration functions on the MPI Timeline.
They are the result of messages with slow delivery times.
This simple example showed the basics of how to examine the
relationships between MPI functions and messages.
|
|
