Introduction
Part I of this series explained how the Solaris Operating Environment (OE) kernel provides a set of functions and data structures for device drivers and other kernel modules to export module-specific statistics to the outside world. This infrastructure, referred to as kstat, provides the Solaris software developer with:
- C-language functions for device drivers and other kernel modules to present statistics
- C-language functions for applications to retrieve statistics data from the Solaris OE without needing to directly read kernel memory
- Perl-based command-line program /usr/bin/kstat to access statistics data interactively or in shell scripts (introduced in the Solaris 8 OE)
This article will explain what tools are available to access kstat
information using shell scripts. Included are a few examples to introduce the
kstat(1m) program and the Perl language module it uses to extract kernel statistics.
The Solaris 8 OE introduced a new method to access kstat information from the command line or in custom-written scripts. You can use the command-line tool /usr/bin/kstat interactively to print all or selected kstat information from a system. This program is written in the Perl language, and you can use the Perl XS extension module to write your own custom Perl programs. Both facilities are documented in the pages of the online manual (see Resources).
The kstat Command
You can invoke the kstat command on the command line or within shell scripts to selectively extract kernel statistics. Like many other Solaris OE commands, it takes optional interval and count arguments for repetitive, periodic output. Its command options are quite flexible:
The first form follows standard UNIX command-line syntax, and the second form provides a way to pass some of the arguments as colon-separated fields. Both forms offer the same functionality. Each of the module, instance, name, or statistic specifiers may be a shell glob pattern or a Perl regular expression enclosed by "/" characters. It is possible to use both specifier types within a single operand. Leaving a specifier empty is equivalent to using the "*" glob pattern for that specifier. Running kstat with no arguments will print out nearly all kstat entries from the running kernel (most, but not all kstats of KSTAT_TYPE_RAW are decoded).
The tests specified by the options are logically ANDed, and all matching kstats
are selected. The argument for the -c, -i, -m, -n, and -s options may be
specified as a shell glob pattern, or a Perl regular expression enclosed in "/"
characters.
If you want to pass a regular expression containing shell meta-characters to the command, you must protect it from the shell by enclosing it with the appropriate quotation marks. For example, to show all kstats that have a statistics name
beginning with intr in the module name cpu_stat, you could use the script in
Example 1.
Example 1 - Example of Escaping Meta-Characters from the Shell
$ kstat -p -m cpu_stat -s 'intr*'
cpu_stat:0:cpu_stat0:intr 878951000
cpu_stat:0:cpu_stat0:intrblk 21604
cpu_stat:0:cpu_stat0:intrthread 668353070
cpu_stat:1:cpu_stat1:intr 211041358
cpu_stat:1:cpu_stat1:intrblk 280
cpu_stat:1:cpu_stat1:intrthread 209879640
The -p option used in the previous example displays output in a parseable format. If you do not specify this option, kstat produces output in a human-readable, tabular format. In Example 2, we leave out the -p flag and use the module:instance:name:statistic argument form and a Perl regular expression.
Example 2 - Default, Human-Readable Output Format
$ kstat cpu_stat:::/^intr/
module: cpu_stat instance: 0
name: cpu_stat0 class: misc
intr 879131909
intrblk 21608
intrthread 668490486
module: cpu_stat instance: 1
name: cpu_stat1 class: misc
intr 211084960
intrblk 280
intrthread 209923001
Sometimes you may just want to test for the existence of a kstat entry. You can use the -q flag, which just returns the appropriate exit status for matches against given criteria. The exit codes are as follows:
- 0: One or more statistics were matched.
- 1: No statistics were matched.
- 2: Invalid command-line options were specified.
- 3: A fatal error occurred.
Suppose that you have a Bourne shell script gathering network statistics, and you want to see if the NFS server is configured. You might create a script such as the one in Example 3.
Example 3 - Testing for kstat Existence Using the -q Flag
#!/bin/sh
# ... do some stuff
# Check for NFS server
kstat -q nfs::nfs_server:
if [ $? = 0 ]; then
echo "NFS Server configured"
else
echo "No NFS Server configured"
fi
# ... do some more stuff
exit 0
Real-World Example Using kstat and nawk
If you are adept at writing shell scripts using editing tools like sed or awk, here is a simple example to make a network statistics utility with kstats.
The /usr/bin/netstat command has a command-line option -I interface that allows you to print out statistics about a particular network interface. Optionally, netstat takes an interval argument to print out the statistics every interval seconds. Example 4 illustrates that option.
Example 4 - netstat Interface Statistics Output
$ netstat -I qfe0 5
input qfe0 output input (Total) output
packets errs packets errs colls packets errs packets errs colls
2971681 0 1920781 0 0 11198281 0 10147381 0 0
9 0 7 0 0 31 0 29 0 0
4 0 5 0 0 24 0 25 0 0
...
Unfortunately, this command only accepts one -I flag argument. What if you want to print out statistics about multiple interfaces simultaneously, similar to what iostat does for disks? You could devise a Bourne shell script using kstat and nawk to provide this functionality. You want your output to look like Example 5.
Example 5 - Desired Output from Statistics Script for Multiple Network Interfaces
$ netstatMulti.sh ge0 ge2 ge1 5
input output
packets errs packets errs colls
ge0 111702738 10 82259260 0 0
ge2 28475869 0 61288614 0 0
ge1 25542766 4 55587276 0 0
ge0 1638 0 1075 0 0
ge2 518 0 460 0 0
ge1 866 0 7688 0 0
...
Example 6 is the statistics script. Note that extracting the kstat information is very simple, and most of the work goes into parsing and formatting the output. The script uses kstat -q to check the user's arguments for valid interface names and then passes a list of formatted module:instance:name:statistic arguments to kstat before piping the output to nawk.
Example 6 - nawk Script to Print Statistics for Multiple Network Interfaces
#!/bin/sh
# netstatMulti.sh: print out netstat-like stats for
# multiple interfaces
# using /usr/bin/kstat and nawk
USAGE="$0: interface_name ... interval"
INTERFACES="" # args list for kstat
while [ $# -gt 1 ]
do
kstat -q -c net ::$1: # test for valid interface
# name
if [ $? != 0 ]; then
echo $USAGE
echo " Interface $1 not found"
exit 1
fi
INTERFACES="$INTERFACES ::$1:" # add to list
shift
done
interval=$1
# check interval arg for int
if [ X`echo $interval | tr -d [0-9]` != X"" ]; then
echo $USAGE
exit 1
fi
kstat -p $INTERFACES $interval | nawk '
function process_stat(STATNAME, VALUE) {
found = 0
for(i=1;i<=5;i++) {
if(STATNAME == FIELDS[i]) {
found = 1
break
}
}
if ( found == 0 ) return
kstat = sprintf("%s:%s", iface, STATNAME)
if(kstat in b_kstats) {
kstats[kstat] = VALUE - b_kstats[kstat]
} else {
b_kstats[kstat] = VALUE
kstats[kstat] = VALUE
}
}
function print_stats() {
printf("%-10s",iface)
for(i=1;i<=5;i++) {
kstat = sprintf("%s:%s",iface,FIELDS[i])
printf(FORMATS[i],kstats[kstat])
printf(" ")
}
print " "
}
BEGIN {
print " input output "
print " packets errs packets errs
colls"
split("ipackets,ierrors,opackets,oerrors,collisions",
FIELDS,",")
split("%-10u %-5u %-10u %-5u %-6u",FORMATS," ")
}
NF == 1 {
if(iface) {
print_stats()
}
split($0,t,":")
iface = t[3]
next
}
{
split($1,stat,":")
process_stat(stat[4], $2)
}
'
Using Perl to Access kstats
The previous example illustrates how simple it is to extract the information you need from the kernel; however, it also shows how tedious it can be to format the output in a shell script. Fortunately, the Perl extension module that
/usr/bin/kstat uses is documented so that you can write custom Perl programs. Because Perl is a "real programming language" and is ideally suited
for text formatting, you can write solutions that are quite robust and
comprehensive.
The Tied-Hash Interface to the kstat Facility
Access to kstats is made through a Perl extension in the XSUB interface module
called Sun::Solaris::Kstat. To access Solaris kernel statistics in a Perl program, you import the module with: use Sun::Solaris::Kstat;
The module contains two methods, new() and update(), correlating to the libkstat C functions kstat_open() and kstat_chain_update(). The module provides kstat
data through a tree of hashes based on a three-part key, consisting of the
module, instance, and name (ks_module, ks_instance, and ks_name are members of the C-language kstat struct). See Example 7 for a synopsis.
Example 7 - Synopsis of the Tied-Hash Interface to the Solaris kstat Facility
Sun::Solaris::Kstat->new();
Sun::Solaris::Kstat->update();
Sun::Solaris::Kstat->{module}{instance}{name}{statistic}
The lowest-level "statistic" member of the hierarchy is a tied hash implemented in the XSUB module and holds the following elements from struct kstat:
ks_crtime, creation time, which is presented as the statistic "crtime"ks_snaptime, time of last data snapshot, which is presented as the statistic "snaptime"ks_class, the kstat class, which is presented as the statistic "class"ks_data, kstat type-specific data decoded into individual statistics (the module produces one statistic per member of whatever structure is being decoded)
Because the module converts all kstat types, the programmer does not need to worry about the different data structures for named and raw types. Most of the Solaris OE's raw kstat entries are decoded by the module, giving you easy access to low-level data about things such as kernel memory allocation, swap, NFS performance, etc.
The update() Method
This method updates all of the statistics you have accessed so far and adds a
bit of functionality on top of the libkstat kstat_chain_update() function. If called in scalar context, it acts the same as kstat_chain_update(): It returns 0 if the kstat chain has not changed and 1 if it has. However, if called in list context, it returns references to two arrays. The first one holds the keys of any kstats that have been added since the call to new() or the last call to update(); the second holds a list of entries that have been deleted. The entries in the arrays are strings of the form module:instance:name. This is useful for implementing programs that cache state information about devices, such as disks, that you can dynamically add or remove from a running system.
Once you access a kstat, it will always be read by subsequent calls to update(). To stop it from being reread, you can clear the appropriate hash. For example:
$kstat->{$module}{$instance}{$name} = ();
64-Bit Values
At the time the kstat tied-hash interface was first released on the Solaris 8 OE, Perl 5 could not yet internally support 64-bit integers, so the kstat module approximates these values.
Timer Values
ks_crtime and ks_snaptime in struct kstat are of type hrtime_t, as are values of timer kstats and the
wtime, wlentime, wlastupdate, rtime, rlentime, and
rlastupdate fields of the kstat I/O statistics
structures. This is a C-type definition used for the Solaris high-resolution
timer, which is a 64-bit integer value. These are measured by the
kstat facility in nanoseconds, meaning that a 32-bit value would
represent approximately four seconds. The alternative is to store the values as
floating-point numbers, which offer approximately 53 bits of precision on
present hardware. You can store 64-bit intervals and timers as floating-point
values expressed in seconds, meaning that this module rounds up time-related
kstats to approximately microsecond resolution.
Counters
Because it is not useful to store these values as 32-bit values, and because
floating-point values offer 53 bits of precision, all 64-bit counters are also
stored as floating-point values.
Getting Started
As in our first example, Example 8 shows a Perl program that gives the same output as calling /usr/sbin/psrinfo without arguments.
Example 8 - Simple kstat Perl Script Emulating the Solaris psrinfo Command
#!/usr/bin/perl -w
# psrinfo.perl: emulate the Solaris psrinfo command
use strict;
use Sun::Solaris::Kstat;
my $kstat = Sun::Solaris::Kstat->new();
my $mh = $kstat->{cpu_info};
foreach my $cpu (keys(%$mh)) {
my ($state, $when) = @{$kstat->{cpu_info}{$cpu}
{"cpu_info".$cpu}}{qw(state state_begin)};
my ($sec,$min,$hour,$mday,$mon,$year) =
(localtime($when))[0..5];
printf("%d\t%-8s since %.2d/%.2d/%.2d %.2d:%.2d:%.2d\n",
$cpu,$state,$mon + 1,$mday,$year - 100,$hour,$min,$sec);
}
This produces the following output:
$ psrinfo.perl
0 on-line since 07/09/01 08:29:00
1 on-line since 07/09/01 08:29:07
The psrinfo command has a -v "verbose" option that prints much more detail about the processors in the system. The output looks like Example 9.
Example 9 - Verbose Output from the Solaris psrinfo Command
# psrinfo -v
Status of processor 0 as of: 08/17/01 16:52:44
Processor has been on-line since 08/14/01 16:27:56.
The sparcv9 processor operates at 400 MHz,
and has a sparcv9 floating point processor.
Status of processor 1 as of: 08/17/01 16:52:44
Processor has been on-line since 08/14/01 16:28:03.
The sparcv9 processor operates at 400 MHz,
and has a sparcv9 floating point processor.
All of the information in the psrinfo command is accessible through the kstat interface. As an exercise, you might want to modify the
simple psrinfo.perl example script to print out the verbose
information as in Example 9.
netstatMulti Implemented in Perl
The Perl script in Example 10 has the same function as our previous example using the kstat and nawk commands. Note that we have to implement our own search methods to find the kstat entries that we want to work with. Although this script is not shorter than our first example, it is certainly easier to extend with new functionality. Without much work, you could create a generic search method, similar to how /usr/bin/kstat works, and import it into any Perl scripts that need to access Solaris kernel statistics.
Example 10 - Perl Script to Print Statistics for Multiple Network Interfaces
#!/usr/bin/perl -w
# netstatMulti.perl: print out netstat-like stats for multiple interfaces
# using the kstat tied hash facility
use strict;
use Sun::Solaris::Kstat;
my $USAGE = "usage: $0 ... interval";
######
# Main
######
# process args
my $argc = scalar(@ARGV);
my @interfaces = ();
my $fmt = "%-10s %-10u %-10u %-10u %-10u %-10u\n";
if ($argc < 2) {
print "$USAGE\n";
exit 1;
} elsif ( !($ARGV[-1] =~ /^\d+$/) ) {
print "$USAGE\n";
print " interval must be an integer.\n";
exit 1;
}
# get kstat chain a la kstat_open()
my $kstat = Sun::Solaris::Kstat->new();
# Check for interfaces
foreach my $interface (@ARGV[-($argc)..-2]) {
my $iface;
if(! ($iface = interface_exists($interface)) ){
print "$USAGE\n";
print " interface $interface not found.\n";
exit 1;
}
push @interfaces, $iface;
}
my $interval = $ARGV[-1];
# print header
print " input output \n";
print " packets errs packets errs colls\n";
# loop forever printing stats
while(1) {
get_kstats();
print_kstats();
sleep($interval);
$kstat->update();
}
#############
# Subroutines
#############
# search for the first kstat with given name
sub interface_exists() {
my ($name) = @_;
my ($mod, $inst) = $name =~ /^(\D+)(\d+)$/;
return(exists($kstat->{$mod}{$inst}{$name})
? { module => $mod, instance => $inst, name => $name }
: undef);
}
# get kstats for given interface
sub get_kstats() {
my (@statnames) = ('ipackets','ierrors','opackets',
'oerrors','collisions');
my ($m, $i, $n);
foreach my $interface (@interfaces) {
$m = $interface->{module};
$i = $interface->{instance};
$n = $interface->{name};
foreach my $statname (@statnames) {
my $stat = $kstat->{$m}{$i}{$n}{$statname};
die "kstat not found: $m:$i:$n:$statname" unless defined $stat;
my $begin_stat = "b_" . $statname; # name of first sample
if(exists $interface->{$begin_stat}) {
$interface->{$statname} = $stat -
$interface->{$begin_stat};
}else { # save first sample to calculate deltas
$interface->{$statname} = $stat;
$interface->{$begin_stat} = $stat;
}
}
}
}
# print out formatted information a la netstat
sub print_kstats() {
foreach my $i (@interfaces) {
printf($fmt,$i->{name},$i->{ipackets},$i->{ierrors},
$i->{opackets},$i->{oerrors},$i->{collisions});
}
}
In the subroutine interface_exists(), you cache the members of the key if an entry is found. This way, you do not have to do another search in get_kstats(). You could fairly easily modify the script to display all network interfaces on the system (rather than take command-line arguments) and use the update() method to discover if interfaces are added or removed from the system (with ifconfig, for example). This exercise is left up to you.
Resources
About the Author
Peter Boothby is a staff engineer with Sun Microsystems' Market Development Engineering group. He works with independent software vendors to make sure their applications run well on Sun systems. He has 18 years of industry experience developing, performance-tuning, benchmarking, and troubleshooting commercial computer applications.
December 2001
| 
Print-friendly Version
