MDeBug: An MDB-Based Debugging Tool for the Solaris 8 and 9 OS

Print-friendly Version

MDeBug is a tool that can be used to perform system crash dump analysis for the Solaris 8 Operating System (OS) and above, and application core analysis for the Solaris 9 OS. This tool uses mdb (modular debugger).

The target audience for this tool is developers and system administrators who might be interested in gathering information from core files to debug a hang, a panic, or an application crash.

Here are links to the relevant files. (Note: You can save these without the .txt extension):

• mdebug_readme.txt -- The tool
• outfile_adv.txt -- Output from an advanced crash dump analysis
• outfile_appcore.txt -- Output from an application core dump analysis
• outfile_basic.txt -- Output from a sample basic crash dump analysis

Some of the salient features of this tool are that it:

• Has an option to do a basic or a more advanced analysis.
• Uses mdb (modular debugger), which is the latest kernel crash dump analysis utility (ed note: as of this writing in 2002).
• Can be run on a 64-bit OS to analyze a 32-bit crash dump file. (Please note that the reverse is not possible; one cannot debug a 64-bit crash dump while booted off 32-bit OS.)
• Uses only mdb as opposed to a mix of adb, crash [which is being EOL'd (end of life'd)], and direct commands run on the system on which the crash dump file was generated.
• Captures some very useful information such as prtconf, modinfo, /etc/system entries directly from the crash dump files.
• Provides an optional findleaks macro output, which can provide useful information to debug kernel memory leaks.
• Can perform an analysis on an application core file to print the thread stack and shared objects mapped.

MDeBug provides an option to run a basic or an advanced analysis for the kernel crash dump.

Basic analysis: Captures some of the basic information such as time of crash, panic string, panic message buffer, CPU structures, process table, and so on.

Advanced analysis: Captures all the information captured with basic analysis in addition to some advanced information such as /etc/system entries, prtconf, modinfo outputs, kernel memory allocation statistics, and so on. It will also collect the threadlist macro output (in case of a system hang) and optionally detect kernel memory leaks (if kmem flags are set to an appropriate value).

1. MDB packages:

The following packages should be present on the system on which you want to use this tool:

SUNWmdb -- Modular Debugger

SUNWmdbx -- Modular Debugger (64-bit)

You can verify this by typing the command:

%pkginfo | grep -i mdb

2. OS version:

For the best results, you should run this tool on the same system on which the crash dump files or application core were collected.

If this is not the case, the alternate system used must at least have the same OS version.

1. Copy the mdebug shell script to the location of your crash dump files or your application core.

2. Run the script as:

%sh ./mdebug

3. Choose the appropriate option in the startup menu.

The script output for both basic and advanced analysis is in the file by name:

outfile.`date |cut -f4 -d" "`

where `date |cut -f4 -d" "` gets replaced by the current time on the system.

In case of advanced crash dump analysis, the script creates two additional files:

threadlist.txt ==> In case the script detects that the crash dump files are from a hung system (forced crash dump).

memleak.txt ==> In case the script detects that kmem_flags is set to equal to or greater than 0xf, and if you answer "yes" to the question of whether you would want to run the findleaks macro.

For application core dump analysis:

outfile.appcore ==> This file contains the output of an application core dump analysis.

All the files are generated in the same directory as the crash dump or core file location (current working directory).

All errors are expected to be due to the utility in use, namely mdb. Hence, one needs to refer to the man pages of mdb(1) for more information on mdb.