appcert: A Static Application Checking Tool

Print-friendly Version

Introduction

As Solaris interfaces evolve and migrate from one release to the next, application breakage can plague developers who have come to depend on the stability of the underlying system. In the past, developers have had to adjust for binary incompatibility issues by modifing their programs to keep up with the evolution of the Solaris libraries. This was a difficult process and application testing was often an overwhelming task.

To alleviate the problems and challenges facing application developers, Sun has defined the Solaris Application Binary interface (ABI).

This article reviews appcert, the ABI utility that performs a static check of an application, and describes the technology behind appcert.

Defining the Application Runtime Interface

An application runtime interface definition is a way to identify the making of procedure calls to interfaces in Solaris shared libraries. To track these interfaces, Sun developed an application binary interface that keeps an inventory of all the libraries and symbols, and works to ensure that they are provided in an upward compatible manner.

appcert maintains a complete list of the interfaces in a flatfile database model for each successive release of the operating system, from Solaris 2.3 to Solaris 8. In Solaris 2.6, these definitions were embedded in the libraries themselves.

Once an application passes appcert, the application is guaranteed to run on newer Solaris versions.

appcert Technology

The appcert utility is a Perl script that examines ELF object files for Solaris symbol usage. All interfaces in Solaris are versioned to ensure upward compatibility, and labeled as public or private, depending on the shared object. Public interfaces are those that are intended for use by applications or middleware running on top of Solaris, while private interfaces are those only intended to be used by other Solaris interfaces.

Figure 1: Appecert Technology (Click image to enlarge)

The Three appcert Components

There are three main appcert components:

• Profiling
• Comparison
• Reporting

Profiling Component

In the profiling component, all bindings and associated symbols that are used directly by the application are extracted. appcert depends on the Solaris runtime linker to identify these dependencies, using the ldd command, which lists the path names of all shared objects that are loaded when the file is loaded. (Refer to the ldd man page for more usage information.)

appcert sets the environment variable LD_DEBUG to "files, bindings" to induce debugging output from the dynamic linker, which in turn can be parsed to find path names of the objects being examined and symbol bindings into Solaris shared libraries. This is an accurate binary footprint of the application since the runtime linker is used to resolve dependencies and make the bindings before the application is run. Another utility used during the profiling phase is the dump command (specifically dump -Lv), which dumps dynamic linking information the object depends on.

Comparison Component

In the checking (or comparison) component, the profiled objects are ready to be used and checked against the Solaris ABI. Thus, comparison is relative to any given release of Solaris, while profiling is not. The pvs utility can extract the versioning information from the libraries themselves, assuming they are versioned. Using the -dos option, pvs displays any internal versioning information contained within a Solaris shared library.

Although the most problematic cases of binary incompatibility result from the use of private symbols, other issues can arise from unstable practices. This includes static linking of Solaris shared object to the application at compilation time, since Solaris interfaces may change over time. As these interfaces evolve, the interactions between the shared object being statically linked and other Solaris libraries become at risk for binary incompatibility as well. Thus, private symbol usage by the library that is statically linked is reported as if it came from the application. This can cause confusion in the report since the private symbol usage did not come directly from the application code itself, but from interactions between two Solaris interfaces.

Therefore, in the checking phase, when static linking is noted in the profiles of the objects, the checking phase will report this as dangerous. In addition, heuristics are used for checking partial static linking by looking for particular symbols. Three of the most problematic cases are static linking of libc, libsocket, and libnsl.

Reporting Component

In the reporting component, the output from the checking section is read and additional output for the object is contained in an overall report. The evaluation contains a count of the number of private symbol usage and returns a formatted table detailing the number of private symbols and where the private symbol usage occurs. This aids application developers who are examining their product's release-to-release stability.

Obtaining appcert

The unbundled version of appcert can be downloaded. The site provides complete installation instructions. As of this publication, the current appcert version is 2.3.1.

Running appcert

Once installed, invoke appcert using the following command:

The appcert utility recursively searches to locate object files and checks for binary unstable practices. It must be run in the same environment in which the application being checked normally runs. Differences in the environment could result in false positives for potential binary incompatibility or unresolved references to interfaces in the Solaris libraries. For example, if the LD_LIBRARY_PATH is not set correctly, appcert will likely report additional unbound symbols.

Another good build practice to ensure accurate appcert reporting is to record dynamic library dependencies at compilation time using, for example, -z defs. Using -z defs allows the object to be self-contained and better defined, forcing a fatal error if any undefined symbol remains at the end of the link. This is the default when an executable is built.

Compatibility

The appcert utility cannot examine programs written in Java or C++. Since Java applications do not run directly on Solaris, but rather on the Java virtual machine, appcert is not applicable to Java programs. In addition, it cannot run on C++ applications since no ABI is currently defined for the C++ runtime interface (lib.so). Although there exists a standard for the C++ API, no standard exists for the C++ ABI. Note that appcert will check the C interface aspects of a C++ application (for example, calls into lib.so, libxII.so, etc).

Other objects that appcert cannot examine include:

• Partial or complete statically linked files
• Executable files that do not have execute permission set
• Object files that are setuid root
• Non-ELF file executables, such as shell scripts

Limitations of appcert include using dlopen(), rather than linking with -llib, which can cause unresolved symbol use. This is because the application is accessing a shared object that does not have its Solaris dependencies recorded. Since the dynamic linker is never invoked on the shared objects and there is no other dependency information, appcert cannot resolve symbol usage.

Navigating through the Working Directory

If no working directory is supplied on the command line with the -w option, then appcert will use /tmp/appcert.# (where # is the process ID of the appcert run) as the working directory. The working directory will contain the following files (this list excludes temporary files used by appcert during checking):

• Index - mapping between the checked binaries and the subdirectory in /tmp/appcert.#/objects/ where the binary output is located
• Report - copy of the roll-up report printed on stdout at the end of the appcert run
• Skipped - list of the files that were skipped and reasons why

In addition, there is per-object information in the subdirectories under /tmp/appcert.#/objects that includes:

• check.demoted_symbols - List of symbols suspected to be demoted (removed) Solaris symbols
• check.dynamic.private - List of private Solaris symbols to which the object makes direct bindings
• check.dynamic.public - List of public Solaris symbols to which the object makes direct bindings
• check.dynamic.unbound - List of symbols not bound by the dynamic linker when running ldd -r
• summary.dynamic - Printed summary of dynamic bindings for the examined objects

Sample C Program and Output

A sample C program and corresponding appcert output appears below. The sample program makes calls to a private symbol.

% more private.c 


/* 
* Copyright (c) 2000 by Sun Microsystems, Inc. 
* All right reserved. 
*/ 


#pragma ident "@(#)private.c 1.1 00/08/16 SMI" 


#include <stdio.h> 
#define IPV6_RTHDR_TYPE_0 0 


/* 
* main() 
* additional function calling to private symbol found in libsocket.so 
*/ 


size_t 
main() { 
size_t space_needed; 
space_needed = __inet6_rthdr_space(IPV6_RTHDR_TYPE_0, 0); 
return (space_needed); 
} 


% cc -o private_file private.c -lsocket 

Sample appcert Summary Report

% appcert private_file 
finding executables and shared libraries to check ... 
profiling: private_file 
determining list of Solaris libraries ... 
checking binary objects for unstable practices ... 
checking: private_file
----------------------------------------------------------
Summary: Potential binary stability problem(s) detected. 
A total of 1 binary objects were examined. 

The following (1 of 1) components have potential stability problems: 
private_file (1 private symbols) 

Summary of Private symbol use in /usr/lib/libsocket.so.11 binaries used 
libsocket.so.1, 1 of these used private symbols__inet6_rthdr_space 1 

Additional output regarding private symbols usage and other data is in the 
directory: 
/tmp/appcert.3937 
see the appcert documentation for more information. 

Interpreting appcert Output

The appcert output report lists the profiled and checked files that correspond to the phases in appcert that occurred during the investigation of the objects. A rollup summary appears at the end of the report indicating how many binaries were examined and how many reported potential binary stability problems occurred. The rollup report can also be found under $working_dir/Report.

Once the report is generated, all potential binary problems should be investigated as soon as possible. The recommended course of action for each specific output report error is listed below. If no potential binary problems occurred, the application is guaranteed to run on successive releases of Solaris with no binary compatibility problems.

Courses of Action:

Private Symbol Usage

If this symbol has not changed in the current release, then the application can still run. However, future use of this symbol is not guaranteed and could cause problems. If this private symbol has been changed or altered already, then use of this symbol should be eliminated immediately.

Static Linking

If this library contains a shared object counterpart, then dynamically link the counterpart instead. Otherwise, continue to use the static library until the shared object becomes available.

Demoted Symbol

Eliminate use of these symbols. Return of demoted symbols in a later release is possible, but there are still some releases on which the application will not run.

Unbound Symbols

Check environment variables such as LD_LIBRARY_PATH to make sure it is set correctly so all the binary objects can find all of the libraries they depend on. Refer to the man page for more information on how to scope the possible problems resulting in unbound symbols.

No Bindings Found

This could be due to any one of the limitations of appcert. If a result is expected, then check the environment or permissions setting.

Obsolete Library Usage

These symbols will not change and are frozen until the library is removed. Eliminate use of these symbols as soon as practical.

Use of sys_errlist[ ] or sys_nerr

This is a potential binary compatibility issue since it presents a risk in which reference may be made past the end of the sys_errlist array. Use strerror(3C) instead.

Use of Strong versus Weak Symbols

Strong symbols are reserved as private, so their semantics or behavior could change in the future. Reference the weak symbols instead.

Summary

The appcert utility is able to extract binding information and resolve dependencies in a systematic way to identify a wide range of problematic cases of binary incompatibility.

Additional Documentation

• The appcert man page provides information on additional uses of appcert.
• "The Solaris Linker and Libraries Guide"

Back to Top