Testing Wireless Java Applications

Print-friendly Version

Wireless applications written in the Java programming language (wireless Java applications), like all other types of software, must be tested to ensure functionality and usability under all working conditions. Testing is even more important in the wireless world because working conditions vary a lot more than they do for most software. For example, wireless Java applications are developed on high-end desktop machines but deployed on handheld wireless devices with very different characteristics.

The aim of this article is to help you test your wireless applications. The article:

• Provides an overview of software testing
• Describes the challenges in testing wireless applications
• Presents a tutorial on testing wireless applications
• Furnishes testing checklists for user interface, networking, and other areas
• Discusses certification programs for applications targeted at the Java 2 Platform, Micro Edition (J2ME applications)

Overview of Software Testing

Software testing is a systemic process to find differences between the expected behavior of the system specified in the software requirements document and its observed behavior. In other words, it is an activity for finding errors in the software system. There is no one agreed-upon goal of software testing. One school of thought describes the goal of testing as demonstrating that errors are not present. Dijkstra (1930 - 2002) describes the goal of testing as showing the presence of faults but not their absence. The ultimate goal, however, is to find errors and fix them so users can be confident that they can depend on the software.

Errors (also known as bugs or glitches) in software are generally introduced by people involved in software development (including analysts, architects, designers, programmers, and the testers themselves). Examples of errors include:

• Interface specification: Mismatch between requirements and implementation.
• Algorithmic faults: Missing initialization, branching errors, or missing tests for null.
• Mechanical faults: The user manual doesn't match actual conditions or operating procedures.
• Omissions: Some of the features described in the requirements documents are not implemented.

Many developers view the subject of software testing as "not fashionable," and as a result too few of them really understand the job software testers do. Testing is an iterative process and should start from the beginning of the project. Software developers need to get used to the idea of designing software with testing in mind. Some of the new software development methodologies such as eXtreme Programming stress incremental development and testing. eXtreme Programming is ideally suited for some types of applications, depending on their size, scope, and nature. User interface design, for example, benefits highly from rapid prototyping and testing usability with actual users.

One way to make testing simple is to design applications with testing in mind. Organizing the system in a certain way can make it much easier to test. Another implication is that the system must have enough functionality and enough output information to distinguish among the system's different functional features. It is now common to describe a system's functional requirements (features that the system must provide) by using the Unified Modeling Language (UML) to create a use case diagram, then detailing the use cases in a consistent written form. Documenting the various uses of the system in this way simplifies the task of testing the system by allowing the tester to generate test scenarios from the use cases. The scenarios represent all expected paths users will traverse when they use the features that the system must provide. Developers distinguish these functional requirements from system requirements not related to particular functionality, constraints related to performance, configuration, and usability.

Testing Activities

The testing that needs to be performed can be split into two classes: functional (black-box) testing and structural (white-box) testing. In black-box testing, each of the components -- and ultimately the system as a whole -- is treated as a black box, and testers verify that it supports all the features identified (often as use cases) in the requirements documents. Black-box testing activities include:

• Unit (or Class) Testing: In this testing activity, components are tested separately. Because some objects may depend on other objects that are not yet available, you may need to develop test drivers and test stubs. A test driver simulates the part of the system that calls the component under test. A test stub simulates a component called by the tested component.
• Integration Testing: In this activity, objects are integrated in increasingly large and complex subsystems. This is an incremental testing process.
• System Testing: In this activity, the system is tested as a whole. Testers employ various techniques at this stage, including functional testing (testing actual behavior against documented requirements), performance testing (testing nonfunctional requirements), and acceptance and installation testing (testing against the project agreement).

Black-box testing concerns itself with externally visible behavior, and ignores the source code.  In white-box testing, the focus is on the code that produces the behavior.  One common white-box testing activity is path testing, also known as code coverage. Its goal is to identify faults in the implementation by exercising all possible paths through the code at least once. Testers check that every branch in the code has a test that exercises that branch.

		Note: The starting point of path testing is a flow graph consisting of nodes representing executable blocks, and associations (or edges) representing flow of control. The minimum number of tests necessary to cover all edges is equal to the number of independent paths through the flow graph. This is known as the cyclomatic complexity (CC) of the flow graph, which has the formula: CC = number of edges - number of nodes + 2

Fortunately, you don't have to draw flow graphs for your code by hand, as several code coverage tools are readily available.

Challenges of Testing Wireless Applications

The wide variety of Java technology-enabled devices such as wireless phones and PDAs results in each device running a different implementation of the CLDC and MIDP. Varying display sizes add to the complexity of the testing process. In addition, some vendors provide proprietary API extensions. As an example, some J2ME vendors may support only the HTTP protocol, which the MIDP 1.0 specification requires, while others support TCP sockets and UDP datagrams, which are optional.

To make your application both portable and easy to test, design it using standardized APIs defined through the Java Community Process (JCP), so it will run as-is on devices with different J2ME implementations. If you feel you must use vendor-specific extensions, design your application in such a way that it defaults to the standard APIs if it's deployed on a device that doesn't support the extensions.

Testing Wireless Java Applications

The testing activities described above are applicable to testing wireless Java applications. In other words, you perform unit or class testing, then you integrate components and test them together, and eventually you test the whole system. In this section I provide guidelines for testing wireless applications.

Validating the Implementation

Ensuring that the application does what it's supposed to is an iterative process that you must go through during the implementation phase of the project. Part of the validation process can be done in an emulation environment such as the J2ME Wireless Toolkit, which provides several phone skins and standard input mechanisms. The toolkit's emulation environment does not support all devices and platform extensions, but it allows you to make sure that the application looks appealing and offers a user-friendly interface on a wide range of devices. Once the application has been tested on an emulator, you can move on to the next step and test it on a real device, and in a live network.

Usability Testing

In usability testing (or GUI navigation), focus on the external interface and the relationships among the screens of the application. As an example, consider an email application that supports entry and validation of a user name and password, enables the user to read, compose, and send messages, and allows maintenance of related settings, using the screens shown in Figure 1, among others.

Figure 1: Messaging Application

In this example, start the test at the Login window. Enter a user name and a password and press the soft button labeled Login. Enter a valid user name and password.  The application should display the main menu. Does it? The main menu should display a SignOut button. Does it? Press the SignOut button. Does the application return to the Login screen? Write yourself a note to raise the question, "Why does the user 'log' in but 'sign' out?" Now enter an invalid user name or incorrect password. The program should display a meaningful message box with an OK button. Does it? Press the OK button. Does the application return to the Login screen?

You need to test the GUI navigation of the entire system, making notes about usability along the way. If, for example, the user must traverse several screens to perform a function that's likely to be very popular, you may wish to consider moving that particular function up the screen layers.

Some of the questions you should ask during usability testing include:

• Is the navigation depth (the number of screens the user must go through) appropriate for each particular function?
• Does the application minimize text entry -- painful on a wireless phone -- or should it provide more selection menus?
• Can screens of all supported devices display the content without truncating it?
• If you expect to deploy the application on foreign devices, does it support international character sets?

The MIDP Style Guide provides helpful hints about user interface design.

Network Performance Testing

The goal of the next type of testing is to verify that the application performs well in the hardest of conditions (for example, when the battery is low or the phone is passing through a tunnel). Testing performance in an emulated wireless network is very important. The problem with testing in a live wireless network is that so many factors affect the performance of the network itself that you can't repeat the exact test scenarios. In an emulated network environment, it is easy to record the result of a test and repeat it later, after you have modified the application, to verify that the performance of the application has improved.

Server-Side Testing

It is very likely that your wireless Java applications will communicate with server-side applications. If your application communicates with servers you control, you have a free hand to test both ends of the application. If it communicates with servers beyond your control (such as quotes.yahoo.com), you just need to find the prerequisites of use and make the best of them. You can test server-side applications that communicate over HTTP connections using HttpUnit (a Java API for accessing web sites without a browser. It is ideally suited for automated unit testing of web sites when combined with a Java unit test framework such as JUnit, which I'll discuss in the next section. You can also measure a web site's performance using httperf, a tool designed for measuring the performance of web servers).

Automating Unit Testing

One of the unit-testing tools most widely used by Java developers is the JUnit framework. A similar framework for J2ME unit testing is J2MEUnit. Here is a sample test:

import j2meunit.framework.*; public class TestLoginWindow extends TestCase { public TestLoginWindow() { super("null"); } public TestLoginWindow(String s) { super(s); } protected void runTest() throws java.lang.Throwable { if(getTestMethodName().equals("testLoginWindow")) testLogindWindow(); } public Test suite() { return new TestSuite(new TestLoginWindow().getClass(), new String[] {"testLoginWindow"}); } public void testLoginWindow(){ // test it // use assert(String, boolean) } } }

When using J2MEUnit in your testing, you need to:

• Create a subclass of TestCase, like TestLoginWindow in the example.
• Override the method runTest() as in the example. Because J2MEUnit doesn't use reflection, when you override runTest() you must call getTestMethodName() to check the name of the test method (testLoginWindow() in the example above).
• Override the method suite() so that it returns a TestSuite object containing the name of the class for the test case and the names of the test methods.
• To check a value, call the assert() method and pass a boolean that is true if the test succeeds.

J2MEUnit provides two test runners that allow you to run your tests, collect results, and display them:

1. Text-based: This test runner provides simple-text based output of the results. To use it, add the following main method to any subclass of TestCase:

   public static void main(String argv[]) { String[] {"j2meunit.examples.TestOne"}; j2meunit.textui.TestRunner.main(runnerArgs); }

   
   

2. GUI-based: As the name implies, this test runner provides GUI output. To use it, you must compile and preverify the J2MEUnit framework, then:
   • Create a subclass of j2meunit.midp.ui.TestRunner.
   • Override the startApp() method.

Debugging Information

Adding debugging information in your code is very important. You can display trace points, values of variables, and other information during testing and debugging. One way to minimize the tedium of writing System.out.println() calls is to write a utility method such as the following:

public void debug(String s) { System.out.println("DEBUG: "+s); }

You can easily use the debug() method to display debugging information, then later remove the calls from production code.

The J2ME Wireless Toolkit provides a debugger that's easy to use. If you use the Sun ONE Studio 4, Mobile Edition, see the article Debugging Wireless Applications with Mobile Edition for useful guidance.

Testing Checklists

This section provides checklists you will find useful when testing your application, in both emulation and live environments. These checklists include tests that are usually performed in the Motorola Application Certification Program, described a little later in the article.

Navigation Checklist

• Application name: Make sure your application displays a name in the title bar.
• Keep the user informed: If your application doesn't start up within a few seconds, it should alert the user. For large applications, it is a good idea to have a progress bar.
• Readable text: Ensure that all kinds of content are readable on both grayscale and color devices. Also make sure the text doesn't contain any misspelled words.
• Repainting screens: Verify that screens are properly painted and that the application doesn't cause unnecessary screen repaints.
• Soft buttons: Verify that the functionality of soft buttons is consistent throughout the application. Verify that the whole layout of screens and buttons is consistent.
• Screen Navigation: Verify that the most commonly used screens are easily accessible.
• Portability: Verify that the application will have the same friendly user interface on all devices it is likely to be deployed on.

Network Checklist

• Sending/Receiving data: For network-aware applications, verify that the application sends and receives data properly.
• Name resolution: Ensure that the application resolves IP addresses correctly, and sends and receives data properly.
• Sensitive Data: When transmitting sensitive data over the network, verify that the data is being masked or encrypted. Use the SSL protocol.
• Error handling: Make sure that error messages concerning network error conditions (such as no network coverage) are displayed properly, and that when an error message box is dismissed the application regains control.
• Interruptions: Verify that, when the device receives system alerts, SMS messages, and so on while the application is running, messages are properly displayed. Also make sure that when the message box is dismissed the application continues to function properly.

Other Important Issues

• Successful startup and exit: Verify that your application starts up properly and its entry point is consistent. Also make sure that the application exits properly.
• Classes outside the MIDP and CLDC specifications: Unless you are willing to sacrifice portability and, in some environments, certification, ensure that the application does not use classes not included in the MIDP and CLDC specifications.
• User manual: Verify that all product documentation is accurate, and consistent with the software's actual behavior.

Certification Programs

Several certification programs for J2ME applications are available. The two most widely known at this writing are those sponsored by Motorola and Nokia.

Motorola

The Motorola Application Certification Program is designed to promote consistent user interface and high standards of operation in J2ME wireless applications.

The benefits of the certification program cited by Motorola are:

• Valuable testing and feedback results to the developer
• Higher application quality
• Improved customer experience
• Access to marketing and promotion programs

Note that this certification program is not free (check the pricing information); but it is a good way to form a partnership with an important vendor.

To get Motorola to certify your J2ME application, you must schedule a test by submitting a package -- a MIDlet suite or a single MIDlet -- online at http://www.qpqa.com/motorola/iden. To do so:

1. Fill out an online submission form
2. FTP or email a pre-tested candidate of your application
3. Supply a copy of your documentation in either .pdf or .html

For more detailed information on the certification process and the fees involved, check out the Motorola Certification Application Program.

If your J2ME application meets all the certification requirements, Motorola will provide you with the "Motorola Compatible" logo, and will place your application and user manual on Motorola's application distribution server. If your J2ME application does not achieve certification the first time you submit it, you can fix the problems and resubmit (for an extra charge).

Nokia

The Nokia OK Program is a testing and evaluation process for third-party applications that run on Nokia mobile devices. Once an application completes the process, it is granted the Nokia OK designation, meaning that the application has met the Nokia standards and can be used on Nokia products with confidence.

The benefits of the program according to Nokia include:

• Allows end users to identify third-party applications available for Nokia mobile devices
• Raises your application's profile with the Nokia OK logo
• Provides easy access to Nokia sales channels
• Makes your application visible on Nokia Mobile City, a source of information for operators and consumers that includes third-party solutions that Nokia sees as adding value to its products

The certification process is similar to Motorola's. It is less expensive, but not free. For more detailed information, visit the Nokia OK Program.

Conclusion

Testing is a systematic approach to finding errors in programs. It is a very challenging process in the wireless application market because an application is developed on one platform and deployed on ones that are vastly different. The process is complicated even further by the availability of a wide range of devices that implement different versions of the KVM, CLDC, and MIDP, and that provide extension APIs not available on all devices. To achieve the maximum portability, verify that your applications use the standard APIs.

This article also provided guidelines and checklists to help you test your wireless Java applications.

For more information

• J2ME Wireless Toolkit
• Wireless Software Design Techniques
• MIDP Style Guide
• iDEN Update
• Motorola Application Certification Program
• Nextel Developer Program
• Nokia OK Program
• JUnit
• HttpUnit
• httperf
• J2MEUnit

Acknowledgments

Special thanks to Gary Adams of Sun Microsystems for his contributions to this article.