New gaming experiences with OpenGL ES and the Mobile Sensor API

Over the last couple of years, we have seen ever more advanced games for mobile devices. 3D graphics are no longer something you can only experience on the cutting edge devices, but are available on many mid-range mobile phones.

For game developers focused on Java ME, there are two APIs available for creating 3D graphics, the Mobile 3D API (JSR-184) and the OpenGL ES bindings for Java ME (JSR-239). The Mobile 3D API is basically the Java ME equivalent to the Java 3D API we know from Java SE, while JSR-239 is a Java implementation of the OpenGL ES standard managed by the Khronos Group (there is also a Java SE implementation of the standard OpenGL API called JOGL). It is my belief that the JSR-239 will be more successful in the long run, since it is a more widespread standard than JSR-184 and is also better prepared for the new 3D graphics chips. We have seen the same behavior with Java3D versus JOGL, where developers abandoned Java 3D since it was too difficult to extend, and make use of new graphics features.

3D graphics can make very cool games, but unless you can interact with the game using more sensitive controls than binary buttons, the gaming experience will be limited. A touchscreen would be one way to raise the gaming experience, but there is another solution that I believe is likely to be even more successful. Several mobile devices on the market today have a built in accelerometer, much like the one used in the controls for the Nintendo Wii. These can be used to detect motion and the current tilt of the device. Since the feedback from the accelerometer is near real-time, it provides a new way to interact with our devices.

We have just recently started to see the use of accelerometers in applications and games, and I believe that built-in accelerometers in mobile phones will become even more common as new mobile games appear. For Java ME, we already have an API for reading these accelerometers, the Mobile Sensor API (JSR-256). This article will describe a very simple game for a Sony Ericsson w910i that uses both the OpenGL ES API for Java ME and the Mobile Sensor API.

The game is a simple space game where you fly straight ahead and try to avoid colliding with incoming asteroids. You steer the ship by tilting the phone in the direction you wish to go. Since I'm no 3D artist, and severely color blind, the graphics are quite crude, but I hope that you'll all see the possibilities of using these APIs together anyway. All the code for the game is available as a download at https://mobilespaceracer.dev.java.net/. Feel free to experiment with this and use it as you wish.

The OpenGL ES is a standard managed by the Khronos Group, which also manages several other graphics and audio standards (OpenGL, OpenAL). OpenGL ES is based on the standard OpenGL API, so if you are already familiar with OpenGL, much of the code shown below will not be new to you. The Java ME implementation of OpenGL ES supports version 1.1, which is based on the standard OpenGL version 1.5. There is also an OpenGL ES version 2.0, but that version requires much more of the hardware and is currently not suitable for our mobile phones.

I will not go into the details of 3D programming; there are plenty of other resources available that will give you a good introduction to OpenGL and 3D programming in general. Here I will focus on the details of initiating the OpenGL ES on Java ME and how to configure it properly for your device.

JSR-239 is basically a direct implementation of the OpenGL ES API for the C programming language. There is also a package named java.nio which contains a partial Java ME implementation of the Java SE NIO API. This implementation is used to create native buffers that are used by the OpenGL ES for vertexes, normals, and textures.

To draw 3D with JSR-239 we use a GameCanvas and draw our 3D scene onto it. First, we need to initialize the graphics and configure things like color depth and other OpenGL-related properties. The following code will initialize the graphics using 5 bits for red, 6 for green and 5 for blue (a total of 16 bits color depth) and a depth buffer with 16 bits. The reason green gets one more bit than red and blue is that the human eye sees more shades of green.

private boolean initGraphics() {
    egl = (EGL11) EGLContext.getEGL();
    if (egl == null) {
        System.out.println("Error: could not initialize OpenGL ES");
        return false;
    }

    eglDisplay = egl.eglGetDisplay(EGL11.EGL_DEFAULT_DISPLAY);
    if (eglDisplay == null) {
        System.out.println("Error: could not open connection to display");
        return false;
    }

    int[] majorMinor = new int[2];
    if (!egl.eglInitialize(eglDisplay, majorMinor)) {
        System.out.println("Error: could not initialize OpenGL ES display");
        return false;
    }
    System.out.println("EGL version: " + majorMinor[0] + "." + majorMinor[1]);

    int[] numConfigs = new int[1];
    egl.eglGetConfigs(eglDisplay, null, 0, numConfigs);
    if (numConfigs[0] < 1) {
        System.out.println("Error: no configurations found");
        return false;
    }

    int configAttributes[] = {
        EGL11.EGL_RED_SIZE, 5, EGL11.EGL_GREEN_SIZE, 6, EGL11.EGL_BLUE_SIZE, 5,
        EGL11.EGL_ALPHA_SIZE, 0,
        EGL11.EGL_DEPTH_SIZE, 16,
        // EGL11.EGL_STENCIL_SIZE, EGL11.EGL_DONT_CARE,   // don't care about stencils
        EGL11.EGL_SURFACE_TYPE, EGL11.EGL_WINDOW_BIT,
        EGL11.EGL_NONE
    };

    EGLConfig eglConfigs[] = new EGLConfig[numConfigs[0]];
    if (!egl.eglChooseConfig(eglDisplay, configAttributes, eglConfigs,
            eglConfigs.length, numConfigs)) {
        System.out.println("Error: could not find a suitable configuration");
        return false;
    }
    EGLConfig eglConfig = eglConfigs[0];

    eglContext = egl.eglCreateContext(eglDisplay, eglConfig,
            EGL11.EGL_NO_CONTEXT, null);
    if (eglContext == null) {
        System.out.println("Error: could not create a rendering state");
        return false;
    }

    g2d = getGraphics();
    gl = (GL11) eglContext.getGL();
    if (gl == null) {
        System.out.println("Error: could not create a 3D graphics context");
        return false;
    }

    eglWinSurface = egl.eglCreateWindowSurface(eglDisplay, eglConfig, g2d, null);
    if (eglWinSurface == null) {
        System.out.println("Error: could not create a drawing surface window");
        return false;
    }

    if (!egl.eglMakeCurrent(eglDisplay, eglWinSurface, eglWinSurface, eglContext)) {
        System.out.println("Error: could not make the context current");
        return false;
    }

    return true;
}

This method must be called before we can start drawing our 3D scene or initialize any textures or other resources that require the use of JSR-239. To start drawing our 3D scene, we create a new thread with a loop that runs as fast as possible (no delays between each iteration). This way we get the maximum number of frames per second. The actual game logic is managed by the game loop (see below). Typically, when designing a 3D scene using OpenGL, we use a scene graph to structure all the 3D objects. This way we can construct a complex object of several smaller ones and move them as a single unit. The code below shows how we tell the JSR-239 to capture the GameCanvas and start drawing our scene. I've removed some of the code in this examples to show the important part related to rendering JSR-239 using Java.

private void renderScene() {
    egl.eglWaitNative(EGL11.EGL_CORE_NATIVE_ENGINE, g2d);

    // clear the scene, reset the transformations and set the lighting 
    ...

    // Start rendering the scene graph
    root.renderNode(gl);

    // Tell the device we have finished drawing 3D for this frame and release the device
    gl.glFinish();
    egl.eglWaitGL();

    // Draw ordinary 2D graphics using the Graphics2D object for the GameCanvas
    g2d.setColor(0x00aa00); 
    ...

    // Flush the graphics of the GameCanvas to display the new frame 
    flushGraphics();
}

The eglWaitNative method must be called before each rendering pass to capture the graphics of the device. We must also release the device once we're done and finally call flushGraphics to display the new frame. Using eglWaitNative and eglWaitGL we can combine ordinary Java ME 2D graphics with JSR-239. It is also possible to combine JSR-239 with other graphics APIs, such as the Mobile 3D API (JSR-184) or the Scalable Vector Graphics (JSR-226).

The Mobile Sensor API (JSR-256) is a general specification focused on communicating with the different sensors that are integrated on a Java ME device. It can be any type of sensor, like a thermometer, light sensor, or, as in this case, an accelerometer. An accelerometer measures accelerations, and the acceleration we are interested in for this game is the most common of all, gravity. Since gravity is constant, it is used to determine the tilt of our phone. In the Sony Ericsson w910i we have a built in three-axis accelerometer, which means that we can measure the acceleration in all three dimensions. If the phone was placed flat, the accelerometer would tell us that the acceleration along the z-axis (up and down) is about 1000 (this value represents 1G). The accelerations along the X and Y axises (sideways) would be about 0 since the phone is sitting still and gravity only works downwards. Flipping the phone over with the screen facing down, the accelerometer would give us the value of -1000 on the Z-axis. Standing on its side, would give us a value of 1000 or -1000 along either the X- or the Y-axis, depending on which side you put it. Putting the phone in a 45 degree angle along the X-axis would give us a value of ±707 on the Z-axis and ±707 on the Y-axis, since gravity cannot affect either axis with its full force (You can easily calculate what the value should be for a certain angle for each axis using the sine and cosine functions). Using the values from the X and Y-axis from the accelerometer, we can determine the position of the phone at any time, and then use that value to move our ship to avoid the incoming asteroids.

The API for JSR-256 is very simple. To get access to the accelerometer, you use the Generic Connection Framework to open a connection given a device-specific URL. You can also use a SensorManager to retrieve information about all available sensors on the device, including the URL for making a connection to it. In the example below, I've used a hard coded string for the URL, but if you wish to make a more generic implementation, the SensorManager should look for available accelerometers and use the one it can find, or give a warning to the user.

When you have a reference to a sensor (implemented as a SensorConnection) you can choose to either retrieve the sensor data by polling the device, or you can register a DataListener to receive updates automatically. My opinion is that the event-driven approach provides a better design and that is also the method I've used in the game. When registering a DataListener to a SensorConnection, you also provide the size of the buffer. This buffer indicates how many data-values should be collected before sending an event to the DataListener. Setting the buffer size to a low value would give you faster updates. However, using a large buffer size you can easier filter out freak values from the accelerometer by calculating the average.

The code below is an implementation of an adapter that is suitable for games. It automatically registers itself as a DataListener on the accelerometer and stores the values in three different integers. There is also a constant storing of the URL to the sensor. This URL is specific to the accelerometer in the Sony Ericsson phones. To make this more generic, it should query for all sensors and connect to the one that is an accelerometer (or display an error message to the user). The getAverage method should be used to determine the average of the collected values, so that we can filter out any freak values that might occur.

public class SensorAdapter implements DataListener {
   private int x, y, z;
   private static final String sensorURL = "sensor:acceleration;contextType=user;model=ST_LIS302DL;location=inside";
   
   public SensorAdapter() {
      x = 0; y = 0; z = 0;
      try {
         SensorConnection sensor = (SensorConnection) Connector.open(sensorURL);
         sensor.setDataListener(this, 10);
      } catch (IOException e) {
         e.printStackTrace();
      }
   }

   public int[] getXYZ() {
       synchronized(this) { return new int[] {x, y, z}; }       
   }
   
   public void dataReceived(SensorConnection sensor, Data[] data, boolean isDataLost) {
      synchronized(this) { 
         for(int i=0; i

The core of this game (and most other games) is the game loop that updates the 3D scene, checks for collisions, and keeps track of the status of all objects in the game. It is what drives the game. Our game loop consists of a thread that keeps running until the user decides to exit the game. Since our game is quite simple, it basically does the following things in this order:

1. Read the data from the sensor.
2. Update the ship position sideways.
3. Move the ship forward.
4. Check for collisions.
5. Update the score.

Once the game loop has completed these steps, we call Thread.sleep to create a delay before starting over. The delay will determine how fast the game runs. Setting it to high will make the game seem sluggish, and a low value will make it impossible to play.

When moving the ship forward we are actually moving, or translating in 3D terms, all the asteroids and leaving the ship in the same position. Since the positions of every object in our 3D scene is determined by a simple translation using three float values, one for each axis, we need to consider the fact that we might end up with a loss of precision if the values gets large enough. If this were to happen, the 3D object would become distorted and things would start looking quite strange. To avoid this we make sure that the “camera” and the ship are located near the origin, and when moving ahead we simply apply a negative translation to all the other objects in the 3D scene. This is actually the way most 3D games are implemented, regardless if it is a simple 3D game on a mobile device or a huge MMO game on your PC.

Collision detection can be very difficult, depending on how good and precise detection is needed. In this game I've used one of the simplest forms of collision detection, axis-aligned bounding box. We create a virtual box around every object and when testing two objects for collision, we use a simple check to see if the boxes intersect.

We now have a simple 3D game for Java ME, controlled using a built-in accelerometer. Feel free to download the source code for the game and make your own changes. Hopefully this can give you ideas for your own game or application, using either JSR-239 or JSR-256.

The use of the accelerometer in this game is very basic. We're not bothering about the values from the Z-axis at all. I'm certain that there are several other usages of the Mobile Sensor API that I haven't thought of and that are much more interesting than the one described in this article. One idea I'm working on is to use the phone as a game controller for my PC, sending the accelerometer data to my computer over Bluetooth. Could be useful to implement a tennis game. That would also give me a reason for doing more advanced 3D stuff using JOGL. I also believe that there are great opportunities besides games for the Mobile Sensor API. A cordless mouse that could be used during presentations is one idea that pops into my mind.

I hope this article has given you some ideas of what is possible, and that you come up with your own cool games or applications using either OpenGL ES or the Mobile Sensor API. Feel free to contact me if you have any questions or would like to share ideas you have.

• Online chapters for JSR-239 by Dr. Andrew Davison
• Official OpenGL ES site at the Khronos Group
• Good article on JSR-256 form Sony Ericsson Developer World
• JavaGaming.org, a very good community forum for games and Java
• Project website for the game

Erik Hellman has been working with Java for a little more than ten years. After working with Java EE on large telecom systems for Ericsson he switched to Java consultant and did several presentations at various conferences on a wide range of different topics. He is currently working as the team leader on the development of the Java SDK at Sony Ericsson in Lund, Sweden.