Getting Started with Data Synchronization Using SyncML

Print-friendly Version

By Qusay H. Mahmoud
September 2004

The ability of applications running on handheld devices, on desktop computers, and in networks to update a single body of information, and to coordinate the updates intelligently, is key to the popularity of the paradigm of ubiquitous computing. The process of making two or more sets of data identical is called data synchronization.

Today, synchronization is supported by a variety of technologies. Database applications such as IBM's DB2 Everyplace and Oracle's Oracle Lite have their own proprietary syncing technologies. To synchronize personal information on your handheld device and your PC you probably use the proprietary technology that comes with the device – and use a different technology to synchronize data on your phone with data on your laptop. Each technology can synchronize one or a few applications, and is limited to a particular type of device or network connectivity. Users waste huge amounts of time and money configuring and installing all these different applications.

Given the large and growing diversity of applications and devices we use today, we need a standard synchronization technology. The SyncML initiative is such a standard, which promises to enable users to buy devices that synchronize with a wide range of data and devices, and to reduce the effort and costs expended by device manufacturers, service providers, and application developers as well.

This article discusses the foundations of SyncML and how it works, so you can understand the SyncML standard, get familiar with the tools available, and start developing SyncML-compatible mobile applications for the Java 2 Platform, Micro Edition (J2ME). The sample code in this article will be most useful to you if you need to synchronize data on mobile devices and on servers.

Change Once, Update Everywhere

SyncML is an open standard that drives data mobility by establishing a common language for communications among devices, applications, and networks. The goal is to enable smooth, efficient synchronization of remote data and personal information across devices, platforms, and multiple networks, both fixed and mobile. This universal language enables devices and applications to synchronize data like email, contact information, calendar, and to-do lists over the network, so that information is consistent, up to date, and accessible, no matter where it's stored: on a mobile phone, a PDA, a laptop, a  PC, or a server. As just one example, a user would be able to read and write email either on a desktop or on a mobile phone, and still maintain consistency as to which messages have been read and replied to.

The SyncML initiative, recently consolidated into the Open Mobile Alliance (OMA), is supported by all the major players, including Ericsson, Nokia, IBM, Motorola, and Symbian. OMA includes working groups that focus on topics outside the scope of this article, as well as on synchronization. For instance, the Device Management Working Group is developing a set of functional requirements for device management, to facilitate over-the-air administration of devices and applications, so as to simplify configuration, updates, and support.

The SyncML specifications were designed to support two primary goals:

• To synchronize networked data with data on any mobile device


• To synchronize data on a mobile device with any networked data

To accomplish these objectives, the data synchronization protocol must have certain characteristics; each can be seen as a benefit of the SyncML standard:

• Operates effectively over wireless and wired networks: Wireless networks in particular suffer from high network latency, limited bandwidth, relatively high packet costs, and low reliability of data and connectivity. To handle these concerns, SyncML features a robust synchronization protocol, WAP Binary XML (WBXML) encoding of data and synchronization commands, and use of a single request-response pair of messages, where the request contains all updates and the response provides the updates with any conflicts already identified and possibly resolved. In addition, the synchronization process is designed to survive inadvertent disconnection, ensuring that the device and networked data stay consistent and progress resumes when the connection is reestablished.


• Supports a variety of network transport protocols: SyncML supports familiar transport protocols, such as HTTP, WAP's Wireless Session Protocol (WSP), Bluetooth's OBEX and IrDA, and SMTP.


• Supports arbitrary network data: SyncML doesn't mandate how data must be represented on the device or within the network repository. It only describes how data formats are represented over the network. Supported formats include:
  
  
  • Common personal data formats such as vCard for personal information
  
  
  • Email and network news
  
  
  • Relational data
  
  
  • XML and HTML documents
  
  
  • Binary data


• Supports data access from a variety of applications: SyncML makes no assumptions about the programming language, and thus is language-independent.


• Addresses the resource limitations of mobile devices: To reduce memory requirements, data exchanged is encoded in binary format.

Potential Problems

As any protocol does, SyncML faces problems that may affect developers and end users. These concern compatibility of clients and servers. For example, a mobile phone that supports SyncML and vCard may have a calendar application whose capabilities differ from those of vCard on the server; For instance, vCard may support an event that recurs every third Thursday except for the 21st of each month, while the phone's calendar application does not. Data compatibility is limited to commonly specified data formats and the applications that support them. Developers of SyncML-based applications should keep such pitfalls in mind, and consider interoperability issues and compatibility testing.

Who Benefits From SyncML?

SyncML represents a common synchronization standard that benefits all kinds of players in the mobility industry:

• Device manufacturers: Because storage on mobile devices is limited, manufacturers can support only one synchronization standard. Adopting a common standard enables the device to interoperate with a wide range of applications, services, and network transmission technologies.


• Service providers: If multiple synchronization technologies are widely  used, service providers must install and configure multiple server infrastructures, one for each synchronization technology, and must update and maintain each to preserve compatibility and performance. Using a common synchronization standard reduces maintenance effort and operating costs.


• Application developers: Developers who adopt SyncML can create applications that connect to broad varieties of devices and networked data. Applications that don't need to support multiple synchronization technologies are less complex and less costly to maintain. Deployment is easier, too – a feature likely to appeal to service providers deciding which of similar applications to offer.


• End users: If platforms and applications use a standard synchronization technology, users will be able to synchronize a wide range of data and devices without spending all those hours configuring multiple, incompatible connections.

The Genesis of SyncML

The SyncML standard for data synchronization consists of two protocols: the SyncML representation protocol and the SyncML synchronization protocol. The former defines the representation format of SyncML messages in XML, and the latter defines the actions between a SyncML client and a SyncML server. Together, they ensure a consistent set of data is available on any device or application at any time.

A SyncML message is represented as an XML document. It must be a well-formed XML document but not necessarily a valid one. For example, a document is considered well-formed if it consists only of the XML body, but is considered well-formed and valid only if it includes the XML header as well. The header, if present, specifies routing and versioning information, and the body contains one or more SyncML commands, defined by a set of request commands and a set of response commands. A small set of common data formats is also identified, and therefore a SyncML document can be interpreted as a Multipurpose Internet Mail Extension (MIME) content type. Indeed, a new MIME media type has been registered: application/vnd.syncml.

Features and requirements defined by the SyncML synchronization protocol include:

• Change logs: Both the client and the server are required to maintain information about changes to the data in their databases. The SyncML specification doesn't specify the format of the change log, but each device must be able to specify the data items that have changed since the previous synchronization.


• Sync anchors: A sync anchor is a string representing a synchronization event. There are two, Last and Next, sent in the initialization stage of the synchronization process. The Last sync anchor describes the last event when the database was synchronized, and the Next sync anchor describes the current sync event, both from the sending device's point of view. The receiving device must store each Next sync anchor it receives until the next synchronization. At that time, the receiving device compares these stored Next sync anchors with the sending device's Last sync anchor. If the Last and Next sync anchors match, the receiving device concludes that no failures have occurred since the last sync; if they don't match, the device can request a special action from another device, such as a slow sync, in which all items in the client databases are compared w.


• Mapping of data items: The client and the server may each have their own unique identifier for each data item in their databases. The client ID for an item is known as the Locally Unique ID (LUID), and the server ID is known as the Globally Unique ID (GUID). Because these IDs can differ, the server must maintain an ID mapping table. For example, for an item called "Car" the client's LUID may be 45, while the server's GUID may be 101451. The server's two-column ID mapping table would show that LUID = 45 is mapped to GUID = 101451. Note that LUIDs are always assigned by the client; if the server sends a new item to the client, the client assigns it an LUID and sends that local ID back to the server for mapping to a GUID.


• Conflict resolution: Conflicts arise when changes to the same item occur on both the client and server databases. These are generally resolved by a sync engine on the server device. The protocol provides the functionality to notify the client of a resolved conflict, supplying one of the status codes for common resolution policies. Therefore, if the sync engine on the server resolves a conflict, it can send information about the conflict and how it was resolved, using the Status element as in the following example:

  Sample Code: Server Notifies Client of Conflict Resolution

  <Status> <MsgRef>1</MsgRef> <CmdRef>2</CmdRef> <SourceRef>2121</SourceRef> <Data>208</Data> <!-- conflict; resolved with data merge --> </Status>

  
  

• Security: The synchronization protocol requires support for basic authentication, and for verification of data integrity using a message digest created with MD5, an algorithm standardized by the Internet Engineering Task Force (IETF). User authentication is defined for three levels: the server, the database, and the object. Here is a sample basic authentication between the client and server; note that the client sends the credentials with the request:

  Sample Code: Client Sends Request, With Credentials

  <SyncML> <SyncHdr> <VerDTD>1.0</VerDTD> <VerProto>SyncML/1.0</VerProto> <SessionID>1</SessionID> <MsgID>2</MsgID> <Target><LocURI>http://www.syncml.org/sync-server</LocURI></Target> <Source><LocURI>IMEI:493005100592800</LocURI></Source> <Cred> <Meta><Type xmlns='syncml:metinf'>syncml:auth-basic</Type></Meta> <Data>QnJ1Y2UyOk9oQmVoYXZl</Data> <!—base64 formatting of "userid:password"--> </Cred> </SyncHdr> <SyncBody> ... </SyncBody> </SyncML>

  
  

  Sample Code: Server Replies to Client's Request

  <SyncML> <SyncHdr> <VerDTD>1.0</VerDTD> <VerProto>SyncML/1.0</VerProto> <SessionID>1</SessionID> <MsgID>2</MsgID> <Target><LocURI>IMEI:493005100592800</LocURI></Target> <Source><LocURI>http://www.syncml.org/sync-server</LocURI></Source> </SyncHdr> <SyncBody> <Status> <MsgRef>1</MsgRef><CmdRef>0</CmdRef><Cmd>SyncHdr</Cmd> <TargetRef>http://www.syncml.org/sync-server</TargetRef> <SourceRef>IMEI:493005100592800</SourceRef> <Data>212</Data> <!--Authenticated for session--> </Status> ... </SyncBody> </SyncML>

  
  

• Addressing: Clients use a URI-based scheme, and the LocURI tag, for addresses of devices, services, databases, and data items.


• Device capabilities: The client and the server can exchange information about themselves and their capabilities in an initialization step. Either may request this exchange. Two types of information are exchanged: device information, such as type, model, and manufacturer, and service information, including representations of the capabilities of the data objects that the client or server supports. Suppose, for example, the server indicates that it supports content in vCard 3.0 format and the client indicates it doesn't; notified of this difference, the server can refrain from sending  vCard 3.0 properties to this client.

Sync Initialization

The synchronization process begins with an initialization stage. The client informs the server which database it wants to synchronize, and which type of synchronization is needed – I'll describe the standard types in the next section. In this request, the client can also include authentication and service capabilities information. What follows is a sample sync initialization package from the client that has been adapted from the SyncML specification. Note the following:

• The databases to be synchronized are indicated in separate Alert commands in the SyncBody.


• Another Alert command indicates the synchronization type; here the 200 refers to a client-initiated two-way sync.


• Authentication information is placed in the Cred element of the SyncHdr element.


• The client can send service capabilities using the Put command in the SyncBody element.

Sample Code: Client Begins Initialization

<SyncML> <SyncHdr> <VerDTD>1.0</VerDTD> <VerProto>SyncML/1.0</VerProto> <SessionID>1</SessionID> <MsgID>1</MsgID> <Target><LocURI>http://www.syncml.org/sync-server</LocURI></Target> <Source><LocURI>IMEI:493005100592800</LocURI></Source> <Cred> <!--The authentication is optional.--> <Meta><Type xmlns='syncml:metinf'>syncml:auth-basic</Type></Meta> <Data>QnJ1Y2UyOk9oQmVoYXZl</Data> <!--base64 formatting of "userid:password"--> </Cred> <Meta> <!--The Meta is now used to indicate the maximum SyncML message size a client can receive.--> <MaxMsgSize xmlns='syncml:metinf'>5000</MaxMsgSize> </Meta> </SyncHdr> <SyncBody> <Alert> <CmdID>1</CmdID> <Data>200</Data> <!-- 200 = TWO_WAY_ALERT --> <Item> <Target><LocURI>./contacts/james_bond</LocURI></Target> <Source><LocURI>./dev-contacts</LocURI></Source> <Meta> <Anchor xmlns='syncml:metinf'> <Last>234</Last> <Next>276</Next> </Anchor> </Meta> </Item> </Alert> <Put> <CmdID>2</CmdID> <Meta><Type xmlns='syncml:metinf'>application/vnd.syncml-devinf+xml</Type></Meta> <Item> <Source><LocURI>./devinf10</LocURI></Source> <Data> <DevInf xmlns='syncml:devinf'> <Man>Big Factory, Ltd.</Man> <Mod>4119</Mod> <OEM>Jane's phones</OEM> <FwV>2.0e</FwV> <SwV>2.0</SwV> <HwV>1.22I</HwV> <DevId>1218182THD000001-2</DevId> <DevTyp>phone</DevTyp> <DataStore> <SourceRef>./contacts</SourceRef> <DisplayName>Phonebook</DisplayName> <MaxGUIDSize>32</MaxGUIDSize> <Rx-Pref> <CTType>text/x-vcard </CTType> <VerCT>2.1</VerCT> </Rx-Pref> <Tx-Pref> <CTType>text/x-vcard</CTType> <VerCT>2.1</VerCT> </Tx-Pref> </DataStore> <CTCap> <CTType>text/x-vcard</CTType> <PropName>BEGIN</PropName> <ValEnum>VCARD</ValEnum> <PropName>END</PropName> <ValEnum>VCARD</ValEnum> <PropName>VERSION</PropName> <ValEnum>2.1</ValEnum> <PropName>N</PropName> <PropName>TEL</PropName> <ParamName>VOICE</ParamName> <ParamName>CELL</ParamName> </CTCap> <SyncCap> <SyncType>01</SyncType> <SyncType>02</SyncType> </SyncCap> </DevInf> </Data> </Item> </Put> <Get> <CmdID>3</CmdID> <Meta><Type xmlns='syncml:metinf'>application/vnd.syncml-devinf+xml</Type></Meta> <Item> <Target><LocURI>./devinf10</LocURI></Target> </Item> </Get> <Final/> </SyncBody> </SyncML>

When the server receives this request from the client, it completes the initialization phase by responding to the client with authentication information, sync anchors, and device information.

Sync Types

The SyncML specification defines seven synchronization types:

• Two-way sync: This is the normal type of synchronization, in which client and server exchange information about any new changes to the data in their databases. This sync type is always initiated by the client, which sends a synchronization request to the server. The server then unifies the client's data and the data in the server's database and sends the unified data to the client. The client then updates its own database and sends back to the server any necessary mapping information.


• Slow sync: In this form of two-way sync, all items in the client's database are compared with all items in the server's database, field by field. Either the client or the server may request such a sync if it loses its change log.


• One-way sync from client: The server gets all changes from the client but doesn't send any changes back.


• Refresh sync from client: The client sends all its data to the server, which is expected to replace all its own data with the client's data.


• One-way sync from server: The client gets all changes from the server but doesn't send any changes back.


• Refresh sync from server: The server sends all its data to the client, which is expected to replace all its own data with the server's data.


• Server-alerted sync: The server notifies the client that there is a need to initiate a specific type of synchronization.

Updating Databases

A client must inform the server of all client data modifications that happened since the previous sync, as in the following example:

Sample Code: Client Sends Modifications to Server

<SyncML> <SyncHdr> <VerDTD>1.0</VerDTD> <VerProto>SyncML/1.0</VerProto> <SessionID>1</SessionID> <MsgID>2</MsgID> <Target><LocURI>http://www.syncml.org/sync-server</LocURI></Target> <Source><LocURI>IMEI:493005100592800</LocURI></Source> </SyncHdr> <SyncBody> <Status> <MsgRef>1</MsgRef><CmdRef>0</CmdRef><Cmd>SyncHdr</Cmd> <TargetRef>IMEI:493005100592800</TargetRef> <SourceRef> http://www.syncml.org/sync-server </SourceRef> <Data>212</Data> <!--Statuscode for OK, authenticated for session--> </Status> <Status> <MsgRef>1</MsgRef><CmdRef>1</CmdRef><Cmd>Alert</Cmd> <TargetRef>./dev-contacts</TargetRef> <SourceRef>./contacts/james_bond</SourceRef> <Data>200</Data> <!--Statuscode for Success--> <Item> <Data> <Anchor xmlns='syncml:metinf'><Next>200005022T093223Z </Next></Anchor> </Data> </Item> </Status> <Sync> <CmdID>1</CmdID> <Target><LocURI>./contacts/james_bond</LocURI></Target> <Source><LocURI>./dev-contacts</LocURI></Source> <Meta> <Mem xmlns='syncml:metinf'> <FreeMem>8100</FreeMem> <!--Free memory (bytes) in Calendar database on a device --> <FreeId>81</FreeId> <!--Number of free records in Calendar database--> </Mem> </Meta> <Replace> <CmdID>2</CmdID> <Meta><Type xmlns='syncml:metinf'>text/x-vcard</Type></Meta> <Item> <Source><LocURI>1012</LocURI></Source> <Data><!--The vCard data would be placed here.--></Data> </Item> </Replace> </Sync> <Final/> </SyncBody> </SyncML>

Sync4j

Sync4j is an open source server and framework that implements the SyncML protocol for data synchronization. It is written entirely in the Java programming language and complies with version 1.3 of the Java 2 Platform, Enterprise Edition (J2EE) specification. It complies with SyncML version 1.1, and supports several types of synchronizations.

Conclusion

Synchronization plays an important role in applications that run on mobile devices. In addition to keeping data in sync on all devices and platforms, it gives the user the illusion of uninterrupted operations in the presence of poor or nonexistent network connections. This article presented a tutorial to help developers get started with data synchronization with SyncML. You should be able to follow the sample applications in this article and adapt them to your own SyncML-aware mobile Java-based applications.Yu Feng and Jun Zhu provide another example of using SyncML in their book, Wireless Java Programming with J2ME. The relevant chapter is posted on the Sun Developer web site: "Data Synchronization for Wireless Applications."

Keep an eye on the evolving JSR 230: Data Sync API, whose objective is a J2ME optional package that provides a generic high-level API for performing data synchronization using underlying implementations of protocols such as SyncML.

For More Information

• J2ME


• SyncML


• Sync4j


• Data Synchronization for Wireless Applications


• SyncML Over Bluetooth


• JSR 230: Data Sync API


• Java API for XML Processing (JAXP)


• SyncML-compliant mobile devices

Acknowledgments

Special thanks to Roger Riggs of Sun Microsystems, whose feedback helped me improve this article.

About the Author

Qusay H. Mahmoud provides Java consulting and training services. He has published dozens of articles on Java, and is the author of Distributed Programming with Java (Manning Publications, 1999) and Learning Wireless Java (O'Reilly, 2002).