Complex Event Processing at Orbitz
TS-6048

This presentation is a case study of complex event processing at Orbitz. The Orbitz Worldwide Data Centers host numerous leading online travel agency web sites that utilize thousands of services distributed across hundreds of Jini network technology connected virtual machines (VMs) to service more than 18 million visitors per month. Monitoring and managing these large-scale, complex applications is a daunting task. Failure happens, and downtime is money! Orbitz has harnessed the power of complex event processing to handle a torrent of monitoring events with minimal application development and hardware costs. The resulting system has improved manageability, by reducing the mean time to resolution (MTTR) for customer-impacting events caused by software availability, reliability, and performance issues.

Orbitz has developed a proprietary Java technology-based instrumentation API named the Extremely Reusable Monitoring API (ERMA). ERMA is as simple to use as a logging API yet flexible enough through configuration to satisfy most requirements for logging, monitoring, analytics, and other event processing needs. It dynamically correlates events across distributed VMs servicing a user request, enabling efficient drill-down root cause analysis for errors and latency as well as bottom-up impact analysis. ERMA has been applied with filters, interceptors, listeners, Spring-AspectJ AOP integration, and custom instrumentation of core Orbitz object models. As a result, Orbitz has access to data for more than one hundred thousand distinct event types with minimal development cost.

Monitoring data corresponding to discrete events is streamed through ERMA from hundreds of VMs to a complex event processing (CEP) engine in real time where it is aggregated and processed with high throughput and low latency. A single two-way commodity computer executing Orbitz's most elaborate event processing application can handle nearly 100,000 events per second. The ability to handle such a large volume of data enables the company to monitor services at a very fine-grained resolution as needed. Also, the hardware cost of adding new monitoring applications is minimal with this technology.

A high-level event processing language provided with the CEP engine makes it possible to develop new monitoring applications quickly and easily. A visual development environment makes it easy to trace event flow and wire in new functionality. The event processing language has been extended by Orbitz with custom Java technology-based functions and operators tailored to the Orbitz environment. For example, Orbitz has developed an operator that can deliver streams of data via SNMP, using the OpenNMS API to integrate with its Service Operations Center infrastructure.

A Java technology-based portal has been developed to visualize the output from the CEP engine. The portal presents tabular and graphical views of vital system statistics and publishes RSS feeds for alarms. Users can subscribe to feeds for particular alarm severities and/or affected applications.

The future of complex event processing at Orbitz includes event pattern monitoring capabilities. The company is developing a solution that will reduce the volume of alarms delivered to the operator by bundling customer-impacting event information with root cause estimation determined by detection of patterns of discrete events. As its business grows, it is imperative that its operations team can manage the system in a scalable manner by relying on automated actionable event detection. Complex event processing is the solution to this problem for Orbitz.

Reference: http://complexevents.com/

Watch The Session
You need to be a registered Sun Developer Network member to view this multimedia session. If you are a registered SDN member, please click on "Watch Multimedia" button to log-in to view the multimedia session. If you wish to join SDN, please click here.