Event Processing and Predictive Analytics

The Digital Essentials, Part 3

Developing a robust digital strategy is both a challenge and an opportunity. Part 3 of the Digital Essentials guide series explores five of the essential technology-driven experiences customers expect, which you may be missing or not fully utilizing.

Get the Guide

II had a conversation with a product manager for a product labeled Complex Event Processing (CEP) but when I looked at the actual product implementation I asked why the product was labeled CEP versus Event Stream Processing (ESP). The product uses continuous queries and does not support Event Condition Action (ECA) rules. They responded that the use cases for CEP can be solved with ESP – i.e. they are essentially the same. Then they agreed that the product really was ESP but the distinction is lost on potential customers.
This technology is difficult enough for potential users to understand that confusing the products not helpful. The vendors also tend to talk about features versus use cases. And, when they do talk about use cases they often overlap – for example fraud detection is listed as a use case for CEP, ESP and predictive analytics.
At high level these products can accomplish the same use cases but in very different ways. Let’s look the use case of predicting product failures, a use case listed for all three products.
With predictive analytics, this use case could be solved by analyzing outages historically and predict future potential outages based on statistical models. These models can be applied against near real-time data to predict the likelihood of an outage. ESP could execute a continuous query against sensor data looking for a set of events that indicates a potential outage. CEP would similarly examine a set of events but apply rules to those events to generate a complex (or logical) event that is an outage alert.
So, what’s the difference? Predictive analytics uses data mining techniques to find out what might cause an outage. This is a necessary step if the rules for what causes an outage are not well understood. CEP uses rules to determine if there is a potential outage. These rules may be well understood, for example a device phones home and communicates the problem state. However, these rules can be more complex and also apply to individual devices. For example, if a sensor detects a low voltage event for a specific device that may predict an outage.
ESP would work well for intelligent devices that specifically phone home for outages. The continuous query would select the outage notification in a stream of status events that would be processed as an outage event.
This use case however would difficult to implement in ESP if there were many dissimilar devices being monitored with different rules about what constitutes a potential outage. For example, monitoring a power plant for a potential outage might involve hundreds of sensors on very different devices. The event stream would need to be pre-processed and made homogenous to indicate specifically that there is a potential outage.
CEP would not need this preprocessing step because each device event could have its own rule base as to what constitutes a potential outage. But CEP is not a machine learning engine. The rules have to be defined so you may have to go through data mining exercise – e.g. predictive analytics.
Predictive analytics would score each event based on a statistical model for the likelihood of a failure. This would require a data store of past events and outages and a data mining activity to define the models.
Each of these products has strengths and weakness. In general, ESP is very fast and works well with simple rules that can be defined in the context of a continuous query. CEP is not as fast, executing rules obviously requires time, but one can built a sophisticated and dynamic rule base over time. If you find a new fault rule you don’t have to change the query, you just add a rule. And, CEP is fast enough for most high-velocity use cases.
Predictive analytics is a necessary step in many cases to figure out the rules. But predictive analytics does not define the rules, for example the event would be scored has highly likely to cause a failure but what do you do about it? One would still have to define what to do in the event of the failure – define the rules and/or workflows.
So, this is all quite confusing. Hopefully I have shed some light on the subject please give me feedback and/or questions. I believe over time these products will converge and we will get closer to true machine learning for these use cases.