Complex event processing (CEP) software handles many low-level events to recognize a high-level event that triggers a business process. Since many business processes do not consider low-level data events, BPM may not seem to need event processing. On the other hand, event processing would not be relevant at all if it did not occasionally trigger a business process or decision. In other words, it appears that:
- CEP requires BPM but
- BPM does not require CEP
The first point is market limiting for CEP vendors. Fortunately for CEP vendors, however, most BPM does require event-processing, however complex. In fact, event processing is perhaps the greatest weakness of current BPM systems (BPMS) and business rules management systems (BRMS), as discussed further below.
BPM usually requires event processing
BPM involving workflow or business to consumer (B2C) interaction is frequently driven by events, however. For example, consider:
- a task being completed,
- an activity, communication, or response by a consumer,
- or, more interestingly, a lack of activity over or after a period of time
For example, consider each of the following governing policies:
- an application taken by order entry should be forwarded to underwriting
- a postal letter received from a customer must be scanned and handled as a fax letter from the customer
- a letter warning of imminent collection should be sent to the primary customer on a delinquent account by certified mail if no communication with the customer has occurred within the last thirty days
Clearly, managing business processes that involve people requires event processing.
- CEP processes low-information events, potentially in high-volume.
- BPM processes events that are more directly relevant to the business.
Event-driven inadequacies of BPMS and BRMS
Note that it is a frequent exaggeration to say that BPM processes events at all. For the most part, BPMS require analysts to define a process to be invoked when events are signaled by program code that has to written and inserted into other systems. BPMS do not typically externalize operational governance and compliance policies as stated above as BRMS do. And, since neither BPMS nor BRMS handle events that arise from a lack of activity over a period of time, the analysis and systems infrastructure to trigger processes arising from a lack of events commonly results in extraordinary development and maintenance costs.
Having pointed this out, let us continue with understanding the relationship between business process management and event processing, however complex.
Does CEP overemphasize complexity?
Complexity in event processing can come from an onslaught of simple events, as in telecommunications or computer networks, where individual events typically have little relevance. When combinations or correlations of such events may be relevant, however, event processing complexity obviously increases further. CEP specifically addresses the onslaught of events and the processing necessary to distill raw data into more relevant and typically less frequent information events.
Examples of such CEP include:
- alarm management in telecommunications and computer networks
- security monitoring in telecommunications and computer networks
- performance monitoring in computer systems
- algorithmic trading of securities
In addition to distilling data into information, complexity can arise in how the event is handled. For example, if a five minute trend indicates that a stock is free-falling, what action should be taken? Or if an authentication for a mailbox has been failing every ten minutes, what should be done?
CEP or BPM decisions
This is were the handling of events crosses the bridge from processing, however complex, into a business process that may be as simple as a single decision about how to respond and then invoking the selected process.
That is:
- event processing triggers business processing
- business processing involves decision making
- decision making requires a business rules engine
Or course, if the decision making is trivial, an event processing platform may suffice. Few business processes have no decision complexity or workflow, however.
GRC and EDM
Enterprises necessarily manage their decision making from governance, risk, and compliance (GRC) perspectives and with an eye to improving business performance with experience and over time. To implement such enterprise decision management (EDM), BPMS typically integrate with BRMS which manages externalized GRC and decision making policies that the BRMS automates using a business rules engine (BRE) that is integrated with a BPMS.
To put all this simply, assuming CEP is not overkill for simple events:
- CEP triggers BPM that uses a BRMS for EDM
Note that decisions are frequently externalized from business processes by embedding BRE in service-oriented architectures (SOA) that are invoked by BPMS.
CEP as BPMS niche
So CEP may appear to be a bolt-on that extends BPM to handle events (or the lack of events) more easily and competently. CEP vendors are not happy with this view, since it limits their market to a BPM add-on. As a result, CEP vendors like TIBCO are moving down the stack to incorporate BPM functionality. On the BPMS front, some smaller vendors will move to incorporate CEP to increase their market share in a defined niche. With few exceptions, one of which is discussed below, BPM market leaders are less motivated to incorporate CEP functionality into their BPMS precisely because CEP is a relative small market.
Where CEP = BPMS + BRMS
As discussed above, CEP addresses niche markets where the volume or complexity of event processing can be bolted-on to a typical business process. But what if high-volumes of events are directly relevant to a business, such as in algorithmic trading at a hedge fund? BPMS cannot handle the event stream and CEP cannot handle the BPM or the complexities of decision making address by a BRMS.
Every CEP vendor recognizes and targets the securities industry for a substantial share of their revenues. More specifically, every CEP vendor targets algorithmic trading. The amount of capital invested algorithmic trading alone is substantial enough that the word niche seems inappropriate. The market for CEP on Wall Street alone is enough to motivate BPM market leaders to incorporate CEP in their offerings. On the other hand, the need for CEP, not BPM, is so specific and intense in algorithmic trading that some CEP vendors can justify focusing on the application rather than moving down the stack to incorporate BPM functionality.
- This contrast seems most apparent between Apama and TIBCO.
Whether or not CEP vendors move to incorporate BPM functionality or focus on algorithmic trading, CEP consumers need the technology to support decisions. As CEP becomes more capable in GRC and EDM, it will also have to become more usable by less technically skilled people. CEP could accomplish all this by following BRMS’ lead.
Mainstream CEP
Twenty five years ago, rules where written in technical languages. Today, BRMS such as the one we developed at Haley, express polices – not just rules – in natural language. BRMS would not have crossed the chasm without these advances. CEP needs to do the same. The most likely path for CEP to cross the chasm is to follow BRMS example.
In CEP, there are two approaches.
- Extend SQL with syntax related to streams and time windows
- StreamSQL from Streambase
- Continuous Computation Language from Coral8
- BEA AquaLogic (recently acquired by mainstream BPM vendor IBM)
- EQL, the event-processing language (EPL) of open-source Esper
- -Define a Java-like programming language for processing events
As with business rules prior to the release of Innovator by Blaze Software (now part of Fair Isaac) and Authority from Haley, the esoteric technical skills required by these CEP languages will prevent CEP from crossing the chasm into more mainstream use, as BRMS has accomplished for business rules and BPMS for processes.
Making CEP more usable
I thank and apologize to Coral8, who are exemplary in their product coverage, for using an example of their excellent technical overview to demonstrate a point. In section 5.1 they show following stream SQL:
Insert Into StreamAvgPrices Select Symbol, Avg(Price) From StreamTrades Keep 5 minutes Where Volume >= 10000 Group By symbol
Personally, I think this is one of the nicer examples of the stream / SQL approach. The code that I have seen from the Java approaches seems more difficult to implement.
Superior CEP expression though it may be, this is not a form of expression that CEP market analysts such as Forrester or BRMS vendors would consider for crossing the chasm. Much more appropriate would be:
the average price of a stock for trades over 10,000 shares over the last 5 minutes
For those who are familiar with my work in BRMS, you can understand how the latter is completely equivalent to the former. I believe, as do BRMS analysts, that such or similarly non-technical expression is a practical market requirement for crossing the chasm. Ilog and other vendors’ approach of authoring pseudo-English statements would clearly be superior to EPL code.
Note that EPLs do not express business processes or address decision making at all. The EPL above corresponds to a noun phrase, not a statement of policy, process, or action.
Understanding CEP and BPM together
Although most BRMS do not understand time or what it means to say “over the last 5 minutes”, CEP and BPM require a deeper understanding of time, as discussed here in a prior post. Given such an understanding and an ontology based on FIX or otherwise covering stocks, trades, shares, and other concepts relevant to algorithmic trading, noun phrases like the one above can replace EPL.
Assuming EPL is replaced by noun phrases, the lines between CEP and BPM fall. Such noun phrases will be used within sentences, obviously! Such sentences will constitute the GRC and decision making policies and knowledge involved in EDM and BPM. The CEP will fall out from the same content that BRMS now manage.
Eventually Eventful BPM
CEP will cross the chasm but its distinction may also fade as knowledge management becomes formal enough to automate requirements, policies, and rules as they are managed using BRMS today. As BRMS become increasing semantic and logically capable, they will also encompass more and more of BPM complexity (e.g., GRC).
And as BRMS and BPM converge with architectures and ontologies that cover long-running processes that involve workflow and events, EPL will be generated automatically just as Haley Authority now generates technical rules from knowledge managed by stakeholders and subject matter experts who have far less technical skill than CEP currently requires.