By no means is this a book about SBVR.  Rather, it is about the core and critical concepts necessary to understand how  to define an enterprise repository of declarative knowledge covering business.  The unfortunately reality is that the work necessarily suffers from legacy concepts and perspectives, but this is a statement of fact and practicality rather than a negative assessment.  Essentially, my problem with much of the increasingly mainstream marketing and practicing of business rules is precisely the term “rule”.  Nonetheless, since the mainstream audience is pre-occupied with the word “rules” rather than the word “knowledge”, Ron’s book is – again – right on the money for practitioners who want to achieve the benefits of separating models and knowledge from implementations in order to increase the agility of systems and more closely align IT with business needs (all of which he discusses in Chapters 2 and 3).

This edition clearly benefits from the progress in the field and reflects current enterprise objectives that have progressed into the mainstream since Ron began practicing and writing many years ago.  Veterans will quickly recognize and benefit from the simple clarity that Ron brings to the subject.

The critical topics that Ron addresses are the core concepts underlying modeling and linguistics that allow non-technical business analysts (even, perhaps, subject matter experts) to capture unambiguous statements of business definitions, requirements, policies and other “rules” in English sentences that are suitable for presentation to and verification by stakeholders.  In addition, in the event knowledge-level standards like SBVR evolve and are adopted by technology (e.g., business rule) vendors, managing such statements will directly impact the operational behavior of business processes, especially where they involve automated decisions, policy enforcement and other forms of governance requiring enforcement.

Although Ron and I have come to a shared view on knowledge capture and management, we have followed different paths.  Ron’s concern is primarily capture.  Mine extends to and necessarily emphasizes implementation and execution.  Nonetheless, our perspectives on modeling and capture are extremely aligned.  Ron’s writing on what he calls “terms” and “wordings” are right on the money.

We have independently arrived at a point where we believe that linguistic expression of business knowledge is much more important (and valuable) than rule-based expressions suitable for execution by rule engines.  In effect, Ron argues and I agree that the knowledge is much more important, durable, and valuable than its executable form or expression.  As a result, I could not more emphatically introduce Ron’s coverage of how to express models and logic in English.

By the end of Chapter 1, Ron defines what he calls a vocabulary and why  managing  vocabulary is a requirement for managing business knowledge (including definitions, requirements, and policies).   Ron’s writing on nouns and verbs and how they form the backbone of expression for business knowledge is seminal and accessible to novice and expert practitioner alike.

As more or an artificial intelligence practitioner, I might quibble with the omissions of mass nouns, such as money or time, from his discussion, but these topics will not be missed by anyone but the most forward-looking strategist.   Similarly, the limitation of current standards to wordings involving verbs eliminates coverage of adjectival phrases and results in too many wordings involving the verb “has”, at least in my opinion.  For example, wordings like “a person has an age” seem arcane, perhaps technical, to me.  Nonetheless, even though my work supports such “phrasings”, most of its users still use “has” phrasings!  Perhaps this is because I have not taken the time and expended the effort to clarify the concepts that Ron covers so well.  In short, Ron’s writing is right on target for practitioners given the current state of the art (and standards).  And I can think of no other expert or author who has reduced current practice to such an accessible form.

Chapter 4 clearly explains and motivates the use of sentences as the independent units of business knowledge.  Ron does a good job of introducing the notion of roles as the components linked together within predicates using verbs.  This is fairly abstract material, so I don’t blame him for avoiding too much depth on the semantics of roles.  Fortunately, other materials, such as the SBVR standard itself, cover these details in greater depth.  So again, the content seems right on for someone entering the field and focused on linguistic modeling, capture and management rather than more formal “semantics”. 

Ron clearly separates statements defining and relating concepts from requirements.  Not only does he present the material linguistically, he reinforces it with clear graphical presentations of the corresponding models.   Personally, I prefer to mix certain requirements with definitions, as in “a person has exactly one mother”,  but this would complicate the graphical presentations.  He also addresses common linguistics, such as passive voice and using verbs (i.e., participles) as adjectives, which I have found to be important concepts that must be brought to practitioners’ attention.  Otherwise, the models become unnecessarily complex or the resulting sentences seem cumbersome, if not stilted.

In places the formality of SBVR shows through.  For example, “a person must not lease a vehicle the person owns”.  This reflects linguistic limitations of available tools.  This phrasing reflects “necessity” within the underlying logical formalism of SBVR.  In addition, the reiteration of “the person” begs for natural language processing that would understand pronouns, such as “(s)he”.  Having said that, this is clearly a higher level statement than can currently be understood and implement by most business rule management systems (BRMS), however.  As such, it demonstrates the power of Ron’s approach and motivates (given most commercial offerings) the separation of capture and management of business knowledge from its expression in business process management systems (BPMS) or BRMS.

Having encountered the notion of logical necessity in the preceding example, it is worth pointing to Ron’s discussion in Chapter 10.  Externalizing business knowledge from systems requires certain architectures for information systems.  For example, the prohibition of a person leasing a vehicle that (s)he owns requires some action not stated here.  In effect,  a violation of requirements must be anticipated in the runtime architecture.  There are various approaches to this, including runtime exceptions.   Ron demonstrates this approach in Chapter 10 and, if the system incorporates workflow for resolution, he advocates using the end-user accessible form of business knowledge for explanation (in his discussion of guidance within Chapter 2).  More advanced approaches, such as meta-reasoning, and detailed architectural approaches are appropriately left out of this work on capture and management.

Interestingly, in Chapter 7, Ron suggests that business rules should not be expressed within an if-then syntax.  Obviously, this clearly separates Ron’s methodology from the production rule focus of most commercial BRMS.  It also exposes a gap between SBVR and operational systems.  Ron discusses behavioral rules within the chapter but the gap between requirement and behavior remains unfilled.  This is my most significant concern for the logical approach to knowledge management.   Without a framework by which declarative knowledge can bridge to imperative action, the logical approach falls short of operational relevance.  This is reflected, in my practical viewpoint, by the lack of operational deployment of SBVR. 

The principal  challenge remaining for SBVR is to cross the gap from expression into operation.  Aside from linguistic limitations, this is my primary concern for the formal logic approach to knowledge capture and management.   The ideas Ron covers so well are necessary for long-term success in automating managed knowledge, but they are not enough.   Unfortunately, most of the business rule vendors do not adequately address the necessary capabilities that Ron’s methodology and toolsets handle.  This leaves us with a continuing need for business rule engineers to bridge the gap, manually.

Most business logic is at least as complicated.  But most business rule metaphors are not up to expressing tax regulations in a simple manner.  Nonetheless, the tax regulations are full of great training material for learning how to analyze and capture business rules.

For example, consider the earned income credit (EIC) for federal income tax purposes in the United States.  This tutorial uses the guide for 2003, which is available here. There is also a cheat sheet that attempts to simplify the matter, available here. (Or click on the pictures.)

What you will see here is typical of what business analysts do to clarify business requirements, policies, and logic.  Nothing here is specific to rule-based programming.  If this seems tedious, just rely on someone else to do it.  There is no avoiding it.  Really!  We are certainly able and willing to help, whether to teach your staff or outsource the effort.  Our objective here is not as much to solicit your business but to help those who are new to this succeed.  Specifically, this is for my friend, PS.

The EIC publication has the following chapters:

• Chapter 1. Rules for Everyone
• Chapter 2. Rules If You Have a Qualifying Child
• Chapter 3. Rules If You Do Not Have a Qualifying Child
• Chapter 4. Figuring and Claiming the EIC
• Chapter 5. Disallowance of the EIC
• Chapter 6. Advance Payment of EIC in 2004
• Chapter 7. Detailed Examples

For tutorial purposes, we are primarily concerned with the first four chapters.

Chapter 1. Rules for Everyone

This chapter discusses Rules 1 through 7. You must meet all seven rules to qualify for the earned income credit. If you do not meet all seven rules, you cannot get the credit and you do not need to read the rest of the publication.

Note. If you meet all seven rules in this chapter, then read either chapter 2 or chapter 3 (whichever applies) for more rules you must meet.

What should we do with these rules if we are trying to capture them in a formal manner that can support decision making, such as using a business rules engine?

First, consider whether these are rules or conditions.  What are their implications?  That is, if we state them as “if-then” rules, what comes after the then?

Try completing the following sentences:

• If you have a valid social security number then …
• If your filing status cannot be “married filing status separately” then…
• If you must be a U.S. citizen or resident all year then…

Each of these have a different problem!  None of them is the beginning of a useful rule.  There is no legitimate conclusion from any of them! 

The problem is that these are necessary conditions, not rules.  You must meet each of them in order to qualify for the EITC.  Meeting any one of them is not sufficient.  In fact, meeting all of them is not sufficient!  The note above indicates that other rules must also be considered.  (Note that the IRS forgot to mention chapter 4!  Think about the staggering amount of money the IRS probably spent on their publication.  Your business requirements documents are likely to have many more errors, unfortunately.)

Rules have consequences!

The consequence of each of these rules is not that you qualify for the EIC.  In fact, these rules do not have consequences!  These are conditions.  To make them rules we need to realize that their conclusions can only be that we do not qualify for the EIC.  Then, we can write these rules as follows:

1. If your adjusted gross income is not below the limit … then you do not qualify for the EIC.
2. If you do not have a valid social security number then you do not qualify for the EIC.
3. If you are married and filing separately then you do not qualify for the EIC.
4. If you were not a US citizen or resident alien all year then you do not qualify for the EIC.
5. If you are filing form 2555 or 2555-EZ then you do not qualify for the EIC.
6. If you had investment income over $2,600 then you do not qualify for the EIC.
7. If you do not have earned income then you do not qualify for the EIC.

Of course the limitations on adjusted gross income are more complicated and need to be dealt with, but we will get to that after discussing what just happened a bit more.

Notice how we had to flip the logic in each of the conditions.  This is a common activity for knowledge engineers.  It can also a common source of error.  It would be much better if we did not have to do this.  Unfortunately, the “if-then” metaphor or most business rules management systems require us to do so.  It would be much better if we could avoid the double negation of negating conditions and negating conclusions.  For example, wouldn’t it be nice to write:

• You qualify for the EIC
  1. only if your adjusted gross income is less than the limit…
  2. only if you have a valid social security number
  3. only if you are not married and filing separately
  4. only if you were a US citizen or resident alien all year
  5. only if you do not file form 2555 or 2555-EZ
  6. only if your investment income was $2,600 or less
  7. only if you had earned income

This is much closer to the original form and much more reliable.  Unfortunately it is wrong!

First, there is no implication here.  Each of these is necessary but not sufficient for the deduction that you qualify for the EIC.  In chapter 4, for example, the IRS gets around to saying that there are limits on earned income!

The fact is that all of these statements must be true in order to qualify, but even if they are all are, you may not be qualified.  This could be more easily expressed as follows:

• You do not qualify for the EIC
  1. unless your adjusted gross income is less than the limit…
  2. unless you have a valid social security number
  3. unless you are not married and filing separately
  4. unless you were a US citizen or resident alien all year
  5. unless you do not file form 2555 or 2555-EZ
  6. unless your investment income was $2,600 or less
  7. unless you had earned income

Here we have avoided negating the conditions but we have retained the negative conclusion.  This is indicated by the fact that none of the conditions is sufficient for the positive conclusion.

Think carefully about whether a statement is a condition or a rule and whether the conclusions of the rule should be stated positively or negatively.

We’ll come back to the AGI limitations, but note that they are forward references to chapter 2!

Chapter 2. Rules If You Have a Qualifying Child

If you have met all the rules in chapter 1, use this chapter to see if you have a qualifying child. This chapter discusses Rules 8 through 10. You must meet all three of those rules, in addition to the rules in chapters 1 and 4, to qualify for the earned income credit with a qualifying child.

Note. You must file Form 1040 or Form 1040A to claim the EIC with a qualifying child. (You cannot file Form 1040EZ.) You must also complete Schedule EIC and attach it to your return. If you meet all the rules in chapter 1 and this chapter, read chapter 4 to find out what to do next.

No qualifying child. If you do not meet Rule 8, you do not have a qualifying child. Read chapter 3 to find out if you can get the earned income credit without a qualifying child.

This chapter is a bit trickier in that it begins to talk about children, which is plural.  The introduction is misleading because we are not simply determining whether we have a qualifying child but which of our children qualify.  For example, our adjusted gross income limitation is higher if we have more than one.  In addition, the relationship, age and residency tests also tricky.  We’ll come back to those after chapter 4.

What are the conclusions that we are trying to reach in this chapter?

Try completing the following sentences:

• If your child meets the relationship, age, and residency tests then…
• If your qualifying child cannot be used by more than one person to claim the EIC then…
• If you cannot be a qualifying child of another person then…

Now it is clearly tempting to try to write the first of these as:

• If your child meets the relationship, age, and residency tests then your child qualifies.

But this is not really true if another person claims your child or you as a qualifying child, right?

Also, what happens if you have two children, both of whom satisfy the relationship, age, and residency tests?  Suppose only one of them can be used as a qualify child by someone else.  How many qualifying children do you have if you cannot be claimed as a qualifying child by anyone else?

This may seem complicated, but it is a common problem.  Let’s look at the answer first and then figure out how to get to it more easily.

In this chapter we are trying to determine which of our children qualify.  That is:

• A child of yours qualifies 
  1. Only if the child meets the relationship test.
  2. Only if the child meets the age test.
  3. Only if the child meets the residency test.
  4. Only if the child cannot be used by more than one person to claim the EIC
  5. Only if you do not a qualifying child of someone else

Note that this expression has eliminated the use of the word “qualifying” within rules 9 and 10.

The IRS has some additional language on who can use a child to claim the EIC that exposes additional flaws in their expression. 

You can choose which person will claim the EIC. If you and someone else have the same qualifying child, you and the other person(s) can decide who will claim the credit using that qualifying child. But if you and the other person(s) cannot agree and more than one person claims the credit using the same child, the tie-breaker rule (explained in Table 2, next) applies. If the other person is your spouse and you file a joint return, this rule does not apply.

And here is the table of tie-breaker rules:

So the IRS would have better stated rule 9 in some other way, but it is hard to formulate!  The IRS introduced the notion of a child who passes the three tests to get closer to a reasonably simple expression but still failed.  Let’s try their approach:

• A child of yours passes the qualification tests 
  1. If and only if the child meets the relationship test
  2. If and only if the child meets the age test
  3. If and only if the child meets the residency test

Note that this is still not right since the IRS anticipates that a child may pass these tests for more than one person.  Also, the relationship test will not simply involve you, but any person that a child might qualify for and, as demonstrated in the tie-breaker table, the residency test also considers who the child resides with. Therefore, we should re-write the preceding inference as:

• A person’s child passes the qualification tests 
  1. If and only if the child meets the relationship test for the person
  2. If and only if the child meets the age test
  3. If and only if the child meets the residency test for the person

Now, we can clarify the IRS’ logic, as in:

• A person’s child qualifies
  1. Only if the child passes the qualification tests for the person
  2. Unless the child passes the qualification tests for another person
     1. Unless the two people file a joint return together
     2. Unless the other person agrees that the first person can claim the child as qualifying
     3. Unless the first person wins the tie breaker for the child versus the other person
• A person wins the tie-breaker for a child versus another person 
  1.  If the first person is the child’s parent
     1. Unless the other person is the child’s parent
        1. Unless the child lived with the first person longer
        2. If the child lived with each person for the same amount of time
           1. Unless the first person has the higher adjusted gross income
           2. If the second person has the higher adjusted gross income

Note the last two, most nested conditions are harmlessly redundant.  (The regulations did not seem clear for two people with the same AGI.) 

Of course, there are other means of expressing the same logic, as in:

• A person wins the tie-breaker for a child versus another person
  1. Only if the first person is the child’s parent
  2. Unless the other person is the child’s parent
     1. Unless the child lived with the first person longer
     2. If the other person has a higher adjusted gross income
     3. If the child lived with the other person longer

All of these forms can be automated by some commercial BRMS while the tables require a lot more intelligence and interpretation to understand.  All the knowledge that it takes to decipher the conditions in one column versus the conclusions in the other and that they are sequential in application top to bottom is too much for today’s software.

Finally, the rules in chapter 2 should infer which of your children qualify.  If you have no children, they will not apply, obviously.  If you have two children, there may be zero, one or two that qualify.  Then, the rules that refer to how many qualifying children you have will work.

Don’t worry about sequencing your rules for now – just realize that inferences build on each other.  Rely on it.

Chapter 3. Rules If You Do Not Have a Qualifying Child

Use this chapter if you do not have a qualifying child and have met all the rules in chapter 1. This chapter discusses Rules 11 through 14. You must meet all four of those rules, in addition to the rules in chapters 1 and 4, to qualify for the earned income credit without a qualifying child.

Note. You can file Form 1040, Form 1040A, or Form 1040EZ to claim the EIC without a qualifying child. If you meet all the rules in chapter 1 and this chapter, read chapter 4 to find out what to do next.

Try the same approach as we have taken in prior chapters.  Think about the conclusions of these rules by trying to complete the following:

• If you are at least age 25 but under age 65 then…
• If you cannot be the dependent of another person then…
• If you cannot be a qualifying child of another person then…
• If you must have lived in the United States more than half the year then…

The conclusions are not that you qualify for the EIC.  As in the first chapter, their logic is negative.  If you don’t meet these conditions and you have no qualifying children, then you are not qualified for the EIC.

• If you do not have a qualifying child then you do not qualify for the EIC
  1. If you are under 25 years old
  2. If you are over 65 years old
  3. If you are a dependent of another person
  4. If you are a qualifying child of another person
  5. If you have lived in the United States less than half the year.

Note that as we experienced in the first chapter, however, we have had to negate the conditions.  Fortunately, I think they read better so I won’t belabor the point.

Chapter 4. Figuring and Claiming the EIC

You must meet one more rule to be eligible to claim the EIC.

The earned income limitations are similar to the adjusted gross income limitations in the first chapter in that they vary by filing status and the number of qualifying children.  This could be easily and appropriately coded in two lookup tables.  Not everything has to be a rule!

Without further ado, I give you the Earned Income Limits:

Then we can write that:

• You do not qualify for the EIC
  1. unless your income is less than your earned income limit
• Your earned income limit is the amount stored in the Earned Income Limits table for the number of your children that qualify and your filing status.

I would repeat this for adjusted gross income, but lo and behold – they are the same!

The EIC in a nutshell

The total logic of the earned income tax credit (excluding the relationship, age, and residency tests for a qualify child) could be expressed as:

• A person’s child qualifies
  1. Only if the child passes the qualification tests for the person
  2. Unless the child passes the qualification tests for another person
     1. Unless the two people file a joint return together
     2. Unless the other person agrees that the first person can claim the child as qualifying
     3. Unless the first person wins the tie breaker for the child versus the other person
• A person wins the tie-breaker for a child versus another person
  1. If the first person is the child’s parent
     1. Unless the other person is the child’s parent
        1. Unless the child lived with the first person longer
        2. If the child lived with each person for the same amount of time
           1. Unless the first person has the higher adjusted gross income
• You do not qualify for the EIC
  1. unless you have a valid social security number
  2. unless you are not married and filing separately
  3. unless you were a US citizen or resident alien all year
  4. unless you do not file form 2555 or 2555-EZ
  5. unless your investment income was $2,600 or less
  6. unless you have a qualifying child
     1. If you are under 25 or over 65 years old
     2. If you are a dependent of another person
     3. If you are a qualifying child of another person
     4. If you have lived in the United States less than half the year.
  7. unless your adjusted gross income is less than your earned income limit
  8. unless your earned income is less than your earned income limit
• Your earned income limit is the amount stored in the Earned Income Limits table for the number of your children that qualify and your filing status.

Qualifying Children

The relationship, age, and residency tests of the EIC are described in laborious detail on pages 11 to 14 of publication 596.  Here’s what the say, albeit with much more “effort”:

• A child passes the relationship test for you
  1. If the child is your son, daughter, adopted child, or stepchild or descendent of any of these.
  2. If the child is your foster child
     1. Only if the child is placed with you by an authorized placement agency
     2. Only if you care for the child as you would your own child
  3. Unless the child is married at the end of the year
     1. Unless you can claim as an exemption
        1. Unless you cannot claim the child’s exemption because you gave that right to another parent of the child
           1. If you completed form 8332 or a similar written statement
           2. If you did so in a written agreement prior to 1985
• A child passes the age test for you
  1. If the child was under 19 years old at the end of the year
  2. If the child was a full time student under 24 at the end of the year who lived with you in the United States for more than half the year.
  3. If the child was permanently and totally disabled at any time during the year.
• A child passes the residency test for you
  1. If the child lived with you in the United States for more than half the year.

And there are further details about what it means to be a full-time student or to reside in the United States (including provisions for US territories, military personnel stationed elsewhere, kidnapping, homelessness, and death).  There is even one universally applicable rule buried in the details:

• A child does not qualify unless the child has a valid social security number unless the child was born and died within the year.

If any reader has a question about how to handle the details within the EIC, please leave a comment or send email. 

I’ll finish up with one finer point.  The phrase “or descendent of any of these” is a bit complicated for the natural language processing logic in current BRMS.  In order to get the expressed logic clear enough that it can be automated with today’s technology, it would be better to rephrase as follows:

A child passes the relationship test for you

• A child passes the relationship test for you
  1. If the child is your legal child.
  2. If the child is a descendent of your legal child
• A child is your legal child
  1. if the child is your son, daughter, adopted child or stepchild.

For those who are familiar with Authority, you can see that these rules are essentially executable once the vocabulary and phraseology is defined.  Although SBVR is not as         expressive today, I hope it develops to be so capable, even if it takes Microsoft’s “embrace and extend” approach.

In his post, James agrees with the Butler Group’s statement:

Possibly the most important aspect of a rules repository, certainly in respect of the stated promise of BPM, Service Oriented Architecture (SOA), and BRMS, is the ability for the developer to re-use rules within multiple process deployments.

I have several problems with this statement:

1. I don’t think it’s the most important aspect.
2. I don’t think the possibility of re-use is being realized.
3. I  think current tools’ facilitation of re-use is limited.

The most important aspect of a repository is the reuse of knowledge, not just rules.  The types of knowledge that are most easily reused are:

1. the vocabulary (e.g., nouns, verbs and adjectives) and what they mean in the domain
2. the taxonomy of concepts and how they are refered to using (primarily) nouns and adjectives
3. the ontology of concepts, which is the taxonomy plus relationships between concepts, and how they are referenced using nouns, verbs and adjectives
4. the definitions of derived concepts in terms of formulas or equations or constraints that define the truth without respect to process, state, or event
5. the rules that specify behavior, regulation, governance, or policy

And their re-use decreases in the same order.  Industry specific standards, like ACORD or MISMO of FIX or XBRL, for example, stop at 2 or 3.  Logical formalism address 4 but are used more rarely than business rules.  The SBVR standard, although it stands for the semantics of business vocabulary and rules, actually stops at 4.  It’s rules are logical definitions and constraints, not business rules that take action, resulting in state transitions, data modifications, or other side-effects.

The use of SBVR or simply ontology is quite limited in today’s BRMS/BPM markets.  There is simply too little emphasis on separating semantic models and business vocabularies using logic and ontology.  Most BRMS/BPM platforms are fixated on behavior and process, rather than truth and knowledge. 

Sure, clever enterprise architects and systems analyts can engineer rules so that they are useful in more than one application, but this is not the typical result of a business rules application.  I do agree that encapsulating decisions within services is a more viable way of realizing re-use, but that is an architectural result (i.e., an SOA benefit), not a benefit of a rules repository.

I have blogged repeatedly (see the requirements category, for example) about the limitations of current tools with regard to vocabulary, separating ontology from implementation, and managing definitions and functional requirements in the natural form.  Current BRMS tools are generally limited to an if-then mentality that makes them inadequate for managing knowledge.  If-then metaphors force knowledge to be expressed and organized relative to how it is used.  This fundamentally and inherently limits its reusability.

Just to drive this home, if you are technically inclined, take a look at the standard developed by the vendors, PRR.  It will become immediately apparent that their focus is on rules at the operational rather than the knowledge level.  RIF, on the other hand, evolved from the logic community to address the requirements of the semantic web.  In this regard it is much more like SBVR.  It makes up for its vocabularly weaknesses with much more ontological and logical strength.  Nonetheless, RIF, like SBVR, is more at level 4 than level 5. 

Another way of looking at this is that the models more than the rules are interchangeable.  This misses that logical rules are very reusable, but it is in accordance with the 80/20 rule.  In BPM, for example, the model is shared across processes.  With decision management, the model should also be shared across decisions.  The easiest rules to share across decisions are those that are true, not those that determine what to do.  

Practically speaking, put rules that determine what to do in a services that share what is true. 

[ratings]

For example, just substitute rules in the historical distribution of requirements failures:^[1]

• 34% Incorrect requirements
• 24% Inadequate requirements
• 22% Ambiguous requirements
• 9% Inconsistent requirements
• 4% Poor scoping of requirements
• 4% Transcription errors in requirements
• 3% New or changing requirements

So the problem is not training analysts to capture rules but helping them understand that rules and requirements are two sides of the same coin, especially when focusing on the functional needs of clients^[2]. Of course, when we focus on functional requirements and rules in the modern age, we expect new or changing rules and requirements to lead to more failure.

Note that these percentages hold from the era of waterfall development. There have been improvements from iterative, spiral, and other rapid application development methodologies. RAD helps correct, expose and disambiguate requirements by exposing development obstacles or functional inadequacies earlier, but RAD does not improve the process of elicitation directly.

Today, especially concerning functional requirements and business rules, change is both pervasive and chronic while agility is critical. That is, maintaining correct, adequate, unambiguous, and consistent requirements has become a real-time challenge. Change occurs asynchronously and more rapidly than even the shortest plausible RAD iteration.

Fortunately, rule-based technology can address change directly – if the rules are accurate, adequate, and unambiguous^[3]. Regrettably, the same cannot yet be said for functional requirements (yet), as discussed here.

Still, key questions and points are apparent, particularly considering agility:

• How do incorrect rules or requirements persist?
• How can inadequacies be exposed in real-time?
• Can ambiguity be avoided in real-time?
• Can inconsistency be exposed in real-time?

In each case, if the answer is “not completely” or, worse, “not at all”, is there a coping strategy?

In the following sections, we examine correctness, adequacy, ambiguity, and consistency. As will become apparent, with some understanding and technique, in addition to use cases, eliciting rules and requirements can be accomplished much more effectively.

Correctness

How can an analyst know if a rule or requirement is correct? In some cases, it may be evident. And in some such evident cases, the analyst will nonetheless be mistaken. Nonetheless, the “self-evident” criterion is useful and generally reliable. But when an analyst is less than certain, how is correctness established?

The fact that incorrect rules or requirements persist into development or production in so many cases is evidence that correctness is typically established poorly. How does this happen?

The answer must be simple. It must be the case that too few knowledgeable people considered the requirement before it passed into development. Alternatively, perhaps some such requirements are too complex for human contemplation? Or perhaps the requirement was ambiguous, which we discuss separately below.

If an unambiguous requirement is incorrect, it must be either too complex or it must not have been considered enough by enough knowledgeable people. If too complex and considered by enough knowledgeable people, however, it would naturally be clarified and broken down. No matter how complex, if enough people understood it, failure would not be at the requirements stage but in implementation.

An unambiguous and yet incorrect requirement is almost certainly a result of inadequate collaboration. There are organizational and technical reasons that typically result in such poor collaboration.

Organizationally, the requirements process may flow from stakeholders and subject matter experts to application developers in a waterfall process. Literally, the organization expects the analyst to understand their needs and communicate them to others, not clarify them, seemingly ad infinitum, back to the stakeholders and SMEs. The organization resists investing their attention into iterative dialogue and review. The organization, perhaps unwittingly, seeks to minimize its up-front investment and ongoing commitment.

Technically, the tools or techniques used by analysts to capture and – more importantly – present rules and requirements back to stakeholders and SMEs alienate them. Ideally, the results of elicitation would be so clearly documented and presentable that reviewers could comfortably read and return marked up copy between sequential meetings or drafts. In reality, analysts tend to produce complex spreadsheets or documents with four or more levels and many cross references.

Such spreadsheets and documents are typically structured in a manner that is not intuitive to the reviewers. And the prose within such documents is typically composed with technical jargon or stilted prose rather than almost axiomatic statements expressed unambiguously in grammar familiar to authoritative and knowledgeable reviewers.

In practice, these organizational and technical realities result in documents that cannot be more than cursorily examined and excessive reliance on conversation and perception.

The result is the persistence of incorrect rules and requirements.

The cure is to ensure that the presentable form of captured requirements is easy to understand and can be reviewed without explanation.

This has many corollaries, of course! I will offer a few here:

1. Minimize the need to present or teach how to interpret what is to be reviewed.
2. Minimize cross-references between sentences, paragraphs, sections, etc.
3. Seek sign off at the level of sentences rather than on a section or chapter.

In addition to making elicited knowledge more perspicuous and subject to more collaborative review, every analyst knows that use cases are facilitate elicitation. Use cases not only assist with correcting rules or requirements, but identifying missing or inconsistent rules and requirements. So we cover the challenges of adequacy and consistency before discussing how use cases can be used more effectively. If you want to focus on this immediately, see my prior comments about requirements in this post.

Inadequacy

A collection of rules and requirements can be inadequate only if they are incomplete. That is, if all relevant rules and requirements are correctly understood, they cannot be collectively inadequate. So avoiding inadequacy is achieving coverage or completeness. Put another way, you will achieve adequacy when you are certain you know everything there is to know (at least “within scope”)!

Perhaps it is intuitively obvious that adequacy is something to seek but that we must cope with inadequacy. We cope with it by trying to identify it and minimize it, but we must also, at times, realize that we cannot resolve a situation at hand, and cope with that fact.

To reiterate:

• Try to identify incompleteness before moving to development or production
• Cope with incompleteness when it is identified, especially during production
• Once identified, seek the rule(s) or requirement(s) to eliminate incompleteness

But how can we “identify” incompleteness?

Two approaches come to mind:

1. Use an exhaustive, technical approach that covers EVERY possibility.
2. Determine when the rules and requirements produce no outcome for a use case.

In the first case, a technical approach is to use nested tables to map every possible combination of input variables to an outcome value. This is equivalent to a decision tree on a vector of singly-valued inputs.^[4] Alternatively, a spreadsheet metaphor does not feel as constraining but requires programmatic coverage analysis.^[5] In each case, expressiveness has been sacrificed in order to analytically ensure completeness.

If the stakeholders and SMEs agree that all their rules and functional requirements (e.g., business policies and regulations) are based on a finite collection of singly-valued variables, such authoring and analytic techniques may be effective. Unfortunately, doing so violates the previous guideline about presentation being effective without training. Nonetheless, for fairly simple, closed problems, such approaches may pan out.

The problem with starting out with a limiting metaphor is that it may become an obstacle during elicitation. Consider whether it would make sense to tell stakeholders that you were going to reduce everything that they told you to flowcharts for their review. Why would you be more comfortable telling them that you are going to reduce everything they tell you to a forest of decision trees or a catalog of decision tables?

I recommend that analysts keep using the tools they understand.^[6]

For the most part, this means Microsoft Word or, perhaps, a requirements management system, such as IBM’s Rational ReqPro, many of which use Word as the front-end to a collaborative, repository-based capability.

Unfortunately, Word does not support analysts in their work and RMS have not advanced to the point of business rules management systems.^[7] Specifically, RMS do not address the problems of ambiguity or transcription errors, such as by enforcing grammaticality using defined vocabulary as in BRMS. BRMS arguably support analyst tasks more effectively than Word or RMS, but they do handle only rules, not requirements!^[8]

The second alternative mentioned above for identifying inadequacy is to identify cases for which the rules and requirements gathered produce no result. As mentioned above, use cases effectively expose incorrect or inconsistent rules or requirements in addition to missing (i.e., inadequate) requirements.

The key in each case is to determine the outcome or outcomes, if any, indicated by the rules and requirements elicited so far. Ideally, this will be done well in advance of development or deployment into production.

Unfortunately, determining the outcome(s) for use cases is typically a mental exercise rather than automated. These mental exercises require comprehension and consideration hundreds or thousands of rules and functional requirements at once. They are clearly error prone. In many situations, their use is limited to the development phase, during which discrepancies between outcomes facilitate additional requirements or corrections.

In order for use cases to be more effective at eliciting more adequate and correct requirements, it is necessary for simulation to be more automatic than mental and to occur more before development or deployment than is typical. Simulation before programming is a capability of some BRMS^[9] but is not facilitated by Word or RMS.

BRMS can simulate use cases without requiring code to the extent that use cases and rules are “understood”. If the form they understand is presentable without explanation to stakeholders and SMEs, a BRMS may be an adequate RMS for use by analysts.^[10]

I don’t think the market is there yet, however. The market needs more natural language analysis applied to requirements, in particular. Specifically, the vocabulary and phraseology used in rules and requirements needs to be managed.^[11] Without understanding the meaning of the words used in rules or requirements it is clearly not possible to understand the sentences that use them. Beyond the meaning of words or phrases, more natural language analysis of clauses and sentences is required to discern the plausible logical interpretations of each captured rule or requirement.^[12]

Ambiguity

Most analysts do not apply natural language processing technology as discussed above to ensure that the rules or requirements they have captured are grammatical and use only defined vocabulary. Until they do, ambiguity will be extremely difficult to eliminate.

Microsoft Word may appear almost good enough here. It does a good job of flagging grammatical errors based on a grammar of English and knowledge of the parts of speech a large vocabulary. Unfortunately, Word’s grammar check cannot be limited to the domain of discourse relevant to a particular problem. You can’t stop someone from introducing requirements involving gorillas using Word. And Word won’t complain about grammatical nonsense, such as “He at piece of Pennsylvania”.

Analysts need (B)RMS tools that identify rules and requirements that are not grammatical or that use undefined vocabulary (including misspellings) or that use vocabulary or phrases within sentences that are not understood. Such sentences may or may not make sense. If they do make sense, the tool needs to acquire the meaning of that construct before it will be able to deduce the plausible meanings of sentences that use it.

The plausible meaning or logical interpretations of a sentence are what it might mean in a formal, rigorous, “interpretable” sense. Ambiguity arises when the sentence(s) that express a rule or requirement have more than one plausible interpretation. If an analyst has taken the first step of applying natural language analysis this second step will flag requirements that have zero or multiple plausible interpretations.

If a rule or requirement has zero plausible interpretations it cannot be simulated automatically. In practice, this may result in a delay in verifying its correctness, as discussed above, or in detecting inconsistencies, as will be discussed below.

If a rule or requirement has multiple plausible interpretations any or all of them can be simulated automatically using a logic or rules engines. Typically, this execution is limited to rules using a rules engine. Authority, for example, understands rules expressed in English, and will complain if they are grammatical using only defined vocabulary and phrasings, but ambiguous nonetheless.

The technology to extend Word-like grammar checking to Authority-like understanding may not be obvious but is nonetheless straightforward. This step will substantially improve requirements and modeling processes.

In the event that a rule is expressed in grammatical sentences that are unambiguously understood, Authority can generate and interpret rules using Haley’s Eclipse rule syntax and engine. By also capturing use cases in Authority, the correctness, adequacy and consistency of rules can be determined based on outcomes that are simulated using automation rather than mentally.

The technology to extend Authority-like simulation of rules to include requirements involves either more production rule generation or the use of logical theorem provers. These steps will substantially improve the requirements process by identifying incorrect, inadequate, and incomplete requirements before implementation.

RMS will eventually incorporate disambiguation but it will be longer before they incorporate automatic simulation. BRMS are better positions to move in this direction but are not adequate for direct use by analysts in capturing requirements at this time. In addition, there is too much coupling between the RMS capabilities of rules vendors and their proprietary business rules engines (BRE). Until BRMS vendors actually support (rather than simply endorse) emerging standards, my recommendation to separate RMS and BRE decisions will remain in force.

I truly hope that Ruleburst continues in the direction I set out in Authority and extends its natural language understanding to requirements in addition to rules. Regrettably, I did not have the opportunity to demonstrate the generation of logic or the use of theorem proving for requirements before they bought the assets of “my” former company. Therefore, I suspect that they will remain in just a BRMS and leave this opportunity.

Wouldn’t it be great if IBM did this for Rational ReqPro?

Inconsistency

In this article we are addressing the failures in eliciting requirements first cited above. Incorrect, inadequate, and ambiguous requirements were cited as 80% of the problem and we have discussed that the impact of changing functional requirements and rules was underestimated. According to the Pareto Principle, therefore, we might ignore inconsistency.^[13] However, the framework we have discussed above addresses inconsistency so directly that it can be tackled with little additional effort.

The challenge for an analyst during elicitation shifts from an initial emphasis on simply acquiring correct rules and requirements, where almost anything new is welcome, to focusing on inadequacies where there is enough content that fleshing it out rather than building it up takes priority. As the content containing rules and requirements becomes substantially correct and increasingly adequate use cases and simulation become increasingly important to identify residual errors and inadequacies, as well as inconsistencies.

As discussed above, relying exclusively on mental simulation results in more incorrect requirements and less adequate requirements entering development or production. Iterative development methodologies reduce but cannot eliminate the resulting increases in costs and time to market that risk projects and lower ROI.

Simulation without Implementation

Using automatic simulation of use cases, as first discussed above, the outcomes for each case can be determined by interpreting rules and functional requirements prior to development or deployment into production. For each simulated use case, the outcomes fall into one of the following:

1. No outcome is determined.
2. A single outcome is determined.
3. Multiple outcomes are determined.

For the purpose of discussion, assume that the business process at hand involves a number of decisions and that the expected or correct decisions for each use case are known by the business analyst or can be determined given the results of simulation by the stakeholders or SMEs.

In the first case, the absence of a decision, inconsistency is not the problem. An incorrect rule or functional requirement is possible but it is more likely that one is missing. In this case, simulation will facilitate elicitation.

In the second case, the outcome is either expected or unexpected. In either case, it may be correct or incorrect. If authorities determine, upon reflection given a use case, that an expected outcome is incorrect, one or more rules or requirements are usually incorrect. Otherwise, there is an inadequacy (i.e., an omission). Quite frequently, authorities determine that an unexpected outcome is in fact incorrect, typically arising from an incorrect or missing rule or requirement.

In either case, the ability to review the chain of reasoning that produced an errant decision for a use case with authorities is a very effective and focusing elicitation tool.

Regrettably, most analysts have little experience with such effective elicitation until the latter iterations of traditional software development processes. This is one of the early improvements that can be realized from augmenting analysts with tools to capture and simulate functional requirements and rules.

In the third case, where simulation produces multiple outcomes for a single decision given a use case, the outcomes are typically in conflict. That is, they are inconsistent and, by implication, rules or requirements must be inconsistent.

Theory versus Practice

Before diving in to the use of simulation and elicitation to avoid inconsistency, we should discuss the notion of resolving inconsistencies. Unlike avoidance, resolution implies tolerance for inconsistency. Resolving inconsistency is practical. Avoiding inconsistency, however desirable, is not – in general – practical.

Most commercial business rule engines (BRE) are based on production rules. Conflict resolution has been addressed since the early beginnings of this technology.^[14] Conflict resolution has since matured to a notion of deliberation in which decisions are considered before action is taken.^[15][16]

Avoiding inconsistent rules and requirements requires extreme and onerous precision in the specification of conditions and exceptions. Resolving inconsistencies typically involves some type of scoring function, which may be as simple as a priority, or preferences, which may be levels of stratification, pair-wise, or even higher level rules and requirements (sometimes called “meta” knowledge).

For example, if a customer qualifies for multiple loans from various lenders, clarifying conditions and exceptions such that only one loan from a single lender is selected would be impractical. Instead, meta-knowledge about the respective benefits of alternative loans might order specific recommendations or result in a single, high scoring recommendation.

Technical analysts tend to place far too much emphasis on reducing decisions to a single outcome without deliberation. There are requirements and then there are requirements. Or, some rules are made to be broken. OMG goes as far as to anticipate this with the notion of an enforcement level on requirements captured using its SBVR standard.

Inconsistency in Practice

Let us now return to the analyst given a use case for which simulation produces multiple outcomes for a single decision. Each of such outcomes can be reviewed as if it was the only outcome. That is, each outcome is either correct or incorrect. Any outcome that is incorrect would indicate incorrectness or inadequacy as previously discussed.

Perhaps, after eliminating all errant outcomes through further elicitation, there remains only a single correct outcome will be reached by subsequent simulation. If subsequent simulation results in no outcome, we are back to the first case discussed above, which may involve further incorrectness or inadequacy. Alternatively, if subsequent simulation still results in multiple outcomes, none of which is judged as objectively wrong, there may be heuristics or preferences rather than strict rules or requirements to be elicited.

Organization and Architecture for Agility

I have discussed previously that practical applications cannot avoid incomplete or inconsistent knowledge. This is further discussed and reflected in the references on conflict resolution, deliberation, and stratification given in the footnotes cited above. It also needs to be reflected in an agile, iterative organizational process and in the architecture of information systems.

Take it is a given that a new, modified, or missing rule or functional requirement will arise after a system is in production. Take it as a given that a case will arise during production use for which no outcome, an errant outcome, or multiple, possibly conflicting outcomes result given correct automation of rules and requirements reasonably deployed. What are the risks and the processes of improvement?

Rules or requirements mapped into code typically result in action without deliberation. That is, if multiple outcomes might be indicated, only the first will result. In other words, the implementation of rules and requirements is typically incapable of detecting inconsistent requirements.

By using an engine to interpret requirements deliberatively in an architecture that logs outcomes, including multiple potential outcomes, elicitation of the knowledge that will avoid such inconsistencies in the future becomes possible. In cases that produce no outcome, elicitation of the additional knowledge necessary (including clarifications of incorrect knowledge) can also be facilitated. And by auditing cases for which single outcomes are reached, discovery of additional or errant knowledge can be accomplished at a controlled level of effort.

The ability to facilitate elicitation has development and organizational implementations.

To the extent that newly elicited rules and requirements can be automated without programming, as is commonly the case for rules using BRMS, elicitation post production becomes more viable economically.

In addition, the auditable use of rules and requirements can be combined with performance metrics to analyze and tune or learn rules, requirements, heuristics and preferences, some of which may produce enough insight to lead to new business models.

Such architecture effectively becomes a source of use cases but the organization has to remain engaged in the analytic and elicitation process after the rules and requirements have gone into production to receive a business benefit beyond the development impact.

^[9] one of the advances we made circa 2000 at Haley and since by Ilog, among others

^[10] Unfortunately, as discussed previously here, BRMS do not handle requirements (yet).

^[11] OMG’s Semantics of Business Vocabulary and Rules (SBVR) recognizes this but, excluding Haley, I am not aware of any (B)RMS that manages the vocabulary and phraseology used in rules or requirements and their formal, ontological meaning.

^[12] Ruleburst still presents a white paper on how we did this at Haley for rules here.

^[13] i.e., the 80/20 rule

^[14] Production system conflict resolution strategies

^[15] Soar: A Comparison with Rule-based Systems or Cognitive Theory, SOAR

^[16] there have been similar pragmatic developments in the logic community, as in stratification and courteous logic, albeit without tolerance of inconsistent logic

Consider these words from Forrester:

One might argue that Word documents, email, phone calls, and stakeholder meetings alone are adequate for managing rules. In fact, that is the methodology currently used for most projects in a large number of IT shops. However, this informal, ad hoc approach doesn’t ensure rigorous rules definition that is communicated and understood by all parties. More importantly, it doesn’t lend itself to managing the inevitable rules changes that will occur throughout the life of the project. The goal must be to embrace and manage change, not to prevent it. ^[2]

But note that Forrester used the word “requirements” everywhere I used “rules” above!

The fact is, BRMS vendors target business analysts, promising that they can capture and maintain rules externalized from application infrastructure. But BRMS vendors do not support capturing and maintaining requirements.

A better vision would unify rules with requirements in a single framework of understanding and within a single management tool. Otherwise, rules will remain downstream from functional requirements and use cases in the development process. Consequently, business rule management systems (BRMS) would always be at least one step removed from business analysis – and, therefore, end users.^[3]

Until BRMS vendors do support requirements, business analysts will continue to use word processors, spreadsheets, or requirements management software. And for the most part, they will do so instead of – rather than in addition to – using a BRMS.

There are many thoughts on how to elicit business rules and requirements.^{[4][5][6][7][8]} 

Surprisingly, there is substantial confusion on the relationships between rules and requirements.

For example, some have written: ^[9]

1. rules are not requirements^[10]
2. rules cannot be managed using requirements tools
3. rules must reference requirements which must not reference rules

A distinction should be made between functional and non-functional requirements (as in ISO 9126^[11] and according to NASA^[12]). When we use rules we are most concerned with functional requirements.^[13] The positions stated above seem to reflect a perspective: that rules are an implementation of functional requirements.^[14] For example, that rules exist to clarify how to comply with requirements or how to behave when requirements are not satisfied (e.g., during a business process^[15]).

In order to realize their marketing promise, BRMS need to advance to include direct support for functional requirements. Ideally, this support will eliminate the need to translate such requirements into rules. At a minimum it should support mapping back and forth between related rules and requirements, perhaps by supporting the OMG mappings between concepts like means (e.g., directives, including policies and rules) and ends (e.g., objectives), as in the Business Motivation Model.^[16]

Supporting relationships between rules and requirements should not require relationships between them, however. Sometimes a policy may exist without a clear underlying requirement. Although theoretically, perhaps, there should be some underlying requirement for every rule, uncovering it may not be productive. Moreover, as discussed below, the distinction between rules and requirements is artificial in many cases.

When integrating rules and requirements into a single framework, care should be taken to support all the natural results of business analysis. In addition to rules and requirements, business analysts also uncover definitions of vocabulary and underlying concepts.

Vocabulary definitions are mostly ontological^[17], such as mapping nouns onto taxonomic concepts or defining what it means when an adjective is used with a noun. This is why I designed Haley Systems’ Authority to manage such vocabulary and ontologies. Regrettably, the software did not yet eliminate the need to translate requirements into rules before it was acquired by Ruleburst. With it, Ruleburst is ahead of other vendors who do not manage ontologies and vocabularies explicitly, but there is still a problem.^[18]

Current BRMS technology is not able to implement requirements directly. Current BRMS need a human to analyze a requirement and translate it into sometimes many rules that use the words “if” and “then” instead of words like “must” that are so common in requirements. This leads rule technologists to say things like, “a rule may reference a requirement” but “a requirement is not a rule”, as cited above.

One would think that OMG standards in rules could be supported by the BRMS vendors who contributed to them. Surprisingly, what SBVR^[19]calls a rule is beyond the capabilities of current BRMS (as also discussed here). For example, most rules technologists would have trouble “coding” the SBVR rule: “each rental car always has exactly one vehicle identification number”. The rules technologist might say that this was a definition, not a rule or requirement.

Business analysts, such as those involved at OMG, are alienated by such discrimination.

And they are right to be alienated. What if a service request provides two VINs? Or zero? Does the statement change from a definition to a requirement or some rules?

Vendor support for requirements (or simply SBVR) rather than just rules involves – at a minimum – generating rules that invoke an exception mechanism when a requirement is violated (as also discussed here). Generating rules that avoid premature exceptions is a significant challenge, however, particularly when information may be missing (including when it is accumulated over time, such as across multiple submissions of web forms).

For example, given the requirement that “an ad on the back page of the New York Times Magazine must be full-page color”, it may be preferable to deduce the (permitted) color or size of a back page ad rather than indicate an exception if the size or color is unspecified. Today’s BRMS do not assist users with such rules nor do they relate them to such requirements. All such translations and any such mappings are strictly manual.

The looming battle in RMS involves the clarification and automated implementation of requirements and less codification of rules.

As you might surmise, I find this area presents some very interesting opportunity…

The line between requirements and rules needs to be clarified. Perhaps it needs to be erased. Consider that not every requirement must use the word “must”. During analysis, for example, a business concept might be similar to the concept of an orphan. A business analyst could capture this as any of “an orphan has no parents”, “a person without parents is an orphan”, “if a person has no parents then the person is an orphan”, “a person is an orphan if the person has no parents”, or “an orphan must have no parents”, among other alternatives. Some of these appear to be definitions, some requirements, and some rules. Yet they are all the same from the standpoint of stakeholders and less technical analysts.

Should software help clarify and unify such typical results of business analysis or should the burden be borne by humans in a waterfall process that distances natural expression of “requirements” from technical definitions and implementing “rules”?

Obviously, I believe the former.

In addition to previous recommendations to manage rules in a BRE-independent manner, I hope the following provides additional help in planning for the integration of requirements, definitions, and rules as the market evolves:

1. Contemplate the extent to which your rule analysts are distinct from your requirements analysts and discern why if that difference is significant.
2. Adopt methodologies that are consistent with best practices in requirements capture, analysis, and management rather than over-emphasizing rule-specific approaches.
3. Understand where rule/requirement convergence and vendor independence is headed. Specifically consider SBVR (e.g., look at the video or document here). To a lesser degree consider PRR and semantic web activity.
4. Consider managing rules and requirements outside a BRMS using requirements management software or as discussed here. Also consider the functionality of requirements management software (e.g., Rational) or BRMS that is not provided by an unassisted word processor or spreadsheet (e.g., as Ruleburst still presents here).

1. Business Rules Management Systems (BRMS) typically produce forward chaining production rules that are interpreted by^[1] a business rules engine (BRE) based on the Rete Algorithm.
2. BRMS typically generate rules that are interpreted by a BRE.

First, dropping the word “production” before “rules” loses information. BRMS do not typically generate rules that are not production rules. Consider, for example, the BRMS vendors involved in the OMG effort produced the Production Rule Representation (PRR) standard. The obvious question is:

• What is different about production rules?

Second, dropping the words “based on the Rete Algorithm” loses information. The dominant rules vendors and open-source engines are all based on the Rete Algorithm.

• Why does the Rete Algorithm matter?

Third, dropping the word “chaining” before “rules” loses information. Chaining refers to the sequential application of rules, as in a chain where each link is the application of one rule and links are tied together by their interaction. But:

• Why does chaining matter?

Fourth, dropping the word “forward” before “chaining” loses information. Forward chaining reacts to information without requiring goals. This begs the question:

• Don’t goals matter?

What is different about production rules?

In a system where rules are not productions, new knowledge must be positioned within the existing knowledge. In the worst case, the cost of adding a new rule is proportional to the existing number of rules. That is, the cost of acquiring new knowledge increases as knowledge accumulates. The ability to improve at constant cost is the basis of agility that comes from using production rules to implement business rules.

The term comes from automata theory, in which the rules that define a formal language are called “productions”. The important thing about productions is that the formal language they define does not depend on their order.

For example, changing the order of rules expressed in Backus Normal Form (BNF) will not change the definition of a language (e.g., Pascal). This is distinct from “if-then-else” statements in procedural languages where order is clearly relevant.

Also consider that the order of logical formula does not matter to a theorem prover.^[2]

Procedural rules (including if-then-else statements in procedural languages and decision trees) are less agile. The effort required to codify them increases as the amount of pre-existing code increases.

The irrelevance of order is the critical ingredient of agility.

Focus on truth. Minimize context.

Why does the Rete Algorithm matter?

The agility of production rules would be less practical if performance degraded as more rules were added. For a practical business rule system, performance must scale less than linearly with respect to the number of rules. That is, if doubling the number of rules always cut performance in half, the applicability of rules might be limited. Ideally, adding a rule would have zero impact on performance.

I’ve written about these topics many times^[3] and there is plenty available on the Web^[4].

Essentially, in Rete, the conditions or antecedents of rules correspond to SQL queries. Rather than check rules by running a sequence of queries against data, however, Rete compiles queries into a pattern matching network through which data flows. Performance of the first approach would clearly be linear in the number of rules (queries). The worst case performance of Rete can also degrade linearly as rules are added, but the expected performance is less since Rete effectively ignores conditions that are not satisfied by data.

Decades or experiments and commercial use have demonstrated that Rete typically scales up with performance that degrades far less than linearly. Here is the technically accurate statement:

The expected performance of the Rete Algorithm is asymptotically independent of the number of rules.

In practice, adding a hundredth or ten thousandth rule will have immeasurable impact.

Rete matters only because it implements unordered rules in an efficient and scalable manner. If you can implement rules without ordering them some other way, great – but consider chaining, too.

Why does chaining matter?

Chaining refers to the sequential application of rules, as in a chain where each link is the application of one rule and links are tied together by their interaction.

Without chaining, inference is impractical.

Chaining is a key distinction between implementing rules using an engine versus in procedural code. A rules engine can string inferences together to reach conclusions involving several steps, where each step is expressed in a rule. Procedural languages provide no support for reaching intermediate conclusions and building further upon them.

Without chaining, a system cannot be intelligent.

Chaining is your greatest source of leverage for semantics and intelligence. By writing statements^[5] that are true you can leverage deduction. Doing so makes all your statements less verbose and more reliable.

Don’t Goals Matter?

Backward chaining involves working backwards from goals to the current state of the world. Therefore, backward chaining requires a goal. Problem solving involves goals.^[6]

Forward chaining involves working forward from the current state of the world. Forward chaining does not require goals. Forward chaining is behavioral and – without goals – strictly reactive.

Unlike artificial intelligence and logical rule systems, PRR and all but one BRE based on the Rete Algorithm are limited to forward chaining.

Forward chaining systems are inherently limited to “if” and “then”. When the “if” is satisfied they want to conclude or execute what comes after the “then”. Which “ifs” are satisfied, is determined or driven by data, of course.

Unfortunately, the data-driven aspect of forward chaining is often confused with the data-driven nature of the Rete Algorithm. The fact that the Rete Algorithm is data driven has nothing to do with whether it can support backward chaining.^[7] The lack of support for goals in the business rules market is the issue, not the Rete Algorithm.

I have previously supported backward chaining using the Rete Algorithm. The details are available on-line^[8], from Haley Systems, or by request. More vendors need to step up their support in this area. The open-source community is actually ahead of most!

Support for backward chaining will allow business to use more logical deduction without as much need to indicate when to perform or how to control inference. Subgoaling will focus inference. This will encourage and facilitate increasingly large and sustainable repositories of knowledge.

Even with improved deductive capabilities using backward chaining, BRMS will remain inadequate for requirements, however.

Why can’t BRMS handle requirements?

In order to support emerging standards from OMG^[9] and W3C^[10], BRMS need to handle logic that is not limited to the “if-then” or “A implies B” form of rules. Such statements include definitions and constraints^[11]; or, more generally, requirements.

Requirements do not fit the “if then” paradigm of today’s business rules systems. They do not pose any challenge to logic systems, however. Current rules technology requires an antecedent. A formula in the predicate calculus does not.^[12]

This results in all sorts of distortions in the business rules market. Worst among them, in my opinion, is the distinction between requirements management and rules management on which I will post shortly.