Comments on: Probabilities are Better than Scores

By: snshor

snshor — Sun, 11 May 2008 17:23:28 +0000

I think FICO score still proved as areliable tool in measuring relative probabilities. Another thing, the crisis started in subprime mortgages - by definition the sector with a higher probability of default. IMHO, considering the width of the problem, it is not the model’s fault, it is the much more general systemic fault of knowing the limitations of the model by the people responsible for the decisions. As is the case with Newtonian physics, for example.

The statistical models are vulnerable to the “black swan” events as a very timely Nasim Taleb’s book has pointed out. It is a bit ironic, that the people putting “past performance” clause into all the fine print, could not apply the same principle to the statistical models that always look at the back mirror. It is too much to expect that FICO score, solely based on individual’s credit behaviour, could predict housing market meltdown, that happened a few years after the most of the subjects happily granted and received their motgages, packaged it and sold to investors as the “the bulletproof” instrument. Last year Goldman and otehr were explaining their losses, as caused by 25-sigma event. Anyone with a slightest bit of statistical knowledge can judge this statement, as being either admission of using the low-touch-with-reality model, or even worse, not understanding it.

By: paul@haleyAI.com

paul@haleyAI.com — Fri, 09 May 2008 12:46:03 +0000

I would not disagree with you, James, that the FICO score was not intended as a probability, but practically speaking, that is how it has been used. Dr. Breeden’s presentation specifically addresses how lenders do not account for the variation between FICO scores and probabilities through interest rate and real estate cycles. My thesis here is not that scores are useless but that probabilities are better and more direct.

Yep, Vantage is a nasty, effective move by the credit agencies against Fair Isaac’s prior monopoly. I feel for Fair Isaac on that one. But Dr. Greene, FICO CEO, explicitly acknowledged the continuing negative impact of Vantage on revenues.

So, I think my facts are in order, I’m afraid. But I truly understand how upsetting the recent wounds from sloppy lending practices and unfair competition are to those who have enjoyed working with Fair Isaac.

As for Ilog’s announcement, I was going to blog on that today! I’ll pass now that you’ve covered it.

Respectfully, Paul

By: jamet123

jamet123 — Fri, 09 May 2008 06:27:48 +0000

Paul
Normally I find your posts very accurate but today, I fear, I have to take issue:
- The FICO score is not, never was and was never said to me a measure of the likelihood of default on a mortgage. Any lender who took it as such is probably beyond help - they would have taken any other analytic measure and misused it too.
- Scorecards are a technique for PRESENTING an analytic that is defensible with regulators and easy to explain. Scores can be built using any and all analytic techniques and often are. Lots of other techniques, like Neural Nets, can create scores. What they struggle to do is explain them. Scorecards get used because they can be used to explain, not because of any particular predictive power. This has not changed in the current meltdown.
- Vantage is impacting scoring revenues because the bureaus are practically giving it away, necessary to promote adoption, and the lenders can obviously use this to create price pressure on Fair Isaac. I don’t know but I suspect that every Vantage user is also using the FICO score. Competition, yes, but just another version of the same kind of model created to try and capture some of Fair Isaac’s revenue not to create any new kind of score.

I do agree that there is a huge gap in the analytics world figuring out how to bring non-technical (in this case non-statistical) people into the process to enable effective collaboration. The rules community has done a lot here, the process community a little, the analytic community almost nothing. This needs to change.

I actually started the comment simply to point out that ILOG’s recent release of a scorecard modeler (blogged about here) is another sign of convergence but you had made some unusually sloppy statements that I felt needed correction.