{"id":102,"date":"2008-05-08T12:24:25","date_gmt":"2008-05-08T17:24:25","guid":{"rendered":"http:\/\/haleyai.com\/wordpress\/2008\/05\/08\/probabilities-are-better-than-scores\/"},"modified":"2018-07-23T09:26:33","modified_gmt":"2018-07-23T14:26:33","slug":"probabilities-are-better-than-scores","status":"publish","type":"post","link":"http:\/\/haleyai.com\/wordpress\/2008\/05\/08\/probabilities-are-better-than-scores\/","title":{"rendered":"Probabilities are Better than Scores"},"content":{"rendered":"<p><a href=\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/2008\/05\/strategicanalytics2007mortgagemeltdown.jpg\" title=\"Strategic Analytics slide from Fair Isaac Interact on 2007 mortgage meltdown\"><img loading=\"lazy\" decoding=\"async\" align=\"right\" width=\"372\" src=\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/2008\/05\/strategicanalytics2007mortgagemeltdown.jpg\" alt=\"Strategic Analytics slide from Fair Isaac Interact on 2007 mortgage meltdown\" height=\"271\" style=\"width: 305px; height: 244px\" \/><\/a>During a panel at Fair Isaac&#8217;s Interact conference last week, a banker from Abbey National in the UK suggested that part of the credit crunch was due to the use of the FICO score.\u00a0 Unlike other panelists, who were former Fair Isaac employees, this gentleman was formerly of Experian!\u00a0 So there was perhaps some friendly rivalry, but his point was a good one.\u00a0 He cited an earlier\u00a0<a target=\"_blank\" href=\"http:\/\/www.fairisaac.com\/fic\/templates\/myinteract\/myinteract_streamfile.aspx?file=SF08_riskmanage\\The_Mortgage_Crisis-Implications_for_a_Global_Economy.pdf\">presentation <\/a>by the <a href=\"http:\/\/www.strategicanalytics.com\/company_biobreeden.php\">founder<\/a> of Strategic Analytics that touched on the divergence between FICO scores and the probability of default.\u00a0 The panelist&#8217;s key point was that some part of the mortgage crisis could be blamed on credit scores, a point that was first raised in the media last fall.<\/p>\n<h3><strong>The FICO score is not a probability.\u00a0 <\/strong><\/h3>\n<p>Fair Isaac people describe the FICO score as a ranking of creditworthiness.\u00a0 And banks rely on the FICO score for pricing and qualification for mortgages.\u00a0 The ratio of the loan to value is also critical, but for any two applicants seeking a loan with the same LTV, the one with the better FICO score is more likely to qualify and receive the better price.<\/p>\n<p>Ideally, a bank&#8217;s pricing and qualification criteria would accurately reflect the likelihood of default.\u00a0 The mortgage crisis demonstrates that their assessment, expressed with the FICO score, was wrong. \u00a0Their probabilities were off.<!--more--><\/p>\n<ul type=\"disc\">\n<li>Was the FICO score a useful metric of creditworthiness before the crisis but not during?<\/li>\n<li>Is the FICO score a reliable metric going forward?<\/li>\n<\/ul>\n<p>In these mid-crunch days, Fair Isaac is reminding its customers that the FICO score is a ranking not a probability.\u00a0 The underlying point they seek to make is that the relationship between the FICO score and the probability of default is more complex and dynamic than their banking customers understood last year.\u00a0 (Another <a href=\"http:\/\/haleyai.com\/wordpress\/2008\/04\/29\/super-crunchers-predictive-analytics-is-not-enough\/\">post<\/a>\u00a0on predictive analytics also discussed stationarity.)<\/p>\n<h3><strong>It&#8217;s the probability that matters, not the score!<\/strong><\/h3>\n<p><a href=\"http:\/\/haleyai.com\/wordpress\/2008\/04\/29\/super-crunchers-predictive-analytics-is-not-enough\/\">In his keynote, Ian Ayres<\/a> also focused on the inadequacy of scores.<\/p>\n<blockquote><p>He was explicit that bankers need the probability of default and, further, that they need to know how reliable such probabilities are.\u00a0 As an example, he cited polls where one candidate is leading by 6 points within a margin of error of 3 points as almost meaningless. \u00a0More meaningful would be the probability that the leading candidate will win.\u00a0 Even better would be an estimate of the probability of default along with an assessment of the reliability or accuracy of that probability.<\/p><\/blockquote>\n<p>Bankers increasingly understand that the FICO score is not the probability of default that they need when originating and underwriting credit.\u00a0 As a result, bankers increasingly understand that there is no adequate external source of the probabilities they need in order to optimize their portfolio performance.\u00a0<\/p>\n<p>This realization has several ramifications:<\/p>\n<ul type=\"disc\">\n<li>A market opportunity for predictive analytics in credit has opened on Fair Isaac&#8217;s turf.<\/li>\n<li>Scorecards have lost much of their &#8220;solutions&#8221; luster, becoming just another technique.<\/li>\n<\/ul>\n<p>But several things also became clear as I talked with numerous practitioners last week.\u00a0 Fair Isaac doesn&#8217;t have much competition.\u00a0 In fact, it is shocking how little competition they have in such a large and lucrative market.\u00a0<\/p>\n<h3><strong>What decisioning market?<\/strong><\/h3>\n<p>Although there is a market opportunity for more rigorous decisioning solutions, there is no significant challenger to Fair Isaac. \u00a0I expected to hear more about the <a target=\"_blank\" href=\"http:\/\/www.experiangroup.com\/\">Experian Group<\/a>, but the only direct competitor identified by more than one person was <a target=\"_blank\" href=\"http:\/\/www.austinlogistics.com\/\">Austin Logistics<\/a>. \u00a0Several people indicated that they were using statistical tools directly, especially SAS, and Fair Isaac itself is placing a great deal of emphasis on its own predictive analytic tools, especially <a target=\"_blank\" href=\"http:\/\/www.fairisaac.com\/fic\/en\/product-service\/product-index\/model-builder\/\">Model Builder<\/a>.<\/p>\n<p>Note that <a target=\"_blank\" href=\"http:\/\/www.vantagescore.com\/about\">Vantage Score<\/a> is really having an impact on Fair Isaac scoring revenues, as reflected in their most recent <a href=\"http:\/\/seekingalpha.com\/article\/74489-fair-isaac-f2q08-qtr-end-3-31-08-earnings-call-transcript?page=9\">earnings call transcript<\/a>.\u00a0 So there is more competition than may seem apparent to the audience that attends Interact.<\/p>\n<h3><strong>Another chasm to cross<\/strong><\/h3>\n<p>Generally speaking, this market needs the benefits of broader machine learning techniques, such as statistics, and a more rigorous understanding and emphasis on probabilities.\u00a0 The audience, however, is not technically sophisticated enough to become aggressive adopters, despite recent harsh lessons.\u00a0 In the same panel, every banker in turn solicited risk analysts to join their organizations, headquartered in Asia-Pacific, London, Canada, and on the west coast.\u00a0 They also agreed that it is easier to learn finance than analytics.<\/p>\n<p>The market for analytics is crowded with sophisticated tools and intellectually demanding techniques that are simply too hard for most people to understand and use, let alone to use effectively and reliably.\u00a0 This is precisely the circumstances that decision management was in during the late nineties when business rules technology started going mainstream.\u00a0 In 2000, we crossed that chasm by introducing natural language business rules (see Haley&#8217;s Authority).\u00a0 At the same time, Blaze Advisor, now owned by Fair Isaac, was crossing that chasm using a form based approach called &#8220;Innovator&#8221;.<\/p>\n<p>Similar advances will be forthcoming in analytics.\u00a0 As with business rules, this will not eliminate the need for highly skilled consultants, but their criticality and marginal value will diminish as analytics becomes more effective in the hands of non-experts (and as better solutions develop in key markets, such as in credit, risk, fraud and other criminality or terrorism).<\/p>\n<h3><strong>Until it&#8217;s easy, use expertise<\/strong><\/h3>\n<p>If you are in this market and could use some help with modeling, analytics, or <a href=\"http:\/\/haleyai.com\/wordpress\/2008\/04\/15\/adaptive-decision-management-for-business-performance-management\/\">adaptive decision management<\/a>, feel free to get in touch.\u00a0 We have some excellent capabilities and partners in these areas.\u00a0 We are happy to help recommend approaches or products, or simply to make referrals.\u00a0 Of course, there are also highly specialized consultancies, such as <a target=\"_blank\" href=\"http:\/\/www.strategicanalytics.com\/articles.php\">Strategy Analytics<\/a> that can give excellent implementation-agnostic advice.\u00a0<\/p>\n<p>One thing worth noting, but only in passing for now, is Fair Isaac&#8217;s <a target=\"_blank\" href=\"http:\/\/www.fairisaac.com\/NR\/exeres\/AAF3DD63-ABE9-4FA7-B432-91E5B671224E,frameless.htm\">acquisition<\/a> of Dash Optimization.\u00a0 This\u00a0 reflects the increasing trend towards broader and deeper application of technology within credit decisioning.\u00a0 it is also a response to the decline of scoring and the increasing need for decision optimization, which is a broader subject than decision management, with or without predictive analytics and adaptation.\u00a0<\/p>\n<blockquote><p>Nonetheless, optimizing portfolios will not optimize profits if the scores used are not reliably correlated with probabilities.<\/p><\/blockquote>\n<p>It is also interesting how Ilog and Fair Isaac continue to converge from a technological perspective.\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"<p>During a panel at Fair Isaac&#8217;s Interact conference last week, a banker from Abbey National in the UK suggested that part of the credit crunch was due to the use of the FICO score.\u00a0 Unlike other panelists, who were former Fair Isaac employees, this gentleman was formerly of Experian!\u00a0 So there was perhaps some friendly &hellip; <\/p>\n<p class=\"link-more\"><a href=\"http:\/\/haleyai.com\/wordpress\/2008\/05\/08\/probabilities-are-better-than-scores\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Probabilities are Better than Scores&#8221;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,18],"tags":[],"class_list":["post-102","post","type-post","status-publish","format-standard","hentry","category-decision-management","category-predictive-analytics"],"_links":{"self":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/102","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/comments?post=102"}],"version-history":[{"count":1,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/102\/revisions"}],"predecessor-version":[{"id":1542,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/102\/revisions\/1542"}],"wp:attachment":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/media?parent=102"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/categories?post=102"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/tags?post=102"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}