 |
Alexander Samuel, VP
Capital Markets Group
|
|
A World of Models
Wednesday, July 07th, 2010.
The world is inundated with models in every realm of life. In the physical sciences, models operate within a controlled environment where an experiment can be repeated over and over again. Models can be built to simulate an experiment and can be perfected to significant precision. However, building models in the social sciences to simulate an experiment presents additional challenges. Given the same scenario over and over again, human beings will not necessarily act in the same manner. As a result, modeling that involves elements of human behavior (such as prepayment and default) is quite difficult, and testing these models is even more difficult. Despite these challenges, modeling and model testing have a prominent place in real world applications from measuring risk to asset valuation to financial forecasting.
The mortgage world is commingled with many elements of human behavior which makes mortgage analytics very challenging but also highly useful. Not just the mortgage world but the whole financial system is driven by models. Models are used everywhere from forecasting defaults to estimating counterparty risk, valuing complex derivates, and analyzing a variety of risks. They are critical in the decision-making and risk assumption processes inherent in our financial system. Models that are not appropriately tested can bring the entire financial system down. As they say, an untested model is nothing more than a mere conjecture. Although models pertaining to social science have many obvious challenges that arise because human behavior is involved, model testing plays a vital role in the ensuring the usefulness of these models in the financial world.
Risk Evaluation Needs Useful Models
Model testing should help the management of financial enterprises avoid making very risky decisions that have the potential to put a company’s survival in jeopardy. Model testing should help the company by revealing whether the model is right (and the right model is being used) to measure risk appropriately.
Since 2008, more than 200 banks have failed. Additionally, many other banks and financial services firms received funding from the government during the same period in order to avoid a similar fate. In addition, the Federal Reserve and other bank regulators embarked on a comprehensive assessment of the capital held by the 19 largest U.S banks and ordered 10 of them to raise a total of $75 billion in extra capital (Treasury’s Supervisory Capital Assessment Program, or “SCAP”, completed in the spring of 2009).
What is the reason for these failures? Is it because these institutions failed to appropriately measure the risk of their highly levered investments? Is it because they relied on unregulated investments wrought with enormous risk to drive growth? Is it because some large banks used leverage of 20 to 1 or 30 to 1 to invest in subprime loans that were packaged into mortgage-backed securities? Or is it because they were speculating? Even if they were speculating, wouldn’t it still be prudent to measure the risk exposure under a stressed scenario? Why did these large banks and financial firms fail to stress test their liquidity risk model? Failure to understand the stressed scenario, in a highly leveraged world, results in significant exposure, and depending on the magnitude of the risks taken, it can affect not only a firm’s balance sheet but, as we’ve seen, the stability of financial markets as a whole. Hopefully, after enduring this long and painful credit crisis, the financial world has a new appreciation for the importance of stress testing as a prudent risk management practice.
These disasters may have been avoided or partially avoided had there been an adequate level of stress testing enabling a better understanding of the risk exposure in a stressed scenario. But to have good stress testing, valid models must exist to perform the stress testing correctly. On the other hand, some may argue that models are not robust enough to yield reliable results under stressed scenarios. As the OCC said in their Model Validation of Credit Risk Rating Models Conference in February 2006, “All models are wrong but some models can be useful.” Said differently, all model outputs have error, yet the best models are those that minimize this error.
The Purpose of Model Testing
Model testing, then, is designed to make the errors as small as possible, thereby making the model more useful and reliable. What does model testing entail? Model testing consists of two components:
- Model Verification: A process to ensure that the model is right
- Model Validation: A process to ensure that it is the right model
Although model verification and model validation are both part of the model development process, the second component, “model validation”, is an ongoing, interactive process. Model validation is a process to make sure that the model does not become stale over time, because the right model today may not be the right model tomorrow. Effective models continue to evolve in the market place as regulations change or as the market itself evolves. MIAC views the model verification and model validation process as the key to continued innovation in all of its tools. It is the first and foremost way of ensuring that MIAC continues to offer the right models to our clients. As such, MIAC looks both internally and externally at the results of the model validation process around each of its tools as a way of establishing development priorities and delivering the right models to its clients and to the broader market.
What is Model Verification?
Model verification is a process to ensure that the model is right. In other words, model verification ensures that the model correctly simulates the process it is intended to simulate under a variety of scenarios, both typical and atypical. This is accomplished through verifying the following:
- Is the model mis-specified?
- Is the model programmed correctly?
- Is the model implemented correctly?
Model verification is not a perfect science and there is no such thing as a model that is completely verified, especially one that involves human behavior. However, as more and more cases are tested, the degree of certainty improves. In other words, through repeating the model verification process, model errors are discovered and corrected, and we thereby improve the reliability of the model. This evolutionary process results in making the model as accurate as possible.
What is Model Validation?
Model validation is a process to ensure that the right model is used. In other words, it demonstrates that the chosen model reasonably represents the actual real world process. This is done through testing three main components and reviewing five ancillary components. The testing of the main components is done to reduce the model risk; and the reviewing of the ancillary components is done to reduce operational risk. The three main components to be tested are:
- Model Data and Assumptions: To ensure that the data is logical and accurate and that the assumptions used are sound. The soundness of the assumptions can be gauged through back-testing, benchmarking and/or through the use of various statistical measures.
- Model Parameters: To monitor the sensitivity and the volatility of the parameters and update the parameters as additional empirical data becomes available.
- Model Output: To ensure the results are reasonable and reliable through back-testing, benchmarking and/or through the use of various statistical measures. In addition, stress testing, sensitivity analysis and scenario analysis must be performed to understand the complete spectrum of the risk exposure.
The ancillary components to be reviewed include:
- Model Governance: Model Governance should be well balanced. A very extensive program can be as bad as (or could be worse than) a program that barely meets some minimum requirement of functionality. Among other things, model governance should provide guidance on:
- What is a reasonable definition of a model that requires model validation?
- How to measure the model risk?
- How often to perform model re-validation?
- What is the minimum level of independence required of the validator?
- At what point can a model go into production?
- What is the retention policy for inputs, reports, and documentation?
- Model documentation: This should contain:
- A technical manual that describes the underlying mathematical theory behind the model, including its strengths and weakness.
- A user manual that describes how the model is used by the end user.
- Model inventory: This is a catalog of all models used enterprise-wide, providing, among other things, the following information:
- The purpose of the model
- The date the model was created and name of the model developer
- Line of business responsible
- Model risk defined by the criticality and complexity of the model
- Status of model validation and summary of any major issues
- Model change control: This should provide guidance on
- Frequency of model changes
- Model change approval process
- Requirements that trigger model re-validation
- Parallel-run requirement before the new model is put into production.
- Model security: This should provide guidance on
- Who should have access to the model and the model code?
- Where and how often should a model be backed-up?
- How secure should the model and the model location be?
By performing testing of the three main components and by reviewing the five ancillary components, the evaluator discovers if the right model is being used. It is important to remember that model validation is not a onetime process.
Industry practices on model testing
A vast majority of financial industry participants use the word “validation” to mean both validation and verification. Very rarely does anyone in the industry make the distinction between the two processes. There are three main model validation articles published by US regulators that are widely cited by all financial institutions:
- OCC 2000-16 Bulletin
- FHFA Advisory Bulletin 2009-AB-03
- Model Governance published by FDIC in its Supervisory Insights (Winter 2005).
Although the definitions of model validation as given by these three governing articles are slightly different, they are all consistent with the above description of model verification and model validation.
- OCC 2000-16 defines validation as a process that not only increases the reliability of the model but also promotes improvements and a clearer understanding of a model’s strengths and weaknesses among management and user groups.
- FHFA Advisory Bulletin 2009-AB-03 defines model validation as a process of determining that a model’s results accurately reflect the intended use of the model
- FDIC Supervisory Insights defines model validation as an activity that assesses how effectively a model is operating.
Despite the wording differences among these three definitions, all three articles governing model validation prescribe the same list of activities to be performed during model validation. Following is the list of activities prescribed by all three model validation governing articles:
- Independent review of model’s logical and conceptual soundness
- Review inputs
- Review output
- Review model code
- Replicate model
- Sufficient model documentation to facilitate model validation, replication and training
- Prescribe frequency of re-validation
Given all this literature on model validation, it would be an interesting exercise to see how the industry participants perform model validation. Is there a gold standard in model validation? Or is the standard still evolving? The Risk Management Association (RMA) conducted a best practices survey on model validation across 45 large, global financial institutions in 2009, and the results of the survey were published in the 2009 RMA Journal. It states that 82% of the participants have an independent model validation unit within their institution. 84% have a model validation policy. About 75% validate a model or model change before putting the model into production. 100% of the survey participants review each model’s logical and conceptual soundness, however significantly less perform all of the other aspects of model validation including reviewing model documentation (75%) and performing scenario or stress testing (34%), and so on.
MIAC Practices on Model Verification and Validation
Maintaining the longstanding and widespread market acceptance of MIAC’s proprietary software is the first priority of MIAC’s model verification and model validation process. Implicitly this means that MIAC tools have been (and will continue to be) independently verified and validated by a variety of market participants. In addition, MIAC continues to perform every aspect of model verification and model validation internally; and, on many occasions, MIAC tools have been subjected to rigorous testing by various third-party mortgage modeling experts.
What are some of the ways MIAC verifies and validates its proprietary software? First, we begin with verifying and validating each specific enhancement or change request.
Additionally, we perform regression testing to ensure that changes made to the model do not inadvertently “break” other parts of the model and that the changes made hold true in an exhaustive set of scenarios, both typical and atypical. Specifically:
- Developer tests coding changes and builds beta version of the software
- Automated scripts are run, looking for discrepancies in results against the prior version, or “regression” testing
- Quality Assurance tests the changes against the specifications provided
- A Business Analysts also tests the changes with a broader view of scenarios
- Updated programs are run by a select team for a period of time for further model validation
- Testing results are verified by a model validator
- Verified and validated programs are put into production.
MIAC makes a variety of tools available to its clients for model verification and model validation, which allows clients to replicate substantially all of the calculations inherent in its cash flow model. Further, we offer change tracking and permission tools which give clients confidence in their ability to control their processing environment, helping to ensure quality output. In addition, an independent Statement on Auditing Standards No. 70 (SAS70) audit is performed routinely at MIAC to evaluate the suitability of the process controls in place and to certify that those controls were operating effectively during the audit period.
Model enhancements and model changes are tracked and monitored in a software project tracking tool, and routine meetings are held between the business and development teams at MIAC to review, assess, and prioritize all model changes. At MIAC, every model enhancement and model changes go through the rigor of model verification and model validation process. MIAC not only takes pride in its rigorous model verification and model validation process but also on its ability to quickly respond to enhancement requests (from internal and/or external sources) that arise from the model validation process.
Verification of third party vendor models is usually difficult as these models tend to be proprietary in nature. Vendors, in most cases, provide minimal information to their clients making their model a true black-box. Difficultly in doing model verification does not liberate the end user from performing model verification. Even in a black-box situation, model verification is still a necessary evil, even when assistance and tools from the vendor to assist in the process are minimal. MIAC’s goal is to provide our clients all of the tools necessary, including access to business professionals, developers, and verification tools, to verify and validate all MIAC proprietary software and configurations.
MIAC strongly believes in getting the model right and getting the right model to its clients. In the recent years, MIAC has increased its investment in model verification and model validation to meet the following three objectives:
- Ensure with high certainty that MIAC’s proprietary models are right and that they are the right models for valuing any financial product;
- Assist MIAC’s clients in validating all MIAC proprietary software and its configuration; and
- Provide third party, independent financial model verification and model validation for any financial institution needing this assistance.
MIAC proprietary software continues to evolve in accordance with the current needs of the market and the changing regulatory environment. The mortgage world is changing at an extremely fast pace, and MIAC understands the importance of being a leader in adapting to the evolving needs of the mortgage market and its clients.
Conclusion
Although model verification and model validation is a must for a prudent risk practice, it is not a be-all end-all risk management exercise. The real world is very fast paced, and there is no substitute for human judgment and intelligence. A trader might not be able to wait until all the model validation and verification is done before a trade can be executed. When new information becomes available in the market, the trader often must use innate intelligence and experience to immediately execute a decision based on best judgment.
In the truest sense, a fully verified or validated model does not exist in the financial realm. When a model is verified and validated, it means that particular model has gone through a battery of tests to increase the usefulness and reliability of the model. Model verification and model validation enhances the understanding of the model’s capabilities, limitations, and proper usage. But the cycle of model validation and verification never ends, particularly in our increasingly complex and dynamic financial market.
So what is most important? No matter how perfect the model is, if it is not the right model, it is of no use. It is necessary to have a model that is right, but that alone is not sufficient to guarantee success. What is sufficient is having the model right and the right model.
[BACK TO MAIN PAGE]