Evaluating Geithner’s Stress Test

With the release of the stress test methodology(pdf), let’s take a look at what the stress test is, what it isn’t and how it’s done.

What the Stress Test Is, is a test of the whether the 19 bank holding companies with more than 100 billion in assets have enough income and reserves to survive till the end of 2010, with enough reserves left at that time to make it through 2010. Banks were required to report their expected income for the next two years, then that was compared to their reported expected losses for the duration.

The Stress Test Is Not a test of whether or not, if a given bank holding company were liquidated right now, it would be worth more than 0 dollars. As such, it is not, primarily about mark-to-market accounting. The question is now “are the banks solvent”, the question is “can they keep operating and if not how much money do they need to keep operating?” In household terms, think of it as “can they pay their bills”, not “what is their net worth?”

The assumption is thus that all loans will be held to maturity and not sold on the market. What matters, then, is the income on those loans and how likely those loans are to default. Income is thus loan income minus defaults, taking into account any ability to recover losses. (For example, if a homeowner defaults on a mortgage, how much will the bank receive when it seizes and sells the house?)

Something over 150 people worked on the test on the government side. They were divided into teams by asset and income areas, such as “Commercial Real Estate Loans”, “First and Second Lien Mortgages” and “Credit Cards and Other Consumer Loans”. Each of these teams evaluated the submissions off all the Banks for that area.

Using the Banks Own Models

An initial criticism of the Stress test was that it used the banks own financial models-the same models which didn’t predict this mess in the first place. That’s mostly, but not entirely correct. The banks run the numbers based on their models, but they have to give the assumptions underlying those models to the supervisory teams. If the teams don’t agree with the banks model, they can insist on changes. How much they have done so, we don’t know, but there is some indication in the methodology that the teams did their own analysis of likely losses. For example, when referring to mortgages, the methodology reports that:

Certain attributes, in particular FICO, LTV bands, vintage, product type, and geography, were found to be strongly predictive of default. These attributes were used to further evaluate submissions by the firms, and where necessary, loss estimates were adjusted to better reflect portfolio characteristics in a consistent way across firms.

There are also indications that the teams found some substantial differences in underwriting standards between firms, and have taken that into account as well. Nonetheless, given that this was done in a 2 month period, with a little over a 150 people reviewing 19 massive banks, the teams would not have been able to develop their own models and could only have tweaked the bank models.

Economic Scenarios

The losses taken into account were based on two economic scenarios, a scenario based on median economists forecasts at the time the Stress Test was first planned, and a more adverse scenario. The first scenario has already been superseded by events, as the economy is performing worse than mainstream economists expected. The more adverse scenario has not yet been exceeded, but, for example, it models 8.9% unemployment in 2009, and current unemployment is running at 8.5. It is highly unlikely, in my opinion, that unemployment will not rise more than .4% in the rest of 2009 unless it’s for technical reasons like people despairing so much of finding a job that they just stop looking entirely.

What this means is that even if one accepts the models might be accurate after the examiners fiddled with them, losses will probably be higher than expected.

Banks that don’t pass will be required to raise enough capital to make it through the time period. They can try and do that on private markets, or the government may step in and provide capital. In addition, they can ask the government to convert its preferred shares into common stock, which will reduce the company’s expenses.

Concluding Remarks

The stress test is flawed, but not worthless. The economic forecast used was overly optimistic, and given that mainstream economic forecasts have been consistently off throughout the crisis, this should have been expected. The staffing may be sufficient to do superficial analysis, but this is by no means a real audit, in which hundreds of examiners swarm over each bank to discover whether or not the top end numbers they are supplying are accurate or if their accounting and underwriting has been weaker than declared or if there has been outright fraud.

While examining underwriting standards is useful, examining actual random cases in a professional audit to see whether or not underwriting standards had actually been followed would have been far more useful and predictive of future losses. To some extent, looking at comparative loss rates between banks can substitute for this, but only partially, as it is backwards looking rather than forward looking.

Bank default and valuation models are highly suspect, as well. As a rule the models used during the collateralization process did not sufficiently account for default clustering (i.e. for the fact that in a recession or depression a lot of people default in a short time period) or for the fact that there were housing or securitization bubbles. Those models have to be corrected for each particular class of securitized loan, since each one had its own model. Crude approximations were no doubt put in place by the teams and perhaps by the banks, but there’s still plenty of reason to question the models being used.

Given these flaws the stress test is only indicative, not final. Certainly any bank which fails them definitely needs money, but it may need more money than indicated by the test. Likewise banks that pass will likely not be viewed as out of the jungle, unless they pass with flying colors.

Likewise, the methodological paper was quite vague. The stress test will not be trusted without more specifics, and when results are released we will need firm numbers, not just the final numbers “needs X amount of money” or “passed” but the assumptions on default rates broken down by year, location of loan, type of loan and so on so that independent analysts can come to their own conclusions. Failure to do so will mean that the banks and Treasury are saying “trust us”, and unfortunately, at this point, no one does. Given the known flaws of the stress test, independent verification will be required