The future of stress testing – realism, relevance and resources
Keynote speech by Andrea Enria, Chair of the Supervisory Board of the ECB, at the European Systemic Risk Board (ESRB) Annual Conference
Frankfurt am Main, 26 September 2019
In 1628, the Vasa – the pride of the Swedish navy – was the most powerful warship in the Baltics. Until she sank about half an hour into her maiden voyage, that is. So what happened? Well, nothing special when it comes to sailing ships: the wind simply picked up.
The problem was that the Vasa’s design was rather unstable. She had been built for the shallow waters around Stockholm, so not too much of the ship was below the waterline and a lot of weight was concentrated in her upper structures. Thus, when the wind picked up, it pushed the ship so far over on its port side that water poured in through open gunports. And that was the end of the Vasa – and a huge embarrassment for the King of Sweden, Gustavus Adolphus.
Now, why am I telling you the story of the Vasa? In my view, it shows how beneficial it is to spend a moment or two thinking about what might happen if the wind picks up. This is true when it comes to building ships and, in a sense, it is also true when it comes to managing banks. How will lenders fare in a storm? Are they stable enough to weather a storm? Or do they need additional weight below the waterline, in the form of more capital.
This idea sounds straightforward, doesn’t it? Yet, it is only recently that stress tests for banks have entered the picture. It was the IMF that started such exercises as part of its regular Financial Sector Assessment Programs. But it took another 20 years before stress tests would become a key tool for banking supervisors.
In 2009, the United States used stress tests as a means to fight the financial crisis; Europe followed suit in 2010. Since then, stress tests have become a key instrument in the supervisory toolbox. So let us take a closer look at the evolution of stress testing and its objectives, and how it might need to be adapted in the future.
Goals and benefits of stress testing banks
The general goal of a stress test is quite simple: to assess how the ship would weather a storm and what shape it would be in afterwards. In terms of banks: how would the bank fare if the economy took a turn for the worse? Would it survive? And would it still be able to provide loans to help the economy recover?
Answering these questions has been the core intent of the stress tests since they were first introduced in Europe. But over time, they have also served other topical purposes. Stress tests were used during the last financial crisis to gauge the size of the capital holes in bank balance sheets, and they were key elements in determining how much additional capital needed to be provided. This helped to reduce uncertainty and calm the markets – not least because some banks took upcoming stress tests as a cue to pre-emptively build up capital.
At the time, there was the difficult challenge of addressing the risks triggered by the sovereign debt crisis. And so the choice was made to provide extensive disclosure to the markets with a view to helping them correctly assess the risks of individual banks. This, in turn, helped to bring about more effective market discipline.
Then, in 2014, stress tests featured heavily in the comprehensive assessment that preceded the establishment of European banking supervision. Here, they helped to level the playing field for banks from 19 countries with different accounting rules and supervisory practices. It was these stress tests, for instance, that first relied on the benefits of the Single Rulebook, which provides fully harmonised definitions of capital and non-performing loans as developed in the Capital Requirements Regulation and the EBA’s standards.
Since then, stress tests have become a regular transparency and benchmarking exercise. But their focus has shifted. Rather than measuring the actual size of capital holes against a supervisory yardstick, stress tests now help us to spot vulnerabilities in banks. Also, the dialogue between supervisors and banks has become more important as the stress test is seen as a tool to strengthen banks’ own risk management.
Ever since they were first used, stress tests have helped to make the banking sector more transparent. In Europe, disclosure has always been a key feature of the exercise. Transparency helps to foster market discipline in parallel with supervisory judgement.
Over the years, stress tests have remained an important tool for supervisors. What we learn from them guides our Supervisory Review and Evaluation Process, or SREP. The stress test results are used, for instance, as a starting point when setting the Pillar 2 guidance, or P2G. Likewise, the insights we gain from stress tests can also inform other supervisory actions such as follow-up on-site missions or internal model investigations.
No matter which goals they have served, stress tests have so far proved very useful. But we have to acknowledge one thing: the attempt to align different, sometimes conflicting objectives has led to a fairly complex and resource-intensive exercise. And this, in turn, has led to a certain amount of discontent among supervisors and banks alike. So ten years after the crisis, and five years into European banking supervision, it is time to rethink the design of stress tests.
Rethinking stress tests: realism, relevance and resources
When discussing the future of stress tests in Europe, we can rely on three yardsticks to gauge how best to proceed: realism, relevance and resources. We need a stress test that paints a realistic picture of individual banks; we need a stress test that is relevant for supervisors, banks and markets alike; and we need a stress test that balances costs and benefits.
All these things are connected, of course. But for the sake of today’s discourse, I’ll try to disentangle them for you, starting with realism.
First, let’s look at the methods we use to calculate the impact of the stress scenarios. Currently, we rely on a constrained bottom-up approach. Banks use their internal models to calculate how their capital positions would change in the stress scenario. However, we impose some constraints on these models. Static balance sheets are one of them: we assume that management would not attempt to mitigate the impact of the stress.
Such constraints act as safeguards. They discourage banks from engaging in a beauty contest where they strive to look good instead of real. They make the results of the stress tests easier to compare and more conservative. And they also make quality assurance less complex.
All this comes at the cost of realism, however. But what exactly does realism mean for an exercise that is based on hypothetical scenarios? To me, it means that the results of the stress test accurately reflect how the hypothetical stress would actually affect a bank’s balance sheet. In other words, if it rains on the economy, how would the water trickle down or pour into the balance sheet?
So, if we want to make stress tests more realistic, the constraints might be one of the levers to pull. We might ask, for instance, whether it is realistic to assume static balance sheets. And the banks do ask, believe me. Selectively relaxing the constraints would allow us to better account for bank-specific factors and management actions – and this would make results more realistic.
Also, constraints might be seen as being the main driver of the results, rather than a backstop to ensure their credibility. In this case, banks would tend to consider the exercise irrelevant from an internal risk management perspective. This view often makes banks reluctant to publish final results which do not reflect their own views on risks.
But the realism of the exercise is also challenged because it easily turns into a “beauty contest”: banks direct their efforts to “model the stress away” in order to look good to supervisors and investors. And the experience from the first rounds of European stress tests shows that banks indeed often try to compensate losses in the adverse scenario. They do so by either being overly optimistic when estimating their income in the adverse scenario or by being very positive on what management can achieve in turbulent market conditions.
We also see banks conspiring to game stress tests, often with the help of external advisers. Data are collected from banks ahead of their submission to supervisors, and each bank is informed of its position vis-à-vis its peers. This helps them to align before and during the exercise in order to collectively adjust the results and minimise the impact of the stress scenario. We see this, we don’t like it, and we will not tolerate it.
Naturally, realism is closely linked to the second point on my list: relevance. The more realistic the results are, the more relevant they should be. But the notion of relevance may differ according to the objective that the stress test aims to achieve.
For banks, the stress test might be most useful for risk management purposes when the methodology and the scenario are tailored to the specific business model and risk profile of each bank. Thus, the ideal would be to have a tailor-made stress test for each and every bank.
Supervisors, too, care about the ability of the stress test to capture the risk profile of each bank. But they also need the results as input when setting capital buffers – the Pillar 2 guidance. As this is an administrative measure, consistency across banks is crucial: each bank should be subject to significant stress, which challenges its resilience and provides comparable input for the Supervisory Review and Evaluation Process.
Macroprudential authorities want to assess how resilient the system as a whole is. They want to see how a single shock that hits banks at a certain point in time would spread throughout the system and potentially hamper its ability to serve the economy.
Finally, there are the markets. For them, relevance is closely linked to transparency. In other words, does the stress test provide valuable additional information on individual banks? I believe that the European stress test is one of the most transparent in the world. We publish a huge amount of data bank by bank and based on common definitions.
But there is one thing that markets cannot see: the link between stress test results and supervisory action. While we are very transparent on the results, we remain quite opaque on how they translate into capital add-ons. I am very much aware that both banks and supervisors have concerns when it comes to enhancing transparency in this area. They are worried, for instance, that if stressed capital guidance as defined by supervisors was to be disclosed, it might be misunderstood. It might be seen as a rigid minimum requirement and not as a buffer to be used under stress.
But to be honest, I still think that we need to seriously consider disclosing Pillar 2 guidance at some point. This is particularly true in a bail-in world, where private investors and not taxpayers are supposed be first in line when it comes to picking up the bill following a crisis. More transparency on the supervisory outcome of stress tests would sustain their relevance for banks and markets alike. Stress tests must have clear consequences.
Let’s now turn to the final item on my list: resources. A stress test is a complex exercise. As I’ve said before, it is extremely resource-intensive for both banks and supervisors. This might be justified as long as the tests produce realistic and relevant results. Still, we need to strike a balance between costs and benefits. And currently, the impression seems to be that the amount of resources deployed is quite high when set against the value of the information that is generated.
I believe that all stakeholders agree on the basic idea that if we want to take stress testing from infancy to maturity, we need to progress in all three dimensions. We need to make it more realistic, more relevant and less resource-intensive. Yet, views differ significantly as to the best way forward, even among supervisors. So how can we square the circle?
Squaring the circle – the way forward
Ladies and gentlemen, when discussing the future of stress testing in Europe, we must keep in mind that “the future” means post-2020; the upcoming stress test will certainly follow the current approach. Nevertheless, the time for debate has come. The report recently published by the European Court of Auditors on the European stress test provides important input into this debate.
So far, I have discussed three yardsticks that stake out the bounds of possible futures: realism, relevance and resources. While I’ve taken them one by one in my remarks today, in reality they’re all linked. Unfortunately, these links sometimes take the form of trade-offs. Thus, there are many parts which we can arrange in different ways, depending on how we want to balance them.
Imagine we decided to relax constraints in the bottom-up approach. Banks would be free to depart from a common methodology and scenario to better reflect their individual business models and risk profiles. The results of the stress test would become more realistic and more relevant for banks, which is good. However, supervisors would have a hard time using the results of such an exercise to determine capital buffers in a consistent way across banks. They would need to invest more time and energy in quality assurance, which would require additional resources.
So what should we do about it? Well, there are voices arguing that supervisors should focus more on top-down stress tests, as is done in the United States. Thus, they would run the exercise with their own models and fully control the consistency of the results. But then, much less information would be provided to the markets. After all, banks would not accept the publication of very granular risk parameters which would not reflect their own risk management practices.
I think that the root of the problem lies in the fact that we are trying to do too much with too little. If we aim to achieve several goals, some of which conflict with each other, and if we aim to serve different customers, the stress test is bound to disappoint all of them.
So I wonder whether the time is ripe to consider some bolder solutions. Why not split the microprudential stress test into a bank view and a supervisory view? The bank view would stem from a largely unconstrained bottom-up approach. There would be no quality assurance, but banks would have to explain where and why they deviate from the constraints. This would allow each bank to account for its individual circumstances. And each bank would have an added incentive to invest in risk management. The stress test would become more realistic and more relevant as a result. It could provide useful, granular information to the markets.
At this point, some might argue that banks could get over-excited about the freedom they would gain, and that this could lead to a lack of conservatism. This is where the supervisory view comes in. It would rely on a constrained bottom-up approach, and top-down models would be used to provide quality assurance for the results and focused benchmarking. So it would be very similar to the approach followed in 2018. However, some methodological constraints, including the static balance sheet assumption, could be relaxed somewhat to increase realism. Likewise, the exercise could be streamlined and thus become less resource-intensive. First, there would be fewer quality assurance cycles, as there would be no need to align the bank view and the supervisory view. Second, the outcome of the supervisory view would only be published in terms of capital depletion, making it less granular. Third, some design features of the bank view and the supervisory view could be aligned, which would avoid duplicating efforts. This too would save resources. At first, top-down stress tests would continue to support quality assurance. But in the longer run, as accuracy and reliability improve, we might consider relying even more on top-down stress tests.
The bank view and the supervisory view would then be published next to each other so that markets could form their own view.
To sum up, splitting stress tests into two components could help make them more realistic and relevant without taking up more resources.
Stress tests have become an important tool for supervisors in the wake of the financial crisis. And in Europe, we have benefited greatly from their use. I am indeed a great believer in stress testing, but I do also see the need to refine the exercise and adapt it to the post-crisis world. I have outlined a few ideas for you today, but they are only ideas at this stage. Nevertheless, if we all agree that we need to make stress tests more realistic and relevant without becoming too burdensome, then it is these ideas that we need to discuss.
This brings us back to the Vasa. The twist in the story is that the captain did design a rudimentary stress test. To gauge the stability of the ship, he had 30 of his men run from one side of the deck to the other in order to make her roll. And roll she did. However, a representative of the King was on board, and he stopped the test. His main concern was that the stress test itself might sink the ship.
My final message is that we should be open-minded in the conversation about the future of stress tests. And the bottom line is this: they should remain serious, challenging and credible exercises; they should help supervisors to ensure the safety and soundness of banks.
Thank you for your attention.