ESG ratings: Don’t throw the baby out with the bath water | MIT Sloan (2024)

Fueled by lackluster returns and a deepening partisan divide, ESG investing is facing a growing backlash. Along with it comes skepticism over the ratings services and scores used to measure environmental, social, and governance-related business practices.

ESG has come under fire for being overly broad and underperforming as a predictive measure of robust financial returns. Many are calling out the prevalence of greenwashing, where companies exaggerate the environmental impact of their actions. Others are backing away from ESG due to an increasingly polarized and politicized landscape.

One of the biggest concerns related to ESG investing is inconsistency across ESG indexes and ratings, which makes it difficult for fund managers, investors, and consumers to draw reliable insights and comparisons across firms. ESG ratings agencies have been criticized for a lack of standardization in what should be measured and how, as well as a lack of transparency concerning procedures used in the rankings process.

Some opponents have called for ESG ratings to be overhauled or abandoned, while last year The Economistmade the case that financial institutions should retreat from ESG and focus only on the environmental dimension, specifically carbon emissions.

A team of experts at the MIT Sloan Sustainability Initiative contend that such proposed changes are too dramatic and far-reaching. The group’s deep dive into ESG ratings, the Aggregate Confusion Project, aims specifically to identify and document the inconsistencies among ratings organizations, with the goal of improving the quality of ESG measurements.

In their latest paper, “The Signal in the Noise,” the researchers argue that abandoning ESG now would essentially be throwing the baby out with the bath water. They maintain that ESG, however flawed, is currently the best way to measure the ethical behavior of companies and that ESG data, when deployed with greater transparency, can be an important source of information for investors.

They maintain that the push to standardize ratings or to disregard ratings altogether is a mistake. “Given the complexity of what ESG measurement entails, we believe that the only solution to gathering, analyzing, and aggregating the data runs through commercial ESG rating agencies and ESG data providers,” write authors Florian Berg, Jason Jay, Julian Kölbel, and Roberto Rigobon, co-founders of the Aggregate Confusion Project.

“The standardization of ESG ratings would not be an appropriate solution, as this would set in stone an imperfect measure, prone to be manipulated by firms and disincentivizing all research for further improvements,” they write.

Credit: MIT Sloan Sustainability Initiative

Unlocking the value of ESG ratings

The key to reducing inconsistencies and making ESG ratings more useful is elevating the signal in the otherwise noisy data, the researchers say. They assert that with the right methods, it’s possible to locate a signal — a clear relationship between a company’s ESG ratings and behaviors and its stock returns — amid the noise of conflicting and inconsistent standards and measures.

To do so, the Aggregate Confusion Project has leveraged a common taxonomy along with mathematical modeling techniques to create a consistent framework that helps make sense of the different ratings approaches.

The research identifies and explores the three factors driving ratings divergence: scope, when ratings are based on different attributes; measurement, when agencies measure the same attributes with different raw data; and weights, when the different ESG agencies have different views on the importance of the attributes. The Aggregate Confusion Project has also come up with a noise-correction procedure that tackles the bias caused by noisy ESG data and ratings divergence.

Based on their findings, the researchers make the following arguments:

There are signals worth finding. The researchers say there is a valuable relationship between stock returns and ESG scores — if one factors in some noise along with the underlying true ESG performance.

“Think of a real-life situation, such as when you try to listen to a lecture with a lot of background noise due to construction work,” the authors write. “The noise will drown out the signal and make the lecture harder to understand; however, the knowledge is still being imparted.”

To disentangle signal from noise, the researchers instrumented the scores of one ratings agency with up to seven others, performing a two-stage regression modeling exercise. The effort uncovered a substantial amount of noise in the ESG measures — up to 60% of the total score. Yet this also means there is a clear signal in the ESG data that can be effectively drawn out using noise-correction procedures like the one developed by the Aggregate Confusion Project.

CO₂measures aren’t enough. Abandoning ESG in favor of a narrower carbon emissions score simply because it seems easier to measure than the social and governance dimensions is shortsighted and not nearly as effective, the MIT researchers argue.

Carbon emissions are measured with different degrees of precision — for example, we might know a lot about a firm’s past emissions but have far less clarity into future emissions based on current operating decisions. There’s also very little insight into the emissions of partner companies used up and down the supply chain — what’s commonly referred to as scope 3 emissions. The authors say CO₂ data has to be put in context to get a realistic gauge of the societal impact.

Instead of ignoring social- or governance-related dimensions — such as the treatment of disadvantaged groups — because they are undoubtedly difficult to quantify, the investor community needs to recognize the limitations of all measurements. “This is a delicate balance that is difficult to navigate,” the authors write. “On the one hand, if an issue is important, it should be measured — regardless [of] how hard or uncertain the measurement is. On the other hand, what is done with the measurement is a matter of understanding its precision and accuracy.”

Standardization and aggregation aren’t the answer. Settling on one aggregation rule for an all-encompassing ESG rating is misguided because aggregation is fundamentally about preferences, the authors say. Some entities will prioritize climate change while others will lean into anti-discrimination efforts, making it impossible for a single score to capture the heterogeneity in preferences. In fact, ratings agencies are proposing different aggregation rules, generating another source of discrepancy, the researchers found.

A competitive market would help

Instead of a nonstrategic overhaul or complete abandonment, the researchers advocate for an approach that holds ESG ratings to a higher standard. They suggest creating a competitive market among ratings agencies that is centered around the quality of measurement, which would enable the agencies to harness economies of scale and competition to drive down costs. To do so, they say, regulators should standardize ESG disclosure requirements, not the actual ESG scores, to enhance transparency around methodologies and encourage compatibility between ratings systems.

In short: ESG data and procedures have problems, but all is not lost. “The existing shortcomings are not a reason to resign,” the researchers say. “Instead, they call for redesign.”

“The Signal in the Noise” was authored by Florian Berg, MIT Sloan research scientist; Jason Jay, senior lecturer and director of the MIT Sloan Sustainability Initiative; Julian Kölbel, assistant professor of sustainable finance at the University of St. Gallen; and Roberto Rigobon, MIT Sloan professor of applied economics.