Quantitative vs Qualitative Impact

This is the third blog post in a collaborative series between early-stage climate VC Amasia and Malk Partners, preeminent advisors to private market investors on impact and ESG.

Both quantitative and qualitative impact are important to Amasia, which tries to incorporate both in its impact measurement process and its interactions with portfolio companies. Similarly, Malk considers both quantitative and qualitative data valuable for its industry leading ESG diligence, portfolio management, and advisory work. This blog post serves as an introduction to and comparison of the two types of metrics used to inform impact and ESG, and how Amasia and Malk view them in their work with portfolio companies and advisees respectively. 

Quantitative Impact

As outlined in the IMP’s 5 Dimensions of Impact, quantitative metrics explain the scale of impact made.

Quantitative impact metrics are useful in the following ways:

• They succinctly capture a company’s biggest value-add, impact-wise

• They make it easy to track the progress that companies make over time with regard to impact, and this process of tracking holds companies accountable

• Quantitative metrics are the easiest to communicate and understand

• They are useful for ESG diligence and monitoring, by providing a common language or framework with which to compare a company’s ESG performance with peers and/or the industry average

Amasia tracks quantitative impact metrics with its portfolio companies guided by its impact screen, developed in collaboration with Malk. Amasia works with its portfolio companies to establish these metrics, and companies track these metrics on a quarterly basis. For instance, Amasia’s portfolio company Topanga, whose software powers large-scale packaging reuse programs, currently tracks a number of metrics. The number of single-use plastic containers diverted from landfills, water saved, and carbon emissions avoided due to reuse are instantaneously calculated and made available to users to track their impact. 

Fig. 1: Impact data on Topanga dashboard; read about Amasia’s investment in Topanga here

There are multiple advantages to Topanga tracking these metrics and displaying them to users. Users are able to keep track of their reuse habits and compare themselves to others in the community, which empowers and encourages them to continue deepening their reuse habits. Promisingly, Topanga has noted that the availability of this data seems to encourage reuse among its users. 

Despite its clear strengths, however, we know that quantitative data cannot effectively capture impact, which is multidimensional, context-dependent, and dynamic. Stating the number of jobs created in an impact report, for instance, says nothing about whether the jobs are stable, safe, or making a meaningful difference in the job holder’s livelihood. Likewise, stating the number of trees planted doesn’t tell you about species, carbon capture potential, the length of time for which the tree will be left standing (and therefore storing carbon), etc.

These limitations also apply to ESG diligence: quantitative data alone can fail to capture the entire story. For instance, a company with few employees could have a higher safety incident rate than peers simply due to a one-off accident, due to the smaller sample size. In this case, quantitative metrics without narrative color would fail to tell the entire story. 

For this reason, Malk endeavors to integrate nuance into its diligence reports, both through context from the internal operations of a company (e.g., narrative description of how a Management team approaches a risk area) and through sector-based benchmarking. In the latter case, Malk leverages ratings and analytics through its partner data platform, Integrum ESG, which can create custom, normalized benchmarks (e.g., by region, sector) in order to create the most apt understanding of ESG performance.

Additionally, the uncertainty of new business strategies frequently entails many assumptions and generalizations, making it difficult to accurately quantify the environmental impact of all parts of a business idea.

Qualitative Impact  

While quantitative metrics can communicate impact concisely, qualitative metrics capture a wide range of impact that cannot be distilled into numbers without losing vital nuance. For example, a 1 cm sea level rise for one country doesn’t necessarily have the same effects as for another – and these critical differences would only be made known with qualitative data.

Qualitative metrics have the following strengths:

• They capture the contextual nuance of an isolated event and the consideration of forward trends

  • While quantitative metrics provide replicable and comparable benchmarks across ESG areas, qualitative descriptions often informed by qualitative data points, allow investors and companies to tell their ESG stories with nuance. For instance, a Company with significant renewable energy procurement or GHG reduction efforts might supplement a quantitative metric (e.g., % energy from renewable sources, total GHG emissions) with a qualitative description (e.g., initiatives to decrease energy consumption and subsequent GHG emissions).

• They validate a company’s theory of change/impact narrative (depth of impact) and what specifically a company is contributing to

  • Importantly, they could also reveal cases in which the impact thesis does not pan out as expected; ideally, this information comes from stakeholders, offering a feedback loop and accountability mechanism that allows companies and funds to respond and recalibrate

• The process of capturing qualitative metrics necessitates closer attention to and interaction with stakeholders, which is inherently valuable and can generate much-needed insight

• They involve open and close-ended questions that can help researchers understand the reasoning behind motivations, frustrations, and actions – this is especially important for Amasia, which is specifically interested in behavior change

  • Qualitative impact data allows Amasia to understand how a company affects behavior. For instance, finding out how the provision of a certain service such as ride-sharing affects an individual’s commuting behavior.

Amasia’s portfolio company Clarity is in the process of collecting qualitative data from end-users to understand use cases, motivations, obstacles, and so on. This not only helps Clarity understand how to improve its product but also how it has shaped the decision-making of countless stakeholders, strengthening their impact narrative and informing them on how to support users’ use cases further.

Fig. 2: Network of over 400 of Clarity’s air quality measurement devices across all 33 London boroughs. Source: Clarity

At the same time, however, qualitative methodologies might also be biased due to social norms and power dynamics; for example, a survey question could be phrased in a way that encourages a particular response, introducing response bias. It’s therefore important to integrate local communities in data collection and co-creation. Similarly, in the ESG universe, qualitative metrics unsupported by any quantitative ones could run the risk of greenwashing, as they might enable a Company to exaggerate ESG performance or diminish adverse outcomes.

Balancing Both Qualitative & Quantitative Metrics 

Both qualitative and quantitative impact metrics are necessary to build a comprehensive account of a company’s impact, and Amasia and Malk both acknowledge this in their practice.

Amasia strives to establish a “north star” metric for each of its companies: a quantitative metric that neatly reflects the core of the company’s impact and is feasibly tracked over the long term. For portfolio company TreeDots, this metric is food waste saved from ending up in landfills, and for Commons, its carbon emissions offset and/or reduced by users. Alternatively, Malk strives to couple qualitative metrics with quantitative data when telling a company’s ESG story in order to provide the most accurate and holistic account possible and to signal operational areas to monitor. For instance, Integrum ESG captures both qualitative and quantitative performance metrics to convey programmatic mechanisms qualitatively and the related performance and success of those mechanisms quantitatively, providing a wholesome picture of performance and indicating potential areas for improvement.

Of course, the numbers themselves don’t tell the full story; there are ways in which Amasia’s portfolio companies supplement these metrics. On the Commons app, for instance, user behavior and activity like community engagement, and pledges to adopt eco-friendly practices, help the Commons team better understand the impact of the app on users’ lives and their decision-making.

It’s not always easy to balance both qualitative and quantitative data; nevertheless, it’s valuable to consider both as much as possible within the constraints of time and data availability. Amasia’s impact screen takes a qualitative framework and quantifies it: this allows Amasia to assign numerical scores from 1 to 3 on positive impact, intentionality, scale, depth, and additionality when evaluating a company. The impact screen thus allows for an efficient and easily communicated rating system that simultaneously incorporates more nuanced qualitative elements.

Likewise, in ESG monitoring, Malk leverages quantitative metrics to inform qualitative assessment and vice versa: you can’t have one without the other. For example, the qualitative metric of whether a company performs anti-discrimination and harassment training or not can inform how Malk interprets a company’s discrimination and harassment incident rate or even its female and racial/ethnic representation.

Conclusion

As investors, we should recognize the value of data that cannot be succinctly captured in figures, including when it comes to impact. Amasia is at the start of its qualitative impact journey and will share learnings and insights as its impact approach evolves. As a trusted ESG advisor to private markets, Malk understands that qualitative and quantitative metrics are simply two sides of the same coin for assessing and improving ESG performance; you cannot have one without the other.