Blog Post

Guide for OSS Viability: A CHAOSS Metric Model

By December 19, 2023No Comments
Photo by William Bout on Unsplash

This guide is part of a three part series. This is part three. Read part one or two for context and a deep dive into the metrics respectively.

In the last two posts of this series, we covered the existence of the CHAOSS Metrics (Super) Model for Viability. We then covered what exactly comprises that metrics model, and gave brief impressions of why and how they comprise a whole.

In this guide, we’ll talk about what’s possible with the CHAOSS tools, and how we can comprise a Viability metrics model. Namely, we’ll focus on GrimoireLab and Augur.

Consider the chart below to see the breakdown of what is available for which service.

Breakdown by Category

CategoryMetricGrimoirelabAugur
StrategyProgramming Language DistributionAvailableAvailable
StrategyBus FactorAvailableAvailable
StrategyElephant FactorAvailableAvailable
StrategyOrganizational InfluenceAvailableAvailable
StrategyRelease FrequencyNot AvailableAvailable
CommunityClonesNot AvailableNot Available
CommunityForksAvailableAvailable
CommunityTypes of ContributionsNot AvailableNot Available
CommunityChange RequestsAvailableNot Available
CommunityCommittersAvailableNot Available
CommunityChange Request Closure RatioAvailableAvailable
CommunityProject PopularityAvailableAvailable
CommunityLibyearsNot AvailableAvailable
GovernanceIssue Label InclusivityAvailableAvailable
GovernanceDocumentation UsabilityNot AvailableNot Available
GovernanceTime to CloseAvailableAvailable
GovernanceChange Request Closure RatioAvailableAvailable
GovernanceProject PopularityAvailableAvailable
GovernanceLibyearsNot AvailableAvailable
GovernanceIssue AgeAvailableAvailable
GovernanceRelease FrequencyNot AvailableNot Available
Compliance / SecurityOpenSSF Best PracticesNot AvailableNot Available
Compliance / SecurityLicense CoverageNot AvailableAvailable
Compliance / SecurityOSI Approved LicensesNot AvailableAvailable
Compliance / SecurityLicenses DeclaredNot AvailableAvailable
Compliance / SecurityDefect Resolution DurationAvailableNot Available
Compliance / SecurityLibyearsNot AvailableAvailable
Compliance / SecurityUpstream Code DepenciesNot AvailableNot Available
A Summary of Available CHAOSS metrics and their fit to Viability across Grimoire and Augur

Breakdown by Tool

Augur Summary  
CategoryAvailableNot Available
Community50.00%50.00%
Compliance / Security57.14%42.86%
Governance75.00%25.00%
Strategy100.00% 
Grand Total67.86%32.14%
Grimoirelab Summary  
CategoryAvailableNot Available
Community62.50%37.50%
Compliance / Security14.29%85.71%
Governance62.50%37.50%
Strategy80.00%20.00%
Grand Total53.57%46.43%

While we can’t get every metric for every service, we can get a good majority of what we need through a mix of Grimoire Lab, and Augur. We intend to continue building the ability to get this data into services like Grimore and Augur, then update the CHAOSS metrics wiki to reflect how we’ve done it.

Augur provides the most metrics overall for three categories, while Grimoire is best for Community management. Grimoire also provides sigils, which create panels for you as a user for a good amount of metrics you may want to use. Augur also has a tool supported by RedHat that visualizes metrics within it.

How Does this Guide My Decisions?

Depending on your use case, you may find different opportunities to use the Viability model. It was originally developed for use evaluating using open source products, and your thresholds for each model category will vary based on your assumption of risk.

For example:

  • Organizations starting their journey in governing Open Source Software at their organization usually start with Compliance and Security, cornering vulnerabilities and licensing information to choose their software.
  • Large companies may consider strategy to be the next-most important. Given that many organizations build software that is in use for years, the importance of the strategy in a project — and indeed who maintains that strategy — can be a critical decision.
  • Community is important for cutting-edge or newer implementations of technology. While older technology will likely have a less volatile community, where maintainers and flow of new contributions can be judged over time, a new project may need a stronger community with more vigilance on it’s competitors to ensure that a software stack isn’t abandoned.
  • Governance is crucial for organizations that intend to engage the open source community; or contribute to a project to shape new functionality. If an organization is willing to commit time and resources to maintaining a project — the Governance of that project becomes important to consider

Getting Started

Consult the documentation of GrimoireLab and Augur for more details on how to get started. Based on what your team needs or cares about, consider choosing the tool that has the highest coverage, or use them both to maximize your results. If you find that you can trace some metrics that I’ve gotten wrong here, I’d love to know! Drop by our OSPO working group, metrics working group, or somewhere else to publish your contributions!

Until then, you can find me on CHAOSS community slack, as Gary White. Thanks for reading!

Author