This guide is part of a three part series. This is part three. Read part one or two for context and a deep dive into the metrics respectively.
In the last two posts of this series, we covered the existence of the CHAOSS Metrics (Super) Model for Viability. We then covered what exactly comprises that metrics model, and gave brief impressions of why and how they comprise a whole.
In this guide, we’ll talk about what’s possible with the CHAOSS tools, and how we can comprise a Viability metrics model. Namely, we’ll focus on GrimoireLab and Augur.
Consider the chart below to see the breakdown of what is available for which service.
Breakdown by Category
| Category | Metric | Grimoirelab | Augur |
| Strategy | Programming Language Distribution | Available | Available |
| Strategy | Bus Factor | Available | Available |
| Strategy | Elephant Factor | Available | Available |
| Strategy | Organizational Influence | Available | Available |
| Strategy | Release Frequency | Not Available | Available |
| Community | Clones | Not Available | Not Available |
| Community | Forks | Available | Available |
| Community | Types of Contributions | Not Available | Not Available |
| Community | Change Requests | Available | Not Available |
| Community | Committers | Available | Not Available |
| Community | Change Request Closure Ratio | Available | Available |
| Community | Project Popularity | Available | Available |
| Community | Libyears | Not Available | Available |
| Governance | Issue Label Inclusivity | Available | Available |
| Governance | Documentation Usability | Not Available | Not Available |
| Governance | Time to Close | Available | Available |
| Governance | Change Request Closure Ratio | Available | Available |
| Governance | Project Popularity | Available | Available |
| Governance | Libyears | Not Available | Available |
| Governance | Issue Age | Available | Available |
| Governance | Release Frequency | Not Available | Not Available |
| Compliance / Security | OpenSSF Best Practices | Not Available | Not Available |
| Compliance / Security | License Coverage | Not Available | Available |
| Compliance / Security | OSI Approved Licenses | Not Available | Available |
| Compliance / Security | Licenses Declared | Not Available | Available |
| Compliance / Security | Defect Resolution Duration | Available | Not Available |
| Compliance / Security | Libyears | Not Available | Available |
| Compliance / Security | Upstream Code Depencies | Not Available | Not Available |
Breakdown by Tool
| Augur Summary | ||
| Category | Available | Not Available |
| Community | 50.00% | 50.00% |
| Compliance / Security | 57.14% | 42.86% |
| Governance | 75.00% | 25.00% |
| Strategy | 100.00% | |
| Grand Total | 67.86% | 32.14% |
| Grimoirelab Summary | ||
| Category | Available | Not Available |
| Community | 62.50% | 37.50% |
| Compliance / Security | 14.29% | 85.71% |
| Governance | 62.50% | 37.50% |
| Strategy | 80.00% | 20.00% |
| Grand Total | 53.57% | 46.43% |
While we can’t get every metric for every service, we can get a good majority of what we need through a mix of Grimoire Lab, and Augur. We intend to continue building the ability to get this data into services like Grimore and Augur, then update the CHAOSS metrics wiki to reflect how we’ve done it.
Augur provides the most metrics overall for three categories, while Grimoire is best for Community management. Grimoire also provides sigils, which create panels for you as a user for a good amount of metrics you may want to use. Augur also has a tool supported by RedHat that visualizes metrics within it.
How Does this Guide My Decisions?
Depending on your use case, you may find different opportunities to use the Viability model. It was originally developed for use evaluating using open source products, and your thresholds for each model category will vary based on your assumption of risk.
For example:
- Organizations starting their journey in governing Open Source Software at their organization usually start with Compliance and Security, cornering vulnerabilities and licensing information to choose their software.
- Large companies may consider strategy to be the next-most important. Given that many organizations build software that is in use for years, the importance of the strategy in a project — and indeed who maintains that strategy — can be a critical decision.
- Community is important for cutting-edge or newer implementations of technology. While older technology will likely have a less volatile community, where maintainers and flow of new contributions can be judged over time, a new project may need a stronger community with more vigilance on it’s competitors to ensure that a software stack isn’t abandoned.
- Governance is crucial for organizations that intend to engage the open source community; or contribute to a project to shape new functionality. If an organization is willing to commit time and resources to maintaining a project — the Governance of that project becomes important to consider
Getting Started
Consult the documentation of GrimoireLab and Augur for more details on how to get started. Based on what your team needs or cares about, consider choosing the tool that has the highest coverage, or use them both to maximize your results. If you find that you can trace some metrics that I’ve gotten wrong here, I’d love to know! Drop by our OSPO working group, metrics working group, or somewhere else to publish your contributions!
Until then, you can find me on CHAOSS community slack, as Gary White. Thanks for reading!