Have you ever been in a position in your company or community where you would like to start getting a sense of the health of a project – but you don’t know where to begin?
People often struggle to get started with measuring project health in a way that allows them to draw meaningful conclusions without becoming overwhelmed. Measuring key aspects of project health is an essential first step toward understanding how an open source project can be improved and deciding where to focus improvement efforts. The following four metrics are great ways to get started.The Starter Project Health Metrics Model was published by the CHAOSS project to address this very issue.
Time to First Response Determine the amount of time between when an activity was opened (e.g. Issue or Change Request) and when it received the first response from a human. A quick response helps contributors feel welcome and appreciated.
Change Request Closure Ratio Measure the ratio between the total number of open change requests during a time period versus the total number of change requests closed in that same period. This helps to determine whether your project has enough maintainers to keep up with incoming contributions.
Bus Factor Determine the smallest number of people that make 50% of contributions to understand whether your project would be in jeopardy if one or more key contributors left.
Release Frequency Determine the frequency of project releases (including point releases with bug fixes) to make sure that security fixes, new features, and bug fixes are available to your users.
CHAOSScon 2023 Europe is now complete! Thanks to everyone who helped put on another wonderful event and thanks to everyone who took the time to attend and participate! It was really great to connect with everyone in the beautiful city of Brussels.
This CHAOSScon, we led several discussions centering on two key questions. We asked participants to reflect on the questions in small groups and report back. This post will highlight a few of the key takeaways — and also provide captured comments that maybe didn’t make it into this post.
What challenges exist for using metrics within your OSPO or Community?
The Consistent Use of Metrics within an Organization or Community
Many of the responses centered on the difficulties associated with the consistent use of metrics within an organization or community. For example, with the sheer size of many organizations and communities, the deployment and interpretation of metrics can vary a lot between people. This leads us, in the CHAOSS Community, to a 2023 goal of developing ways to communicate simple metric strategies and results that can be easily shared between people.
The full set of recorded comments for the first question with general categorizations is here:
There isn’t a central place for metrics discussions within an organization.
There isn’t a common taxonomy for how to speak about metrics.
Fragmentation of metrics within an organization can make consistent use difficult.
The size of an organization can make consistent use of metrics difficult.
The size of a community can make consistent use of metrics difficult.
Different stages of project maturity can make consistent use of metrics difficult.
There is minimal guidance on replicating CHAOSS structures locally.
How to determine business value from a project?
How to measure the value of participation in a project?
How to determine business risk from a project?
How to determine company impact on a project?
How to measure the cost of non-participation?
Metrics themselves can be quite complex
What should the CHAOSS project be working on in the future?
Building Collaborative Communities and Helping People Communicate Better
There are many, many things that we could work on within the CHAOSS project in 2023. Recurring themes from CHAOSScon include (1) helping people connect with others in similar contexts to discuss health-related concerns (i.e., corporate open source program offices) and (2) developing ways to help people communicate about metrics within their respective organization or community.
The full set of recorded comments for the second question with general categorizations is here:
Enable all contributions to be seen in a dashboard.
Add social, design, and educational material use (like a social calendar) into a dashboard of contributor metrics to cross-visualize data about the ecosystem beyond code.
Continue to support the newcomer experience.
Assist others with the interpretation of results.
Have mechanisms to help interpret the data in non-biased ways / manipulating it towards certain predetermined hypotheses.
Provide badging to validate business metrics strategies.
Provide metrics and metrics model validation.
Support user groups that need specific metrics and metrics models to help in a variety of contexts.
Develop personas for groups of people with similar interests.
Build better starting points for people who may not want to build metrics.
Have social justice built into the metric development process, not as charity, but to address systemic issues continuously.
Provide Pathways into the metrics, more context on where they have been used in the past (e.g. to answer the question what do people like me use?).
Think about people who work in different languages – is CHAOSS accessible for them?
Talk with other communities to understand different views of metrics.
Tell user stories of how others are using the metrics and metrics models in practice.
Provide a framework for how to apply metrics.
Provide ways to represent and talk about metrics in business meetings.
Push a goal-oriented approach.
Articulating why metrics are important to help the business case.
Provide a set of five metric models to use that do five different things.
Software license compliance between repos
Test coverage in repos.
Ensuring a broad community of voices.
Build productivity and efficiency metrics and metrics models.
Need to understand the signal-to-noise ratio.
More human-centric metrics, the kind that help to signal/prevent burnout.
An estimate of how difficult it might be to implement. A signal of whether tech experience is required to implement a particular metric, or if it can be done by a quick survey, or if it can be identified automatically, or if it requires big program management lift.
Thanks again to everyone for an amazing CHAOSScon 2023 in Brussels! We hope to see you soon!!