Have you ever been in a position in your company or community where you would like to start getting a sense of the health of a project – but you don’t know where to begin?
People often struggle to get started with measuring project health in a way that allows them to draw meaningful conclusions without becoming overwhelmed. Measuring key aspects of project health is an essential first step toward understanding how an open source project can be improved and deciding where to focus improvement efforts. The following four metrics are great ways to get started.The Starter Project Health Metrics Model was published by the CHAOSS project to address this very issue.
Time to First Response Determine the amount of time between when an activity was opened (e.g. Issue or Change Request) and when it received the first response from a human. A quick response helps contributors feel welcome and appreciated.
Change Request Closure Ratio Measure the ratio between the total number of open change requests during a time period versus the total number of change requests closed in that same period. This helps to determine whether your project has enough maintainers to keep up with incoming contributions.
Bus Factor Determine the smallest number of people that make 50% of contributions to understand whether your project would be in jeopardy if one or more key contributors left.
Release Frequency Determine the frequency of project releases (including point releases with bug fixes) to make sure that security fixes, new features, and bug fixes are available to your users.
CHAOSScon 2023 Europe is now complete! Thanks to everyone who helped put on another wonderful event and thanks to everyone who took the time to attend and participate! It was really great to connect with everyone in the beautiful city of Brussels.
Photo Credit: Sean Goggins
This CHAOSScon, we led several discussions centering on two key questions. We asked participants to reflect on the questions in small groups and report back. This post will highlight a few of the key takeaways — and also provide captured comments that maybe didn’t make it into this post.
What challenges exist for using metrics within your OSPO or Community?
The Consistent Use of Metrics within an Organization or Community
Many of the responses centered on the difficulties associated with the consistent use of metrics within an organization or community. For example, with the sheer size of many organizations and communities, the deployment and interpretation of metrics can vary a lot between people. This leads us, in the CHAOSS Community, to a 2023 goal of developing ways to communicate simple metric strategies and results that can be easily shared between people.
The full set of recorded comments for the first question with general categorizations is here:
Organization/Community:
There isn’t a central place for metrics discussions within an organization.
There isn’t a common taxonomy for how to speak about metrics.
Fragmentation of metrics within an organization can make consistent use difficult.
The size of an organization can make consistent use of metrics difficult.
The size of a community can make consistent use of metrics difficult.
Different stages of project maturity can make consistent use of metrics difficult.
There is minimal guidance on replicating CHAOSS structures locally.
Metrics/Metrics Model:
How to determine business value from a project?
How to measure the value of participation in a project?
How to determine business risk from a project?
How to determine company impact on a project?
How to measure the cost of non-participation?
Metrics themselves can be quite complex
Photo Credit: Sean Goggins
What should the CHAOSS project be working on in the future?
Building Collaborative Communities and Helping People Communicate Better
There are many, many things that we could work on within the CHAOSS project in 2023. Recurring themes from CHAOSScon include (1) helping people connect with others in similar contexts to discuss health-related concerns (i.e., corporate open source program offices) and (2) developing ways to help people communicate about metrics within their respective organization or community.
The full set of recorded comments for the second question with general categorizations is here:
CHAOSS Software:
Enable all contributions to be seen in a dashboard.
Add social, design, and educational material use (like a social calendar) into a dashboard of contributor metrics to cross-visualize data about the ecosystem beyond code.
CHAOSS Operations:
Continue to support the newcomer experience.
Assist others with the interpretation of results.
Have mechanisms to help interpret the data in non-biased ways / manipulating it towards certain predetermined hypotheses.
Provide badging to validate business metrics strategies.
Provide metrics and metrics model validation.
Support user groups that need specific metrics and metrics models to help in a variety of contexts.
Develop personas for groups of people with similar interests.
Build better starting points for people who may not want to build metrics.
Have social justice built into the metric development process, not as charity, but to address systemic issues continuously.
Provide Pathways into the metrics, more context on where they have been used in the past (e.g. to answer the question what do people like me use?).
Think about people who work in different languages – is CHAOSS accessible for them?
CHAOSS Communication:
Talk with other communities to understand different views of metrics.
Tell user stories of how others are using the metrics and metrics models in practice.
Provide a framework for how to apply metrics.
Provide ways to represent and talk about metrics in business meetings.
Push a goal-oriented approach.
Articulating why metrics are important to help the business case.
Provide a set of five metric models to use that do five different things.
Metrics/Metrics Model:
Software license compliance between repos
Test coverage in repos.
Code quality
Pipeline status
Ensuring a broad community of voices.
Build productivity and efficiency metrics and metrics models.
Need to understand the signal-to-noise ratio.
More human-centric metrics, the kind that help to signal/prevent burnout.
An estimate of how difficult it might be to implement. A signal of whether tech experience is required to implement a particular metric, or if it can be done by a quick survey, or if it can be identified automatically, or if it requires big program management lift.
Thanks again to everyone for an amazing CHAOSScon 2023 in Brussels! We hope to see you soon!!
This week, one of our design contributors at CHAOSS Africa came up with a great idea that could enhance one of our ongoing projects. This idea wasn’t listed among the open issues in our GitHub repository. The contributor was unsure of how to proceed with her contribution, which is a common scenario for many designers who are not familiar with GitHub.
If you’re a designer interested in making open-source contributions, here are two key things you should know about contributing to projects in any GitHub repository:
Pick a design issue you’d like to work on and talk to the design maintainer. Once you understand what needs to be done, you can start working on the issue. You can also start a conversation using the comment section on GitHub and seek guidance from others involved in the project.
Look at current projects and see how you can help improve them. Check the open issues in a repository to find tasks that address a problem you’d like to solve. If there isn’t an issue related to your contribution, initiate a discussion with the project maintainer on any communication channel they are available. Then make sure that an issue is added to GitHub before you work on your idea.
Why is it important to open an issue before making a contribution? Identifying a problem in an open-source project doesn’t necessarily mean you have to solve it. Another member of the project may have the technical or design skills to resolve the problem. Also, some ideas may be out of the scope of the project. Either way, identifying a problem or idea is a valuable way to contribute, and done by opening a GitHub issue.
Think of it like testing a mobile app and discovering a usability issue or bug. You don’t have to be the UX designer or the Software Engineer to solve the problem, but your discovery can still help improve the app.
In conclusion, GitHub can be a great tool for project management, including assigning tasks, monitoring progress, and tracking contributions from members of a project. Don’t be intimidated by its technical nature. As a designer, your contributions are valuable and can help enhance projects in your organization’s repository.
I would love to hear from other open-source design contributors on any points I may have missed. Please share your thoughts and experiences on CHAOSS Discourse.
As many of you know, CHAOSS has been working with a team of DEI advisors for the past 2 years via an internal audit to help our project center diversity, equity, and inclusion in all that we do. This process included a Community Survey, which we launched in October of 2022.
This survey was designed to get a general idea of the experiences of all our community members (Chaotics), no matter where they were in their CHAOSS journey. We also wanted to surface places for improving our policies and procedures to make a more welcoming and inclusive community. We asked questions about demographics, the ways in which they were contributing, their experiences with being a part of the community, and their suggestions for the future.
We will write another blog post reporting back on the results and our plans for addressing them, but in the meantime, we thought it would be helpful to share the survey structure with other open source communities who might benefit from a survey of this kind.
We chose to use Limesurvey to administer our survey because not only is it open source, it is a highly customizable and accessible tool. (The free version does have a few limitations.) We found that a tool like this one allowed us to fulfill the requirements that were important to us:
Anonymous responses allowed
Different kinds of response types allowed
Hosted in a location that was GDPR-compliant
Flexible presentation of questions
Restricted access to data
We will be sharing some lessons learned from launching this survey at this week’s FOSDEM and CHAOSScon, so if you’ll be there, let’s connect! We will also take the lessons learned from launching this survey to our Chaotics, and iterate on this survey for next time, but for now, you can find links to the survey structure here:
Every year seems like a big year for CHAOSS, and this year was no exception. Here are some of the milestones we reached in 2022:
Our Slack community grew to almost 1000 members! (Huge welcome to all our new faces. We are so happy you are here!)
We launched CHAOSS Africa and Ruth Ikegah is serving as the CHAOSS Africa Community Lead! (Ruth has done a fantastic job bringing newcomers to CHAOSS and open source in general.)
Our CHAOSS Asia-Pacific community became instrumental in helping build metrics models and working with OSPOs (Shoutout to Xiaoya Xia and Yehui Wang for leading that community to where it is today.)
To date, we have issued nearly 100 DEI event badges, and we doubled our team of Badgers! (We are so appreciative of our Badging Team and all those who filled out an application to get their event badged.)
We officially released our first Metrics Model: Code Quality Guarantee! (This group has been working hard to define what should be included in our Metrics Model definitions and what those look like.)
We are so incredibly grateful to all those who have attended community and working group meetings, contributed code, created design assets, participated in our community survey, greeted newcomers in Slack, helped review old metrics, improved our documentation, offered suggestions, helped another Chaotic find an answer, or any of the myriad of other ways our contributors have participated in the global CHAOSS community. You are what makes CHAOSS move forward and we couldn’t do much without the strength of our community. Here’s to an amazing 2023— our future has never looked brighter!
CHAOSS is very excited to be working more closely with the TODO Group as an Associate Member, to collaborate around open source community health metrics specifically for OSPOs. We often enjoy participation from folks working in OSPOs at all stages of their development, but previously we didn’t have a dedicated working group designed to address their specific needs. Coincidentally, we noticed over time that OSPO-relevant topics would pop up in our Value Working Group occasionally, and this was the place where OSPO managers would often enter the CHAOSS community.
As a result, we decided to formally pivot our Value Working Group from focusing on measuring the value a community offers individuals and companies, to an OSPO Working Group that would focus on the specific needs and challenges of those in the OSPO space, regardless of industry. We also wanted to give OSPO managers a place to connect with each other to talk about their challenges with building and maintaining a successful open source program.
With this new direction of our working group, we are looking to the larger open source community to help guide and build this team. We would love to get feedback from those working in the OSPO space, whether it be in a company, for a university, or any other place. What kinds of things are you measuring? What would you like to measure? What have been your successes or barriers to getting the data you need?
If you are interested in joining this conversation, we invite you to do 2 things:
A core value of CHAOSS is centering diversity, equity, and inclusion in everything we do. In order to make our community more welcoming and inclusive and continue to center DEI, we have been working with our DEI Audit team to develop a community survey. With this survey, we hope to increase our understanding of community members’ experiences within CHAOSS, and surface areas for improving our policies and practices.
We highly encourage all CHAOSS community members (past and present) to share their thoughts and experiences by completing this survey.
This survey:
has 14 questions in 3 sections
should take around 10-15 minutes to complete it
is completely anonymous and no personal information will be collected
is GDPR compliant
will be open until October 12
It is important to stress that this survey will remain completely anonymous. We will only share the results in an aggregated format, and high-level insights will be shared publicly by the DEI Audit Team. The results of this survey will be used to help the Audit Team develop recommendations for improving DEI within CHAOSS.
If you have ever considered yourself part of the community, we want to hear from you! You can take the survey until October 12 by following this link.
In early spring of 2021, the CHAOSS project embarked on a 9-month journey to reflect on its own practices and policies surrounding diversity, equity, and inclusion within the project. We were fortunate enough to receive a grant from the Ford Foundation to complete this work with the idea that we could not only improve DEI in our own project but also help other open source projects who are wanting to do the same. Centering DEI in an open source project creates a healthier and more productive environment for current members, it helps lower the barrier to contribution for others, and it fosters a more diverse and stronger community. This audit was a fantastic experience and in 2022 we will be continuing to implement ideas that stemmed from this work.
It’s been a year since the introduction of the CHAOSS DEI (Diversity Equity and Inclusion) badging initiative in the September of 2020, and we are happy to announce the release of CHAOSS DEI V3. This version introduces new and well-defined metrics and improved guidelines for reviewers. The CHAOSS DEI badging Initiative will also be issuing badges to open source projects in this new version.
Open source projects and ecosystems organize events to bring together the community members. These events provide the space for collaboration, deepening relationships, and making new friends.
The CHAOSS App Ecosystem Working Group describes a set of metrics for event organizers. The metrics are designed to support four goals that open source event organizers may have:
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
Cookie
Duration
Description
cookielawinfo-checkbox-analytics
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional
11 months
The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy
11 months
The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
