Category

Blog Post

Unveiling the Impact: DEI Metrics Overcoming Social Barriers in Open Source

By Blog Post

For decades, the topic of inclusion and marginalization of underrepresented groups in Open Source software communities has been a prevalent topic of discussion. There have been recurring studies and reports on the state of open source participation that spotlight persistent issues that plague these communities; Marginalization of certain individuals based on their race, gender, ability, skills, etc.  For instance, we see today that representation of women constitutes an underwhelming 18 to 23% of the tech industry, while in open source, women, along with other gender minorities, make up less than 10% of the population. Another common disparity that is often overlooked is the underrepresentation of neurodiversity groups, which is less than 11% in the tech industry.

The question of how an Open Source community can break down these barriers to participation and foster more inclusion and belonging for all individuals, regardless of abilities, skills, experiences, and differences, is a key to Open Source community health.

DEI metrics and initiatives have been introduced to play a crucial role in addressing this question. In October 2022, the Linux Foundation’s Community Health Analytics in Open Source Software or CHAOSS Diversity Equity Inclusion (DEI) Working Group launched an interview campaign to explore how CHAOSS DEI metrics are perceived and to identify ways to enhance the efficacy of these metrics in addressing these barriers. In this article, we present findings from these interviews that shed light on the role of DEI metrics in overcoming social barriers in contributing to open source. These interviews highlight the common challenges of DEI initiatives and their impact on open-source projects. We will explore how these DEI metrics can contribute to overcoming social barriers and create more welcoming environments for contributors from all backgrounds.

The State of Diversity, Equity, and Inclusion in Open Source

To gain a better understanding of how CHAOSS DEI metrics can impact Open Source community health, let’s dissect recent studies and findings that showcase the state of diversity, equity, and inclusion within open source.

Social Barriers to Diversity & Inclusion
  • Exclusive Language and Behavior: The use of exclusive language, threats, disrespectful behavior, and online harassment can alienate potential contributors, making them less likely to participate. During our interviews, some participants admitted that they often experience microaggression on day-to-day interactions within their respective communities. Additionally, reports from the Linux Foundation 2021 research also indicate that Women, non-binary, LGBQ+, and people with disabilities were twice as likely to have experienced threats of violence in the context of an Open Source project. While transgender respondents were three times as likely.
  • Lack of Representation: Some Open Source communities still struggle with the challenges of homogeneous leaders and members. A research paper published by IEEE states that the lack of gender diversity remains an ongoing issue among Open Source projects. However, many of the projects also suffer from a lack of female inclusion in leadership positions. When underrepresented groups do not see individuals who look like them in leadership roles or as active contributors, it can be discouraging, making them less likely to get involved. The Linux Foundations report also indicated that ​​although 82% of participants felt welcomed in open source, demographic segmentations show varied sentiments. The 18% of those who do not feel welcome were from disproportionately underrepresented groups: people with disabilities, transgender people, and racial and ethnic minorities. Also, during our interviews, some participants pointed out that in some communities, while minority groups are encouraged to participate and contribute, they are often overlooked for leadership roles.
  • Socioeconomic Barriers: The 2023 open source maintainer research study indicated that open source contributors and maintainers are mostly students looking to build on their skills and experiences. Hence, access to resources such as a stable internet connection, a powerful computer, and free time to contribute can pose a challenge and create disparities in participation. When the question “Can you describe how your unique attributes, traits, characteristics, skills, experience, and background have affected your participation in open source?” was asked, an interviewee even  stated that “In some communities with lots of different people from various backgrounds,  the presence of various backgrounds doesn’t always translate to equity — in the sense that people who already have access to resources and advantages end up getting more, making the rich even richer.”
  • Education on Diversity, Equity and Inclusion: Recognizing and addressing common disparities that exist in our communities is essential for progress. It is nearly impossible to effectively address these matters without first acknowledging their existence and comprehending their impact. However, during these interviews, we discovered that over 40% of the interviewees were not knowledgeable about CHAOSS DEI metrics, and most of them were just hearing about them for the first time. The Open Source maintainer research study further revealed that fewer than one-third of interviewees recognized the presence of formal DEI programs within their projects. This highlights the urgent need to improve awareness and drive initiatives that foster diversity, equity, and inclusion within open source.
  • Accessibility: A critical aspect of Open Source software that often remains overlooked is accessibility. Unfortunately, a significant number of Open Source projects do not prioritize accessibility in their software, communication platforms, or technical documentation. This oversight has a profound impact on contributors with disabilities, who often experience a sense of exclusion when attempting to engage and collaborate within these communities. In our interviews, participants also mentioned that from their experiences,  accessibility is thought of as a feature to be added only later, after product releases.
  • Implicit Bias: Unconscious biases are also common factors that affect the way contributors from underrepresented groups are welcomed and how their ideas are received. These biases play a significant role in influencing the decision-making within a community, in turn affecting inclusive participation. Although some of the actions of our biases are not intended, they end up hindering the progress of diversity and inclusion efforts in an Open Source community.

What are DEI Metrics?

CHAOSS metrics are indicators used to assess Open Source community health. The DEI metrics aim to evaluate and reveal the diversity, equity, and inclusion efforts within open-source projects and communities. These metrics are used to:

  • Quantify Diversity: By collecting and evaluating data on the demographic makeup of contributors, DEI metrics initiatives provide a clearer picture of the community’s diversity. This data can help identify gaps and track progress over time. This data can help our leaders make informed decisions that create safe, welcoming, and equitable developer environments.
  • Reveal Disparities: DEI metrics could reveal/identify, to some extent, incidents of both explicit and implicit bias that might exist in a community. For instance, in communities where members with disabilities, women, or people of color are not considered for leadership roles, it could signal a bias in promotion or recognition. 
  • Monitor Inclusive Practices: Metrics observe incidents of harassment, the use of inclusive language, contribution participation, and other behaviors that affect the community’s inclusivity. They ensure access to resources (e.g., code of conduct) and understanding whether barriers to entry, like socioeconomic factors, limit participation. For instance, in a project with many non-native English contributors, the lack of translations in key languages for code, documentation, and guides poses a challenge. Another known barrier to open source participation is that much of it relies on voluntary labor, while the majority of unpaid care work falls on those in marginalized groups. Metrics can help us understand how we support diverse users and accommodate their needs.

Open source projects can leverage CHAOSS DEI metrics to proactively address biases, expand the talent pool, and ensure belonging, ultimately breaking down social barriers and promoting diversity in open source communities at large.

Why an interview campaign?

Since underrepresented groups have the most potential to benefit from diversity, equity, and inclusion (DEI) metrics in open source, we launched an interview campaign to help us evaluate actual DEI activities in OSS and how they relate to the metrics we presently measure at CHAOSS through interviews.

This study aimed to help us comprehend the impact of current DEI metrics in real-world circumstances, especially for underrepresented groups in open source. Identify new metrics and methods for measuring DEI and also identify gaps to be addressed in current DEI metrics. 

Do Metrics Address all of these Challenges?

While conducting interviews, we discovered that DEI metrics, though effective, may only provide a partial solution to some of the DEI challenges within our communities. Some participants shared valuable insights about the use of diversity metrics.

For example, some interviewees expressed concerns about relying solely on metrics to gauge progress. They argued that merely tracking the increase in the number of diverse individuals, such as going from 40% women in 2022 to 50% women in 2023, might not be a fully positive way to measure change or growth. Instead, the focus should be on whether the product or Open Source community has remained strong or improved since that time. If the answer is yes, then this improvement may be attributed to an increase in diversity.

Another participant emphasized that people from diverse backgrounds have unique perspectives and mindsets. They suggested that general metrics might not be inclusive of these diverse perspectives because they are often created by a group of individuals from one community without perspectives from the world.  Since diversity can mean a lot of things for different people, it’s hard to say these metrics will address all of these challenges from individual perspectives. Hence, using the same pool of individuals for the creation of DEI metrics could influence their credibility. 

Some participants raised concerns that relying on metrics might lead to unfair judgments of people’s abilities and skills. While others shared their experiences of being assigned more unpaid work because they were the only diverse representatives for an organization. Additionally, some feedback questioned the effectiveness of DEI metrics in communities or groups with a higher proportion of individuals who are neurodivergent, stating that it can lead to challenges in understanding and adhering to the customary social norms or expected behaviors within that community.

In terms of successfully implementing these metrics, we understand that DEI metrics alone may be unable to address all DEI challenges if the barriers of these DEI are not recognized and tackled. Overall, feedback from interviewees emphasized that it’s not just about creating metrics but empowering people to understand and use them. So, even if you have metrics, they will only be effective if people are aware of them and use them to make changes for the better.

Impacts of CHAOSS DEI Metrics on Open Source Communities

CHAOSS DEI metrics have found application in various scenarios, spanning software development, governance and policies, events, communities, and beyond.

The interview campaign comprised an anonymous survey with over 180 participants and one-on-one interviews with 19 participants. 

During the interviews, we got to ask participants to share their experiences with Diversity, equity, and inclusion in open source. To help us understand how their differences may have influenced their participation and how communities can leverage metrics to address some of these barriers.

Based on these interviews, we gained insights from participants who shared real-time examples of how these metrics have been instrumental. They shed light on how CHAOSS DEI metrics have positively impacted individuals and communities.

CHAOSS DEI Metrics Scope
  • Some participants mentioned that they have greatly benefited from the CHAOSS DEI metrics at events they attended in the past, and these metrics were enforced for a more inclusive event culture. Notably, they appreciated open-source events that considered attendees’ safety, diverse speaker line-ups and live captioning for some talks, which was especially valuable to attendees who may be hard of hearing. Additionally, they indicated that the availability of Diversity Access tickets eliminated concerns about their participation cost.
  • For event organizers, the CHAOSS metrics on event diversity offered valuable insights into better connecting with attendees and speakers, handling conflict cases, and improving the overall event experience. They emphasized that DEI metrics were pivotal in encouraging active participation and enforcing a code of conduct—for example, the State of Open Con23 and Open Source Summit Europe 2023.
  • Participants indicated that these DEI metrics were utilized in their project to ensure smooth collaboration and healthy communication among team members. They also added that these metrics were helpful in assessing their progress, particularly in areas like leadership and mentoring, and in supporting contributors with diverse backgrounds and skills, ensuring inclusive contributions. Example the Apache Traffic Control project.
  • A participant said the DEI metrics were being utilized in the  DEI research that was conducted by the Apache Diversity and Inclusion project. They said these metrics were being incorporated to help determine actionable steps to take based on their research outcomes.
  • Some participants also said the CHAOSS metrics have been instrumental in the development of a code of conduct and implementing inclusive practices in their communities and projects—for example, The Good Docs Project. 

Harnessing the Power of DEI Metrics in FOSS

Following the conclusion of our interview campaign, it became certain that CHAOSS DEI metrics have the potential to address various challenges and help address barriers to contribution within respective communities.

Diversify Recruitment: 

  • Metrics on community activity can be used to diversify the recruitment process and offer equal opportunities to contributors regardless of their differences.
  • Metrics on labor investment can also be utilized by open-source leaders to understand how much value a potential candidate provides to a project as they contribute. 
  • Metrics on contributor onboarding, mentorship, and contribution attribution can help community leaders identify a person suitable for a leadership role or full-time hire and subsequently craft a position that aligns with their strengths.

Diversify Leadership: 

  • Metrics can be used to diversify the leadership board and serve as a key performance indicator for community health.
  • Metrics on board and council diversity can help in promoting inclusive representation within communities.
  • By improving the power structure within communities, most Open Source communities can avoid dictatorships among leaders.
  • Organizations and community leaders can use DEI metrics to brainstorm strategies for diversifying their communities and to assess whether they are effectively fostering inclusive participation.

Inclusive Event Participation:

  • Metrics are helpful for their event, promoting inclusive events and cultures. 
  • Metrics on public health and safety can be used to ensure participants feel safe to attend in-person events regardless of their differences.
  • Metrics on the event code of conduct are helpful to organizers in figuring out better ways to relate with attendees.
  • Metrics on diversity access tickets also help participants from underrepresented groups in attending events.

Psychological Safety Awareness:

Inclusive Contribution & Retention:

  • Metrics on mentorship can be employed to help mentors and mentees collaborate better. 
  • Metrics on leadership and mentorship can help encourage more folks to participate, also giving equal opportunities to contributors regardless of the differences.
  • Metrics relating to contributor onboarding and conversion rates can be used to discern the progression of new community members as they evolve into enduring and engaged contributors.
  • Metrics on mentorship address the common barrier of entry for contributors, especially those from underrepresented groups.

Accountability: 

  • Through the CHAOSS project badging initiative, metrics can be used to hold Open Source projects accountable for their diversity and inclusion efforts. By quantifying the DEI status quo, communities are more likely to take action to address disparities.
  • These can be used to tackle the segregation, language barrier, similarity, and conformation bias that exist within most communities.
  • CHAOSS DEI metrics can cover access to knowledge and education so more people are aware of these differences and how to respect and abide by the code of conduct.

Data-Driven Decision-Making: 

  • Metrics can be used to gather data on demographics, diversity in contribution roles, and psychological safety.
  • Metrics can help communities in creating  & enforcing code of conduct.
  • Metrics enable communities to make informed decisions to address specific DEI challenges.

Conclusion

CHAOSS DEI metrics are becoming invaluable in addressing diversity and inclusion challenges within Open Source communities. While they hold the potential to transform these communities into more inclusive and equitable spaces, it is essential to acknowledge and address the associated challenges, foster understanding, and champion the adoption of DEI metrics across the Open Source landscape. Only through collective effort and ongoing commitment can Open Source communities continue to evolve and thrive in the spirit of diversity, equity, and inclusion.

Acknowledgments

A heartfelt gratitude to the numerous individuals who have contributed to the successful completion of this research article.

First and foremost, appreciation goes to Matt Germonprez, Sean Goggins, Foundjem Armstrong, and Vinod Ahuja, who provided invaluable insights, critical feedback, and unwavering support throughout the research process. Your dedication and expertise were instrumental in shaping the quality of this work.

A special shout-out to the individuals who generously participated in these interviews and shared their knowledge and experiences, enriching our findings significantly. Your openness to sharing was pivotal to the success of this interview campaign.

Finally, a big shout-out to the CHAOSS community members for their ongoing efforts and dedication to advancing open-source community health metrics and analytics. And for supporting the successful completion of this interview campaign.

Pile of intertwined measuring tapes of various colors

Demonstrating OSPO Value and How CHAOSS Can Help

By Blog Post No Comments

Recently, I’ve been thinking about how Open Source Program Offices (OSPOs) can demonstrate value within their organizations and how CHAOSS metrics and software can help. As a result, I’ve given presentations, had discussions on a podcast, and wrote a blog post about this topic. I wanted to write a summary blog post here to highlight the work in one place while tying some of those conversations together into a broader narrative.

I recently attended OSPOlogy Live in Frankfurt, which had presentations and engaging roundtable discussions about a wide variety of topics relevant for OSPOs. My presentation was about Getting More Value from your OSPO, and I talked about how OSPOs can take a more strategic approach by fostering alignment between individual contributors, business unit leadership, and the communities where employees contribute. The presentation also spent quite a bit of time on how an OSPO can demonstrate value toward accomplishing the overall goals of an organization while using metrics to make improvements and demonstrate success toward meeting those goals. Sean Goggins presented on the topic of Selecting the Right Collections of Sustainability Metrics with a focus on how OSPOs can be data providers that can help an organization make sense out of the mountains of data generated by open source software. CHAOSS Metrics Models help OSPOs focus on collections of meaningful data to create something that provides insight and wisdom about their open source efforts. Ulrike Fempel from SAP’s OSPO wrote a great wrap-up of the Frankfurt edition of OSPOlogy Live if you’d like more details. If you haven’t already attended an OSPOlogy Live event, it’s a great place to discuss challenges and solutions with your OSPO peers!

Building on my presentation about showing the value of an OSPO and Sean’s talk about metrics models as collections of metrics, I wrote a blog post about Measuring Open Source Project Health for Opensource.net that focused mostly on the CHAOSS Starter Project Health Metrics Model. OSPOs can be overwhelmed by the mountains of data and metrics available to understand open source projects, so this blog post and metrics model are designed to help new OSPOs (or ones new to metrics) get started with a few relatively easy metrics. Not only are these metrics relatively easy to gather, they also make it easy to understand how to take action on the data to make meaningful improvements to the health of open source projects. The goal is to get OSPOs started on their journey into using data to learn and improve with the idea that they can expand on this and start measuring other things that matter to an OSPO.

Another thing that OSPOs care deeply about is the long-term viability of the open source projects that their organizations’ rely on for infrastructure along with the products and services that they deliver to their customers. It ultimately comes down to a complex assessment of risk vs. reward across many dimensions, including security, community, and governance, just to name a few. We recently released a podcast about Open Source Software Viability and Project Selection where Matt Germonprez, Sophia Vargus (Google), Gary White (Verizon), and I talked in depth about how to assess viability and some of the metrics used in those assessments. Gary is also working on publishing some metrics models and blog posts, so watch this space to learn more about measuring open source software viability. 

If you are interested in learning more, we have an OSPO Working Group within the CHAOSS project that we have jointly with the TODO Group. We meet every other Thursday and have Slack channels both within the CHAOSS Slack and TODO Group Slack workspaces if you want to join in these discussions or ask questions.

Photo by patricia serna on Unsplash.

A group of CHAOSS community members taking a selfie on the bridge leading to the Bilbao old town area

CHAOSS at Open Source Summit Europe

By Blog Post No Comments

The CHAOSS crew was well-represented at the Linux Foundation’s Open Source Summit event in Bilbao last week with several talks and panels from CHAOSS community members. 

On Tuesday, we held a panel discussion: Demonstrating OSPO Value with Daniel Izquierdo, Chan Voong, David Hirsch, and me. The idea for this panel came out of the CHAOSS OSPO WG, and during the panel we talked about how to demonstrate OSPO impact using metrics, practical applications for OSPOs, tools, and how to build a narrative for your stakeholders out of your data.

CHAOSS board member, Brian Proffitt, along with his Red Hat colleague, Natalie Pazmiño, held a session about the challenges of Measuring the Impact of Community Events, which can be harder to measure than traditional industry events that rely mostly on lead generation. They talked about creating collateral that can be measured (e.g., whitepaper downloads, landing pages via QR code) and creating opportunities for later participation in a channel that you can measure. They had some creative approaches, so I talked to Brian about the possibility of creating some CHAOSS metrics / metrics models to share their ideas.

Daniel Izquierdo and Yehui Wang had a session about Building SaaS Services with CHAOSS Technology to Evaluate Community Health and Sustainability where they talked about how CHAOSS’ GrimoireLab software is based on 16 years of research, development, and testing in the market, which made it possible for OSS Compass to be built on top of GrimoireLab in just one year! OSS Compass is a SaaS solution implementing CHAOSS metrics and metrics models, and the slides at the link above show examples of how they’ve implemented them. CHAOSS has brought great visibility for GrimoireLab, and the community has been a great amplifier. 

I also gave a talk about Contributor Growth Strategies for OSS Projects where I talked about the challenges that maintainers face and how hard it can be to get more people participating in a project along with some ideas for ways that these challenges can be overcome. I used several graphs from CHAOSS tools to demonstrate how metrics can help maintainers decide where to focus their efforts for growing their contributor base. The slides in the link above have more details about the challenges, solutions, and metrics. 

In addition to the talks from the Chaotics at the event, there were a few others that I found interesting:

  • Nithya Ruff’s keynote about the Evolving OSPO touched on several topics that we’ve been talking about recently in the OSPO WG. She talked about how risk can slow innovation, and how OSPOs are working hard to manage risks that include licenses, AI, security, and regulations.
  • Building On-Ramps for Non-Code Contributors in Open Source by Natali Vlatko and Celeste Horgan echoed many of the conversations we’ve had over the years in the CHAOSS DEI WG with some solid ideas for both maintainers and contributors about how to get more people engaged in your project through documentation, community, project management, and other roles.
  • There were also a bunch of other talks that were relevant for CHAOSS folks, especially some from the Diversity Empowerment Summit, Open Source Leadership Summit, and OSPOCon.

The individual session videos aren’t yet available, but the full day videos for some tracks are available, and this video from the Leadership track contains the talks from Daniel and Yehui (time index 16:20), Natali and Celeste (1:08:50), and my talk (2:05:01).

In addition to the content from the talks, Linux Foundation Research also released 4 new reports: The 2023 State of OSPOs and OSS Initiatives, The World of Open Source Europe Spotlight 2023, The European Public Sector Open Source Opportunity, and Open Source for Sustainability.

Overall, it was great to see many of my CHAOSS friends, some of them for the first time in person. We had great conversations and fun both at the conference and over pintxos, a traditional food in northern Spain’s Basque region.

Emilio and Miguel Angel at the Bitergia booth with an orange bitergia tablecloth and table covered in stickers, coasters, and other materials to give away.
Emilio and Miguel Angel at the Bitergia booth

Explore Compass Lab and CHAOSS Metrics Models

By Blog Post No Comments

The following is a blog post from our friends in the OSS Compass Project. OSS Compass is a SaaS solution, hosted in China, that is built from CHAOSS GrimoireLab software and deploys CHAOSS metrics and metrics models. There is also a video demonstrating how Compass Lab works from our August 29, 2023, CHAOSS Metrics Models meeting. You can check out that video here.


Explore Compass Lab, Making it easy to create an open source project evaluation model!

Hi, open source adventurers! Today, I’m here to introduce the latest marvel from the open source realm – the new darling of the OSS Compass (hereinafter referred to as “Compass”) world! Yes, you heard it right. It’s our new SaaS service – Compass Lab! Wait, you haven’t heard about it yet? Don’t worry, let me explain it all to you step by step. I promise it will leave you dazzled and excited!

In the world of open source, countless projects emerge every day. However, accurately measuring the health of an open source project remains a vexing challenge. Luckily, we have Compass, which is like a health check doctor for open source projects. And recently, a new member has joined – Compass Lab!

01? What is Compass Lab?

Compass Lab is an incubator for open source community evaluation metric models. It might sound sophisticated, but actually it’s essentially a “health evaluation center” that provides a comprehensive checkup for open source projects. This evaluation center not only offers data but also serves as a platform filled with fun and creativity.

Compass Lab gathers an enormous amount of data from over 20,000 open source projects, offering more than 200 project categories and over 100 evaluation metrics. This means you can conduct an all-encompassing health assessment for a project – the possibilities are endless! You can select different evaluation metrics and datasets based on your needs, making it a paradise for assessment enthusiasts!

02? How to “play” with Compass Lab?

It’s actually quite simple! First, you need to approach it with a curious mindset. Then, register and log in to the SaaS service platform of Compass Lab, where you’ll discover a fascinating world. You can choose existing evaluation models or customize one based on your interests.

➤General Steps to Create a Model

1. Open the model creation page:

On the Compass official website, locate the model creation area on the right side of the Lab page and click “Create a model now”or use the button “Create a model”at the top.

2. Fill in basic information:

Provide basic model information, such as ecosystem dimension, model name, applicable industry, and whether it’s public or private.

3. Select datasets:

Choose a dataset from the 200+ datasets that suits your needs, offering flexibility for various domains.

4. Choose evaluation metrics:

Select the metrics you care about from the 100+ metrics provided by CHAOSS and Compass, categorized by code, issues, pull requests, repositories, contributors, etc. And confirm your selection.

5. Adjust weights and thresholds:

Next, modify weight values in the percentage input box or drag the slider to adjust weights (default is rightward only). Revise thresholds in the right input box (ensuring the values stay within the provided range). This process is similar to customizing your own pizza, adding toppings according to your taste!

6. Choose algorithms:

Currently, we only support the default algorithm (as shown below), but don’t worry, it’s a remarkable algorithm crafted by the expert Rob Pike himself! For an in-depth understanding of this algorithm, refer to our previously released article “OSS Compass Scoring System Switch:Watch Algorithms Transforming into Magic!” If you have more thoughts on algorithms, join the Compass Slack channel to discuss with fellow open source adventurers.

7. Publish the model:

After confirming all the information, click “save” to publish the model.

Next, let data and creativity collide here to create your own project health model!

03? Your creativity finds a home here: creating a LLMs projects evaluation model!

Compass Lab not only offers general project evaluation models but also provides a special space for developers with unique ideas. If you have an extraordinary open source project and want to assign it a health score, this is the place for you! Here, you can customize an evaluation model for your project, allowing your creativity to soar!

For example, if you’re interested in open source LLMs projects or have your own open source LLMs project, you can create an evaluation model for it:

➤Steps for creating an LLMs projects evaluation model

1. Give your model a catchy name:

Besides naming, remember to choose whether it’s for a general or specific domain and whether you’re willing to make the model public.

2. Select datasets:

Choose projects under the categories of open source LLMs and open source LLMs tools to facilitate cross-category comparisons. (Our dataset is continuously expanding!)

3. Choose metrics and adjust weights and thresholds:

Select appropriate metrics based on the characteristics of open source LLMs projects, such as organization count, code commit frequency, ongoing contributions, etc. 

Adjust weights and thresholds for each metric following the previously demonstrated steps.

Then, click “save” and the model is created.

4. Analyze and view assessment results:

Click “Analyze” and wait for it to complete – then you can view the assessment report.

5. Add comments:

In addition, we’ve added a comment feature to the dashboard page. Click the comment button in the upper right corner to share your thoughts.

6. Invite friends to collaborate:

You can also invite friends to collaborate and help you create a more perfect model. Just click “Member” on the model page, set collaboration permissions, and send invitations!

04? Explore limitless fun in the open source world, all in Compass!

Through today’s introduction, I believe you now have a deeper understanding of Compass Lab. Here, you can not only explore various dimensions of open source projects but also customize evaluation models for your own ideas, combining creativity and data. This “evaluation playground” in the open source world awaits your participation! 

If you haven’t experienced Compass Lab yet, take action now! Refer to below operation demonstration and create your evaluation model! Let’s explore the boundless fun of the open source world together, and unravel the mysteries of data! 

Remember to stay tuned for more interesting tech news and creative activities. Until next time, let’s embark on an infinite journey of open source creativity together!

Join OSS Compass Community

In the future, OSS Compass will continue to evolve and give back to the broader user base, providing greater convenience for open source project evaluation and assisting in the healthy and sustainable development of open source communities.

We look forward to more developers joining the OSS Compass community and contributing to its development. At the same time, we welcome users to continue using Compass SaaS services and providing feedback, continuously powering the enhancement of OSS Compass’s capabilities.

image2.png
https://join.slack.com/t/oss-compass/shared_invite/zt-1ttt9sv5h-8E~oPP6VJqm8ero5qH9LlA

 

CHAOSS DEI Self-Reflection Summary

By Blog Post No Comments

Several months ago, we ran a survey to better understand the CHAOSS community. Thank you so much to those who participated in that survey! We really appreciate your valuable input. From the survey, we were able to identify five themes that you told us about. We’ll describe those five themes here and discuss what we are doing in the CHAOSS project to help address these themes.

 

 

Theme 1: Clear Communication  
Comments What We’ve Done in CHAOSS
Processes about how to check-in with the wider community would be helpful. Each regional chapter hosts their own Community Meetings and we post those meeting summaries and links to the meeting recordings in Slack and on Discourse, so anyone in the community can quickly stay up to date on what’s going on in other parts of CHAOSS. We will continue to improve cross-chapter and cross-group communication among the entire CHAOSS community through our newsletters and weekly CHAOSS Community meetings.
It is hard to navigate the detail of the focus areas Sifting through the ever growing list of CHAOSS metrics has proven to be challenging for our users and those new to the community. Previously, focus areas were used to categorize metrics into contextual buckets. The user would have to be familiar with which working group developed the metric, and which focus area we decided provided the best fit. This required a level of knowledge of CHAOSS and the way we work that was a barrier for those new to the community.
We have since moved away from displaying metrics in this way, and have moved toward a more intuitive model of categorization. First we assign 1-2 high-level topics to each metric, then we assign a number of keywords, which provide more context. This gives the user the ability to search for more specific criteria.
Also, we improved the UX by no longer displaying metrics in one single document. Now there are visually clear “tiles” on our website that allow users to easily explore our metrics based on the high-level topic they’re interested in.
With the addition of our metrics models, we also help the user identify sets of metrics that naturally come together.
Theme 2: Working Groups  
Comments What We’ve Done in CHAOSS
Sunsetting working groups may be a good thing. The structure of CHAOSS has evolved over the past year to shift away from the concept of “working groups that develop metrics through a particular lens” and more toward the context in which metrics can be used to help answer questions. Because some metric development benefits from specific areas of expertise, we chose to retain our DEI and Risk Working Groups, but have effectively folded the rest into other groups. We now have 3 working groups focused on metric development: Risk, DEI, and Common.
Preference for the real use cases in metrics models With the hiring of our Director of Data Science and the launch of our Context Working Groups, bridging the gap between theoretical and practical use of our metrics and metrics models will be a primary focus of CHAOSS moving forward.
Contextual work can be distinct from metrics working groups, and this is good. As mentioned previously, the evolution of CHAOSS means a shift toward a separation of Context Working Groups and the working groups responsible for metrics development. In this way, we are able to collaborate with our user community to answer the questions that arise in their individual contexts. We now have 3 Context Working Groups where these high-level discussions occur: Corporate OSPOs, University OSPOs, and Open Source Scientific Software. An outgrowth of our continued discussions with CHAOSS’s expanding community may be additional Context Working Groups.
The Common WG seems to be where “things that don’t clearly go anywhere fit”. The Common Working Group has indeed been through a few iterations! It does seem to act as a catch-all, especially with regard to metric development. But as we have pared down the number of Working Groups, it makes more sense that this development would land in Common. CHAOSS will continue to communicate that message and clarify the mission of the Common Working Group.
Theme 3: Software  
Comments What We’ve Done in CHAOSS
Build community around software. This has always been challenging for us, and it’s something we continue to strive for. With the hiring of our Director of Data Science, and conversations with potential partners, we aim to allocate and acquire additional dedicated resources for this effort. This will likely be a focus for us in 2024, as it is critical to the sustainability of CHAOSS software.
Theme 4: Newcomer Experiences  
Comments What We’ve Done in CHAOSS
Improve processes of where to begin, especially when a person comes to CHAOSS with an interest where work is being done in the many places. Newcomer experience has been a primary focus for the past 12-18 months.
With the creation of a newcomers Quickstart, we have a central document with a few action items for newcomers to complete. This allows them to learn about CHAOSS and be more prepared to fully join our community.
We are improving the long term onboarding experience by moving away from a monthly onboarding call (which was overwhelming and not as inclusive as it could be because of time zones) toward building out a set of learning modules for newcomers to work through at their leisure. This will occur in a learning management system and will allow newcomers to move at their own pace, and learn about CHAOSS in digestible chunks.
We also launched a Tour Guide program whereby newcomers can receive individual attention and guidance to find the spaces within CHAOSS where they might have an interest in participating.
Promote new opportunities for people to engage with the community as they come up. As mentioned previously, our Tour Guides act as a link between opportunities and new contributors, but we are also organizing the many sub projects within CHAOSS through a more structured project management framework. By bringing together contributors with project management experience, we can allow them to contribute by keeping these smaller projects on track, clarifying engagement opportunities, and communicating them regularly to our community.
We also promote the use of the “First Timers Only” labels on issues in our many repositories.
Make “where to begin” clearer in documentation. With the development of our Quickstart, and prominent links to this on the CHAOSS website homepage, newcomer Slack channel, and in the newcomer Slack Bot, we now have one central place where we can point newcomers.
Theme 5: Community Experiences  
Comments What We’ve Done in CHAOSS
As the community has grown, it is beginning to feel more fragmented. This is a challenge for any global community with a lot of moving parts. We have many Slack channels and many sub-projects and it’s difficult to keep track of what is going on with everyone.
We implemented a Discourse Forum with the idea that it would be a way to bring the whole community together in a more substantial way than many fragmented Slack channels, but this has not been as well received as we hoped. The CHAOSS community will continue to iterate on and think about ways to combat this issue.
A learning pathways framework would be helpful. As mentioned, CHAOSS is currently building out such a framework using a learning management system. We think this will bring clarity to the onboarding process and help educate potential contributors so they can actively engage with the community in the way that means the most to them.
There are times that focus on tools becomes too central. While tools are a big part of CHAOSS (even appearing in our name), they are definitely not the only thing CHAOSS works on. We hope that with clearer communication on all the sub-projects happening in CHAOSS, it will not feel like our tools take center stage.
There may be too many working groups. As mentioned, we have effectively cut our metrics development working groups down to 3.
Provide more workshops, peer mentoring, and programs on software and metrics (perhaps also newcomer experience) We plan to include some of this type of education in our learning management system modules. Currently, we don’t have the individual resources to consistently host other types of workshops (aside from the occasional CHAOSScon). By making it easier to learn about CHAOSS, we hope to grow our community of leaders who can facilitate these types of workshops. It would be outstanding to be able to provide these experiences for our community!

 

Survey: Help the CHAOSS project improve our tools and metrics

By Blog Post, News No Comments

We know our metrics and tools can be overwhelming, even for experienced open source professionals. As we ramp up our new CHAOSS Data Science Initiatives, we wanted to start by learning more about what works well and what doesn’t for people using CHAOSS tools and metrics now or in the past. Understanding the challenges that people have experienced will help us drive improvements within the project to overcome those challenges.

We are launching a survey of existing and past users of CHAOSS tools and metrics designed to help us better understand the barriers and challenges that make it difficult for people to gain meaningful, empirically-driven community health insights using CHAOSS tools and metrics.

If you have ever used tools based on CHAOSS technologies (e.g., Augur, GrimoireLab, Bitergia, Cauldron) or used other tools to implement CHAOSS metrics, we want to hear from you! 

Take Our Survey

 If you’re interested in learning more about our new CHAOSS Data Science Initiatives or joining the community, you can join our #data-science Slack channel or attend our Data Science Working Group meetings to collaborate on data science work in the CHAOSS community. 

CHAOSScon Africa 2023 Wrap

By Blog Post No Comments

epic moments, inspiring insights, and a whole lot of awesome!

CHAOSScon Africa 2023 Wrap

Let me tell you how the first ever CHAOSScon Africa went down. If you couldn’t make it to the event or did but need a refresher of the beautiful memories? This piece is for you. Ready? Hold my beer.

Sometime in May, plans for the first CHAOSSCon Africa started. Okay, wait, if you’re going to read this, you need to know what the project CHAOSS is about and why we hold these conferences. If you are new to CHAOSS, you should continue reading, otherwise, you can skip this part.

What is CHAOSS??In CHAOSS, we help open source communities understand the pulse/health of their communities. We do this by creating metrics and metrics models. To learn more about CHAOSS, kindly visit our website here.

Why CHAOSSCon??CHAOSSCon is a gathering of the CHAOSS community members and people fascinated by the project, where current happenings in the project are shared, its use cases and more. Interestingly, it’s not only for the CHAOSS community members. It’s also a gathering to learn about metrics and tools used by several open source projects, communities, and engineering teams to track and analyze their development activities, the community’s health, diversity, risk, and value.

CHAOSSCon was first held in Europe and co-located with FOSDEM in 2018. Since then, it had held in Europe and North America up until 2023. The first ever CHAOSSCon Africa was held, co-located with OSCA Fest in Lagos, Nigeria. You should see the list of past events from the CHAOSS community here.

…..on the real deal

Backstory on the planning and execution of CHAOSSCon Africa ?

When planning for CHAOSSCon Africa started, the organising committee had a few things in mind, bringing together developers, designers, community managers, and others interested in contributing or building open-source software to learn and connect with each other. We wanted to have an inclusive, diverse and memorable environment for attendees. And we achieved just that! See the post below for a quick highlight of the event.

Meetings around the event planning officially started on May 9. About a month into the event, and within that short time, we were able to secure funding from both GitHub and Bitergia. Secure the venue hall, and book an amazing food vendor that understood diversity and made the right meal for everyone. Shortlisted our speakers, and our keynote speakers said yes to our proposal. Got a great number of interested attendees. And all of these wouldn’t have been possible without the collaborative efforts of everyone; from the programs team co-led by Busayo and Ruth Ikegah who made sure the event ran smoothly, to the design team led by Kingsley who made sure designs effectively communicate the vision and expectations of the conference were ready and timely. To the communications-focused group led by Oluchi who made sure updates on CHAOSSCon Africa reached our audience and everyone was carried along in the buzz.

How did I forget Maryblessing (yours truly) who majorly was an errand girl for almost all the focused group ?. See the full list of the organising committee members on the event page here.

…..on community

As you already know, CHAOSS is big on community, in it our strength lies. We actively encouraged members to register and attend the conference, including members from other non-opensource communities. We gave out 100% discounted tickets to these communities to further encourage them, which in turn we hope got them excited about open source and not just CHAOSS as a project.

…..on learning

Remember I said we wanted attendees to experience true learning? Yes! We had the best keynote speakers grace the stage and amazing speakers who touched on different topics. Here are highlights of some of my favourite talks:

  1. Navigating the CHAOSS community: from beginner to Pro by Anita Ihuman our second keynote speaker, where she spoke on how new contributors in the CHAOSS community can get involved in the CHAOSS project using her experience as a case study. She touched on so many relatable stuff and for a moment it made me reflect on how far I have also come in the community looking back from the first day I joined.If you’re excited about what we do at CHAOSS, watch this session’s recap here. Please skip ahead to 1:15:42 of the video.
  2. Revitalize Your Community: Prioritizing Health and Well-Being for a Stronger, More Connected Community by Blessing Eloho. I was excited about this talk because it’s all we’re about in CHAOSS, and seeing someone preach more community health strategies was an absolute delight. The examples she gave were refreshing to hear and the points she made are truly essential.If you want to watch the recap, you can access it with the first link and skip ahead to 12:52 of the video.
  3. Lastly is The Open Source Launchpad: A Then & now look at tech careers by Justin Flory our first keynote speaker. It started off by defining what open source is, and I really loved his definition of open source at the conference. Justin’s talk touches on Tech careers and how open source fits into the puzzle. There’s an unending argument of if open source is free or not, and in this talk Justin let us know what the ‘free’ in open source is. Another point he talked about is what the future of building on ‘free’ stuff is.Again, if you want to watch the recap, click this link and skip ahead to 1:03:25 of the video.
  4. I also loved Brayan‘s talk but couldn’t get a lot while his session was ongoing but you should check the recap here. He spoke on Building Trust in AI: The Importance of Explainability in Open Source Projects.

…..on connecting

We had a few goals in mind, one of which is to connect opensourcerers together. When attendees leave the venue, what should they leave remembering? For us, we wanted that total experience package. Leaving nothing!

How do you feel looking at the above pictures? For me, I feel like going back to June 14, it was the best day.

…..on experience

Without a doubt, you could tell that CHAOSScon Africa 2023 was a blast.

I have made a carousel of some tweets from speakers and attendees, you should take a look below. You can see more of this by checking out the event hashtag #CHAOSSconAfrica2023.

lessons learned and future improvements

The conference went great, however, we had some setbacks. We sent out a post-event feedback form and I’ll make an attempt to analyse it.

  1. We asked that people rate their overall experience at CHAOSScon Africa. 83.3% of participants said it was excellent while 8.3% said it was average
  2. We asked if the event schedule and session information provided were clear and timely 50% said yes. 41.7% said it’s on average, while 8.3% said no.
  3. We asked if the sessions at the conference were relevant to their interests and expectations. 100% of participants said yes, it was.
  4. We asked if they had ample opportunities to network with other attendees and speakers. 41.7% said yes, definitely. 53.3% said yes, maybe.
  5. We asked that they give feedback on the conference venue and facilities. 66.7% of participants said it was great. 16.7% said it was good, while 8.3% said it was poor.
  6. We asked that they rate the quality and variety of food and refreshments provided. 75% of participants said it was excellent. 8.3% said it was good, another 8.3% said it was poor.
  7. To wrap it up, we asked if they would be interested in attending CHAOSScon Africa again in the future.100% of participants said yes, definitely!

This is an overview of all the efforts the different teams put in place to ensure the success of CHAOSScon and the results that emerged from this hard work.

Based on this feedback, there are a lot of things we’ll be looking into and making changes/improvements to for future events. The committee shows their heartfelt gratitude to everyone who filled out the feedback form, and if you haven’t yet, please use this link.

Wrapping up…

CHAOSScon Africa 2023 was a resounding success. We are thankful to all who made this a success, we appreciate our sponsors’ support and the dedication of every volunteer.

Without a doubt, the open source community in Africa is on an upward trajectory, accompanied by a surge of interest in open source technologies. The CHAOSS Africa community is committed to fostering a more sustainable future for open source technologies in Africa. In our pursuit of this goal, we are forging strategic partnerships and implementing comprehensive training programs. To stay tuned to these updates, do join the CHAOSS Project community.

Thank you for reading this far, we hope to see you soon!

Welcome Dr. Dawn Foster as the new Director of Data Science for CHAOSS!

By Blog Post, News No Comments

We are unbelievably excited to announce that Dr. Dawn Foster will be joining the CHAOSS project full time in August 2023, helping direct community data science efforts. To date, the CHAOSS project has been focused on developing software, metrics, and programs aimed at helping people and organizations better understand the health of open source communities they care about. While we are proud of our impact to date, with Dawn’s energy, we will be able to strategically focus on data-driven questions that are important for corporate OSPOs, university OSPOs, and scientific software communities among others.

Dr. Dawn Foster

Dawn joins us from VMware where she is the Director of Open Source Community Strategy within VMware’s OSPO. She is a Governing Board member / maintainer for CHAOSS, co-chair of the CNCF Contributor Strategy TAG, and OpenUK board member. She has 20+ years of experience at companies like Intel and Puppet with expertise in community building, strategy, open source software, governance, metrics, and more.

Dawn holds a PhD from the University of Greenwich along with an MBA and a BS in Computer Science. She has spoken at over 100 industry events, including many Linux Foundation events, KubeCon, OSCON, SXSW, FOSDEM and more. In her spare time, she enjoys reading science fiction, running, and traveling.

Developing a CHAOSS data science effort is intended to help organizations ask complex questions, identify data to address these questions, and effectively use the results to inform decision-making processes. We hope that you can join us in welcoming and supporting Dawn as she takes on these challenges to support all with an interest.

Starter Project Health Metrics Model

By Blog Post No Comments

Have you ever been in a position in your company or community where you would like to start getting a sense of the health of a project – but you don’t know where to begin? 

People often struggle to get started with measuring project health in a way that allows them to draw meaningful conclusions without becoming overwhelmed. Measuring key aspects of project health is an essential first step toward understanding how an open source project can be improved and deciding where to focus improvement efforts. The following four metrics are great ways to get started.The Starter Project Health Metrics Model was published by the CHAOSS project to address this very issue. 

  • Time to First Response Determine the amount of time between when an activity was opened (e.g. Issue or Change Request) and when it received the first response from a human. A quick response helps contributors feel welcome and appreciated.
  • Change Request Closure Ratio Measure the ratio between the total number of open change requests during a time period versus the total number of change requests closed in that same period. This helps to determine whether your project has enough maintainers to keep up with incoming contributions.
  • Bus Factor Determine the smallest number of people that make 50% of contributions to understand whether your project would be in jeopardy if one or more key contributors left.
  • Release Frequency Determine the frequency of project releases (including point releases with bug fixes) to make sure that security fixes, new features, and bug fixes are available to your users.

The model is available at: https://chaoss.community/kb/metrics-model-starter-project-health/ 

CHAOSScon EU 2023 Summary

By Blog Post No Comments

By Matt Germonprez and Georg Link

CHAOSScon 2023 Europe is now complete! Thanks to everyone who helped put on another wonderful event and thanks to everyone who took the time to attend and participate! It was really great to connect with everyone in the beautiful city of Brussels. 

Photo Credit: Sean Goggins

This CHAOSScon, we led several discussions centering on two key questions. We asked participants to reflect on the questions in small groups and report back. This post will highlight a few of the key takeaways — and also provide captured comments that maybe didn’t make it into this post. 

What challenges exist for using metrics within your OSPO or Community?

The Consistent Use of Metrics within an Organization or Community

Many of the responses centered on the difficulties associated with the consistent use of metrics within an organization or community. For example, with the sheer size of many organizations and communities, the deployment and interpretation of metrics can vary a lot between people. This leads us, in the CHAOSS Community, to a 2023 goal of developing ways to communicate simple metric strategies and results that can be easily shared between people. 

The full set of recorded comments for the first question with general categorizations is here: 

  • Organization/Community: 
    • There isn’t a central place for metrics discussions within an organization.
    • There isn’t a common taxonomy for how to speak about metrics.
    • Fragmentation of metrics within an organization can make consistent use difficult.
    • The size of an organization can make consistent use of metrics difficult.
    • The size of a community can make consistent use of metrics difficult.
    • Different stages of project maturity can make consistent use of metrics difficult.
    • There is minimal guidance on replicating CHAOSS structures locally.
  • Metrics/Metrics Model: 
    • How to determine business value from a project?
    • How to measure the value of participation in a project?
    • How to determine business risk from a project?
    • How to determine company impact on a project?
    • How to measure the cost of non-participation?
    • Metrics themselves can be quite complex
Photo Credit: Sean Goggins

What should the CHAOSS project be working on in the future?

Building Collaborative Communities and Helping People Communicate Better

There are many, many things that we could work on within the CHAOSS project in 2023. Recurring themes from CHAOSScon include (1) helping people connect with others in similar contexts to discuss health-related concerns (i.e., corporate open source program offices) and (2) developing ways to help people communicate about metrics within their respective organization or community. 

The full set of recorded comments for the second question with general categorizations is here: 

  • CHAOSS Software: 
    • Enable all contributions to be seen in a dashboard.
    • Add social, design, and educational material use (like a social calendar) into a dashboard of contributor metrics to cross-visualize data about the ecosystem beyond code.
  • CHAOSS Operations: 
    • Continue to support the newcomer experience.
    • Assist others with the interpretation of results.
    • Have mechanisms to help interpret the data in non-biased ways / manipulating it towards certain predetermined hypotheses.
    • Provide badging to validate business metrics strategies.
    • Provide metrics and metrics model validation.
    • Support user groups that need specific metrics and metrics models to help in a variety of contexts.
    • Develop personas for groups of people with similar interests.
    • Build better starting points for people who may not want to build metrics.
    • Have social justice built into the metric development process, not as charity, but to address systemic issues continuously.
    • Provide Pathways into the metrics, more context on where they have been used in the past (e.g. to answer the question what do people like me use?).
    • Think about people who work in different languages – is CHAOSS accessible for them?
  • CHAOSS Communication: 
    • Talk with other communities to understand different views of metrics.
    • Tell user stories of how others are using the metrics and metrics models in practice.
    • Provide a framework for how to apply metrics.
    • Provide ways to represent and talk about metrics in business meetings.
    • Push a goal-oriented approach.
    • Articulating why metrics are important to help the business case.
    • Provide a set of five metric models to use that do five different things.
  • Metrics/Metrics Model: 
    • Software license compliance between repos
    • Test coverage in repos.
    • Code quality
    • Pipeline status
    • Ensuring a broad community of voices.
    • Build productivity and efficiency metrics and metrics models.
    • Need to understand the signal-to-noise ratio.
    • More human-centric metrics, the kind that help to signal/prevent burnout.
    • An estimate of how difficult it might be to implement. A signal of whether tech experience is required to implement a particular metric, or if it can be done by a quick survey, or if it can be identified automatically, or if it requires big program management lift. 

Thanks again to everyone for an amazing CHAOSScon 2023 in Brussels! We hope to see you soon!!

Photo Credit: Sean Goggins