Unveiling the Impact: DEI Metrics Overcoming Social Barriers in Open Source

For decades, the topic of inclusion and marginalization of underrepresented groups in Open Source software communities has been a prevalent topic of discussion. There have been recurring studies and reports on the state of open source participation that spotlight persistent issues that plague these communities; Marginalization of certain individuals based on their race, gender, ability, skills, etc.  For instance, we see today that representation of women constitutes an underwhelming 18 to 23% of the tech industry, while in open source, women, along with other gender minorities, make up less than 10% of the population. Another common disparity that is often overlooked is the underrepresentation of neurodiversity groups, which is less than 11% in the tech industry.

The question of how an Open Source community can break down these barriers to participation and foster more inclusion and belonging for all individuals, regardless of abilities, skills, experiences, and differences, is a key to Open Source community health.

DEI metrics and initiatives have been introduced to play a crucial role in addressing this question. In October 2022, the Linux Foundation’s Community Health Analytics in Open Source Software or CHAOSS Diversity Equity Inclusion (DEI) Working Group launched an interview campaign to explore how CHAOSS DEI metrics are perceived and to identify ways to enhance the efficacy of these metrics in addressing these barriers. In this article, we present findings from these interviews that shed light on the role of DEI metrics in overcoming social barriers in contributing to open source. These interviews highlight the common challenges of DEI initiatives and their impact on open-source projects. We will explore how these DEI metrics can contribute to overcoming social barriers and create more welcoming environments for contributors from all backgrounds.

The State of Diversity, Equity, and Inclusion in Open Source

To gain a better understanding of how CHAOSS DEI metrics can impact Open Source community health, let’s dissect recent studies and findings that showcase the state of diversity, equity, and inclusion within open source.

• Exclusive Language and Behavior: The use of exclusive language, threats, disrespectful behavior, and online harassment can alienate potential contributors, making them less likely to participate. During our interviews, some participants admitted that they often experience microaggression on day-to-day interactions within their respective communities. Additionally, reports from the Linux Foundation 2021 research also indicate that Women, non-binary, LGBQ+, and people with disabilities were twice as likely to have experienced threats of violence in the context of an Open Source project. While transgender respondents were three times as likely.

• Lack of Representation: Some Open Source communities still struggle with the challenges of homogeneous leaders and members. A research paper published by IEEE states that the lack of gender diversity remains an ongoing issue among Open Source projects. However, many of the projects also suffer from a lack of female inclusion in leadership positions. When underrepresented groups do not see individuals who look like them in leadership roles or as active contributors, it can be discouraging, making them less likely to get involved. The Linux Foundations report also indicated that ​​although 82% of participants felt welcomed in open source, demographic segmentations show varied sentiments. The 18% of those who do not feel welcome were from disproportionately underrepresented groups: people with disabilities, transgender people, and racial and ethnic minorities. Also, during our interviews, some participants pointed out that in some communities, while minority groups are encouraged to participate and contribute, they are often overlooked for leadership roles.

• Socioeconomic Barriers: The 2023 open source maintainer research study indicated that open source contributors and maintainers are mostly students looking to build on their skills and experiences. Hence, access to resources such as a stable internet connection, a powerful computer, and free time to contribute can pose a challenge and create disparities in participation. When the question “Can you describe how your unique attributes, traits, characteristics, skills, experience, and background have affected your participation in open source?” was asked, an interviewee even  stated that “In some communities with lots of different people from various backgrounds,  the presence of various backgrounds doesn’t always translate to equity — in the sense that people who already have access to resources and advantages end up getting more, making the rich even richer.”

• Education on Diversity, Equity and Inclusion: Recognizing and addressing common disparities that exist in our communities is essential for progress. It is nearly impossible to effectively address these matters without first acknowledging their existence and comprehending their impact. However, during these interviews, we discovered that over 40% of the interviewees were not knowledgeable about CHAOSS DEI metrics, and most of them were just hearing about them for the first time. The Open Source maintainer research study further revealed that fewer than one-third of interviewees recognized the presence of formal DEI programs within their projects. This highlights the urgent need to improve awareness and drive initiatives that foster diversity, equity, and inclusion within open source.

• Accessibility: A critical aspect of Open Source software that often remains overlooked is accessibility. Unfortunately, a significant number of Open Source projects do not prioritize accessibility in their software, communication platforms, or technical documentation. This oversight has a profound impact on contributors with disabilities, who often experience a sense of exclusion when attempting to engage and collaborate within these communities. In our interviews, participants also mentioned that from their experiences,  accessibility is thought of as a feature to be added only later, after product releases.

• Implicit Bias: Unconscious biases are also common factors that affect the way contributors from underrepresented groups are welcomed and how their ideas are received. These biases play a significant role in influencing the decision-making within a community, in turn affecting inclusive participation. Although some of the actions of our biases are not intended, they end up hindering the progress of diversity and inclusion efforts in an Open Source community.

What are DEI Metrics?

CHAOSS metrics are indicators used to assess Open Source community health. The DEI metrics aim to evaluate and reveal the diversity, equity, and inclusion efforts within open-source projects and communities. These metrics are used to:

• Quantify Diversity: By collecting and evaluating data on the demographic makeup of contributors, DEI metrics initiatives provide a clearer picture of the community’s diversity. This data can help identify gaps and track progress over time. This data can help our leaders make informed decisions that create safe, welcoming, and equitable developer environments.
• Reveal Disparities: DEI metrics could reveal/identify, to some extent, incidents of both explicit and implicit bias that might exist in a community. For instance, in communities where members with disabilities, women, or people of color are not considered for leadership roles, it could signal a bias in promotion or recognition. 
• Monitor Inclusive Practices: Metrics observe incidents of harassment, the use of inclusive language, contribution participation, and other behaviors that affect the community’s inclusivity. They ensure access to resources (e.g., code of conduct) and understanding whether barriers to entry, like socioeconomic factors, limit participation. For instance, in a project with many non-native English contributors, the lack of translations in key languages for code, documentation, and guides poses a challenge. Another known barrier to open source participation is that much of it relies on voluntary labor, while the majority of unpaid care work falls on those in marginalized groups. Metrics can help us understand how we support diverse users and accommodate their needs.

Open source projects can leverage CHAOSS DEI metrics to proactively address biases, expand the talent pool, and ensure belonging, ultimately breaking down social barriers and promoting diversity in open source communities at large.

Why an interview campaign?

Since underrepresented groups have the most potential to benefit from diversity, equity, and inclusion (DEI) metrics in open source, we launched an interview campaign to help us evaluate actual DEI activities in OSS and how they relate to the metrics we presently measure at CHAOSS through interviews.

This study aimed to help us comprehend the impact of current DEI metrics in real-world circumstances, especially for underrepresented groups in open source. Identify new metrics and methods for measuring DEI and also identify gaps to be addressed in current DEI metrics. 

Do Metrics Address all of these Challenges?

While conducting interviews, we discovered that DEI metrics, though effective, may only provide a partial solution to some of the DEI challenges within our communities. Some participants shared valuable insights about the use of diversity metrics.

For example, some interviewees expressed concerns about relying solely on metrics to gauge progress. They argued that merely tracking the increase in the number of diverse individuals, such as going from 40% women in 2022 to 50% women in 2023, might not be a fully positive way to measure change or growth. Instead, the focus should be on whether the product or Open Source community has remained strong or improved since that time. If the answer is yes, then this improvement may be attributed to an increase in diversity.

Another participant emphasized that people from diverse backgrounds have unique perspectives and mindsets. They suggested that general metrics might not be inclusive of these diverse perspectives because they are often created by a group of individuals from one community without perspectives from the world.  Since diversity can mean a lot of things for different people, it’s hard to say these metrics will address all of these challenges from individual perspectives. Hence, using the same pool of individuals for the creation of DEI metrics could influence their credibility. 

Some participants raised concerns that relying on metrics might lead to unfair judgments of people’s abilities and skills. While others shared their experiences of being assigned more unpaid work because they were the only diverse representatives for an organization. Additionally, some feedback questioned the effectiveness of DEI metrics in communities or groups with a higher proportion of individuals who are neurodivergent, stating that it can lead to challenges in understanding and adhering to the customary social norms or expected behaviors within that community.

In terms of successfully implementing these metrics, we understand that DEI metrics alone may be unable to address all DEI challenges if the barriers of these DEI are not recognized and tackled. Overall, feedback from interviewees emphasized that it’s not just about creating metrics but empowering people to understand and use them. So, even if you have metrics, they will only be effective if people are aware of them and use them to make changes for the better.

Impacts of CHAOSS DEI Metrics on Open Source Communities

CHAOSS DEI metrics have found application in various scenarios, spanning software development, governance and policies, events, communities, and beyond.

The interview campaign comprised an anonymous survey with over 180 participants and one-on-one interviews with 19 participants. 

During the interviews, we got to ask participants to share their experiences with Diversity, equity, and inclusion in open source. To help us understand how their differences may have influenced their participation and how communities can leverage metrics to address some of these barriers.

Based on these interviews, we gained insights from participants who shared real-time examples of how these metrics have been instrumental. They shed light on how CHAOSS DEI metrics have positively impacted individuals and communities.

• Some participants mentioned that they have greatly benefited from the CHAOSS DEI metrics at events they attended in the past, and these metrics were enforced for a more inclusive event culture. Notably, they appreciated open-source events that considered attendees’ safety, diverse speaker line-ups and live captioning for some talks, which was especially valuable to attendees who may be hard of hearing. Additionally, they indicated that the availability of Diversity Access tickets eliminated concerns about their participation cost.

• For event organizers, the CHAOSS metrics on event diversity offered valuable insights into better connecting with attendees and speakers, handling conflict cases, and improving the overall event experience. They emphasized that DEI metrics were pivotal in encouraging active participation and enforcing a code of conduct—for example, the State of Open Con23 and Open Source Summit Europe 2023.

• Participants indicated that these DEI metrics were utilized in their project to ensure smooth collaboration and healthy communication among team members. They also added that these metrics were helpful in assessing their progress, particularly in areas like leadership and mentoring, and in supporting contributors with diverse backgrounds and skills, ensuring inclusive contributions. Example the Apache Traffic Control project.

• A participant said the DEI metrics were being utilized in the  DEI research that was conducted by the Apache Diversity and Inclusion project. They said these metrics were being incorporated to help determine actionable steps to take based on their research outcomes.

• Some participants also said the CHAOSS metrics have been instrumental in the development of a code of conduct and implementing inclusive practices in their communities and projects—for example, The Good Docs Project. 

Harnessing the Power of DEI Metrics in FOSS

Following the conclusion of our interview campaign, it became certain that CHAOSS DEI metrics have the potential to address various challenges and help address barriers to contribution within respective communities.

Diversify Recruitment: 

• Metrics on community activity can be used to diversify the recruitment process and offer equal opportunities to contributors regardless of their differences.
• Metrics on labor investment can also be utilized by open-source leaders to understand how much value a potential candidate provides to a project as they contribute. 
• Metrics on contributor onboarding, mentorship, and contribution attribution can help community leaders identify a person suitable for a leadership role or full-time hire and subsequently craft a position that aligns with their strengths.

Diversify Leadership: 

• Metrics can be used to diversify the leadership board and serve as a key performance indicator for community health.
• Metrics on board and council diversity can help in promoting inclusive representation within communities.
• By improving the power structure within communities, most Open Source communities can avoid dictatorships among leaders.
• Organizations and community leaders can use DEI metrics to brainstorm strategies for diversifying their communities and to assess whether they are effectively fostering inclusive participation.

Inclusive Event Participation:

• Metrics are helpful for their event, promoting inclusive events and cultures. 
• Metrics on public health and safety can be used to ensure participants feel safe to attend in-person events regardless of their differences.
• Metrics on the event code of conduct are helpful to organizers in figuring out better ways to relate with attendees.
• Metrics on diversity access tickets also help participants from underrepresented groups in attending events.

Psychological Safety Awareness:

• Metrics on psychological safety and burnout are essential for creating a safer environment in open source, where individuals can freely communicate their concerns
• Metrics also encourage the need to provide support to all contributors and maintainers.

Inclusive Contribution & Retention:

• Metrics on mentorship can be employed to help mentors and mentees collaborate better. 
• Metrics on leadership and mentorship can help encourage more folks to participate, also giving equal opportunities to contributors regardless of the differences.
• Metrics relating to contributor onboarding and conversion rates can be used to discern the progression of new community members as they evolve into enduring and engaged contributors.
• Metrics on mentorship address the common barrier of entry for contributors, especially those from underrepresented groups.

Accountability: 

• Through the CHAOSS project badging initiative, metrics can be used to hold Open Source projects accountable for their diversity and inclusion efforts. By quantifying the DEI status quo, communities are more likely to take action to address disparities.
• These can be used to tackle the segregation, language barrier, similarity, and conformation bias that exist within most communities.
• CHAOSS DEI metrics can cover access to knowledge and education so more people are aware of these differences and how to respect and abide by the code of conduct.

Data-Driven Decision-Making: 

• Metrics can be used to gather data on demographics, diversity in contribution roles, and psychological safety.
• Metrics can help communities in creating  & enforcing code of conduct.
• Metrics enable communities to make informed decisions to address specific DEI challenges.

Conclusion

CHAOSS DEI metrics are becoming invaluable in addressing diversity and inclusion challenges within Open Source communities. While they hold the potential to transform these communities into more inclusive and equitable spaces, it is essential to acknowledge and address the associated challenges, foster understanding, and champion the adoption of DEI metrics across the Open Source landscape. Only through collective effort and ongoing commitment can Open Source communities continue to evolve and thrive in the spirit of diversity, equity, and inclusion.

Acknowledgments

A heartfelt gratitude to the numerous individuals who have contributed to the successful completion of this research article.

First and foremost, appreciation goes to Matt Germonprez, Sean Goggins, Foundjem Armstrong, and Vinod Ahuja, who provided invaluable insights, critical feedback, and unwavering support throughout the research process. Your dedication and expertise were instrumental in shaping the quality of this work.

A special shout-out to the individuals who generously participated in these interviews and shared their knowledge and experiences, enriching our findings significantly. Your openness to sharing was pivotal to the success of this interview campaign.

Finally, a big shout-out to the CHAOSS community members for their ongoing efforts and dedication to advancing open-source community health metrics and analytics. And for supporting the successful completion of this interview campaign.