Practitioner Guide: Responsiveness

Primary metrics:

If you haven’t already read the Practitioner Guide: Introduction - Things to Think about When Interpreting Metrics, please pause now and read that guide.

Responsiveness metrics are an important part of assessing project health (Eghbal, 2020), and are especially relevant for new contributor recruitment and retention of existing contributors. Responsiveness, including time to close, is one of the most important factors in attracting newcomers for a project (Fronchetti et al. 2019). A GitHub survey in 2017 found that when thinking about whether to contribute to an open source project, 95% of contributors said that responsive maintainers was very or somewhat important. Contributors can become discouraged when they don’t receive a timely and appropriate response to a contribution but can be encouraged when they get a quick and helpful resolution to their contribution. When projects are responsive, it can make people want to contribute more or continue contributing. Timely, thoughtful, and kind responses to contributors indicate that you appreciate their work.

Responsiveness can also impact overall project operations, especially regarding change requests (pull requests / merge requests). The longer a change request sits around without being merged, the more merge conflicts accumulate to the point where it may become too difficult or impossible to accept. Sometimes, a contribution may never be included due to incompatibilities, conflicts with overall project direction, or other considerations. In those cases, it’s better to let someone know in a kind and thoughtful manner why their contribution won’t be accepted rather than keep them guessing and increase their frustration with a lack of response. You can reduce technical debt, decrease maintainer workload, and improve contributor retention by being responsive and keeping up with contributions.

Step 1: Identify Trends

While there are also complex ways of looking at responsiveness, getting started with responsiveness metrics can be straightforward if you start by identifying the trends. By looking at Time to First Response, Time to Close, and Change Request Closure Ratio together, you can get a sense of whether contributors are getting a timely response and whether you are keeping up with contributions by closing change requests (merged or closed without merge).

Time to First Response

There are many ways to look at Time to First Response, including changes in mean / median. Still, it is helpful to compare the total number of change requests with the number that are responded to within some benchmark that is reasonable for your community, for example, two business days. I like this approach because you can see how the total volume of contributions impacts your responsiveness.

Time to First Response showing that most PRs are closed within 2 business days

This graph looks at time to first response from a human being. This is important because people put a lot of work into a contribution and are likely watching the change request or issue and waiting to see how it’s received. If it sits there with no response for days and weeks, people can get discouraged and feel like they wasted their time. In this case, any response is likely better than no response, even if the response is to thank them and give them an idea of when to expect feedback.

Time to Close

While time to first response shows how responsive you are when someone initiates a contribution, time to close provides insight into how quickly the project takes a request from initiation to completion. Time to close is frequently used for change requests and issues, but could also apply to other types of contributions. Like with time to first response, contributors can become discouraged if they feel like their contributions aren’t making progress and moving toward completion.

One caution with time to close is that the time it takes to complete a request or close an issue varies widely depending on complexity. For change requests, a one line change can often be completed in a very short timeframe, whereas, a complete refactoring of a large component might require a much longer period of time for review and testing before it can be merged. Time to close for issues is even more variable and can take anywhere from a few minutes to many years because some issues may have such a small impact that they are never high enough of a priority for anyone to decide to work on them.

Because of this variability, it’s often a good idea to look at median time to close in addition to average time to close, since median is likely to be closer to how people perceive the amount of time required to move something to completion.

Average time to close of 8.941 days and Median time to close of 4.022 days

Closure Ratio

In addition to looking at the time it takes to get an initial response and move a contribution to closure, it’s also important to understand how well a project is keeping up with contributions. For example, large numbers of open pull requests is sometimes perceived as a sign that maintainers are "not particularly conscientious in dealing with people external to the project, since each open pull request indicates an offer of code that is being ignored rather than accepted, rejected or commented upon" (Dabbish et al. 2012).

Closure Ratio - sustains and keeps up with contributions showing that they are falling behind in July and August

This graph looks at the closure ratio for change requests (merge requests / pull requests) to see if a project keeps up with incoming contributions. This is important for a couple of reasons. Old change requests sometimes stay around for too long because maintainers don’t want to tell someone that it won’t be accepted, often for excellent reasons, like architectural incompatibility or other conflicts with the direction of the project. This leaves contributors hanging and unsure why their request isn’t being merged, which can waste people’s time and reflect poorly on the project. In this case, it might help to provide some training for maintainers about the importance of closing change requests and how to do it with empathy and kindness toward the people who put in the work to contribute. If even good quality, relevant change requests aren’t being merged in a reasonable amount of time, this can indicate that the project just doesn’t have enough maintainers or that the maintainers don’t have the bandwidth to keep up. After a while, it also becomes impractical to merge old change requests because there are often too many merge conflicts to be realistically resolved.

Step 2: Diagnosis

As mentioned in the Practitioner Guide Introduction, you should start by showing the data to some people intimately involved in the project so that you can look together at the trends identified in these graphs and interpret them in light of the other things that might be happening in the project. For example, the time to first response and closure ratio graphs in Step 1 show a more significant gap in July and August, which might worry me more if it was another time of the year. In this case, it might mean that people were taking summer vacations, which is likely healthy for the project maintainers.

Closure Ratio - sustains and keeps up with contributions showing that the project has gradually stopped keeping up

If the trend shows that responsiveness is gradually declining over several months, then it might be time to figure out why. In the graph above, you can see that up until June, the project was keeping up with their pull requests every month, but more recently have fallen behind. It’s only over a few months, so this might not be an issue, and the issue could have been caused by a number of things that might not indicate underlying problems with responsiveness:

  • The spike in contributions from August might have created a backlog they are still working through.
  • Because the volume of pull requests is low, they might have received several complex pull requests that are taking longer to merge.

However, other things could cause this that might be a concern. For example, maybe one or more of the key people don’t have enough time to devote to the project, or maybe the increased number of pull requests is more than they can manage.

Lead time showing time to close gradually increasing

When diagnosing time to close, it can help to look at time to close trends over time. In this project the time to close trend is increasing and taking longer to close issues, but it’s increasing gradually. It’s likely that the project has become larger and more complex in the 5 years shown in this graph, so this might be an acceptable increase. It would be good to look at what happened during the timing of a few of those big spikes to see what was happening and decide whether there might be something you could do differently to better handle those spikes.

Step 3: Gather Additional Data if Needed

The examples used in Steps 1 and 2 provide a starting point that can be expanded to look at similar responsiveness metrics across a variety of channels (e.g., issues, change requests, reviews)

CHAOSS has other metrics related to responsiveness that can help diagnose specific problems within your community.

Metrics:

Step 4: Make Improvements

It can be tempting to attempt to solve issues with responsiveness by putting more pressure on existing maintainers by asking them to respond more quickly and resolve more contributions, but this rarely solves the long-term problem. It might result in short-term gains, but it could be damaging to the community and the project over time if you are burning out your maintainers by not resolving the underlying problems that are causing the lack of responsiveness in the first place.

If you see responsiveness declining, it might be time to move more contributors into leadership roles and promote some contributors to become maintainers for your project. You can start by looking at the contributors within your project to find people who are contributing regularly but aren’t already maintainers. A good first step is to promote some contributors into reviewer roles where they can reduce the load on maintainers by reviewing contributions from other community members. More people reviewing contributions should provide a faster time to first response and an improved change request closure ratio since maintainers will have less work to do for each contribution, and they can focus on the things that require more expertise within the project. You should also consider whether existing contributors are ready to take on a maintainer role within the project, and they can start by being responsible for a single area within the repository using OWNERS or CODEOWNERS files to manage access. If you don’t already have good documentation about leadership roles within your project, you might also need to define better the roles and responsibilities for contributors, reviewers, and maintainers, which can help with recruiting new people into these roles using something like a contributor ladder. Please see the Contributor Sustainability Practitioner Guide to learn more about recruiting contributors and moving them into leadership roles.

It can also help to set clear expectations for when someone can expect a response. For example, a contributing.md guide can inform contributors of an average or expected response time. You might also consider including details about time off or other things that might cause delays in responsiveness with your project milestones or roadmap to show that a response may be delayed.

Another way to reduce reviewer friction and make it easier for maintainers to review contributions is by using issue and change request templates to help contributors make good contributions that require less work from maintainers. Here are a few examples of the types of things you might want to include in these templates and in your contributor guides:

  • Don’t send changes over X lines of code (500-800 might be an appropriate threshold)
  • Don’t send changes that touch more than X number of files (ideally one file, but if you must change multiple related files, keep it scoped to one feature/issue per change request)
  • Keep change requests in separate feature branches.
  • Clearly communicate what checks will be run on a change request before they can be merged (e.g., style checkers, linters, test suites, signed commits, developer certificate of origin checks).
  • Clear policies and examples for naming, tagging, and labeling of change requests.
  • Provide expectations for reviews (e.g., we try not to leave issues / change requests open for more than X days, we do backlog grooming every X days / weeks / months.)

The caution with issue and change request templates is that you need to find a balance between getting enough information to make it easier for maintainers to review requests, while not asking for so much information that you scare new contributors away before they can contribute.

Talk to your maintainers and find out what else they spend time on. If maintainers spend a lot of time on things like community management, documentation, or other non-coding tasks, then recruiting people to help in those roles is time well spent. In other cases, improved documentation might help reduce maintainer workload; for example, if maintainers spend a lot of time onboarding contributors or answering questions about the contribution process, maybe better onboarding documents or contributor guides would free up some time to focus on responding to incoming contributions.

Step 5: Monitor Results

How you monitor the results will depend on what improvements you decided to make. Continuing to monitor time to first response, time to close, and change request closure ratio are a good start. If you used other data from Step 3, you should also monitor those metrics.

If you are recruiting new approvers and maintainers, it might take a while to see the results of those efforts, so you might not see significant improvements for 3 to 6 months.

Responsiveness can be a challenging metric to improve because many things happening within your project can impact responsiveness, and it’s common for people to focus on responsiveness and see short-term improvements that disappear again when people stop focusing on it. This means that responsiveness is probably a metric that you should always be monitoring whether you are focusing on it, or not.

Cautions and Considerations

  • Responsiveness is hard to diagnose because many things can impact responsiveness, so don’t be discouraged if your first attempt doesn’t result in improvement.

Additional Reading

Feedback

We would love to have feedback to learn more about how people are using the CHAOSS Practitioner Guides and how we can improve them over time. Please complete this short survey to provide your feedback.

Contributors

The following people contributed to this guide:

  • Dawn Foster
  • Chan Voong
  • Remy DeCausemaker
  • Luis Cañas Díaz
  • Joachim Noreiko

References

CHAOSS Practitioner Guides are MIT licensed, living documents, and we welcome your feedback and input. You can suggest edits to this document at https://github.com/chaoss/wg-data-science/blob/main/practitioner-guides/responsiveness.md