News — 14 April, 2021

How Users’ Experience Impacts Project Completion and Retention

Gloria Urrea, University of Colorado at Boulder, and Eunae Yoo, University of Tennessee, partnered with HOT to conduct research that has produced valuable insights for HOT and other organizations using the Tasking Manager. Here, they summarize their research project that examines the impact of users’ experience on project completion and retention.

Based on the number of prior contributions, users on the Tasking Manager are classified as belonging to one of three experience levels: beginner, intermediate, and advanced. We conducted an academic study that examines the relationship between users’ experience levels and two outcomes of interest to HOT and organizations that use the Tasking Manager: (1) project completion and (2) retention.

Project Completion

We considered a project to be completed when all tasks have been mapped. Since projects should not only be completed but also completed quickly, we ran models on projects’ completion rate. This was measured as the daily count of mapped tasks.

Our first analysis examines a project’s number of mapped tasks on a given day as a function of the count of beginner vs. intermediate vs. advanced users that contributed to the project on the same day. According to our model, we find that a project’s completion rate improves by:

• 5% with the addition of a beginner user

• 26% with the addition of an intermediate user

• 27% with the addition of an advanced user

These results indicate that beginner users are not as productive as users at higher experience levels. This is not necessarily surprising given that beginner users have less expertise. What is striking from our results is that the effect of adding an intermediate user is statistically equivalent to adding an advanced user. Thus, intermediate and advanced users are equally valuable to the completion rate of projects.

We further explored how the results above change depending on projects’ priority. We found that the completion rate for High and Urgent projects compared to that of Low and Medium projects:

• Decreases by 2% with the addition of a beginner user

• Increases by 19% with the addition of an intermediate user

• Increases by 10% with the addition of an advanced user

Once we accounted for projects’ priority, intermediate and advanced users are no longer equally valuable. Instead, increasing the count of intermediate users has, by far, the largest impact on projects’ completion rate. These results also provide insight into the type of project, as defined by priority, that users at different experience levels are most productive in.

Retention

Experience levels are essentially statuses that signal the users’ amount of expertise on the Tasking Manager. We analyzed to what extent experience levels incentivize the retention of users. That is, do users return more quickly to contribute to projects when they become closer to reaching the next experience level? We assessed retention by measuring how long it takes users to resume contributing to projects on the Tasking Manager. We label periods that users are actively working on projects as “sessions”.

Our analyses are summarized in the figure above, which plots the number of days between successive sessions and the number of prior contributions by users. Each line represents the best fitting line for each experience level. In line with the Tasking Manager, users become promoted to the intermediate level after mapping 250 tasks and to the advanced level after mapping 500 tasks.

We found that the time between successive sessions decreases as beginner and intermediate users approach the next experience level. This provides evidence of an incentivizing effect by experience levels. In contrast, the line for advanced users is relatively flat, indicating no change in retention for users with no further experience level to work towards.

When comparing the lines for beginner vs. intermediate users, the line for beginner users is noticeably steeper. What this tells us is that the incentivizing effect of experience levels wears off as users move up in the hierarchy of experience levels. That is, beginner users that need X tasks to become intermediate return more quickly to the Tasking Manager than do intermediate users that also need X tasks to become advanced.

Key Takeaways

• The addition of an intermediate or advanced user is more beneficial to projects’ completion rate. However, this is easier said than done since intermediate and advanced users are a scarce resource. In our data, over 98% of the users were beginners.

• To raise completion rate, users may be channeled based on their experience towards projects of certain priority. For instance, beginner users can be nudged towards Low and Medium priority projects.

• Users are incentivized to return to the Tasking Manager by the prospect of reaching the next experience level. Consider building additional experience levels that users can work towards.

Next Steps

As we wrap up this study, we are collaborating with HOT on other exciting research initiatives. We hope to continue sharing what we learn in future blog posts. In the meantime, if you would like to learn more about the research we described in this post, please visit: https://ssrn.com/abstract=3784152. You can also contact us directly at eyoo@utk.edu and gloria.urrea@colorado.edu.