W9 - Developer Community

Author: J. Konrath

Developer Community - Collaborative Editing in Social Q&A Communities

At some point in their lives every programmer will have googled a question that popped up when coding. Searching an abstracted conceptual question or just copy pasting an error message into the search bar will often lead to an online Q&A community where someone else might have had the same issue or question before.

However what can we do when we are the first to come across a specific issue? If no one has wondered about this specific question before us? Of course we could post a new question to such a community and hope that we will receive a helpful response. But what if the question itself is stupid and that's why nobody else posted it on the internet already? What if we phrase our question in a bad way and all the responses will just tell us that we don't know how to ask a "good" question?

Many online communities have somewhat "hidden" rules. A set of principles, well known among senior members but not immediately obvious to newcomers. A community where these rules are then enforced by some public show of rejection can lead to new contributors being discouraged from interacting further or keep them from joining the community in the first place.

Including new Users

As described in Multiple online communities have explored ways to encourage new users to contribute or highlight novice users to others to foster a more welcoming environment. Quora launched a Spanish version of their site to include the large number of Spanish speaking internet users and GitHub states that "One of the best ways to grow your community is to welcome new contributors" and started showing a "First-time contributor" badge to maintainers of a repository when they were reviewing a pull request from a user that had not contributed to the project previously to encourage them to write more lenient and helpful comments .

Stack Overflow

One of the most famous online Question and Answer communities is Stack Overflow. After being created in 2008 it quickly gained popularity and as of today is well established as a reliable source for answers to programming related questions. Stack Overflow is part of the Stack Exchange Network which also envelops numerous other Q&A communities, all modelled after Stack Overflow.

Stack Overflow has also gained an infamous reputation of being somewhat hostile towards new users, since the community is very quick to enforce its norms by means of public downvotes or direct edits to new questions. The best example for such an established standard which is often overlooked by new users is that there is a broad consent on Stack Overflow that questions should not contain any salutations or greetings .

The Experiment

Mentoring

One way of making it easier for new users to be welcomed into such a community is pairing them with more experienced users in a mentoring system. This was also found in other research such as. This was implemented on Stack by offering new users who were about to post their first question to the site the option to join a chat room where a more experienced user would assist them in rewriting their question in a way that would follow the established rules of the forum. An important part of this experiment was, that the mentors should always be human instead of a chat bot to encourage organic conversations between the mentors and novices.

Mentors were shown a live preview of the question the new user had been about to post and could chat with the user in real time, but the user had to make all the changes themselves. At any point the user could chose to copy the current version of the question back to the original site where they could proceed to post it. This way mentors could not just change the draft to what they felt was a perfect question and tell the newcomer to post it, but the novice had to make all the changes themselves and - hopefully - gain a deeper understanding of why the changes made sense.

As part of these conversations mentors would instruct users to provide Minimal Reproducible Examples or format the code they had already provided as code blocks to improve readability.

Results

The Experiment found that questions that had gone through the mentoring process would be more well-received by the Stack Overflow community than questions posted by eligible users who had chosen not to interact with a mentor. states that the score of questions who had gone through the mentoring process was 50% higher than the comparable questions who been posted without review.

The programme was received very positively both among participating mentors and novice users and many of the newcomers who had participated indicated that they now felt more welcome on Stack Overflow

Limitations

The study also notes some limitations of these results, some of which are related to the implementation of the chat room but an interesting one is that mentors would tend to look up how their "mentored" questions were doing and would sometimes upvote it themselves which introduces a bias to the score statistic references above, because the mentors might not have seen the questions at all without the mentoring interaction.

Additionally a possible Self-Selection bias is mentioned, stating that users who voluntarily elect to have their question reviewed by a more experienced user might have been more likely to write a good question from the start when compared to those who ignored the prompt.

Finally it seems difficult to scale this successful setup to a permanent and platform-wide feature given how much it depends on volunteer mentors spending their time on supporting new users, however with some sort of incentive structure such as reputation points (With a users' reputation score already being an established part of Stack Overflow) for mentoring sessions this might actually be feasible.

References:

Jonas Konrath

Msc Computer Science Student at ETH Zürich