People who know me know that I like to do surveys about things that I’m interested in and passionate about. It’s a whole thing. Mastodon is something that I have been very into, so naturally I get curious. When you use a social network you tune into currents and eddies, and I wanted to know how my limited view fit into the big picture.
Something I love about running surveys is that basically people love to tell you about themselves and how they feel, and I love to absorb all the information into my brain and mash it about and produce something coherent from it.
Recently I have been blogging about my experiences as a marginalised person as part of a FOSSdudebro-dominated social network and as part of the development community behind it, and I felt like maybe I should get some solid data on those topics. Or, you know, any data at all, since my other articles are 99% anecdote.
I’m not gonna say that my survey is 100% neutral and unbiased, since I threw it together very quickly after having woken up way too early and I’m not trying to be extremely Science here. I’ve never pretended to be anything other than an amateur survey enthusiast.¹ But I still think the results are interesting, and I know that there’s going to be a bunch of people who took part who are curious to see what I’ve learned.
What the survey response was like in general
It asked different questions depending on whether you had used Mastodon, had looked into joining, or had only heard of it. Everyone who had used Mastodon was asked to give it a score on a scale of 1–5, and other than that the questions were more relevant to how much you had used Mastodon, or why you hadn’t used Mastodon. At the end I asked everyone for some standard demographic information, such as age, race/ethnicity, gender, and (alright, less standard) coding ability.
I mainly promoted on Mastodon itself (123 boosts) and on Twitter, plus a little on Reddit (/r/samplesize mainly) and Tumblr. I tried to get more people to take the survey by not limiting it at all — the promo stuff all said “come take my quick survey about a social network, I want your data even if you’ve never heard of it before!” I think that helped hook in people who had left Mastodon or who had considered joining but decided against it.
Around 210 unique instances were named, and here’s the top 5 to give you an idea of which bit of the fediverse the survey post reached:
- mastodon.social (163)
- cybre.space (44)
- witches.town (35) [this instance is now closed but a lot of participants were members, so perhaps it had a high retention rate?]
- octodon.social (25)
- mastodon.art (22)
Participation
In the end there were 885 responses. 482 (a little over half) had ever used Mastodon, of which 46 joined but stopped using it. 78 had at least heard of Mastodon but not joined. You can see the spreadsheet of all results here. If you don’t want to wade through all my queries and conditional formatting and charts and stuff you can start from scratch; just make a copy of the whole lot and then delete all the sheets except “Form Responses 1”.
Age
Participants were mostly in the 16–35 range, not surprising for English-speaking social media as far as I’m aware. The ages represented on Mastodon are pretty similar to the ages represented on other social media profiles of a similar nature.
The 1–5 scores
Everyone who had ever used Mastodon was asked to score their experience on a scale of 1 to 5, 1 being the best and 5 being the worst. I compared scores with demographic information and coding ability, and I inverted the score so that it makes more intuitive sense — the higher the number, the better the score.
Here’s a bit of a visual summary.
It’s worth noting that the “disability: other/not sure” group at the bottom was only 24 people, but the score was so significantly low that I felt it should be included in these lists.
Coders vs. muggles
People who have at least some coding ability were in the majority by a long way in this survey. A lot of those people categorised under “other” wrote transferrable skills into the textbox, too.
As you can see below, coders are having a slightly better time on Mastodon than non-coders, but not notably so overall.
- Coders were almost twice as likely to say they have found community in Mastodon and to say it was easy to understand, and were much more likely to say their experience was positive, welcoming, supportive, and nicer than Twitter.
- Coders were over six times as likely to feel that their contributions to development were welcome and appreciated.
Interestingly, there were not such striking differences in the negative words, except that coders were more likely to feel isolated than non-coders. To put that in perspective, 65% of coders said they had found community on Mastodon, and 10% said they felt isolated — more extreme than non-coders on both counts.
Disability
A lot of people who chose “other” wrote in various things that some would class as a disability, such as mental health issues or autism. I decided that since I’m looking for a difference in perspective and life experience it would be practical to combine those two groups.
These results don’t include people who left the question blank.
- People who aren’t disabled are a little more likely to feel a sense of community and to feel welcome on Mastodon compared to people who are.
- People who are disabled are much less likely to feel able to contribute to development, and more likely to feel isolated.
People who are disabled (76, or 18% of respondents) are scoring Mastodon more poorly than any other demographic group, but the positive/negative words and phrases used to describe Mastodon are pretty much matched between disabled and non-disabled respondents, with one possibly predictable exception.
Disabled people were over three times as likely to describe Mastodon as inaccessible, which makes sense, because disabled people are pretty good at judging accessibility and are also less likely to be able to use something made for and by people who are not disabled.
Overall the difference was predictable but not quite so stark as some of the other results. It is worth noting that most of the people whose answers are taken into account in this section are disabled people who still use Mastodon. Disabled people who are unable to use Mastodon for accessibility reasons are significantly under-represented here.
Race and ethnicity
Race and ethnicity were a single textbox, because I didn’t feel confident in my knowledge of race issues to have a dropdown box. This analysis was always going to be a little clumsy and incomplete, because it’s just not my area. The spreadsheet of survey responses is available if you are interested in playing with the data, and if you publish anything please do let me know so I can link to it here.
A rough outline of the stuff that’s useful to know: the race question was optional, there were no validation rules, you could write anything or nothing. I counted “white people” as anyone whose answers included “white” or “caucasian”, and anyone whose entire answer was “european”. I counted blanks separately as a third category, and I counted everyone else (write-ins that were not categorised as white) as “not white”.² I had to make some small changes to responses to make sure that, for example, someone who wrote “mixed black/white” wasn’t counted as white, and to make sure that some non-answers like “Earthian” or “don’t want to say” were counted as blanks, but all the original answers are still clearly readable in the spreadsheet.
Firstly, I had heard a lot of stories about how white Mastodon is and how tough it is to be on Mastodon as a person of colour, so I was expecting to see over half of respondents describing themselves as white or variations thereupon.
However, I was pretty surprised when people who wrote something not-white in the race/ethnicity box rated Mastodon more highly on average.
- People who were not white were twice as likely to use words and phrases from the positive end of the spectrum (such as “community” and “accessible”) to describe their experience on Mastodon compared to white people…
- … but they were also a little more likely to use their corresponding negative words to describe it, too. People of colour were also more likely to say that Mastodon was inaccessible, and more than twice as likely to say that their Mastodon experience felt isolated, compared to white people.
Something else of note is the way the blanks were counted. I was expecting the people who left the box blank to be white people thanks to the way people in privileged groups tend to think of themselves as standard/default, but I didn’t feel comfortable making an assumption and lumping the blanks in with another group, so I counted them as a third category and put all three on a graph to see which group the blanks were most similar to.
I love how sometimes doing surveys like this turns my assumptions upside down and gives them a good shake!
As you can see here, the blanks were very close to people who were not white in terms of the way they scored Mastodon. The tempting conclusion to draw is that the people who left the box blank were mostly people of colour, but I really don’t know enough about data analysis and race, and there could be any number of factors at play here to explain this.
Gender
This was the one demographic question that really made me go “oh wow”. Like, I knew sexism was a thing and it was bad, but, oh wow.
There were more men (238) than women and nonbinary people combined (219), which is concerning on its own. Here’s how the top genders break down:
- Man: 49.4%
- Nonbinary: 24.9% (this is a heck of a lot compared to the general population)
- Woman: 20.5%
- Blank: 5.2%
Gender was easily the strongest indicator of whether someone was going to have a good or bad experience on Mastodon. Of all the demographics I collected, men scored their experience the most highly by a long way. And of all the demographics I collected, women and nonbinary people scored it among the worst, second only to disabled people.
This chart shows that men are much more likely to describe their experience with the best score, and women and nonbinary people are more likely to describe their experience with the lower scores.
On the positive and negative words and phrases, notable details include:
- Women in particular felt that their contributions to development were unwelcome, and women and nonbinary people felt much less able to contribute to development.
- Men and women were equally likely to describe their experience as “critical”. (Perhaps unsolicited advice and criticism are gender neutral issues, or perhaps there are other factors involved?)
- Nonbinary people were especially likely to feel isolated and more likely to describe Mastodon as “less nice than Twitter”.
- Nonbinary people felt significantly less community than men, and women even less.
Overall, women and nonbinary people were more likely than men to describe Mastodon with negative words and phrases, and less likely to use positive ones.
The gender question had four checkboxes (man, nonbinary, woman, other) and “other” had a textbox. The textbox had 28 unique answers. My favourites were:
- aaaaaaaaaaaaaaaaaaaaaaaaa (answer unclear; ask again later)
- do a better job with your gender questions³
- Something between Man and „fuck genders“
In related news, “unnecessary question” is a special snowflake gender if I ever heard one, come on dude∙tte, get real.
Turn-offs
There were many checkboxes. There were also “other” checkboxes with textboxes.
These groups were a relatively small subsection of respondents, but I think the write-ins would be very valuable to someone working on increasing Mastodon adoption. It’s not analysis that I am up to right now for various reasons, but the spreadsheet of results is public and I hope it can help someone.
Anyway, here’s a bit of a summary of the statistics that were easier to organise.
Why did they leave after trying it out?
Of 46 people who left:
- My friends weren’t there: 70%
- I got bored: 50%
- I didn’t understand how it worked: 26%
… closely followed by “it was too difficult to use”.
Notably, 9% of people lost their instance and login details and that was enough that they didn’t come back.
Why do they use it less than they used to?
Of 55 people who joined and still use Mastodon but only occasionally:
- My friends aren’t there: 64%
- I find it boring: 16%
- I disagree with the way development is going: 11%
One reason has taken a strong lead, there.
Why didn’t they join in the first place?
This includes people who have heard of Mastodon, and people who looked into joining Mastodon and then decided against it, 78 in total.
- My friends weren’t there: 58%
- I didn’t understand how it worked: 29%
- I couldn’t choose an instance: 28%
My thoughts
Obviously there is a strong theme of people not being able to follow their friends and thus having no reason to join or come back. The draw of making new friends is not strong enough.
I feel like just trying to pull in as many people as possible isn’t the answer. Unless those new people stick around that’s not going to change, and those people aren’t going to stick around unless it becomes clearer to non-technical users what the heck is going on.
Maybe it becomes clearer by gradually being adopted more and more until the nature of federation becomes common knowledge. Or maybe it becomes clearer with another leap in UX goodness, the same way Mastodon was a huge leap up from GNU Social. Ultimately any decentralised system like this will be inherently more complicated than a centralised network like Twitter, but human ingenuity has a way of making complex systems easy to digest, sometimes in unconventional ways using skills from unexpected vocations. I look forward to finding out what the next generation of OStatus/ActivityPub looks like.
Thanks!
Thank you to everyone who took the time to tell me about their experiences, and to everyone who boosted the signal out to more people! And thank you for reading all the way to the end. :)
- Not that it stops people telling that my surveys are extremely unscientific and that I should get an ethics board in to review things and stuff, gosh.
- I’m fully expecting to get some criticism for that, it’s a very rough and messy way to deal with data that’s pretty culturally sensitive, and I’m very aware that race and ethnicity are not questions I normally ask and I have no idea what I’m doing. I was only looking for rough data to investigate the folk wisdom that white people have a better time on Mastodon, and that’s what I got.
- I run a survey of genders beyond the binary every year called Gender Census, the last one got over 11,000 responses, I’m pretty proud. It’s much more complex than this gender question, so “do a better job with your gender questions” might enjoy that.
