G2 Crowd scores Salesforce Chatter top social collaboration tool

I saw the following tweet from Salesforce CEO Marc Benioff:

This was interesting enough for me to follow the link to see why Salesforce Chatter was rated the best social collaboration system.  Barry Levine at Venture Beat had a nice article about G2 Crowd’s grid on social collaboration.  Here are the first two paragraphs:

Salesforce’s Chatter is the social collaboration tool with the highest customer satisfaction.

That’s a key takeaway from G2 Crowd’s new Grid for Social Collaboration tools, the first Grid report for this category. It scored 14 business products based on G2 Crowd’s customary metrics of customer satisfaction and business presence, in this case based on over 325 reviews from business professionals.

This was really interesting to me.  G2 Crowd allows people to rate these systems and then translates those ratings into a 2×2 grid similar to Gartner’s Magic Quadrant.  Venture Beat also mentions the 325 reviews used to generate the ratings.

Choosing a Global Software Development Partner to Accelerate Your Digital Strategy

To be successful and outpace the competition, you need a software development partner that excels in exactly the type of digital projects you are now faced with accelerating, and in the most cost effective and optimized way possible.

Get the Guide

While the G2 Crowd rating system looks good and in theory produces crowd-based results, when you dig a little deeper, you find that G2 ratings are really based on much smaller sample sizes. For example, here is the latest grid as of today:

In looking at the Leaders Quadrant, you find that Chatter had 97 reviews and Microsoft Lync had only 11 reviews.  You can click on an icon to see the number of reviews:

I don’t dispute that Chatter is an excellent social collaboration tool and Salesforce should be proud to share these kinds of results.

However its hard to rank these tools on customer satisfaction alone from such small numbers of reviews. First, I imagine you could easily get 11 reviewers to enter lower scores for any product. Likewise, I’m sure you can ask 11 people to go rate a product very high.  I’m not suggesting anybody is doing this on the G2 Crowd site to impact the rankings, its just a possibility.

Second, and more importantly, assume that the total population of users for all these tools were 1,000,000 users.  That’s probably very low, but its a nice round number.  To have any confidence in our survey, we’d need a sample size of over 1,000 users responding to the G2 Crowd reviews. And that would be 1,000 random users.  With only 11 reviews or only 97 reviews, statistics tell us that we should not be very confident in the results.

The G2 Crowd information is interesting and it would be great if we could get the right number of reviewers to make the data statistically meaningful.  But, really, don’t rely on this information alone to make your purchasing decisions.  That’s why Gartner, Forrester and Perficient use a variety of factors when rating these systems.