Jump to content

The fault in the stars

Recommended Posts

Recently, Fiverr announced a transition from its traditional reviews system to a rating system.

Rating systems can be effective when they aggregating the views of experts or authorities; because they allow for the quick observation of consensus among the highly informed. They can also be useful in the analysis of survey / focus group data that will be used internally for a company.

Otherwise, rating systems are a notoriously problematic substitute for reviews. Even when provided side-by-side with reviews, it encourages bad/short-cut behavior from buyers that is much worse than simply having the reviews.

Furthermore, just as the creation of an effective survey / focus group is a serious endeavor that requires careful construction, a public rating system cannot hope to work when the construction behind it is rushed or flawed.

Sadly, the Fiverr rating system is neither flawless nor functional.

You’ll understand why as you read on.

Before presenting some research evidence on reviews vs. rating systems, let’s highlight some of the specific issues with the current system as presented by Fiverr.

Currently Fiverr is asking buyers to rate sellers on three criteria: Response Time, Service as Expected, & Buy Again or Recommend?

This particular list of three “criteria” almost couldn’t be WORSE if you tried



"Seller Response Time" - Even Ebay (the business that patented their review process) has removed a buyer from being able to give feedback on delivery time provided the seller used tracking and the third party tracking confirms what was posted initially into the listing.

Why? Well, because “Response Time" is a subjective area.

It's even worse with online interactions. People online expect things to happen in minutes. Online makes an “on-demand” mentality. We’ve all seen it (and if we’re brutally honest we’ve probably done it). Perhaps you’ve sent a message only to send a follow up 15 minutes later—sometimes asking, “What’s taking so long?”

Of course, in most cases, “Response time isn’t an issue”.

Fiverr already DISCLOSES the response time in hours, what more is being “disclosed” or “revealed” by stars?

So, when it’s not an issue to the buyer or simply never came up in the transaction, how can someone ever judge it as “excellent” 5-stars? Isn’t “satisfactory” (3-stars) a more honest review?

Unfortunately, it’s even worse than that, because when response time does come up and is NOTICED by buyers, it’s usually when they feel response time was “off” or somewhat delayed. Even if the “delay” happened because of power outages, world-wide time zones, or simply being a part-time seller on Fiverr supplementing a full time job.

In short, the REVIEW process already allows someone to address time, and the current Fiverr system already shows a seller’s average response time. Why do we need the community to review what is objectively provided to all members of the community?

Next we have "Service as Expected".

This is absolutely the wrong question to be asking.

Service as described is much closer, though I oppose these “specific questions” as anything worth reviewing can be reviewed in a—review. But even if there is a reason to ask it, it’s better as a “yes / no” question (thumbs up/down) not a rated one (stars).

Now let’s dig a little deeper.

(In the first reply to this thread because our forum has a word limit LOL)
Link to comment
Share on other sites

There is an economic problem with asking buyers to review expectations.

It hurts Fiverr, and everyone else involved, because the best marketers will be the most effective at getting prospects to imagine the benefits and joys of buying.

When you couple this “marketing effect” with the fact that Fiverr continues to allow buyers to review gigs – and even prompts them perpetually unless buyers actively turn off those prompts – even YEARS after the initial purchase…

Think about it; are YOU in the exact same life circumstances we were a year ago?

Of course not, things change.

Memories of an impulse purchase fade. And, someone excited about losing 20 lbs in 20 weeks can suddenly be disillusioned when they realize the method is to eat less, move more, and drink water.

There’s an old joke that says,…

"Do you know why divorce is so expensive?"

Answer: Because it's worth it.

This is a joke about getting out of one of the most "expectation" laden relationships we can have.


Because, over a lifetime, it's hard to be prince charming / princess snow white and live "happily ever after". You cannot let buyers come back 9 months later and leave feedback. Buyer’s remorse is already something we struggle with…the rating process makes it immediately worse.

And of course, what of someone who has unrealistic expectations?

Do we really want to give them a way to “leave negative feedback” even though they admit the delivery of their purchase was sufficient/adequate?

Finally, there’s the "Buy Again or Recommend" question.

The "or" suggests that Fiverr may have considered the most obvious problem with merely asking about repeat purchase.

Simply put, many gigs are one-offs.

However, even if something would lend itself to repeat purchases, most of us lack the imagination nor the inclination to correctly forecast future needs when doing what should be the rudimentary task of reviewing their order.

It’s not obvious why I’d buy the same Mother’s Day gift for my Mum. Few would think, oh, you know ...even though I would never need to buy this again for my mum, I could see myself getting one for Auntie Rita.

So, in an attempt to address the first problem, buyers are asked to consider if they would refer.

Well, again, the problem is that we lack imagination—especially while completing tasks we’re pressured to do by green messages on a to-do list.

Even people who can / do refer rarely "plan on it".

Rather, most referrals are made because a synchronistic event happens where they can provide another person with an answer / solution that came up naturally in a conversation which couldn’t have been anticipated.

Once again, the thumbs up / down system wins.

Why this time?

Because a thumbs up is an acknowledgement that “I might refer in the future” whereas, if you disliked the service you’d be disinclined to refer it to others.

Additionally, there are people who aren't Mavens and don't see themselves as referrers. In fact, few people really are.

Having addressed each criterion separately, there’s another problem globally: 5-Star rating is being titled “exceptional”. Outsides who have not yet rated a gig, that excited new buyer, won’t know that 5 is exceptional (meaning the exception and not the rule, that is, rare) will consider anything less than 5 stars an indication of a serious and significant deficiency in the gig on the part of the seller.

Additionally, someone who takes words seriously will struggle.

If there was no reason for seller contact, how can one rate the performance of such? If the buyer was curious and had no "expectations" per se.

How can something which didn’t exist be “excellently” (5-stars) met/exceeded?

Finally, if the gig is not the type of gig that would be repurchased, and I’m the kind of person who doesn’t refer… How do I honestly give a 5-star rating?

For that matter, how many of us have been told, even promised future work and business only to have that buyer never return? That is, the rating is meaningless in BOTH directions (high ratings or low ones); but only good for a seller if it’s a 5-star one.

My final critique of the “entire system” is the buffoonish way in which it is being rolled out.

People keep posting that the rating system has been abandoned and the review system is back BECAUSE from time to time the old review system appears.

However, CS has stated:

“We are still rolling out the new feedback system, and not all users will have the option to use the option to rate with stars just yet.”

I’m befuddled that Fiverr CS thinks this is an ameliorating feature.

This doesn’t make anything better, it just makes everything confused.

It doesn’t make me at all pleased. It just makes me more anxious each time that I deliver a gig – I wonder if my buyer will get the sound system or the insane one? This is such a problem I’ve suspended all my gigs.

Having leveled all this critique…

I want to be transparent about something.

I am a HUGE fan of Fiverr.

I have defended many of their controversial decisions or crafted positions. I spoke reassuringly to the sellers who felt Fiverr 2.0 would be the “death of the site”. I resisted the “add more steps like order approval” to the sales cycle.

And I flatly reject the idea that Fiverr “only cares about the buyers”.

Do I think buyers have an edge in a general way? Absolutely.

However, I also think it SHOULD be that way (but, only very slightly).

Do I think Fiverr is also a special place with privileges that inure to sellers?

Again, absolutely.

So, I flatly reject all the people saying this is some kind of continued “plot” against sellers.

It is, as a few other things have been,…

…a clumsily implemented and unfortunate “upgrade”.

To make it worse, it was a wasted chance to improve the Fiverr experience by focusing on one of the better features of the site (the review system), when there are major issues that really deserved the time and attention this travesty consumed (such as notifications).

So, even though I’m a huge fan and reject some of the anti-seller noise,…

A bad system is a bad system.

It doesn’t matter if Fiverr has some “nice things” in store for sellers soon. Why? Because this rating system is bad for the entire community of buyers and sellers alike.

For sellers, it’s a pin. For buyers it’s unnecessary noise that doesn’t aid good decision making.

This has been my consistent position. I voiced these concerned when the covert testing of the star system first started. And, I explored the problems with rating systems when some members suggested it might be a nice change.

Now Fiverr wants us to believe their tests were positive.

I don’t care to dispute that they may have seemed that way.

However, covert tests that were described as having no immediate impact on buyers does not have the same incentive structure as does a public system that is going to be reported and subsequently shown on a buyer’s gig page.

And while no “tested and measured results” have been released by Fiverr, there is an effect called the “Decline Effect” (Initial results tend to show better correlation of what a research is looking for because of a host of biases. Results then disappear when retested by third parties or merely regress to the mean upon a bigger launch; eventually showing themselves to be random results).

Now, does this mean every result ever should be disregarded?

No! Of course not.

But when one is met with such an immediate, nearly universal, and acerbic response by the community it purports to serve…

I leave you, the gentle reader, to form your own conclusion.

(Continued in the next reply because of the word count limit)
Link to comment
Share on other sites

In an effort to equip you to form valid conclusions, let’s talk about a scant bit from the large body of research that makes it clear reviews are better than rating systems.

There is a report titled: Price Reviews and Ratings-How consumers evaluate a hotel room purchase in a social world which you can Google.

The methodology tested three factors. Price (low/high), Rating (2.8/4.8 out of 5), & Reviews (Positive / Negative)

Fiverr is mostly for $5 services (as the name suggests), and even though gig extras can get large, they remain well below the market conditions. So let’s ignore price.

The report found that reviews mattered much more than rating. Reviews are the better communication of value.

In fact, the report reads from its first published conclusion.

Our findings suggest the following takeaways:
Reviews are the most powerful quality and value indicator for consumers: Our research overwhelmingly indicated that consumers look to the reviews over aggregate ratings to form quality and value perceptions.

This runs counter to some theoretical arguments that suggest that consumers are “information misers”, preferring a metric that’s easier to consume (like an aggregate rating), as opposed to the information-rich review.

We hypothesize that the uncertainty associated with the hotel experience leads consumers to want to gather as much information as they can to mitigate this uncertainty. Hotel managers must not only understand their review sentiment, but also that of their direct competition, in order to successfully position themselves in a highly competitive market.

Competing on price alone is not a winning strategy. While consumers prefer to pay the lowest price, they will look closely at your UGC (User Generated Content / Reviews), and that of the competitors, when making a purchase decision. This means hotels can’t undercut (or raise) price simply based on the price movements of the competitors. Managers must also understand how their UGC compares, evaluating consumers’ total value perception of their hotel versus the competition.

It’s hard to overcome “bad” UGC: Our results indicated that lowering the price of a badly rated, and negatively reviewed, property drives no additional value in the minds of the consumer. If a hotel happens to be in that unfortunate position, they should keep the price up, and take what they can get – which according to our results won’t be much. Our recommendation for these properties is to focus on fixing the problems with the property instead of worrying about how it is priced!

In my own summary, the thumbs up review – reviews with personally written words from buyers (not automatically generated ones by the system) - is the superior method for the community as a whole.

You can also search for the Harvard Business School paper titled: Reviews, Reputation, and Revenue: The Case of Yelp.com by Michael Luca.

On page 19, it reads:

How do consumers update beliefs based on information obtained from consumer reviews?

On the one hand, a standard model of Bayesian learning predicts that the market would react more strongly when ratings contain more precise information and when prior beliefs are less precise. On the other hand,…consumers [will] use the average rating as a simplifying heuristic. This may cast doubt on the sophistication of consumer response.

That means that it’s the casual buyer who is most likely to rely on the rating and ignore the nuance (that would also include a seller’s response to a bad review). Does it make sense to optimize the site with the casual buyer in mind?

No, because the casual buyer will only buy when they see a 100% seller rating, complete with a short response time, etc.

I want to address one more suggestion by Fiverr CS before ending this detailed response.
Fiverr CS has suggested this system won’t lead to abuse, and rather, will lead to useful information to help sellers fix/improve their offerings.

This is a huge assumption and one which even if true is meaningless because it’s not how it will be FELT.

Imagine I ask a buyer why they rated me 3 stars on the response time area when we never had need to connect, only to find out they “wished” I had made a little time to chat with them…been a little more “neighborly” with them before delivering their order in under an hour..

Guess what?

I’m not changing any in my Gigging because of that. There’s no way I can know in advance which people will want to chat. (Not to mention that I don’t want to and that should count for something) What I think is a fair assumption is people want what they purchased in an efficient way.

So the way I run my Giggle will not change, but something has changed. My rating has changed. I’ve been harmed because the next person who sees that three will be perplexed at how I’m both “not good at responding” yet have an average response time of 3 hours; according to Fiverr.

And what will the confused prospect do?

At BEST (and this is, in my opinion, is still very bad) they will ask me questions I have to answer before a sale. (Remember, I wasn’t really wanting to chat?)

At second to the best, the buyer will skip to another seller who hasn’t yet been crippled by the rating system (I don’t mind spreading the wealth).

But at worse, they will simply not buy from the site.

And that hurts all of us. The buyer, the seller, and Fiverr.

Fiverr.com, you are better than this.

Fiverr sellers, you deserve better than this.

Fiverr buyers, you also deserve to have the better of the systems proven to help careful people make informed purchasing decisions.

Let’s all just put this behind us and get back to doing what we love.
Link to comment
Share on other sites

A great and detailed post anarchofighter. Read it all and I couldn’t agree with you more on this. We just “LOVE” to have our old system back where we had lots of fun and excitement working as freelancers. Really hope that the Fverr staff will do the right thing here 🙂 Let’s keep our fingers crossed and hope for the best!

Link to comment
Share on other sites

I agree with this post wholeheartedly.

For the past few days I’ve had no motivation whatsoever to complete orders. It is depressing to know that fiverr cares so little about sellers to implement such a horrible feature with the caveat of ‘future upgrades for sellers’.

How about telling us what you plan on changing? What happened to transparency, openness and fairness?

Stop dangling the proverbial carrot and tell us what you plan on doing. The backlash from the community (who, not forgetting, made fiverr what it is) has been incredibly strong; and long may it continue until fiverr realise how archaic/barbaric this change is.

Link to comment
Share on other sites

I glean no information from the new rating system. I have had 100% rating since the beginning. I’ve had several ratings with the new system. Only 1 did not give full stars. I looked at my gig description and the communication I had with the buyer and nothing indicates any reason for marking me down. Everything was delivered on time, and exactly as the customer requested. I feel like everyone’s expectations are different, so it’s best to let the customer leave their feedback in their own words. I’ve suspended my gigs. Is there anything else we as sellers can do?

Link to comment
Share on other sites

Reply to @calocedrus: You’re spot on.

And you’re also lucky! I’ve resurrected a couple of my gigs and I deliver with messages like this:

Either way, if you have questions or concerns of any sort let me know otherwise, please remember to leave superlative feedback! It's the most important part of my fiverr activity.

If you get a "star" rating system, please help us keep Fiverr a review based site and after selecting 5 stars for all categories, please put in your review that you want to change back to the thumbs up review system.

You can of course not rate the gig at all if you prefer. However, if you are unable to rate 5-stars, let me know so I can refund your order. As long as this doesn't happen often, I can continue to help buyers.

Basically, I'm leaning on people's sense of fair play and knowing that while they can screw me out of my work (at one level that's nothing new) I'm letting them know that they're harming others and themselves in the future.
Link to comment
Share on other sites

  • 1 month later...
anarchofighter said: So the way I run my Giggle will not change


This made me "giggle". LOL


Nice post - really broke it down. I want to bring my gigs back, but I just can't do it with the star system still in place. I work two jobs outside of Fiverr. The people that expect work in 24 hours or less will eat me alive. Not to mention the ones that require infinite amounts of re-work, no matter what the gig states or what Extras are offered.


Anyway. I'm glad to see someone so articulate stating what's wrong with the star system. :)


Now if we could just get them to LISTEN...

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...