Testing improvements to the rating & review system

[excelmagics]

I'm not happy with this EMOJI's thing at all. The stars were much much better.
now with emojis, they will just select middle one thinking work was good or the second last thinking work was very good., they don't know they're going to give you 3 - 4 stars for no reason. Only few of them will think work was Exceptional, even we do the BEST for all of them, not all of them will give us that Exceptional tag,
as I'm writing this now, I've already gotten 4 stars from a buyer , he wrote nice words , and I did really great job for him but sill he marked it as "Very Good" which is 4 stars according to new review system.

[priyank_mod]

16 hours ago, rapidtech1898v2 said:

Thats what i don´t understand - how can a seller finally gets 5 stars with the new system?

Maybe that's the whole point of ALL recent updates to the review system in 2023 including cancellation reviews. 🙃

They are trying to do away with "having too many 5-star sellers" but rather have multiple sellers at every incremental 0.1 between 3.5 to 5.0 where being 5.0 will become almost impossible!! 😇

Seen many relatively young/new sellers having breakdown on the forum after receiving a 4.5 or 4.7 for an order!! People just want to have futile perfect scores because that's what most of us have been taught at schools!! 🫠

[rapidtech1898v2]

They only thing i want to know:

How can a buyer give a 5 star rating to a seller now?

[priyank_mod]

18 minutes ago, rapidtech1898v2 said:

They only thing i want to know:

How can a buyer give a 5 star rating to a seller now?

As sellers, it shouldn't bother us. Not like we can tell/guide a buyer about giving it. 🙅‍♂️

We can leave that between Fiverr & Buyers. Let Fiverr guide them about how to rate effectively & accurately!! 😇

[rapidtech1898v2]

I am not only a seller - i am also a buyer.
I simply want to know what i have to do - so a seller get a maximum rating from me.
Its still unclear for me.

When i gave 2 excellent votings and a short text that i am fully happy with the delivery i expect that the seller will get 5 stars.

So far as i see it now that is not the case and the seller only gets 4 or 4.5. stars in the system  - and this is simply what i would like to get clarified.

[emmaki]

From reddit: 

Question: will this lead to an uptick in "feedback manipulation" account disabling (positive or false positive)? 
Question 2: Is Fiverr CS actively correcting what the comment refers to as a "mistake"?
Question 3: Does this mean the beta test is not working correctly, giving everyone the wrong impression and making the emoji review system even more complex, convoluted, and confusing than it already appears to be? 

Or is this just a test to see if this is a way to overload Fiverr CS with tickets  transparent way to break that 5 star dominance as suggested above? In which case, if CS is apparently changing up reviews, it doesn't seem terribly effective. Unless the goal is to make CS work harder so Fiverr gets more bang for its buck. In which case, genius move. 

I have no new and fiendish ideas for bad review systems today. 
 

[vickieito]

2 hours ago, rapidtech1898v2 said:

How can a buyer give a 5 star rating to a seller now?

I think this is what Fiverr is trying to avoid - it looks like Fiverr doesn't want its buyers or sellers to be thinking in terms of 5-star reviews anymore. And if they do, the seller can get a warning for review manipulation (even though it's the buyer reaching out to CS, not the seller). I wonder if the platform will also flag accounts with 1 or 2-star average reviews.

With the new rating system, Fiverr wants the average to be 3-star reviews. It wants stratification of reviews. It will prompt buyers to categorize their sellers, so good sellers will fall into the "Average," "Very Good," or "Exceptional" categories.

Since the old and new review systems will be merged, those with a lot of 5-star reviews already will have an advantage over the newer sellers or those with very few rated orders. These established sellers will be seen as "Exceptional." Otherwise, if you are a very good seller, your average will be 4 stars. 

Fiverr should also change the threshold for ratings, or else all the "very good" sellers will be demoted every month. It already lowered the threshold to 4.2-star averages when it allowed reviews on canceled orders, but Fiverr needs to lower it to 4.0 with this new review system.

[katakatica]

2 hours ago, emmaki said:

Does this mean the beta test is not working correctly, giving everyone the wrong impression and making the emoji review system even more complex, convoluted, and confusing than it already appears to be? 

THIS is what I'm betting on. (as in, the system is somewhat broken(?) OR working well (without us really understanding what 'well' is.) I understand the point behind it, but I've seen a lot of posts (both here and as a lurker on reddit) that something seems up.

Maybe the transition isn't as smooth, and some metrics are getting mixed up.

I genuinely think that if I give someone two 'happy-af-emojis' they should get a 5-star review. Period. As a buyer, I need to be aware of what I'm doing (though many aren't, but...) 

3 hours ago, priyank_mod said:

futile perfect scores because that's what most of us have been taught at schools!! 🫠

My concern is that many people do actually perceive only a perfect 5.0 as, well 'perfect'. I was one of the kids scolded for not doing perfectly and definitely ended up being someone who scrutinizes ratings as well. In the end, if buyers are aware that there is no 'perfect', I think this can work? But otherwise, I'm still worried... 

Must be weird to see only 5 stars, go away for the holidays and (potentially) come back to a whole new system...

[zeecreations]

4 hours ago, priyank_mod said:

People just want to have futile perfect scores

Not the people, Fiverr has always been a 5-star review platform. Anything less and you're going down the algorithm. The system has always been designed that way.

[zeecreations]

@Kesha After this update, will you guys also be lowering the 4.2 ratings threshold. I doubt anyone would be able to maintain that with this Emoji-infected of a system. 

[priyank_mod]

8 minutes ago, zeecreations said:

Not the people, Fiverr has always been a 5-star review platform. Anything less and you're going down the algorithm. The system has always been designed that way.

I remember reading on the forum long back that, "If everyone is 5 star, then no one is 5 star!!" ⬆️

And it's not just Fiverr - 5-star-based rating systems are common across many e-commerce & restaurant aggregator platforms around the world. And best-performing products/restaurants/services are often between 4 to 4.5; ratings between 3.5 to 4 are usually acceptable and achieving anything above 4.5 is almost IMPOSSIBLE!! 

Edited December 12, 2023 by priyank_mod

[zeecreations]

1 minute ago, priyank_mod said:

best-performing products/restaurants/services are often between 4 to 4.5

You just described a conventional rating system, not the one designed by Fiverr. Anything less than 5 isn't a good rating according to their algorithm. 

[eledigiac]

14 minutes ago, zeecreations said:

Not the people, Fiverr has always been a 5-star review platform. Anything less and you're going down the algorithm. The system has always been designed that way.

I agree. The discourse is not wanting to hold on to the 5-star school ratings, to me that touches on the issue from afar. The issue is that the algorithm relies on being above a certain rating to give you visibility, impression, clicks and conversion. The life of the freelancer cannot be regulated by an algorithm only to the negative, I think instead of winking so much at the buyer fiverr needs to start an internal policy of protecting their best freelencers. And by best I mean: available, professional and reliable in the long-term. That is what should get visibility on fiverr and be incentivized. 

[donnovan86]

51 minutes ago, zeecreations said:

You just described a conventional rating system, not the one designed by Fiverr. Anything less than 5 isn't a good rating according to their algorithm. 

Not really. There are many people with lots of 4 star reviews that are very high in searchy. The problem is when you get a bad private review, that's when things become problematic. And that's what affects ranking.

[Kesha]

22 hours ago, vibronx said:

You can leave a private review without leaving a public review. I have given a seller a bad private review once without giving them a public review, as I did not like their work. It would have been equivalent to a 3-star public review or similar, but I did not feel like leaving that in public—the reason being that I have been harassed by sellers in the past for leaving a negative review. One even went as far as to try to find Instagram profiles that he thought looked like me and presented them to me, likely to threaten me. None of them looked like me, so I got him to stop by asking him if he thinks all white people look the same.

Sometimes, you just don't want the hassle of a seller going crazy over a fair review. For this reason, I like the private review. However, I do feel it should be presented at the same time as the public review. As you say, it's annoying to have to leave feedback twice. 

Hi @vibronx! You make some points and we value your feedback concerning the private review system. We will be sure to pass along your thoughts.

[Kesha]

On 12/8/2023 at 11:03 PM, greg_vo2020 said:

I appreciate the efforts for an improved rating system.  One concern I have is the "value for money" rating.  I consistently have received good ratings for 'the work' over the years.  But that particular rating is not a rating on work.  It is a veeeerrry subjective view of money.  I personally KNOW my rates are lower than other past outlets but that still isn't good enough for people.  In some countries the value is great for vo service.  In some it's not so much maybe. Not to mention the personal opinions about money.  I don't feel this is helpful. 

@greg_vo2020, you bring a valuable perspective to this new system and we appreciate you voicing your thoughts! We will keep them in consideration for future updates!  

[Kesha]

On 12/10/2023 at 10:46 PM, moikchap said:

I just received one of those "You still have time to review (whoever's) order!" and it reminded me of a long-standing problem I have with the review process: I don't know who this is. I have to leave the review page to check what order it's in reference to. I can have up to ten orders going on at a time, sometimes I receive multiple deliveries a day. I don't know these names. I need to be shown the delivery or at least the gig. I have no idea what order it's in reference to sometimes when all I get is a name.

Thanks for this suggestion @moikchap! We will pass it along to the rest of the team for consideration.

[ocastriota]

This new review system is not helpful at all. You're converting emoji's to numbers and these changes are making it more complicated and my buyers have even said "the new reviews are so different and complicated, I did my best." Are you planning to change how sellers review buyers as well? You're going to end up with way more top rated sellers with all 4 star reviews instead of well deserved 5's if you continue like this. 

[visualstudios]

Overall this new system seems to be aimed at lowering ratings across the board. Some points on this:

1 - Yes, a marketplace with all 5's everywhere can seem suspicious. However, I'm not sure Fiverr would rather have 4's everwhere, do you think that will increase sales? Specially for long time users, who are already used to how the system works, and know by now to avoid any gig with a 4 star review average, as that has always been bad?

2 - How will this be applied to different sellers? Imagine that going forward, with the new system, the average rating everyone will get is 4.5, because the system is more punitive. If a seller has 10 thousand reviews on a gig, they'll go years while maintaining a 5 star public review average, since the new 4.5 reviews they will be getting going forward will take a really long time to bring the average down. Therefore they'll be unfairly given an advantage vs a seller with fewer reviews, not because their work is better (they're both getting the same review average from now on), but because 95% of their work was reviewed using a different, more lenient system, so they have a huge headstart of perfect 5 star reviews, that were very easy to get before. This is a major problem. Yes, their private score will be the same, so they will be impacted equally in terms of search results, etc., but from the buyer's perspective all they will see is a seller with a 4.5 gig and a seller with a 5.0 gig - even if their current performance is exactly the same.

Edited December 12, 2023 by visualstudios

[Kesha]

On 12/6/2023 at 6:33 PM, terrygrantvo said:

@Kesha Is this graphic accurate, in the sense that answering 'exceptional' to both of the multiple choice questions only results in a 4.5 star rating?

Also:

Can you elaborate a bit on how this new rating system is going to help with either of these ideas?

 

Thanks!

 

HI @terrygrantvo! The graphic provided is just for illustration purposes, it is just a demo. If someone marks ‘exceptional’ on all new form questions, the rating will be 5 stars. 

Concerning your second question, this new system will help collect more reliable and constructive feedback, allowing you to improve and achieve greater customer satisfaction. It will also provide a modernized streamlined process for your customers.

[Kesha]

On 12/11/2023 at 8:56 AM, bishnubiswas908 said:

@Kesha Is there still going to be a private review system that we have no insight into and will it continue to carry more weight than the public reviews?

Hi @bishnubiswas908 Yes, This test will not affect the Private Review form. We appreciate your feedback and will pass this along to the product team. 

[Kesha]

On 12/11/2023 at 9:11 AM, bishnubiswas908 said:

On the Image 1st and 2nd chosen exceptional but the final result showing 4.5 Rating.
👉Is this just demo or sample?
👉Or Clients Choose both exceptional so the Rating make 4.5?

 

Hello! This is just a demo; the images provided are just for illustration. If someone marks ‘exceptional’ on all new form questions, the rating will be 5 stars. Hope this clarifies some things for you!

[Kesha]

23 hours ago, eledigiac said:

With the new system, I just got a 5-star review, so I guess just check all the boxes.

But honestly despite the fact that I feel great about the platform and that I've been working on it for years now, there's an underlying policy that I can't love. The freelancer always feels, or at least it happens to me, a bit crushed by the algorithm.

I know it's the surest way to ensure that the quality of work is high, so let's just say that over time I've learned to deal with it. I mean to deal with the score going down if you don't respond to new clients within 2 to 3 hours at the most, if you don't deliver the product on time, if the client throws a tantrum and in spite of agreements made asks for a thousand revisions, which in the end I always more than willingly do because for me it is crucial that the client is satisfied with the final work.

But this beta testing of reviews I find terrible. I think there needs to be more clarity for the seller as well, I think the review should be only one and public and only then it can improve the client's work, and I think if everyone's score is going to go down (as it will because some of the options in the boxes are unrealistic to tick), then the average to be top rated (always be above 4.7) should also be revised.

Also, an important update for me could be to limit the customer's ability to request revisions according to the number of revisions actually purchased and not indefinitely. This would allow more deliveries in a short time and more objective reviews by the customer. Which then, I actually wonder why the stars were not, no one ever forced a customer to click 5 stars rather than 1...

Hi @eledigiac! Thanks for your commitment to our platform. I appreciate you expressing your feedback and suggestions. I will be sure to pass them along to the rest of the team! 

[emmaki]

Yes, it's a wall of text. 

@katakatica, the whole problem with Fiverr's 5-stars or you're toast pretty much started when Fiverr switched from the thumbs up, thumbs down system to the "Advanced Star Rating" - Fiverr's term, not mine - system in 2014 (all the talking points here applied then). IMO. 

The whole point was obviously to make it less binary, but that never happened. This was pre-levels, BTW - those weren't introduced until 2018. The Levels system was done to make the 5-star problem less of a problem! Now we're at the stage of how to make the review system less of a problem for the levels system. Oh, and the very long wait-list of people waiting for TRS. Which was also increased by dropping the $20k earnings req to $10k, IIRC? 

And the Pro backlog (which notably doesn't claim to be the top 1% of freelance talent anymore, but the way in which so many Pro sellers were made a few months ago, well...no shade to the promoted, but Pro was almost certainly downgraded to mold to the needs of the "new" pro marketplace). And the need to rebalance what may be private reviews leading to less public reviews, which of course makes the whole monthly evaluation thing... less efficient? In the past year, more than a few buyers have mentioned to me they left a review, but I see nothing. Sure, private reviews matter more! I assume they're why I get Fiverr's Choice orders. 

Anyway, all that waffle amounts to one thing: this new review system is just going to kick the can of worms further down the road. The review system is broken. Everyone knows that. I think I'm safe saying we all want it to be fixed. Properly. Not with emojis that don't add up and give CS extra work. 

As it is, Fiverr is now trying to go upmarket and really sell its new, professional image of pro work for pro buyers  on a pro (or certified) marketplace with... an emoji rating system. Cognitive dissonance, anyone? Unless they're planning to only use emojis for the regular marketplace and keep stars for the pro marketplace. Hey, why not? 

The reality is that most consumers don't leave a review unless they're super-happy or super-mad. Or being paid. Plenty of buyers are aware that their review can have a crazy high impact on sellers' income months into the future, so simply don't leave reviews unless one of those outcomes happens. This new emoji review system hardly fixes that when nobody can figure out how to leave a review that has the potential to demote someone for two evaluation periods in a row. 🙂 - and that's not including the other types of no-review buyer mentioned somewhere above. 

Any review of uh, the review system needs a review of the monthly evaluation system. There are few other places in the world where a company's most successful performers are evaluated on virtually the exact same criteria as its unranked/worst performers. Not just monthly, but even weekly/daily. Not to mention that the less orders one gets (hello big ticket sellers who aren't farming out their work off-platform to evade Fiverr arbitrage Community Standards/chronically ill people/etc), the more severe the impact. I mean, the levels system was only on Fiverr in the first place to gamify the system with the monthly evaluation tacked on. The whole point was "fun". Now that fun has turned into a complete mess of a review and ranking system through lazy, poorly-thought out fixes that don't ultimately fix anything.  

Fiverr put zero thought into its original 5-star rating system, and at the time invested very little time, if much, into the community - anyone who was a mod here at that time knows that I'm 100% correct in that statement. While I can't prove this directly, watch some of Shai Wininger's (co-founder of Fiverr) interview videos about Fiverr's earliest days and it seems that Fiverr grew too fast, hired too fast, and then office politics raised its ugly head as one camp - the make money and get IPO camp - won over the other boring and less profitable "we should vet users more carefully and be at one with our community like hippies" camp. It's only since after the IPO that Fiverr has really started to invest in hearing the community, but it still isn't really listening. Old habits break hard, and of course, profits are perhaps even more important now. 

(Wininger also mentions in one of these interviews that Fiverr was only called Fiverr because he was obsessed with the number 5 at the time. Go figure. Kaufman's input into all this was to realize that $5 was cheap enough that few people would bother cancelling. There's more to it than this, of course, but that's the somewhat uncharitable ELI5)

Yes, I know they frequently consult with top sellers about things that could be changed and blah blah blah, but how many of the actual good ideas have come to fruition - and how many have been replaced by zany and (comparatively) easy-to-implement things like "guys, what about... smiley face reviews instead of numbers?!?!" 

...Which no buyer or seller on this platform who is at a level that Fiverr is personally reaching out to consult with them is going to recommend. OK, so sometimes they do stuff like this on the forum, and after 780 pages of posting by sellers and buyers who don't like the new thing very much, it goes ahead anyway, with some minor fixes. 

Case in point: this announcement is on the forum. The announcement was made on Thursday evening; or, just in time for what is pretty much the start of the Israeli weekend (prime time to bury bad news in the PR and media world...). And maybe I'm wrong here, but doesn't Fiverr usually announce its big beta tests with a notification? Or an email? But what we effectively have is Kesha picking the occasional post - I appreciate that she's probably limited in what she can do - and thanking for input, passing on, etc on a post that a fraction of the community has seen and is really only slowly waking up to because of where is muh five star review. 

Still, it bodes well for the good old thesis-antithesis-synthesis strategy. Tell people about a new thing that everyone invariably hates. When it is showered with rotten vegetables and durian fruit, come back with the new and improved idea that's a bit less rubbish but what you wanted all along. Fiverr can tell everyone how it listens to feedback and adapts without having to change any of the real issues behind the feedback system.  

I don't think sellers here have any real issue with the idea of not getting a 5-star review. The problem is that Fiverr pretty much demands a 5-star review, whether you do it with stars, emojis, or cup and ball games. Sure, we all know there's lvl 1/2 whatever sellers who make more than TRS/Pro, but it's not about that: it's about the (very) effective gamification. 

Fiverr needs to depend less on AI for ranking and reviewing sellers with CS mopping up the ever more complicated messes that are starting to require in-depth guides or experts for when it is "safe" and "not safe" to do perfectly normal contractor things like cancel an order when things go wrong. Or, even just signing up for an account and getting instabanned because you used an image from your Insta portfolio and there's no guidance on site or via the DENIED autoresponder that you can email CS to sort that out. Or a Help Center page that details this. 

I would personally be refocusing on what Wininger said, probably a decade ago. AI just can't achieve that at the moment, and I can't be the only one who has noticed the crazy increase in false positives account actions lately. Makes me wonder about the big Bangladeshi account ban-o-rama at the beginning of the year... 

I could really go on about the myriad issues that Fiverr has, but it hardly matters since nothing will change unless "the people that have the true power on Fiverr," as disgruntled sellers like to say when Fiverr has displeased them, actually show that power. Unionization is actually in Fiverr's annual report as a risk, yet Fiverr will promote the Freelancer's Union (not an actual union - surprise, surprise!). But a safe risk, since there's no hope of sellers really getting together for long enough to do anything, never mind set up a virtual picket line. 

Am I saying Fiverr sellers should unionize? No, because I'd be wasting my breath. Anyway, back to inane review systems! 

For those not in the know, the video below is Thatcher's famous "opposites day" speech. I think the Fiverr review system should copy it. 5 Thatchers = 1 star and 1 Thatcher = 5 stars. You could also use Kinnocks to express toothless disappointment. Numerically, spin the bottle could be used to determine how much a Thatcher or a Kinnock is worth, with an algorithm deciding whether the end number is fair or not then making up a new number based on Fiverr's stock market sentiment. As you can see, I've thought through this carefully. Sadly, no emojis, but maybe little favicons would work instead? We could also have a JFK and Trump/Biden rating system for Americans, where Americans could pick which is good or bad based on their political leanings. Ditto all other countries of the world. There, I even pre-complicated it! 

It's OK Kesha - I shall thank myself for my input and pass it onto the relevant team myself! It all yields the same result anyway... 

[visualstudios]

In addition, the way the emoji scale is described is weird.

"How was the quality of the delivery in relation to your expectation?" is not the same as what shows up in the review - "quality of delivery". Those are two different things.

What happens now, with this new system is very simple:

An ok seller that delivers an ok result, but has a buyer that (either because the seller conditioned him to, or for price reasons) has very low expectations, is likely to get an exceptional (5 star) result. It's easy to beat expectations where there are next to none, and it's easy to beat expectations for 5 bucks.

A professional seller, that delivers a good result, but that is exactly what the buyer was expecting (which is what should happen, and is in any case better than the case above), is likely to get a 3 or 4 star result - because you can't beat sky high expectations.

So, better end result matching expectations is worse than worse end result that beats expectations that happen to be low.

This will make it less attractive to position yourself as an high end, quality seller, and will encourage people to always undersell. This means lower pricing and less attractive gigs (the better the gig looks, the higher the expectations any reasonable buyer will have, and under this system high expectations are dangerous) will be better reviewed.

Edited December 12, 2023 by visualstudios