Testing improvements to the rating & review system

[Kesha]

We’re continuing our efforts to build a more modern, reliable rating & review system that will encourage freelance business growth on the Fiverr marketplace, empower customer decision-making, and enhance trust between freelancers and customers.

Our next step in this effort is to test some changes to the Public Review form, based on extensive research and analysis of our current system. These changes are aimed at cultivating more honest feedback for greater transparency. Some (but not all) of you will come across these changes, so here’s what to keep an eye out for:

1. Adjusted questions in the new form relevant to the quality of delivery and communication. The new form will have 3 parts:

• Feedback on how the delivery’s quality met expectations (for example: “How was the quality of the delivery in relation to your expectations?”) 
• Feedback on communication (for example: “How was it working with [freelancer]?”).
• Leaving a written review

2. A streamlined customer review process with an upgraded design and experience.

We are confident that these changes will both help you assess your performance with more clarity and bring more customers to Fiverr seeking your skills and success.

Please note: The data collected from the adjusted questions will be included in the rating scores shown on your profile and Gigs. Your rating breakdown will begin to reflect scores adjusted by the new form submissions in the coming weeks.

We will keep you posted on any further changes and conclusions from the test here in the forum.

[markauther]

hello!!

sellers would be able to see the answers of these questions or these answers will be kept private?

[priyank_mod]

Thank you for sharing the update @Kesha

Few queries:

1. How will answers to the first two questions reflect mathematically? (in terms of net ratings)

2. How will this feedback merge with parameters in the existing review matrix?? (ie communication level, recommendation to friend and service as described)

3. In the 3rd form, will the buyer enter an overall rating manually i.e. number of stars or will it be the cumulative total of what they chose in the previous forms??

Edited December 6, 2023 by priyank_mod

[uk1000]

20 minutes ago, priyank_mod said:

In the 3rd form, will the buyer enter an overall rating manually i.e. number of stars or will it be the cumulative total of what they chose in the previous forms??

My guess is it will work sort of like it does now - that the review will be the average of the multiple ratings the buyer gives - but now they have 2 but before they had 3 (but now they also have tags that can be checked, like "value for money"). So they've roughly got rid of the "recommend to a friend" bit - the other 2 aren't that different from before. But in the example that user has given 2 exceptional ratings but the rating it shows is 4.5 stars 😕  - which would be expected to be 5 stars if it was an average of those 2.

It would be interesting to know what will be done with the tags the buyer can check too. Is that only for Fiverr's internal use or will it show some of those on the gig page? Maybe those tags (or some) only show to the buyer if they rate 5 stars for that rating section? And maybe some of those results could show on the gig page, eg. if a high percentage of reviewers of a gig selected a particular tag.

Edited December 6, 2023 by uk1000

[emmaki]

Pretty simple chaps. Private feedback may go bye-bye because buyers hate filling out 20 zillion forms just to complete an order, sellers hate it because it's not transparent and it's not fair and completely defeats the spirit of feedback (of the constructive kind, anyway). 

Now they're just going to roll all this into a form that ticks off metric boxes to feed in - one would guess - the algo and Fiverr Neo. Beware, sellers fluent in languages they are not fluent in. And the non-innovative selling non-innovative services? There's some strange checkboxes in there. 

Wildcard assumption: 5-star review system to be replaced by emojis. 😡 to 🥰 with 🥱in the middle. 

Anyway Kesha, looking forward to seeing the changes, but unless private feedback goes and hidden metrics are unhidden, at least for sellers' eyes only, nobody at Fiverr has any business believing that this is an increase in transparency. It's not. It makes far more sense that this is being used to help Fiverr's AI systems AI. If only I wasn't in the waitlist for Fiverr Neo so I could see what questions it asked...

It's also not terribly clear exactly how these new changes manifest in the review overview thingy for sellers in the beta program? It looks just like the current one? IIRC when Fiverr switched from the thumbs system to 5-stars it just ignored the old system in favor of the new, since sellers were already steaming about the complexity that 5-star ratings over 3 categories would bring to the poor, shrivelled up brains of the buyers (the poor dears back then got confused so easily). Oh, if only we'd know then what we know now!

This post was created with the power of wild speculation and reading too much into things. It's my superpower. 

[Kesha]

2 hours ago, markauther said:

hello!!

sellers would be able to see the answers of these questions or these answers will be kept private?

Hi @markauther!

Yes, these reviews are still part of the public review form and will therefore be viewable to sellers. As always, sellers will be able to view all their submitted reviews and feedback on the order summary page, as well as their profile page, and gig page.

[visualstudios]

I find that the labels (and emojis) on the rating system are not balanced. While "exceptional" is better than the previous "perfect" (as nothing is "perfect"), it's not a linear progression.

To be consistent, it should go:

Very poor -> poor -> average -> good -> very good.

Why is one step down from average "poor", but one step above average "very good"? That system seems designed to make lower grades more frequent. Also it's strange that the negatives are "poor" and "very poor", while the positives are "good" and "exceptional". The language could be more neutral using the suggestion I provided above. "Very good" should be enough for the top end, if a buyer is truly amazed, they can always leave a tip for exceptional service (going "above and beyond").

Edited December 6, 2023 by visualstudios

[terrygrantvo]

@Kesha Is there still going to be a private review system that we have no insight into and will it continue to carry more weight than the public reviews?

[venonusa]

Just now, terrygrantvo said:

@Kesha Is there still going to be a private review system that we have no insight into and will it continue to carry more weight than the public reviews?

This is an important part to know, since there is little clarity/clear on it.

[markauther]

1 hour ago, venonusa said:

This is an important part to know, since there is little clarity/clear on it.

same question in my mind as well, as I'm fed up of this private review thingy

[Kesha]

2 hours ago, terrygrantvo said:

@Kesha Is there still going to be a private review system that we have no insight into and will it continue to carry more weight than the public reviews?

 Hi @terrygrantvo! This test will not affect the Private Review form. I appreciate your feedback and will pass it along to the product team. 

[Kesha]

5 hours ago, priyank_mod said:

Thank you for sharing the update @Kesha

Few queries:

1. How will answers to the first two questions reflect mathematically? (in terms of net ratings)

2. How will this feedback merge with parameters in the existing review matrix?? (ie communication level, recommendation to friend and service as described)

3. In the 3rd form, will the buyer enter an overall rating manually i.e. number of stars or will it be the cumulative total of what they chose in the previous forms??

Hi @priyank_mod! Thanks for your questions. I would be happy to answer them for you.

1. How will answers to the first two questions reflect mathematically? (in terms of net ratings)Both the previously submitted ratings (from the existing form) and the ratings from the new form will be given equal weight in the total rating score.

2. How will this feedback merge with parameters in the existing review matrix?? (ie communication level, recommendation to friend and service as described The rating breakdown will not change at this stage. We will update you in the upcoming weeks on changes based on the results of this test.

3. In the 3rd form, will the buyer enter an overall rating manually i.e. number of stars or will it be the cumulative total of what they chose in the previous forms? The total star rating will be automatically calculated as an average of the answers to the previous questions in the form. Please note: the image provided is just an example, and may not be an exact representation of the actual form. 

I hope this helps clarify things for you!

[Kesha]

1 hour ago, markauther said:

same question in my mind as well, as I'm fed up of this private review thingy

Hi! This test will not affect the Private Review form but I will be sure to pass the feedback on to the product team!

[markauther]

19 minutes ago, Kesha said:

Hi! This test will not affect the Private Review form but I will be sure to pass the feedback on to the product team!

private review system really sucks!! trust me it is not helping the sellers in anyway, we get to know we are having negative feedback from Buyers when the damage is already been done

Edited December 6, 2023 by markauther

[Kesha]

10 minutes ago, markauther said:

private review system really sucks!! trust me it is not helping the sellers in anyway, we get to know we are having negative feedback from Buyers when the damage is already been done

Thanks for this feedback. Your concerns are heard and being passed along to our product team. 

[Kesha]

3 hours ago, visualstudios said:

I find that the labels (and emojis) on the rating system are not balanced. While "exceptional" is better than the previous "perfect" (as nothing is "perfect"), it's not a linear progression.

To be consistent, it should go:

Very poor -> poor -> average -> good -> very good.

Why is one step down from average "poor", but one step above average "very good"? That system seems designed to make lower grades more frequent. Also it's strange that the negatives are "poor" and "very poor", while the positives are "good" and "exceptional". The language could be more neutral using the suggestion I provided above. "Very good" should be enough for the top end, if a buyer is truly amazed, they can always leave a tip for exceptional service (going "above and beyond").

@visualstudios, Thank you for voicing your thoughts and suggestions. I will be sure to pass them along to the appropriate team.

[terrygrantvo]

9 hours ago, Kesha said:

@Kesha Is this graphic accurate, in the sense that answering 'exceptional' to both of the multiple choice questions only results in a 4.5 star rating?

Also:

9 hours ago, Kesha said:

We are confident that these changes will both help you assess your performance with more clarity and bring more customers to Fiverr seeking your skills and success.

Can you elaborate a bit on how this new rating system is going to help with either of these ideas?

 

Thanks!

 

[katakatica]

I think I'll have to echo some of the others - this feels potentially concerning to me. On one hand, I do think that buyers seem overwhelmed by the review system at the moment (it's not a big deal but if you order in bulk it adds up...) on the other, I'm not sure emojis are the solution. 

I know that certain places (like coffee shops / etc. ) use this system, but I'm worried that it's just different on a relatively smaller scale. 

Very good would (in theory) be a 4 star review now. Somehow when I see the four stars, as a buyer I recognize that as imperfect (as in - I could give them more stars if they did better) but very good is already... Very good? 

If we end up punished for it without the buyers realizing that it's, in fact, a bad rating (granted that things stay the same!) I feel a bit worried. 

(Basically, I know it's similar now but when choosing between happy emojis it feels like it'll be easier to rate lower? I'm guessing that's partially the point though...

That being said, I'm not huge on it but I'm sure I'll get used to it! Simplification is a good idea in itself!

[donnovan86]

3 hours ago, Kesha said:

3. In the 3rd form, will the buyer enter an overall rating manually i.e. number of stars or will it be the cumulative total of what they chose in the previous forms? The total star rating will be automatically calculated as an average of the answers to the previous questions in the form. Please note: the image provided is just an example, and may not be an exact representation of the actual form. 

 

A lot of buyers just leave random stars, and adding these emoticons and even more steps to leave a review will just make buyers rate their sellers randomly. I had a 1 star review last year due to the person rating me at 1 AM, even if the text was positive he just randomly left stars. So I can assure you that adding more steps to a review is not going to be helpful, at least it will have a negative impact on sellers. 

Also I saw a 40 word limit to the reviews? Most people don't even bother to leave 2-3 words as a review,  let alone 40.. Maybe it's characters, but even then.. 

20 minutes ago, katakatica said:

I'm not sure emojis are the solution. 

It's a professional platform, so I don't see the use or need for emojis. It makes the entire platform feel less professional in the eyes of a buyer. After all, in a professional setting you don't expect trivial stuff like emojis.

My 2 cents: Fiverr wants to further differentiate sellers and not have everyone with 4.9 or 5 star ratings. Let's face it, few buyers will rate work as exceptional. 

I am also highly concerned how those older reviews will translate to the new system. A few weeks ago I had a 3 star average on my main gig and profile, it was due to an error and I reported to customer support. That being said.. I have over 18k reviews and not that many are 3 stars or less so.. .. I really hope that conversion won't mess things up, not sure if that 3 star average was a test or not, but it wasn't calculating things properly. 

25 minutes ago, katakatica said:

If we end up punished for it without the buyers realizing that it's, in fact, a bad rating (granted that things stay the same!) I feel a bit worried. 

 

That's my concern as well. I am very worried that people would just see it as a very good delivery, yet Fiverr's rating system won't. So that alone will bring certain issues for sellers. As I said earlier, maybe it's a way for Fiverr to differentiate sellers even further, but I don't see emojis helping with that or making the rating system easier. I had lots of people placing those stars randomly so.. I really hope it will all be fine.

[terrygrantvo]

16 minutes ago, donnovan86 said:

 I am very worried that people would just see it as a very good delivery, yet Fiverr's rating system won't. So that alone will bring certain issues for sellers. As I said earlier, maybe it's a way for Fiverr to differentiate sellers even further, but I don't see emojis helping with that or making the rating system easier. I had lots of people placing those stars randomly so.. I really hope it will all be fine.

I hate to say it, but I'm beginning to think that's the point. This is going to result in a whole lot of perfectly satisfied buyers unknowingly leaving 3.5 star reviews, which in turn means eventually no one will have a 5 star rating overall, and I think it's intentional. 

There are probably too many sellers in virtually every vertical on the site for how much demand there actually is, which means they can't possibly make all of us happy. All they can do is try and make the majority of us just happy enough to not quit, and the only way an algorithm can do that is if they have a clear number by which to separate one from the next. Currently, there are a ton of us with 'perfect' 5 star ratings and the system has no idea what to do with that.

 

There's a dead simple way to fix this, if simplifying things is actually the goal here - do away with private reviews all together. Give buyers the option to make the one review they leave (at the time of closing out the order) either public or private, and then just get your value and weight from that and that alone.

 

What they're doing here is combining private and public reviews into a new 'public' review, but then also keeping the private reviews, which are literally the same thing as the new half of the new public review (the emoji nonsense). It's really hard to see this as anything other than an attempt to make the personal scores more granular and thereby make it easier for the system to decide who gets to show up in searches that week, rather than just vetting sellers like they should have been doing all along.

 

It could just be the tin foil hat talking, and I certainly don't want to be right about this, but...

[vickieito]

Hi @Kesha, 

This new system for review is repetitive (especially since it has the same questions as the private review) and doesn't make it easier for the buyer to complete a review. Especially since we still have to fill out the private reviews and then deal with the spammy reminders to tip our sellers (even after we've already left a tip).

Things that would make the review system a little bit easier:

1. Let buyers see the total orders a seller has (not just how many reviewed orders they have).

Some gigs already have this, and I don't see why every seller can't have this number right below their seller picture. Some sellers only have 40-60% rated orders, so it looks like they have fewer orders than they really do. New sellers also wouldn't be so devastated when their buyers choose not to leave reviews on their orders.

2. Make the written review optional, especially for repeat buyers or frequent buyers.

After doing about 6 orders with a seller, my reviews start to look the same. It would be nice to indicate what type of buyer I am on Fiverr's platform ("Occasional buyer," or "Frequent Buyer") and if I'm a "Regular Customer" of that seller. That, with my top reasons of why I chose to work with that seller, would be more helpful to prospective buyers than a 5-star rating scale with a generically written review. The same goes for leaving reviews for my buyers. Sometimes I don't have anything new to say, especially to the repeat buyer who just gives me an order and accepts a delivery. I'd just like to rate my experience on a 5-star scale and be done.

3. Allow buyers the option of hiding their IDs when leaving a review. Keep this all in one rating system.

The review could still be shown as a "Verified Order" but buyers could leave a public review without disclosing their IDs to other buyers. This would encourage more buyers to leave a review, especially those who don't want others in their organization/network to know that they are using Fiverr. It would also encourage buyers to be more honest about their experiences (without having a separate rating system like the BSR score). If we want transparency, make it visible and keep it in one place.

4. Stick with the star rating instead of "labeling" buyers' experiences with words.

Not everyone will have the same interpretation of the words you use for seller ratings. Especially if English isn't your first language. It's so much easier for buyers to quantify their experiences qualitatively with stars than with words. 

Many buyers won't know the impact or difference between saying a seller is "Very Good" or "Exceptional." They won't know that sellers with a "Very Good" average rating may be demoted because their average is 4.0 (not 4.2). Other buyers may think that "Exceptional" is a rare thing, and may never give out such ratings. Especially if a seller is exceptional (and then that becomes the norm for that repeat buyer).

 

Edited December 7, 2023 by vickieito

[leannelrivers]

13 hours ago, visualstudios said:

To be consistent, it should go:

Very poor -> poor -> average -> good -> very good.

This makes more sense. When is anything truly exceptional? There will be buyers who rightly opt for very good, which is a great review, but can impact seller visibility on the platform. 

[emmaki]

I'm just really excited for the storm of controversy as unwitting beta testers suddenly get rubbish reviews. I love how you can only see this announcment if you're logged into the forum. Sure, it's only a beta test, not that important. But it is the explanation for rave 2.2 star reviews. I'm sure CS will love the tickets too. And buyers! Buyers who... er... have less language fluency will click the funny pictures but not bother with the boring word buttons. It's so cute. It's like being back at kindergarten. With the game where you put the shape through the hole that is in the same shape and it makes a funny wheeee noise as it goes down (at least where I grew up)!

Not a lot of transparency about how the system spits out an actual review.

Not a lot of transparency about how this was announced.

Not a lot of transparency about how this will affect the beta tested sellers (mass demotion vs the non-beta group?)

Plenty of Fiverr's favorite things when it comes to the review system: the same old words about integrity, honesty, transparency, but the reality is much more confusion for buyers and sellers doing more of their favorite hobby: complaining at Fiverr. 

Also, what is value for money doing as a review metric on a platform which wants to go upmarket? I get it, it's a way to fish out the sellers who are yadda yadda. But how does this help in situations where the buyer is just your everyday Scrooge in hard economic times and wishes that Fiverr was still Fiverr so, no, this $100 gig that used to go for $5 is not good value for money. 

I rate this system 2.6. I couldn't tell you how I arrived at the numbers. I clicked the happy faces! Why isn't the review happy?

Did anyone involved in the making of this test consider that the star review system is internationally recognized by all but the most... challenged... as you know, 1 star bad, 5 stars amazing? Are we really going to try and call this a transparent an improvement? 

Oh well, I'm on holiday for the next month, so idc. I'll just make sure to get some extra bags of buttery popcorn to enjoy the show. 

No point taking it seriously guys. The private review is being completely ignored, so who cares? I almost lost Pro one time because a few people didn't like me being honest about their horrific ideas. At least, that's what my then-Success Manager said. Thing is, buyers almost always returned saying "actually you were right". And I'm like "well, what a surprise." 

Don't worry Fiverr, all these extra reviews do is make me more anemic and more cautious and more everything that makes my services my services disappear. All so I can get a pretty review and not demoted to Hades because you can't demonstrate value to money or a lot of other things if your tongue isn't wiggling furiously around a back passage, can you? 

Off to buy the popcorn now. And ponder the question: If you thought ChatGPT was already overused by Fiverr sellers to converse for them, you ain't seen nothing yet...

 

[uk1000]

8 hours ago, donnovan86 said:

So I can assure you that adding more steps to a review is not going to be helpful, at least it will have a negative impact on sellers. 

It's less steps in the new one that they have to fill in to leave a review (the tags/check boxes are optional),

so before they had 3 sub-ratings+the review text, now they have 2 sub-ratings+the review text that have to be completed to leave a review.

Though was it all on one smaller screen before and has it now been split?

Though if there was a 3rd sub-rating that was more relevant to every order that could increase the accuracy. Now, after this is release nobody will get reviews like 3.3, 4.7, they'll get 3.5, 4.5 etc.

Edited December 7, 2023 by uk1000

[donnovan86]

Ok, so I think I am a part of this system, at least this is the review I received a few hours ago. Here's how it looks. I am pretty sure this is a part of the beta..right? 

It's strange that the rating breakdown scores will be updated in the coming weeks... but I get it if it's a part of this beta..

Edited December 7, 2023 by donnovan86