new reviews system [ UX audit + redesign ] New Rating & Review System

[vhskid]

Heads up → This is a long one

 

Hi there,

With the recent big changes in the seller / gig rating system, a new order reviews system came with it. 

While both seem like they are flawed on many levels, we (sellers) have a full look only at the new reviews system.

As a Fiverr seller and freelance Product Designer, I decided to express my concerns regarding the new reviews form (in the form of a UX audit) and provide improvement propositions by creating new designs.

 

 

→ Introduction

Unfortunately, the logic and execution of the new reviews look like the effect of a 10-minute brainstorming session plus some heavy hours dedicated to creating a PR ideology around it.

I see many wrong things with the new review form and want to address them all. 

I'll use a lot of YOU pronouns, which are directed to Fiverr executives and teams responsible for the discussed changes.
So this post can be treated as an open letter to those who are in charge and have the influence on implementing the platform's functionalities.

Not like I’m holding my breath for a response, but maybe this will reach a few relevant inboxes. 

I'm attaching also → PDF viersion ← with hi-res images.

 

 

 

→ 1 - Emojis

This design decision causes problems on many levels...

 

Behavioral Science Perspective

A person's emotional state is a complex mixture of influences and codependencies. Many things affect our moods and feelings in the short and long term.
Using the basic emoji faces expresses a specific momentary emotion / mood as a reaction to something.
 

Given that I see a few issues:

• Asking people to rate something with an emoji is forcing them to fight against their momentary emotions and overall mood to provide 'fair' feedback - if I don’t feel excited 😐 then I have reservations against using the exited emoji 🤩 for the 'Exceptional' rate
• Emotions are whimsical, tricky and affect us all the time - not only on a conscious but also subconscious level - so emoji feedback is doomed to be skewed from the get-go 
• Emoji rating also gives the impression that it doesn't have significance and that it won’t affect the freelancer much
   

The reports about the insignificance issue problems are starting to trickle in:

Source →

 

 

→ 2 - The Scale

In what reality on a single-step scale, the next grade after 'Average' is 'Very good'?

With the old rating form, the tooltips visible when hovering over the rating stars were also not balanced (4 stars meant 'Good' and 5 'Excellent') but the stars were much more straightforward so it didn't matter that much. 

But here the labels are deceptive. In user experience / interface language, the word for this kind of diversion is 'dark pattern'.

Few points on why hacking the scale is disruptive, not constructive:

• 5 ⭐ meaning 'Very good' / 'Great' is universal for the customer experience online 
• 'Exceptional' on the other hand has bigger significance and feels more like 5+ because it stands for special / extra / unique / rare
• On occasion, we encounter the 'Excellent' label being used for 5 stars, but it has less weight than the 'Exceptional' as well
• Buyers have been exposed to 'classic' 5-star ratings too long and often, to anchor an exception in their minds for your modified system and create Fiverr-specific new habit
• The ‘Exceptional’ scenario should be an additional distinction because people will just not get used to considering the 5th star / maximum rate as something better than “Very good”
• There is just too big of a semantic gap between 'Average' and 'Very good'
• The cultural differences will limit the use of the 'Exceptional' rate as "the best"

We didn't have to wait long for the cultural differences issue to cause problems:

Source →

 

 

→ 3 - Perfect 5 / 5 Being "Not Trustworthy"

In the new review system looks like your goal was to people not rating the highest when it should be "We want to create the review system that reflects the real buyer experience". 

In the responses on the forum, your staff keeps mentioning research to support the reasoning behind the changes. But this looks rather like a research bias scenario. 

I get the company's premise about highlighting exceptional work…

Source →

 

…but that can be done differently.
More on that later.

 

As for "relieving the pressure of aiming for a perfect 5-star rating":

• The pressure will always be there - in a competitive marketplace sellers need every advantage they can get, especially with such an extensive algorithm that Fiverr has
• You increased the pressure because buyers are now rating lower, due to the confusing new rating form
• You increased the pressure because we know now that our gig scores are dependent on how our competition is doing 
  • If my performance as a seller is consistently great but not continually improving, then my score can go down because others' can go up
• You increased the pressure by causing confusion due to the way you rolled out and are handling the new levels / rating (pre)launch
• The approach "4⭐ is the new 5⭐" is misleading users (in the rating form) and buyers in general (in the published reviews) because you changed the rules of the game while the past reviews and new ones are in one pot
  • The sudden drop in ratings looks like a drop in the seller's work quality

 

 

→ 4 - Optional Questions

The way the supplementary questions (dependent on the selected rate) and their answers are displayed is misleading:

• The answers look and sound the same but their selection has different effects depending on the selected rate option
• There are no visual distinctions whether the question asks about negatives or positives
• There is also no indicator that the question is optional

 

 

The way this is implemented will cause scenarios like this:

1. The user reads the first optional question and automatically assumes that the others have a bit different answers, but concern the same instance - positives for example
2. The user is under the impression that is selecting answers in all situations for positive metrics, while they really concerned the negative ones

The same can happen if the user won't read the supplementary questions at all and will select all the answers as positives. 

There are already reports about this:

Source →

 

→ 5 - Real-life Consideration

All the above issues are enhanced when the buyer is in a rush, which is a common scenario. 
Who isn't busy today? Clicking fast through the "unimportant" questionnaire or pop-ups to be over with them while not paying much attention - who among us hasn't done this?

The earlier shared comment about emojis rating insignificance also mentions quickly clicking through aspect: 

Source →

 

 

→ Solutions

A better design approach is actually in the new levels' landing page:

 

 

RATING SCALE

A simple scale with colors looks balanced and intuitive:
 

 

 

QUESTIONS

And now the tricky part with supplementary questions. 

 

Wording

When considered negative, the answers should imply that with their wording:

 

 

 

 

Visual Indicator

The selected state should show visually if the response has a positive or negative connotation:

 

 

 

This is way more intuitive and straightforward. 

 

 

SCORE PREVIEW

Before sending the rating, there should be a noticeable score preview that with clarity shows what will be published:

 

 

 

BONUS ROUND - "making exceptional work stand out"

The "exceptional" aspect of order delivery can be determined in a non-direct way. 

 

Spontaneous Reactions

The Amount Approach

When all 3 main questions are rated to the highest, then the number of selected positive (optional) question responses could additionally affect the score. 
 

 

So let’s say 50% of selected responses for a single question could add 0.1 ⭐ to the score. The threshold size and star fraction value would need a careful assessment. 

 

The Meaning Approach

Another approach could take into account the occurrence of specific response selection. The "Went above and beyond" itself indicates exceptional work.

In the above scenarios, the 'exceptional' rate distinction would be spontaneous (yet measurable) and the lack of it won't interfere with the 5-star rating standard / habits / previous system scores. 

 

Labeling

With either approach, there will be 5+ ⭐ rating possibility and there could be a label / badge indicating this:

 

 

 

If the overall rating of the gig would be 5+⭐, then the 'Exceptional' badge could be included with the gig main stars score (in all the applicable places).

 

 

 

→ Wrapping up

The devil is in the details and I feel like we need exorcisms here.

I hope that the above concerns won't fall on deaf ears. A guy can dream.

Signed
Concerned Seller

 

CC: @Kesha @Lyndsey_Fiverr @ran_success

 

 

____________________________________________________

FAQ section for community
 

1. Why didn't you mention private reviews? 
   - This is a separate can of worms.
    
2. Why didn't you mention the "Value for money" rating question / issue?
   - This topic deserves a separate thread considered not only as a question in the rating form but also as a gig score metric.

All gig metrics should have their own respective threads because these subjects are just too big to discuss them all at once.
____________________________________________________

Edited February 24 by vhskid

[priyank_mod]

@vhskid This is incredibly insightful and fascinating how you integrated comments of other sellers to drive the recommendation! ✅⬆️

PS Love the recommended UI - it is clean, minimal and genuinely transparent.

[visualstudios]

This is an excellent post. Great analysis, and good suggestions. 

[mariashtelle1]

I love how one fiverr seller created a better thought design taking into account  psychological and behavioural aspects with a clean design that a few departments on Fiverr couldn’t do. 
 

I hope Fiverr will notice this thread and take notes for a change. 
I really like thumbs up and down under the scores (and “exceptional rating tag on reviews”) especially for Fiverr, it still keeps it “gamified” as Fiverr likes but also keeps it more clear. 
(however then we are loosing ability to pick what went well in the good and average scores) 

[emmaki]

Excellent analysis and post! 

 

[charlsmcfarlane]

This deserves the “exceptional” tag

[carineb]

8 hours ago, vhskid said:

RATING SCALE

A simple scale with colors looks balanced and intuitive:

You deserve 5 stars and a bonus but I really don't agree with the idea of colors.

Colors have different meanings in different countries, cultures and religions. A color that's positive in your country may mean something very different in another. I'll just mention red, which is commonly used to represent something negative in my country, whereas red represents something extremely positive in Asia!

The questions asked to clients are used to assign star ratings to sellers. So clients have to assign stars, not anything else.

Arbitrarily deciding that a face is worth a certain number of stars is neither honest nor TRANSPARENT!
If Fiverr assigns a number of stars instead of the client, it looks like review manipulation!

I understand that Fiverr wants the ratings to be more heterogeneous to make it easier for clients to choose. It's a very good idea, but the way it's done is really BAD!

Edited February 24 by carineb

[coerdelion]

17 minutes ago, carineb said:

You deserve 5 stars and a bonus but I really don't agree with the idea of colors.

Colors have different meanings in different countries, cultures and religions. A color that's positive in your country may mean something very different in another. I'll just mention red, which is commonly used to represent something negative in my country, whereas red represents something extremely positive in Asia!

The questions asked to clients are used to assign star ratings to sellers. So clients have to assign stars, not anything else.

Arbitrarily deciding that a face is worth a certain number of stars is neither honest nor TRANSPARENT!
If Fiverr assigns a number of stars instead of the client, it looks like review manipulation!

I understand that Fiverr wants the ratings to be more heterogeneous to make it easier for clients to choose. It's a very good idea, but the way it's done is really BAD!

Colours are a great idea ... except for people who are colour blind - there's a lot of that about. 

Much of the confusion could be sorted out by making the emojis bigger, so that people could see more clearly what they're choosing!

 

[vhskid]

3 hours ago, carineb said:

Colors have different meanings in different countries, cultures and religions. A color that's positive in your country may mean something very different in another. I'll just mention red, which is commonly used to represent something negative in my country, whereas red represents something extremely positive in Asia!

I considered the cultural differences aspect when contemplating the design ideas but my conclusions were in favor of the color pills approach:

• This is a progressive scale with text labels and in this context and design form, colors aren't impactful enough to steer the user in the wrong direction 
  • It's not like there would be scenarios such as "Oh no, the red color lured me and I chose the wrong option"
• Traffic light colors are universal enough around the world to make use of red and green as starting points / base here

Cultural differences are important, but going that pathway of tiptoeing around them to the highest extent, when considering color for globally accessible interfaces, we'd end up with gray-only websites and apps.

The culturally sensitive use of colors in design is more crucial in larger contexts with photos, illustrations, key visuals, and bigger storytelling like videos.

 

3 hours ago, carineb said:

The questions asked to clients are used to assign star ratings to sellers. So clients have to assign stars, not anything else.

Fiverr wants to move away from stars, so by not using them I'm playing a bit of devil's advocate.
Will this approach (in general, not with emojis) be more accurate here than stars, when measuring buyers' satisfaction? Only a bigger user testing with a large control group could really tell.
I myself am on the fence right now. 

Having said that, comparing these 2 non-star approaches, my design solution uses a more straightforward progressive scale and translates better to stars than emojis do. 

[vhskid]

3 hours ago, coerdelion said:

Colours are a great idea ... except for people who are colour blind - there's a lot of that about. 

Much of the confusion could be sorted out by making the emojis bigger, so that people could see more clearly what they're choosing!

My approach still uses a progressive scale with text labels, so the deficiencies in color perception are not a factor that would interfere with the user experience and won't skew the results of the survey. 

Emojis on the other hand are very much troublesome due to the reasons I mentioned in my first post.

[carineb]

59 minutes ago, vhskid said:

Fiverr wants to move away from stars,

They don't want to move away from stars. Gig ratings are stars!

Remove the text under the colors.
Mix the colors and ask someone to sort them. You'll be surprised at the results...

Why would pink be worth 2 and not 5 or 1?
Why is yellow worth 4 and not 3?

In fact, for me, yellow is worth 3.
Why is that? Because in all color scales, the place of the colors is always the same, and yellow is always in the middle between orange and light green.
To help users understand things at a glance and improve UX, you need to use scales that are commonly used and that everyone understands without text or explanations.
The star or numbered color scale system is used (and perfectly understood) all over the world.

Do a google search on "color wheel". Number each color on the wheel in order and you'll better understand the subject of color order...

I'd have plenty more to say on this point but I'll stop here so as not to pollute this post with this topic when there are 10 other topics in your excellent post.

[vhskid]

25 minutes ago, carineb said:

They don't want to move away from stars. Gig ratings are stars!

They do want to move away from stars in the reviews form, to have more nuanced feedback. It's like you wrote yourself earlier...

 

5 hours ago, carineb said:

I understand that Fiverr wants the ratings to be more heterogeneous to make it easier for clients to choose.

 

 

25 minutes ago, carineb said:

Remove the text under the colors.
Mix the colors and ask someone to sort them. You'll be surprised at the results...

Sure, because that's how user testing works for web / app interfaces.

 

25 minutes ago, carineb said:

Why is that? Because in all color scales, the place of the colors is always the same, and yellow is always in the middle between orange and light green.
To help users understand things at a glance and improve UX, you need to use scales that are commonly used and that everyone understands without text or explanations.
The star or numbered color scale system is used (and perfectly understood) all over the world.

Do a google search on "color wheel". Number each color on the wheel in order and you'll better understand the subject of color order...

I'm a UX / UI designer with over 15 years of experience, but what do I know?

 

 

Edited February 24 by vhskid

[carineb]

18 minutes ago, vhskid said:

It's like you wrote yourself earlier...

Where and when?

[coerdelion]

1 hour ago, vhskid said:

My approach still uses a progressive scale with text labels, so the deficiencies in color perception are not a factor that would interfere with the user experience and won't skew the results of the survey. 

Emojis on the other hand are very much troublesome due to the reasons I mentioned in my first post.

Similar saturation in all the colours makes the buttons appear a uniform grey to those with colour blindness.  This is not a personal criticism of you - or even your design, really. 

It's important to me, however, that visual impairments of all kinds are taken into account in design.  Colour blindness, like deafness, is not often considered in design of anything.  The reason I think about it is a result of various male relatives who are colour blind - their world is grey, so it's texture and movement that brings it to life. 

Also, accessibility in design for all folks with all kinds of challenges is important these days - ADA (Americans with Disabilities Act) still exists and people are still suing companies under its terms.

 

 

[vhskid]

27 minutes ago, coerdelion said:

Similar saturation in all the colours makes the buttons appear a uniform grey to those with colour blindness.  This is not a personal criticism of you - or even your design, really. 

It's important to me, however, that visual impairments of all kinds are taken into account in design.  Colour blindness, like deafness, is not often considered in design of anything.  The reason I think about it is a result of various male relatives who are colour blind - their world is grey, so it's texture and movement that brings it to life. 

Also, accessibility in design for all folks with all kinds of challenges is important these days - ADA (Americans with Disabilities Act) still exists and people are still suing companies under its terms.

There are different degrees of color blindness, so it's not like every person affected doesn't see any color at all.

While I agree that creating accessible interfaces for those with disabilities is important, designing one to be efficient, useful, attractive, and accessible to everyone is impossible.

That's what contrasting / accessible separate interface versions should be for. This being additional cost makes it rare, and people with disabilities don't have it easy online as well.

 

 

 

Edited February 24 by vhskid

[coerdelion]

8 minutes ago, vhskid said:

While I agree that creating accessible interfaces for those with disabilities is important, designing one to be efficient, useful, attractive, and accessible to everyone is impossible.

That's what contrasting / accessible separate interface versions should be for. This being additional cost makes it rare, and people with disabilities don't have it easy online as well.

 

 

 

I get it that you're proud of your design idea - agreed it's lovely. 

It is, however, fairly straightforward to adjust for accessibility - I had a fairly popular gig to do just that when everyone was panicking about ADA a few years back. 

For visual issues it's often just a question of increasing contrast, reducing flashing and movement, adding texture,  differentiating saturation levels between colours and making interesting use of shapes.

Frankly, the whole emojis thing could easily be solved by going back to stars ...

Adding a screen reader doesn't hurt

 

[vhskid]

6 minutes ago, coerdelion said:

It is, however, fairly straightforward to adjust for accessibility

As a UX / UI designer with over 15 years of experience, I don't agree. 

Colorblind and vision loss interface versions are much less pleasant to look at and use for "regular" folks.

I don't believe in forcing the majority to use something accommodated to the minority. 

I'm more "let's have stairs but also elevators and ramps" kind of guy. Thus the separate / accessible interface version argument.

 

[emmaki]

re: accessibility, Fiverr already has this, which would probably help with colors for people with rubbish eyes: 

Before anyone gets mad about me calling their eyes rubbish I have rubbish myopic eyes. And yes, I am writing this in tiny text. 

Edited February 24 by emmaki

[coerdelion]

7 minutes ago, emmaki said:

re: accessibility, Fiverr already has this, which would probably help with colors for people with rubbish eyes: 

Before anyone gets mad about me calling their eyes rubbish I have rubbish myopic eyes. And yes, I am writing this in tiny text. 

Yes, Userway solved a lot of problems

[emmaki]

6 minutes ago, coerdelion said:

Yes, Userway solved a lot of problems

And created a surprising amount of posts from people who couldn't figure out why their computer was suddenly talking to them! Overall though, I think it's a net positive that Fiverr has it. 

[coerdelion]

14 minutes ago, vhskid said:

As a UX / UI designer with over 15 years of experience, I don't agree. 

Colorblind and vision loss interface versions are much less pleasant to look at and use for "regular" folks.

I don't believe in forcing the majority to use something accommodated to the minority. 

I'm more "let's have stairs but also elevators and ramps" kind of guy. Thus the separate / accessible interface version argument.

 

As you say, that's not a popular choice with those paying the bills.  However, excluding 8% of the population out of ... let's say "determination", isn't a great solution either. 

 

[vhskid]

41 minutes ago, coerdelion said:

As you say, that's not a popular choice with those paying the bills.  However, excluding 8% of the population out of ... let's say "determination", isn't a great solution either. 

 

This isn't up to the designer to decide about.

In my experience, most of the time companies are going for interfaces that are "useful and pretty" for the majority of the population. 
Forcing the accessibility increase aspects on the main design version would cause (for me) the project to get rejected or flooded with revisions.

If I had a choice and budget I would be creating 2 versions of the interfaces. But the reality is different. 

Thankfully at least the are widgets like Userway and browser extensions.

[coerdelion]

I suggest you familiarise yourself with the Web Content Accessibility Guidelines (WCAG) under ADA.  Here's the link: https://adasitecompliance.com/color-blind-website-accessibility/

Litigation under these guidelines is increasing, according to the compliance people and is costing companies a great deal of money every year. 

Ignoring the guidelines during the design process opens your clients up to being sued.  Who do you think they will blame if they get sued due to your design not taking colour blindness into account?

[vhskid]

51 minutes ago, coerdelion said:

I suggest you familiarise yourself with the Web Content Accessibility Guidelines (WCAG) under ADA.  Here's the link: https://adasitecompliance.com/color-blind-website-accessibility/

Litigation under these guidelines is increasing, according to the compliance people and is costing companies a great deal of money every year. 

Ignoring the guidelines during the design process opens your clients up to being sued.  Who do you think they will blame if they get sued due to your design not taking colour blindness into account?

I know what needs to be known here. 

Please stop assuming that I don't and suggesting that I don't raise these topics with my clients at all.

This starts to look like your personal crusade against me.

 

Edited February 24 by vhskid

[kingpirux]

5 stars and recommend for sure!!!

right now i have the feeling that Fiverr will not even read the first lines. Worse i have the feeling that some fiverr AI team will pop up and make a very random AI generated thanks letter to say that your idea will not be taken because they are perfect with the new system lol