Heuristic Evaluation
Discussion Forums of Coursera.com
Role: UX Researcher
Timeline: 3 days
Coursera.com is an online learning platform that enables the learners around the world to participate in degree or non-degree programs and earn certificates after completing each course. The Discussion Forums in Coursera.com is where the learners communicate questions, comments, or participate in designated class activities by generating posts. One class can have multiple forums (e.g., one separate forum for each week). The front page of the forum looks like this:
The goal of this project is to evaluate the forum functionality based on the 10 usability heuristics established by the Nielsen Norman Group and rate each violation on a severity scale of 1-4, each given a weight when scoring:
1 - cosmetic problem (x 0.5)
2 - minor usability problem (x 1)
3 - major usability problem; important to fix (x 1.5)
4 - usability catastrophe; imperative to fix (x 2)
Severity ratings are assigned after all the violations are recorded so that the scale can be calibrated accordingly. Each heuristic started off with a full score of 10. Scores are subtracted when a violation is detected under each rule. Violations that appeared on multiple pages are combined into one comment. If one issue violates multiple heuristics, they are listed together. Limitations are discussed at the end.
KEY FINDINGS
Below is a table showing the total subtraction of scores under each heuristic after weighing in the severity, and a final overview of the heuristic evaluation results. As can be seen, the Discussion Forums of Coursera.com could really benefit from some improvement on user control and freedom, as well as flexibility and efficiency of use. The two are usually seen to go hand-in-hand: a system that cannot be controlled and manipulated in a flexible way inevitably dampens the efficiency of use.
More specifically to the forum, it is extremely hard to keep track of threads and locate information under a certain topic; there are also actions that, surprisingly, cannot be carried out when constructing a post. See below for more detailed findings.
The rest of this report breaks down this result by findings, including description about the caveats, what heuristic(s) each finding violates, where they were found, the severity ratings, as well as specific recommendations for improvement.
Finding 1: Redundant “upload” step when attaching images
Description:
When attempting to embed an image in a new discussion thread, the user has to "upload" the images, choose from the uploaded images, and then "continue" to make it show on the draft of the post. Typically, the users will rarely choose an image (say, from their local drive) but decide not to “upload” it, especially when there is another round of adding/deleting right before posting. Therefore, the step of manual uploading adds a redundant action before execution, causing extra effort from the users.
Severity:
3/4
System Location:
"New Thread" --> Embed Images Option
Heuristic(s) violated:
#7 Flexibility and efficiency of use
Recommendation:
Images should be directly uploaded after being selected from the local drive.
Finding 2: Reposition/resizing corrupts images
Description:
This happens after the user successfully attaches an image. Once the user tries to drag the picture attempting to change its location or to push on boarders attempting to resize it, the picture would suddenly disappear. If the user tries to recover the lost image, the system will not respond to the intuitive shortcut “Ctrl+Z” or “Cmd+Z”. Instead, the place where it was supposed to be a picture now becomes a slim rectangle which one would have to click around to spot it first, delete it, and insert the picture all over again. This can be quite frustrating when happening in combination with Finding 1 because there are too many repetitive steps.
Severity:
3/4
System Location:
"New Thread" --> Embed Images Option
Heuristic(s) violated:
#1 Visibility of system status
#5 Error prevention and detection
#9 Help users recognize, diagnose, and recover from errors
Recommendation:
Add a resizing or repositioning function to inserted images OR prevent the user from attempting to make changes to it. The latter can be done by getting rid of the blue boarder when clicking on the image in the first place (shown below). As is now, the border seems to be inviting the users to interact with it.
Finding 3: No “preview” step before posting
Description:
There is no way to "preview" the entire post before publishing it. Previewing is especially necessary in this case because the text editor is not very intuitive and flexible (see Finding 1 and 2). During testing, it turned out that the layout, including how many texts in each line, and the size of the images differed in the final post compared to how it looked like in the editor. This matters less for posts that contain only texts but makes a big difference aesthetically when the post contains picture.
Severity:
3/4
System Location:
"New Thread"
Heuristic(s) violated:
#3 User control and freedom
#5 Error prevention and detection
Recommendation:
Introduce a “preview” step before finally “publishing” the post. Align the appearance of the message in the editor and the final post.
Finding 4: Not able to “view my posts” or search by users' name
Description:
There is no place to see all the threads posted by the user, neither is there way to search by a user’s name (one can only search key words in contents), making it extremely hard to track the threads that the user started unless there is a reply.
Severity:
3/4
System Location:
"Discussion Forums" screen
Heuristic(s) violated:
#3 User control and freedom
Recommendation:
A third tab named "My Threads" can be created next to "Forums" and "All Threads" (as shown below) to collect all the threads and replies the user has generated. Allowing one to search by names would be helpful as well.
Finding 5: Unclear naming scheme of category tabs
Description:
The Discussion Forums page introduces two types of forums: general and weekly discussion forums. In the current system they are called "discussion forums" and "week forums". The naming of the general forums shares exactly the same name with the overarching name of the webpage, which is very confusing. If not compared to "week forums", the users would never know what the first type of forums is for.
Severity:
2/4
System Location:
"Discussion Forums" screen
Heuristic(s) violated:
#2 Match between system and the real world
Recommendation:
DISCUSSION FORUMS should be changed to GENERAL FORUMS.
Finding 6: Poor presentation of moderator details when hovered over
Description:
Moderators’ initials (or photo?) do not show when hovered over. Only a blank blue badge appeared which does not carry any helpful information at all. This violates the heuristic rule of consistency and standards because it is typically expected that more information is presented when the user hover over something.
Severity:
1/4
System Location:
“Discussion Forums” screen--> "Moderators" panel on the right
Heuristic(s) violated:
#4 Consistency and standards
Recommendation:
This badge can be taken away, leaving only the moderators’ position (Teaching Staff or Instructor), or the initials should be added to keep it consistent with how it appears on the original "Moderators" panel.
Finding 7: Inconvenient access to weekly forums
Description:
The weekly forums are usually accessed more frequently. However, the users must scroll down to see them. This is even worse as if the course has a longer list of general discussion threads.
Severity:
2/4
System Location:
"Discussion Forums" screen
Heuristic(s) violated:
#2 Match between system and the real world
#7 Flexibility and efficiency of use
Recommendation:
To reduce unnecessary actions, the week forum chunk can be brought to the right side of general forum, just so they are listed side by side (see illustration below).
Finding 8: Unclear naming scheme and function of sorting criteria
Description:
Users can choose to sort all the posts by “top”. Does “top” mean the posts with most “views”, most “replies”, or earliest post time? The results showing does not match any of these. In the screen shot below, the thread that got more views AND replies were ranked behind another thread that is lower on both. Later posts were buried under earlier posts. This leads to a sense of frustration and confusion in the users and limits the users’ ability to sort the posts efficiently.
​
Severity:
2/4
System Location:
"Discussion Forums" screen
Heuristic(s) violated:
#7 Flexibility and efficiency of use
Recommendation:
Instead of using “Top”, naming the arranging criteria as “Views” or “Replies” will be helpful. The algorithm also has to be changed to correctly reflect the names of the criteira.
Finding 9: No mechanism identifying invalid links
Description:
When the users type in an invalid link, the system records it without warning the users about the problem. Because the web address is easy to get mistyped, the users would benefit from a better error detection mechanism.
​
Severity:
2/4
System Location:
"New Thread"--> Link icon
Heuristic(s) violated:
#5 Error prevention and detection
#9 Help users recognize, diagnose and recover from errors
Recommendation:
Raise an error message when invalid links are provided: "This is not a valid link. Would you like to proceed?"
Finding 10: Unable to filter searching results
Description:
The "search" function on top of each weekly forum enables the user to search for keywords among all the threads. However, the results of the search cannot be filtered or organized according to criteria such as number of views and replies. For example, in week 6’s forum, I got 877 results if searching for the keyword "sandbox” but will not be further able to see which threads are more recent or got more replies. The users would get very lost and unable to locate the information they need.
​
Severity:
2/4
System Location:
"Discussion Forums" screen--> forum for a specific week
Heuristic(s) violated:
#3 User control and freedom
#7 Flexibility and efficiency of use
Recommendation:
Allow the users to sort searching results according to a few criteria such as "mostly views" or "most replies". However, avoid using unclear criteira like "top" mentioned in Finding 8.
Finding 11: No documentation or troubleshooting channel
Description:
There is no such thing as an FAQ specific to the Discussion Forums function. The website has its overall guideline documented but little information is related to the Forums. Most of the function here are quite self-explanatory and works fine in most cases. However, when the users do get stuck, there is no place to quickly solve it than contacting customer service. In the forum, there are many posts shouting out about system errors or unfair grading that don't seem to have gotten any attention.
Severity:
2/4
System Location:
"Discussion Forums" screen
Heuristic(s) violated:
#10 Help and documentation
Recommendation:
It may be helpful to include some very brief introduction of the overall layout, and capability of the forum (e.g., a 3-minute intro video) and a direct button linking the users to customer service or the course moderator's contacts.
Limitations
Number of evaluators
Ideally, heuristic evaluation is the most effective and accurate when conducted together by 3-5 evaluators. Each evaluator comes up with a unique set of scores which are then averaged in order to mitigate individual bias. This report is not the most accurate conclusion possible because there is only one evaluator.
​
Comprehensiveness
These caveats were detected through a series of actions such as starting a post, embedding images and links, and trying to search for certain threads. These may not be comprehensive enough to cover all the actions that the users could possibly do. There could be other violations that has not been discovered.
​
Scoring criteria
The violations were rated on a severity scale of 1-4 which determined its weight in the number of points subtracted from a full score of 10. This full score was determined after all violations are recorded and weighted, aiming to help the audience to better compare between heuristics. However, since this scale is arbitrary, it may not be consistent with the scores in other evaluations, and a single score makes little sense without comparing it to others.