3.2 Usability Testing
Criteria To Assess
To assess the usability of the project, I have put together a survey to get information on the user's experience whilst using the project. This survey is being used to assess the 5 key points:
1 - Effective
Do you understand the idea of the project?
Do you understand the different parts of the project?
Do you know how to setup the node software?
Did you know how to use the website?
2 - Efficient
Is the website quick and easy to navigate through?
Does the configuration file contain all needed settings?
3 - Engaging
Did you like using the project?
Did you like the style of the website?
Was the messaging demo engaging?
4 - Error Tolerant
Did any parts of the project crash?
Have you found any errors/bugs in the node software?
If so, what are they?
5 - Easy to Learn
Did you know how to use the messaging demo without any help?
Did you know how to setup a node?
Did you know how to use tokens to get access to the dashboard?
Listed below are the results to the questions and why these questions were important, then what the responses to these questions actually means.
The survey can be found here:
User Feedback
Do you understand the idea of the project?
This aim of this question was to ensure that the website/webportal explained the project quickly and easily enough that the users that looked at it understood the idea of project without any further explanation.
As shown it appears that most users (80%) understood the point of the project in some reasonable amount, however, none of the users seemed to understand the project in great detail and one user responded with a disagreement to say that they did not understand the project. This means that the idea of the project was conveyed to some positive extent using the webportal but was not clear enough and needs some more work.
Do you understand the different parts of the project?
Following on from the previous question, this shows that although 80% of the users understand the idea of the project on the whole, once asked about if they understand all the different parts of the project, that number drops significantly. This shows that the separate sections of the project are not explained in enough detail and this needs to expanded so that all users understand the very basic information about all parts of the project and understand which parts are aimed at them and which aren't.
Do you know how to setup the node software?
This question was important because the node software is the more technical side of the project (as shown in the design frame) and hence it is important that technical users understand how to setup and use the node software where as non-technical users do not need to know how to do this. Having said that, the lower the entry requirements to all parts of the project, the better, hence it is still better to have as many users as possible understand how to setup and use a node.
As shown in the data 40% of the users agreed to knowing how to setup the node software, with 40% being "neutral" which likely means they figured it out but with the help of other users or myself, then the final 20% of users disagreeing - showing that they didn't know how to do it. This roughly equates to the estimation of half of the users being more technical and half being more casual.
Interestingly the results for this question are also identical to that of the previous question, showing the suggestion of a correlation between users that understand that separate sections of the project and those that understand how to setup and use the node software.
Did you know how to use the website
This question shows that all of the users either agreed or strongly agreed that they knew how to use the website. This is very important since this part of the project is the key part that all of the users needed to know how to use and understand since it acts as the entry point to every other part of the project.
Hence this strong of a response shows that the website was very effective and easy to use and as such was successful in it's goal to be an effective entry point to the project.
Is the website quick and easy to navigate through?
Since the website is important for all uses of the project, it is important that the website is quick and easy to navigate through so that users can get to any part of the project or information that they want to get to as efficiently as possible.
As shown in the results this was a success, since 40% of the users strongly agreed and 60% of the users agreed meaning that all of the users agreed to some extent that the website is quick and easy to navigate through.
Does the configuration file contain all needed settings?
This is a very specific question aimed at more technical users, since it is not only about the node software, which is used by technical users but also about custom configurations related to the software. This is shown clearly in the results as one of the users agreed the configuration file contained all settings they needed, yet the other 4 responded with a "neutral" meaning that they likely either didn't know how to use the configuration file or didn't need to use it.
This is both a success and a failure, since the configuration file should only be needed by very technical users and such most users shouldn't need to use the file and if they do it should contain all the settings they need which is shown by the responses. However, ideally most of the users would know of the configuration file and know how to use it if they need to because as with the rest of the project, the lower the requirements to use any part of the project, the better.
Did you like using the project?
It is important that users like using the project, as with a project based upon a decentralised network such as this one, the project gets faster and more secure the more users it has. This makes it more enticing to new users so that they then join the project and it continues to scale well.
The responses to this question somewhat positively reflect this, with 60% of the users agreeing or strongly agreeing to liking using the project - which is good - however 40% of the users responded "neutral" to this question meaning that although they didn't dislike using the project they didn't like using it either. Ideally all of the users would at least agree to liking using the project with a decent proportion strongly agreeing to liking using the project.
Did you like the style of the website?
The style of the website is important since a well styled website makes users want to look around and learn about the project, whereas a poorly styled website can scare users away and make them assume that the whole project is old, outdated and badly made.
Therefore it is very positive that all of the users agreed to liking the style of the website, with 40% of them strongly agreeing. Ideally all of the users would strongly agree to liking the style of the website because then they would be more likely to look around and visit the website just for the style and learn about the project in the progress, however all of the users agreeing to liking the style of the website is a positive since it means that users are unlikely to leave or associate the project negatively purely due to the website's look and appearance.
Was the messaging demo engaging?
The messaging demo was the demo created during cycle 12 to show the networking capabilities of the project and was the most up to date example of what the node software could do whilst having low requirements for the users. It is important that this demo was engaging since it represents the kind of program that can be made upon the top of the decentralised networking technology and if the programs built on top of this project aren't engaging then the project itself is not engaging, which is obviously not good.
Bearing this in mind the responses to this question are somewhat positive, since 60% of the users agreed to the messaging demo being engaging, meaning that the majority of the users did indeed find it engaging. However, 40% of the users either "disagreed" or responded "neutral" which means that they either barely found it engaging or didn't find it engaging; ideally all of the users would find the demo engaging so these results are not amazing but the majority still did which is good.
It is also worth remembering here that this demo did not represent all the functionality that the project originally intended to include and hence has a long way to progress in the future.
Did any parts of the project crash?
The project should not crash and hence the results for this question should be as many "None" responses, however, as shown in the responses this was not the case - 60% of the users responded with "some" parts of the project crashed and 40% of users responded with "Lots". This is not of the ideal scenario, which would be 100% responses of "None"
There is a good reason for this however, which will be talked about in the evaluation but essential comes down to a bad choice of programming language for the node software paired with not enough time management given to fully polish the project to prevent crashes such as the ones the users experienced.
Have you found any errors/bugs in the node software?
This leads on from the previous question asking about if any parts of the project crashed and shows that the majority of the errors and crashes in the project were due to the node software and since the node software was the only part of the project that I encountered crashes, I expect that the one user to respond "No" to finding any errors/bugs likely had the software crash but did not understand the error/bug.
However, since the user that gave the "No" response did not expand their answer in the following question "If so, what are they?", these results shall be taken at face value. These results then show that most users found a bug in the software, with one user stating they found multiple bugs, hence it can be stated that the majority of users found some form of error or bug in the node software and therefore the program was not as error tolerant as it would be ideally.
If so, what are they?
These responses show that the same part of the project crashed for all the users which encountered bugs, this is good because it means that their is a clear section of the program that will need to be tested and fixed before the development of the project continues hence there is a clear direction that can be headed in to fix these issues. The downside to this however, is that this means that this part of the project crashed regularly (80% of the users reported bugs related to this demo) and since this demo relies upon lots of different modules and sections within the project on the whole it is likely to be underlying issues rather than issues just due to how the demo was made.
I have a strong suspicion that these underlying issues due in fact exist and in part due to my language choice for the node software, although better code on my half could likely decrease the quantity and severity of these issues, and I will dissect this in the evaluation of this project.
Did you know how to use the messaging demo without any help?
It's important that users know how to use the messaging demo without any help, since this represents the approximate level of complexity of most of the programs which could be built on top of the project and hence if users can understand how to use this demo without any help this a good sign that they could use other programs built on top of the project later on (such as the wallet, transaction tool and marketplace that were originally planned).
Therefore 60% of the users knowing how to use the demo without help is good, however since 40% of the users still did not figure it out, it would likely be good to create a tutorial/guide on how to use the demo in the future.
Did you know how to setup a node?
This is an important part of the project and although the node software is aimed at more technical developers than standard users, the more users that know how to setup a node, the better.
The responses to this question show that all users struggled with this process and either needed additional help or time to figure it out on their own. 40% of the users eventually figured it out without additional help and 60% of the users need some extent of additional help. This shows that this was not a success and some form of solution needs to be introduced to make it easier for users to setup nodes in the future.
Did you know how to use tokens to get access to the dashboard?
This question was regarding the process of clicking the "get token" button on the dashboard login page, then copy and pasting the token code generated from the node's software's logs into the page and submitting it as proof that the user has control over that node.
I assumed this was fairly simply to do and hence users were not given any support on the page itself explaining how to complete this process, but as shown in the results for this question that was not the case. 40% of users figured out how to complete this on their own and 60% required some form of help from myself. This means that the login page needs some kind of instructions or guide to explain to users how to use the token access and luckily this isn't a very complex addition to the project so could be done fairly easily.
Usability Requirements in Success Criteria
Some of the success criteria requirements fall into the section of usability. These have been covered either in the initial survey or will be covered in the additional survey I created to go over these points in more detail.
11
The configuration handler should be accessible and useable to technical users without any help, all users should be aware of it's existence and able to use it with some external assistance.
13
The node dashboard should be easily accessible and usable by all forms of users.
21
Non-technical users must be able to identify what the point/idea of the project is just from the homepage of the website.
22
Users should be able to get to the majority of what they would want to get to within 3 clicks.
23
The web-portal should be available and working on a variety of device sizes including mobile and desktop.
Criterion 11
In order to assess this criterion I included the following three questions in the survey:
This question assess what proportion of the users are actually aware of the configuration handler/file and as shown in the results above, this was met somewhat successfully with 60% of users being aware of it and knowing some of it's uses and the other 40% being aware of it existing but not knowing what it's for. This is evidence that all users are aware of it's existence.
This question shows that the majority of users (80%) likely aren't aware of all the settings within the configuration file as they responded with "neutral" to it containing all needed settings. The reason this assumption can be made is due to users responding "neutral" means that they did not agree or disagree
Knowing what the configuration file is for is another way of ensuring that users are aware of it's existence and could use it if they were given external help. This question shows that: the majority, 60%, of users would be capable of doing that; with 20% of users being aware of it's existence but they would probably require additional external assistance to explain to them what they can use the file for before in more detail; then the final 20% having no idea what the file can be used for and hence would need a lot of external assistance to do anything using it.
Combining the results from all three questions and it can be concluded that all users had at least heard of the configuration file existing, with the majority knowing what it could be used for and some of the more technical users within that pool knowing how to use it without additional help but the majority of users requiring assistance. This meets the criterion's requirements, although if user testing were to continue past this point it would ideally contain a separate standard and technical user group so as to understand if the different stakeholders were able to do what it was aimed for them to be able to do.
Criterion 13
Interestingly, more users agreed to knowing how to navigate to the node dashboard than the amount that knowing what the dashboard was. This suggests that most users were a bit confused by the questions and thus they needed more clarification, after talking to users after completing the survey it was revealed that the majority of users didn't realise that the messaging demo was also the dashboard and after learning this said they understood what it was much better.
Hence this criterion was given somewhat inaccurate results due to the wording of the questions, although it could also be argued that the messaging demo being the dashboard could've been made more clear on the site itself, which is a valid criticism.
Therefore based upon the fact that most (60%) of users knew how to access the dashboard and after speaking to them after this survey they understood what it was very quickly I deem this criterion partially met, since the results are along the right lines as what is required but they need to be furthered on first.
Criterion 21
Since each user that completed this survey used the node messaging demo and were given access to the website without any instructions except to "look around and get a feel for the site" before completing the survey, and the node software does not give any explicit explanations for the reason and ideology of the project, it stands that the users will have gotten most of their understandings on the project itself from the website and it's homepage.
Therefore as 80% of the users that completed this survey agreed to understanding the idea of the project without any additional help except the website it can be stated that this shows evidence of this criterion being met.
Although with 20% of the users disagreeing and no users strongly agreeing, this likely means that the homepage needs some more work in order to make it even more clear what the idea of the project is, however this can likely be pushed into a slightly more long term fix since it does still pass the criterion.
Criterion 22
This shows that All parts of the website can be travelled to within 2 clicks from the home page, hence including the singular click that it takes to get back to the home page, each major page can be accessed from any other major page within 3 clicks. Therefore this shows that this success criterion has be met.
Criterion 23
This device demo shows the website being tested on a large variety of devices, including a standard computer, tablets and phones. Throughout this testing it is shown that the criterion is partially met, as although all parts of the website are functional on all devices and there are no crashes and hence it is available, some parts of the site have visual glitches or bugs that cause the graphics to be partially off the screen or the wrong size. This leads to some text not being readable in specific scenarios which means that in those scenarios I would not classify the website as completely working.
Hence, this success criterion has been partially met and would require some additional tuning on those very specific device sizes to be met, which could be done fairly easily and hence would be good to do in the future.
Last updated