Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Developing a Basic Numerical Metric for Web Usability

By tes in Internet
Tue Apr 16, 2002 at 01:09:57 PM EST
Tags: Internet (all tags)
Internet

In terms of website usability metrics studies, there seems to be little agreement among the supposed experts in the field about a truly good way to numerically evaluate the usability of the website. This article proposes a simple numerical scheme for evaluating the usability of a website.


Background

It is undeniably true that the best way to develop good information about the usability of a website is simply through the thorough testing of users. Are they able to actively use a site as well as find it subjectively appealing? The goal of a web developer should be to ensure that both of these tasks are fulfilled.

The subjective aspects of this are hard to test, of course. A particular color scheme or layout may be visually interesting and exciting to you, but garish and annoying to me. It is largely impossible to judge subjective attributes such as this in a simple numeric fashion.

Instead, we can merely judge the aspects of a page that are strictly quantitative. It is important to remember this distinction as we progress through the proposed model.

Notable Numbers

We need to define a set of numbers that defines the experience of moving throughout a website. I propose the following set of useful numbers that can be obtained through a short count, along with a discussion on how to obtain these numbers.

Total Page Count
How many pages on the site are unique? For a static site this is simple: simply count the number of HTML files in a given directory. However, for a dynamic site this becomes much more difficult. The solution I've discovered is to use a 'bot to navigate your site and then proceed to analyze the weblogs to see how many different URLs were accessed, removing any needed redundancies.

Minimum Span
This refers to the maximum number of clicks it takes to get from one page on the site to another page, regardless of keyboard input. Again, this can be obtained through the use of a 'bot tool by generating a network between pages, then sorting based on number of links needed to travel from one page to another.

Average # Site Internal Links Per Page
How many links does an average page on a site have on it? For a site like kuro5hin, this number is quite high, but many other sites have a much lower number.

Average Page Distance
This number is the average number of clicks it takes to get from one page on the site to another page on the site. A solid 'bot should be able to evaluate this as well.

Average Page Weight
How many pages on the site link to a specific page? On a small site with a typical lefthand navigation bar, this number is often elevated, but as a site grows larger and offers more integrated tools, this number lowers.

Average Page Size
How many kilobytes of information is contained in a given page? The question of including images is a ponderous one; it makes sense to only include them if they are uniquely downloaded for that specific page.

Getting The Data

Obtaining this information could be done through the use of a 'bot, which would navigate a site to generate the numbers for you. Many of the pieces for this theoretical program are already available in tools such as ht://dig and SpiderBot.

Evaluating These Values

Now that we have these numbers, what good are they? We need to evaluate the roles that each of these numbers has on the usability of a website.

Total Page Count
In itself, page count doesn't really deliver a great deal of useful information. A site with 10,000 pages can be just as usable as a site with 10 pages if the pages are sensibly designed. However, a site with 10,000 pages is very likely to have much higher values for minimum span and number of internal links per page.

Minimum Span
Sites that use a common navigational scheme will have lower minimum spanning numbers but will often raise the number of internal links per page. A well-organized site would manage to minimize the span without including an abundance of links on each page.

Average # Site Internal Links Per Page
A complex and large site will need a decent number of internal links simply to allow many avenues to navigate through the information. However, a relatively small site should focus on delivering content and should find ways to minimize this number without increasing the minimum span. This number likely doesn't have the impact that the minimum span has.

Average Page Distance
Somewhere in between lies this value, which is most useful when it is low. A good site will allow you to find most of the information you need as close to any given page as possible. Basically, this value encourages you to make sure that your "leaf" pages are well connected.

Average Page Weight
This number simply indicates whether or not you have piles of redundant links. If each page is overloaded with a large number of links that all lead to the same places, they simply overburden the user. This is another metric that should be low and balanced with the minimum span.

Average Page Size
Having this number low makes natural sense.

In summary, we want a low minimum span, low number of internal links, low average page distance, low average page weight, and low average page size per page on the site. Also, the minimum span and average page distance rank in importance above the others. These evaluations are obviously open to evaluation themselves.

Generating A Useful Metric

My general thought is to put these numbers together into a single useful metric, as follows.

Usability = Average Page Size In kb * Minimum Span * Average Page Distance * sqrt(Average # Site Internal Links) * sqrt(Average Page Weight) ) / Total Page Count

The goal here was to weight the values based on their importance, and then divide their product by the size of the site (simply because larger sites will have to have somewhat elevated numbers).

A site that nears the ideal for this metric is, interestingly enough, kuro5hin. The total page count is quite high, with the other metrics being quite low compared to, for example, Saturn's website, which tanks on the page size metric as well as the links per individual page metric. Saturn's site, though, is itself better than something like this, which pretty much flushes all usability down the toilet.

Conclusion

This article proposes a unified usability metric that can be determined without the use of expensive user testing. Although flawed by its very nature, the metric attempts to unify a large number of sensible web design ideals. Comments are encouraged and eagerly awaited.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
This proposed "web usability metric"...
o generally makes sense, with a few flaws 12%
o sounds pretty good, especially in terms of cost for the evaluation 8%
o is seriously flawed and needs some rethinking 45%
o smells like my grandpa's old sock... where's my Malt O Meal? 33%

Votes: 62
Results | Other Polls

Related Links
o Kuro5hin
o website
o usability
o metrics
o studies
o the thorough testing of users
o kuro5hin
o ht://dig
o SpiderBot
o Saturn
o this
o Also by tes


Display: Sort:
Developing a Basic Numerical Metric for Web Usability | 63 comments (56 topical, 7 editorial, 0 hidden)
Question (5.00 / 2) (#4)
by spacejack on Mon Apr 15, 2002 at 03:14:56 PM EST

for the K5 designers, since the article claims K5 is near-ideal: Did you guys actually do any calculations like these when you designed the site, or did you just use your common sense?

No analysis (5.00 / 1) (#49)
by rusty on Tue Apr 16, 2002 at 10:30:28 AM EST

No, we've never done any kind of usability studies. What we have done is refined, bit by bit, from one design to the next, and tried to incorporate suggestions from all of you. Which could, I guess, be considered "usability testing" in a kind of voluntary and distributed way.

Incidentally, I did do a little actual usability testing on the ad creation process. I had my wife and my dad try it out, and watched what confused them. That did help, in fact, and I'd recommend everyone do something like that for new designs. Even one or two people can provide a lot of useful feedback on a design.

____
Not the real rusty
[ Parent ]

About usability/features (none / 0) (#56)
by linca on Tue Apr 16, 2002 at 07:08:06 PM EST

Where can we put suggestions? email only?

(my only qualm is with the absence of comment number on minimal form; it could be useful in lookin for recent comments...

[ Parent ]
Quantitative/qualitative (5.00 / 6) (#5)
by greenrd on Mon Apr 15, 2002 at 03:15:29 PM EST

-1. Quantitative measures should not be used to reduce a qualitative problem to a single, or a small number of numerical values.

It is undeniably true that the best way to develop good information about the usability of a website is simply through the thorough testing of users

I agree. What is the purpose of your proposed metric again? Oh I see, you mention it in the conclusion:

This article proposes a unified usability metric that can be determined without the use of expensive user testing.

[emphasis added]

You simply cannot avoid user testing in a usability study - that's insane! If you don't test, your "real users" will become your testers by default. I suggest that this approach is in fact, far more risky, and is likely to lead to great opportunity costs, and/or hasty revisions after the project is officially "completed".

Although flawed by its very nature

Quite!

the metric attempts to unify a large number of sensible web design ideals.

Painting by numbers is not the way to develop a successful website.


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes

It gets done all the time (none / 0) (#22)
by Mysidia on Mon Apr 15, 2002 at 08:08:58 PM EST

-1. Quantitative measures should not be used to reduce a qualitative problem to a single, or a small number of numerical values.

Take for example IQ scores.. a quantitative description of a very qualitative thing. People have "intelligence" in different areas, yet some people presume to give a single number to it. People even use metrics to analyze source code.

All part of man's quest to measure the universe... who knows, maybe some day we'll have a function f(a1,a2,a3,a4,...ai) that determines everything in the universe and can predict the future.

You can certainly take metrics of different things, once you do that, you can put them on the scale, average them, and there you go... a fairly superficial value called an "average" that doesn't mean much but looks cool if it's high and bad if it's low.

The grades they give people in school are the same way, a 100% in a course doesn't mean the students know 100% of the material, it just means they were able to answer 100% of the questions posed them in a test, on average.

But a test answering every question they were supposed to know would be impractical, because it would possibly take as long to administer as it did to teach it to them in class.

So metrics are inherently sloppy and leave out material (they sample). Sometimes they even try to justify it by saying the sample's sufficiently large with respect to what's being sampled and it's distributed evenly. On the other hand, with something like an IQ test or a web site measurement, that becomes difficult

Metrics are metrics, they will exist as rough descriptions eventually, and people will find them useful, if not just plain curious.



-Mysidia the insane @k5
[ Parent ]
More bogosity (I love that word) (none / 0) (#47)
by greenrd on Tue Apr 16, 2002 at 09:20:23 AM EST

Take for example IQ scores.. a quantitative description of a very qualitative thing. People have "intelligence" in different areas, yet some people presume to give a single number to it. People even use metrics to analyze source code.

Both pretty much bogus. At least IQ scores can distinguish very crudely between those with severe learning difficulties or mental problems, and highly intelligent people, I'll admit that much - but source code metrics are so open to abuse...

Anyway you don't need an IQ test to work out whether someone has a learning disability - although you may need a specialist.

All part of man's quest to measure the universe... who knows, maybe some day we'll have a function f(a1,a2,a3,a4,...ai) that determines everything in the universe and can predict the future.

I highly doubt it.

So metrics are inherently sloppy and leave out material (they sample). Sometimes they even try to justify it by saying the sample's sufficiently large with respect to what's being sampled and it's distributed evenly.

I was overgeneralising, I admit. I think a Java exam or a mathematics exam are two good examples of things that are quite qualitative (because there are right and wrong answers) therefore it is valid to output a grade of a single number. But web usability is so qualitative, it doesn't make sense to reduce it to a calculation of one single number.


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

They are the metric (none / 0) (#54)
by Mysidia on Tue Apr 16, 2002 at 04:29:41 PM EST

Math/Java exams are part of a complicated quantitative metric designed in the form of testing the particular points (questions) addressed

Scoring 100% on a math test though doesn't necessarily mean you're good at math.

On the other hand, some tests are administered differently... an Essay test may well actually be a qualitative metric converted arbitrarily to a number by the 'scorer'

In that case, a qualitative thing is being measured by a number, an essay isn't "wrong or right" (assuming it doesn't contain lies in it), but it could have things seen as 'bad' and therefore worthy of reducing the score on account of

The situation with a website is similar.. It is possible to "sanely" use quantification to analyze an essay (for example denoting 'weights' to certain attributes), so I think it's possible the same could be done for web sites

The problem of scoring an essay and scoring a site is after all a bit similar to that of scoring a web page, but if your only concern is to usability, then you can dismiss a lot of the qualitative things that have meaning to its usefulness or interest

[Basically, the content of the site doesn't matter, it's how it and its navigational elements are presented that effects its usability]

If you can assign quantities to enough components and find ways to neglect the predominant qualitative elements (by exclusion or approximation), then you can quantify the whole:)



-Mysidia the insane @k5
[ Parent ]
To Use (5.00 / 1) (#58)
by Grammar Avenger on Wed Apr 17, 2002 at 06:53:50 AM EST

It is possible to "sanely" use quantification to analyze an essay...
This should be 'It is possible to use quantification "sanely" to analyze an essay...' to avoid splitting the infinitive. If you split too many infinitives, you'll end up with infinitive bits all over your floor, like my mom's house, because some of the pieces will inevitably become lost. And woe to you if you split an infinitive which happens to be an atomic unit, because in splitting it will trigger a massively destructive nuclear chain reaction.

Whoosh!

Grammar Avenger, away!

[ Parent ]

re: To Use (none / 0) (#62)
by Mysidia on Wed Apr 17, 2002 at 02:54:37 PM EST

It is a common misconception that split infinitives are a bad thing. I think that to properly split infinitives is a good thing for clarity and style.



-Mysidia the insane @k5
[ Parent ]
For Pete's Sakes (4.00 / 1) (#26)
by Hobbes2100 on Mon Apr 15, 2002 at 09:15:47 PM EST

I'm not sure who Pete is, nor what his sakes are, but ...

-1. Quantitative measures should not be used to reduce a qualitative problem to a single, or a small number of numerical values.

I agree that in trying to make the subjective, objective you have an extrememe loss of grain-size. That is, you throw out tons of information. Maybe you even throw out too much.

However, this doesn't mean that simplification is always a bad thing. In particular, you can't say a priori simplification is bad. You do need to exam the correlation between your simplification and "real life" (in this case, how actual samples of users feel about sites that have a "high metric"). If there is a high correlation, you may be on to something useful. Note, it is only a correlation ... not a causal effect. Having a high score doesn't guarantee you'll have good usability. In particular, if you aim at a high score (for the sake of a high score), you'll probably go downhill (or over the top, as the case may be).

Finally, if you can do these studies once (or twice or thrice) and then apply the results, you can save an awful lot of money. If you can save money and get approximate results, you have a winning game plan in my book.

Regards,
Mark

P.S. And, BTW, b/c you have a philosophical problem with this reduction doesn't mean it's not a good story. I'm not saying you didn't have other problems with it, but give another view a chance! [just picking on you green] 'course I didn't read it all; I jumped to the comments after skimming.
Sed quis custodiet ipsos custodes? --Iuvenalis
But who will guard the guardians themselves? -- Juvenal
[ Parent ]

It's a nice enough article, and I'm voting it up (2.00 / 2) (#6)
by BlackTriangle on Mon Apr 15, 2002 at 03:21:17 PM EST

But I think web 'applications' are a passing fad, and it's a mistake to develop metrics for them. I believe that HTTP itself will eventually be nothing more than a carrier for more interesting software like Flash and Java, and not the slightly-embarassing document format we call HTML.

Moo.


this is likely not possible (5.00 / 1) (#7)
by KnightStalker on Mon Apr 15, 2002 at 03:22:21 PM EST

It's probably not possible to assign a single number to a "web site" and use that to compare it to other "web sites" on the basis of "usability". This is mainly because the terms "web site" and "usability" are not very well definable, and because ease of use is a function of the users more than of the page.

Inasmuch as it is even an entity that exists, usability is an emergent property of a group of related pages, and can't really be reduced to a single number. While all of your criteria do affect ease of use in some way, it isn't necessarily meaningful to combine numbers representing them into a single "usability" metric.

does it make me a geek (5.00 / 1) (#8)
by KnightStalker on Mon Apr 15, 2002 at 03:25:13 PM EST

to notice that the time I posted that is palindromic? :-)

[ Parent ]
I must be a geek (4.75 / 4) (#12)
by wolf trap on Mon Apr 15, 2002 at 04:00:40 PM EST

because I feel compelled to point out that technically, "palindromic" applies only to words, phrases, or sentences :-) Dig this palindromic poem by T.S. Eliot.

"To I, Lester"
A cotton eve,
O, trap! Eden!
O, television sad as night.
I wonder if senile sirs tell war days,
Words selfless, drowsy...
A drawl lets rise lines fired
[No!]
With gin:
Sad as "no" is,
I've let one depart.
O, eve--
Not to care...
--T. S. Eliot

--

Il faudrait les inventer, those hooks, on purpose for me alone, for, if you only knew, Alyosha, what a blackguard I am.
[ Parent ]

It isn't in my time zone. (nt) (3.50 / 2) (#46)
by pschap on Tue Apr 16, 2002 at 09:09:43 AM EST



--
"In 1991, we had almost nothing. We'd only begun building cocks. After just 10 years, we have a very robust, active cock."

[ Parent ]
Layout!Layout!Layout! (5.00 / 3) (#9)
by Trickster on Mon Apr 15, 2002 at 03:34:34 PM EST

You completely (well, almost) ignore page layout. It's quite possible to build a site that is ok by your metrics but have all the links stacked on top of one another. Or how about having all the text on single line spanning several horizontal screens? How about bright red text on lime background? And even with all that taken into account you can still end-up with a useless website by ignoring information design completely.

I think, that's the whole problem with measuring usability - you have to take into account page design/layout, info design and rendering/interaction time. That plus I don't think there was a lot of research in this area.

Quantitative analysis is horrible for usability (5.00 / 6) (#10)
by keenan on Mon Apr 15, 2002 at 03:36:47 PM EST

The heuristics that you propose using are simply bogus. I just finished helping work on the overhaul in the design of a major website and these kinds of numbers are totally misleading.

The way you propose to get total page count is ludicrous -- do you include all localized files, different query strings, etc.? Also, too many links (as I've seen in some MLP posts) are horribly distracting, yet if you set "Average # Site Internal Links Per Page" high in your equation, your website is classed as very usable (as long as they are distinct links, which is totally possible on a website). What if Amazon decided to put 8K of links on it's front page without using any images... I think that'd do pretty well with your metric.

In your equation, you solely deal with the average usability of the site as a whole; you fail to account for the stressed importance of the navigation from the home page.

Why would increasing the minimum span increase your usability? Yeah, it takes me 10 clicks to navigate from one page on the site to another, yet this is thought to be highly usable? A hard to find site map page would screw up the minimum span metric as well.

Basically, what we did was have real users test out our site. Even in a non-corporate environment, you can test out your site with friends & family and get some idea of the usability. You could even make multiple prototypes and get an even better idea of what navigation models people prefer.

Keenan

What's the difference between... (2.00 / 1) (#11)
by Trickster on Mon Apr 15, 2002 at 03:39:38 PM EST

...minimum span and page distance?

And the defenition of min span is really confusing...

Marketspeak (3.75 / 4) (#13)
by CaptainSuperBoy on Mon Apr 15, 2002 at 04:01:48 PM EST

Arbitrary formulas that come up with a metric for web usability are, themselves, useless. All this could do is maintain a flawed 'status quo' for usable web sites, resulting in a bunch of sites designed with absolutely 0 new ideas, that are optimized to have low page size and decent navigation. Some independent web designers can go around claiming to develop the most 'usable' sites as long as their goal is to target the spec rather than use their heads.

--
jimmysquid.com - I take pictures.
Is that so bad? (5.00 / 2) (#16)
by davidduncanscott on Mon Apr 15, 2002 at 04:44:12 PM EST

All this could do is maintain a flawed 'status quo' for usable web sites, resulting in a bunch of sites designed with absolutely 0 new ideas, that are optimized to have low page size and decent navigation.
Jeepers, imagine that! A bunch of web sites that could be easily navigated and load fast -- what a fate!

I long for the day when user interfaces change, not only from object to object, but from day to day. I want my toaster to have the artistic freedom to alter its controls each night, so that every breakfast is a new and challenging experience. I want my car to morph each time I drive it, so that one day it might have a tiller, and the next voice controls, and the next not move at all, or plunge randomly from place to place in a sort of frenzied dance.

Cry havoc, and let slip the dogs of design!

[ Parent ]

Some other things not covered... (4.33 / 3) (#15)
by hatshepsut on Mon Apr 15, 2002 at 04:31:09 PM EST

- overuse of unecessary graphics (slows page loading)

- difficult-to-read fonts/colour schemes/embedded flash

- broken links

- optimized for one browser (and anywhere from difficult to useless with any other browser)

There is so much more to usability than the amount of information and the amount of work you have to do to look at the information.





Overuse of graphics IS covered... (5.00 / 1) (#29)
by tes on Mon Apr 15, 2002 at 09:29:54 PM EST

The part about average page size discusses it.



[ Parent ]
This is apparently an active research field. (5.00 / 2) (#17)
by pschap on Mon Apr 15, 2002 at 04:55:18 PM EST

I saw a talk from a perspective faculty member at my university on something fairly similar. It was a project from Berkeley called WebTango in which they were trying to both rate a web site's design and provide suggestions on how to improve that design through the use of an automated tool. They put it this way:

The goal of the WebTango project is to explore the question: How can the web site design process be improved by the application of automated evaluation techniques?
Essentially what she had done was take the results of the reviews from the Webby awards and took pages that were rated as good (mediocre, bad, etc.) and analyzed what common attributes they had to try and come up with some sort of a cluster of features that made a good web site. Then she would take an unknown web site and essentially measure the deviation from that cluster to determine some sort of a numerical rating for the new site. It would then try and help improve the design by giving suggestions as to what attributes it deviated most from the good sites.

The key problem with this method, is that it doesn't (and can't possibly really) take into account the quality of the content, or for that mater even the meaning of the content. When she showed an example of a page being improved by her system the audience immediately noticed problems with the improved version, the most telling problem was that the system complained that certain pieces of text were too small and that the page went on for too long. What it was referring too was the footer of the page, which had the main logo of the site, followed by the small print that basically had the copyright information and the name of the web design firm that made the page. To make this better, the program made the small print almost the same size as the rest of the print on the page, and then made the companies logo go side by side with the fine print. In the process it shrank the company logo because of table formatting issue. So the end result was that you have this really useless text made huge and overshadowing the important logo, simply because the quantitative analysis of the code of the web site did not take into account the purpose of the text that was being formatted.

My suspicion is that any system that claims to measure the usability or the quality of a web page quantitatively is destined to fail unless it can really understand what the relevant parts of the text that it is looking at are.

--
"In 1991, we had almost nothing. We'd only begun building cocks. After just 10 years, we have a very robust, active cock."

Missing Big Pieces (5.00 / 4) (#18)
by blurp on Mon Apr 15, 2002 at 05:13:30 PM EST

Sorry, but you have completely ignored many aspects of site design. It seems to me you have only addressed a sub-section of site navigation. There is so much more involved in designed a Usable site.

It is undeniably true that the best way to develop good information about the usability of a website is simply through the thorough testing of users.

You're right on here. Web design is art, lending itself much more to subjective than quantitative testing.

blurp

This gives me the image of... (5.00 / 1) (#19)
by pschap on Mon Apr 15, 2002 at 05:35:40 PM EST

...some stuffy old art historian trying to come up with a way of numerically catagorizing the greatness of each particular painting in his gallary.

--
"In 1991, we had almost nothing. We'd only begun building cocks. After just 10 years, we have a very robust, active cock."

[ Parent ]
But he isn't talking about design... (5.00 / 1) (#20)
by MikeyLikesIt on Mon Apr 15, 2002 at 07:48:28 PM EST

He's talking about usability.

[ Parent ]
Maybe I don't understand the subtlety of it... (none / 0) (#23)
by pschap on Mon Apr 15, 2002 at 09:04:49 PM EST

Isn't a well designed web site one in which you can easily use it for its intended purpose?

--
"In 1991, we had almost nothing. We'd only begun building cocks. After just 10 years, we have a very robust, active cock."

[ Parent ]
No. (none / 0) (#50)
by loucura on Tue Apr 16, 2002 at 12:04:10 PM EST

Something is well designed if you can use it not only for it's designed purpose, but for any other purpose you can think up.

[ Parent ]
okay fine... (none / 0) (#51)
by pschap on Tue Apr 16, 2002 at 12:08:11 PM EST

...but still, I'm missing how this differs substantially from usability.

--
"In 1991, we had almost nothing. We'd only begun building cocks. After just 10 years, we have a very robust, active cock."

[ Parent ]
The metric (4.83 / 6) (#21)
by Mysidia on Mon Apr 15, 2002 at 07:57:08 PM EST

Is good for measuring one aspect of 'ease of navigation'

And perhaps a good rough estimate of the general 'ease of navigation'; under the assumption that there aren't other problems that hurt or help the navigability; for example, placement and form of navigation links (unusually small or large font sizes, etc).

I think a great number of other things need to be taken into consideration to determine general usability.

First of all, I think you need to measure usability with respect to something, ie a particular abstract user Joe.

  • Is Joe using a text browser, Netscape, or IE5 with all the latest features? How usable is the site in all these environments? :)
  • Does it comply with standards, the ones browsers your user Joe uses will go by, so that the user experiences it in a sane manner?
  • How wide and tall is the content on the page? Are the images, content, and navigation links guaranteed to format nicely for your average user's display resolution and window size?
  • Where are your navigation links on the page? Need a measure of link prominence.
  • How well does the content contrast with the background. Does the background disturb, clash with, or make the foreground text (or even worse, the links) harder to see with your average user's display? With any user's display? To color-blind users?
  • Is the <blink> tag in use? -10000pts
  • Are navigation links prominent and visibly separated from the content? Ie: Is the content well-bounded from the site features or 'standard template' used within the site?
  • Are the links described sufficiently that the user knows what their navigation clicking will get them to? Usually an image isn't enough, are text alternates (or at least ALT= strings) provided for all image links?
  • Are there annoying images (such as animated gifs) in the way of the content? Ie, Does the page contain any rapidly-flashing things of any sort anywhere near the content? -100000000000pts
  • Does the page use any special browser features or plugins that your average user may not have? Ie: flash, pdf, doc, ps files
  • Are the color sequences and layout reasonable? Anything that might hurt your users' eyes hurts usability
  • Are there any banner ads? Banner ads hurt usability by slowing download.
  • Total size of all files loaded by a page (total load-up time for your average user)
  • Any javascript (or java) use? Use of them often hurts usability
  • Any use of ActiveX controls? Always hurts usability.
  • Any use of flash? Ditto, unless the page has a non-flash equivalent:)


-Mysidia the insane @k5
I don't like yours. (4.50 / 4) (#24)
by Mr. Piccolo on Mon Apr 15, 2002 at 09:05:05 PM EST

Here's mine.

Site usability = 0 if site contains Flash
1 otherwise

The BBC would like to apologise for the following comment.


So.... (none / 0) (#30)
by GreenHell on Mon Apr 15, 2002 at 10:28:51 PM EST

What does the <blink> tag do to the usability scale? :)
(Actually, that's my main gripe, nothing about how easy it is to actually view the information. Sure you may be able to get to the page quickly and easily, but that doesn't always mean much)

I agree with you on the Flash though.

-GreenHell
This .sig was my last best hope to seem eloquent. It failed.
[ Parent ]
Damn! (none / 0) (#35)
by Mr. Piccolo on Tue Apr 16, 2002 at 01:22:26 AM EST

My browser doesn't do "blink", I don't think. But for those that render it, I guess I'll have to change it to a C statement:

usability = (flash?0:1) - (blink?1:0)


The BBC would like to apologise for the following comment.


[ Parent ]
fundamental problems with flash (none / 0) (#32)
by AnomymousCoward on Tue Apr 16, 2002 at 12:19:10 AM EST

there is nothing fundamentally wrong with flash.

I feel, though, that your dislike for it comes from bad experiences. This is due to two possible causes:
  1. Bad web designer / developer
  2. Bad browser support
The first is something we've all dealt with. Small flash applications masquerading as websites are annoying. The sounds and flashes are typically more of a nuisance than a feature. This is not a problem with flash, but with the developer. The second issue above is browser support. Most modern browsers support decent flash animations and menus, and this should not be an issue. If so, your OS and browser are situated firmly in the past, and you should upgrade. Quickly.

Vobbo.com: video blogs made easy: point click smile
[ Parent ]
There is no such thing as a decent Flash animation (none / 0) (#36)
by Mr. Piccolo on Tue Apr 16, 2002 at 01:27:09 AM EST

I'm sorry, but Flash as a navigation tool just annoys the hell out of me. Give me normal hypertext any day of the week. Plus it bloats webpages.

P.S. the best browser was Mosaic until they stopped updating it. If it did anything over HTML1.0 it would be perfect.

The BBC would like to apologise for the following comment.


[ Parent ]
Problems with flash (4.00 / 1) (#37)
by carbon on Tue Apr 16, 2002 at 02:23:35 AM EST

1) It's proprietary. Macromedia completely controls the Flash format, and the only official authoring and playing tools. Meaning, the people who actually use Flash have no control over it, only the people who make the most money off of it. This is a Bad Thing.

2) It's closed source and, even worse, closed spec. You all know the reasons why this is a Very Bad Thing For An Internet Technology, so I won't get into it further.

3) It's bloated and doesn't fit any particular use well. Flash is nice because it's vector, making it cheaper then a bitmap movie format like MPEG, but the big problem is, I have yet to see a site that used vector animation in a way that makes much sense. Sites that truly need interactive client side stuff (and almost none do, but there are some, like the Secret World of Numbers site) can use Java, which has it's own problems, but is at least partially open.Sites that use animation simply to look pretty, are stupid, because the web sites are about content, and forcing a OS and browser finicky technology and a 300kb download down the throats of users so that they can click it so it goes away is just plain ridiculous.

The primary good use of Flash is simple animations, but why are these browser embedded anyways? It makes more sense to have the user download these and run them in a seperate viewer, so that they aren't affixed with the web page exclusivlely. There's a reason that people can download or stream music from MP3.com and play it in a seperate player of their choice.

Now, what could happen is the format could evolve cleanly away, as the people who are actually creating content for it could determine, but oh wait, this cannot happen, because it's closed. HTML and CSS evolved as the people who used it and developed for it needed, but this can never happen with Flash. As long as Macromedia makes a profit keeping Flash exactly as it is (and they do, since why expend developer effort and money when people are using it everywhere to annoy their visitors as it is) it won't change.


Wasn't Dr. Claus the bad guy on Inspector Gadget? - dirvish
[ Parent ]
cool link (none / 0) (#39)
by martingale on Tue Apr 16, 2002 at 03:59:55 AM EST

If you're mathematically inclined, you might like to check out Knuth, Vol. 2, Sec. 4.2.4 for a related result.



[ Parent ]
I totally agree that Flash shouldn't be used (5.00 / 1) (#48)
by DeHans on Tue Apr 16, 2002 at 09:27:55 AM EST

however, the rest of your statement is (somewhat) false.
Macromedia completely controls the Flash format, and the only official authoring and playing tools.
and
It's closed source and, even worse, closed spec.
These two are related, since the unavailability of 3rd party tools is related to the closed spec. Only, it isn't closed anymore and hasn't been since early 2000.

You can download both the spec and the source code to the flash player from macromedia (though why you'd want to is beyond me :-). At the developer part of their site, you can also download sample apps written by third parties.

If you have to use vector graphics, try SVG.

[ Parent ]
Numerical Metric (3.33 / 3) (#25)
by Khedak on Mon Apr 15, 2002 at 09:13:24 PM EST

Poll a random sample of n website users. Ask them whether the site was easy to use (yes/no). Your usability is x/n, where x is the number of 'yes' responses. I suggest n >= 1000 for best results. Multiply by a constant and round off if you prefer integer metrics.

Note the sampling technique is the trick here. Just putting the poll online is probably not sufficiently random.

User satisfaction (4.50 / 2) (#31)
by jesterzog on Mon Apr 15, 2002 at 11:24:54 PM EST

Poll a random sample of n website users. Ask them whether the site was easy to use (yes/no). Your usability is x/n, where x is the number of 'yes' responses.

This isn't much better for measuring the usability than the single digit number, imho. You'd get user satisfaction levels, but it's not usability. After a certain obvious point, people usually don't realise when something's designed badly. They just compensate by getting into bad habits and subconscious workarounds.

Just because you're used to typing on a qwerty keyboard doesn't mean it's the most usable system out there. It probably means that you're either used to it or that you can't imagine anything better. It's the same with most things.

There could be lots of changes made that are ergonomically better, improve productivity, improve satisfaction (better than the user already had), and so on. Usually they won't be noticed unless people who are experienced actually sit down and watch other people using something.


jesterzog Fight the light


[ Parent ]
Interesting (none / 0) (#52)
by Khedak on Tue Apr 16, 2002 at 12:30:10 PM EST

Well, if you put it that way then there are countless millions of variables. That's like trying to quantify all microorganisms with a numeric metric. I suggested this metric because it seems ludicrous to think that something as broad and complex as "usability" can be reduced to a numeric metric through inspection.

You make excellent points about the many factors that can effect an objective measure of usability. But at the same time, you have to realize that attributes like "ergonomically better", "improved productivity", and "improved satisfaction" are qualitative measures. If you're looking for something numerical, these don't help.

[ Parent ]
The user and the context (4.75 / 4) (#27)
by jesterzog on Mon Apr 15, 2002 at 09:17:10 PM EST

Everyone has their own pet ideas about what usability is and what's important, which is perfectly okay and I won't argue with that.

My own thing that I haven't yet seen directly mentioned by anyone else is that if usability is a measurement of a user or users to use something, then surely one of the most important parts of any metric is the user.

It's not a big thing to understand that all users are different, and I think what this and many other usability studies try to do is generalise who a user is and what they want. For ordinary web pages and systems this does make some sense, because the web is a diverse platform that's used by lots of different people.

If you know that your particular target users aren't concerned about a particular aspect though (eg. It's only for an intranet and everyone has high bandwidth connections), then obviously everything's going to be weighted differently. Maybe the user even wants it to be harder for beginners, to discourage them from doing things without thinking. Maybe the main goal is to make it ultra-fast for expert users only, without anticipating new people using it.

If you're designing a number system, I think you should make aspects of the user at least half of it, and it should only be presented next to a definition of exactly who the user or users are. Technology changes all the time too, and with it the usability problems and challenges are also continually changing, so keep that in mind and if the number doesn't change with the times, add the technology that it takes into account to the description.

I'm not sure how useful it is to have a single numeric value anyway... which I guess is something that lots of people would say anyway.

Heuristics have their place alongside empirical studies, and they're excellent from a discount usability engineering perspective because they generalise what are currently agreed to be the most important things in the most common domains. But even the usability experts usually recommend that someone who knows what they're doing should carry out the heuristic studies, because when it comes down to it nobody really knows what questions to ask apart from "is it easy to use this?".

We're just not advanced enough at understanding usability yet that it's feasible to have a usability-for-dummies system. Important bits of interpretation that nobody thought might be there can be lost. eg. I could make a page full of black text on a black background and it could still score maximum points in your system, but it would be difficult to use. If you'd thought about that there are still lots of other problems.

Stuffing them into a number just hides those interpreted details even more. I guess there's enough people saying this sort of thing anyway, so just add my opinion to the crowd. :)


jesterzog Fight the light


Hmmm... (4.00 / 5) (#33)
by skim123 on Tue Apr 16, 2002 at 12:21:24 AM EST

If what you are after is a page's use-ability, why not measure the amount of time it takes an average user to use the Web site? That is, if it's a site like k5 you might say to an average user, "Get a listing of all of the stories in the Technology section," or, "Show all posts by user x", or "Update your user preferences to hide ads," or whatever. That way metrics like page size, page load time, page depth, page structure, etc. get incorporated into the end time taken to accomplish a task. After all, isn't a user more interested in how long it takes them to get whatever use they're after?

Money is in some respects like fire; it is a very excellent servant but a terrible master.
PT Barnum


Good idea (3.00 / 3) (#38)
by bugmaster on Tue Apr 16, 2002 at 03:56:05 AM EST

Yeah, we do need a metric that does not rely on actual users personally rating the site. I'd like to see that kind of stuff built into Google search results, for example. This way I have a rough idea of what not to click.

Anyway, I think that a metric needs to include graphics, as well. For example, any animated GIF with less than 4 frames should immediately dump the site into the loser bracket. Same goes for auto-playing midis and 15Mb-sized flash intros.
>|<*:=

An important number. (4.25 / 4) (#41)
by i on Tue Apr 16, 2002 at 05:21:13 AM EST

Average time-to-load with a 56K modem. No usability metric is complete without it.

and we have a contradicton according to our assumptions and the factor theorem

Usability = anchor tag text (5.00 / 7) (#42)
by Alan Crowe on Tue Apr 16, 2002 at 05:43:13 AM EST

<a href="URL"> description</a>

Can the user use the descriptions to select the correct link, or must he click on several different links to find the page he is looking for? The proposed metric fails to penalise labelling your links: stuff, more stuff, yet more stuff.

Most of the sites I see go to a lot of trouble to have pretty labels on the navigation buttons, but go to very little trouble choosing the text. For example, what do you expect from an about link?

  • The biographies of the authors
  • The technology used to make the site
  • A brief statement of the cause the site is campaigning for
  • Instructions on using the navigation features
Pure text downloads very quickly. So use it. Either have more text at the source of the link, so the user doesn't pick the wrong link. Or have pure text at the destination so that the selected page pops up in a ping time, and the user hasn't wasted time clicking to see what is there.

I regard the attempt to rate websites without regard to how well they use language as forlorn

Web usability (2.75 / 4) (#43)
by alfadir on Tue Apr 16, 2002 at 05:50:04 AM EST

This might be slightly off topic, but an interesting note in the discussion about web usability.

The browser is a red herring; it's a dead end. The idea of having batched processing inside a very stupid program that's controlled remotely is a software architecture that was invented about 25 years ago by IBM, and was abandoned about 20 years ago because it's a bad architecture. We've gone tremendously retrograde by bringing in Web browsers. Now we have an infinite variety of computers all around the world and an infinite variety of remote sites all around the world. That's the power. Alan Cooper of Visual Basic fame.

I belive that there are applications that are perfect for the current web. The library capability where you can find the information you need. But there might also be an even better way of solving the information handling.

I have also seen terrible examples where javascripts/flash solutions that has tried to come to terms with the interactive limitations of the webbrowser. So the question is: Is there a better alternative to the webbrowser ??

As a sidenote I can mention that this is the second time I write this comment. The marvelous browser Netscape crashed when checking the link in Preview mode. Maybe another example that another kind of program would be better for Internet.



Is usability really quantifiable? (4.00 / 1) (#44)
by StephenGilbert on Tue Apr 16, 2002 at 09:00:52 AM EST

And can such a metric be universally applied?

These aren't rhetorical questions. I don't really know much about the subject, but it seems rather strange to me, somewhat like standardized tests. What is really being measured here? Different sites have different purposes, and I'm not convinced that a universal "usability number" can be slapped on any given website.

--------------------------------
Wikipedia: The Free Encyclopedia

wrong strategy and wrong conceptualization (3.50 / 2) (#45)
by buridan on Tue Apr 16, 2002 at 09:01:19 AM EST

to quantify usability just will not work, though many people try. What you have above is a measure of information performance, which has little to do with usability for most people, though arguably it matters significantly for certain groups/classes of users.



Just curious (2.00 / 1) (#53)
by Stereo on Tue Apr 16, 2002 at 01:16:17 PM EST

How much would this page score? Do you add extra marks for a page called "SUPER-KL AWESM WEB PAGE"?
-- God will forgive me. That's his job after all. -- Konrad Adenauer
Pepke's Usability Quotient (4.00 / 1) (#55)
by epepke on Tue Apr 16, 2002 at 05:37:05 PM EST

I'm going to be arrogant and name this after myself, because I have never seen it expressed quite this way. If it has, let me know, and I'll include footnotes in future.

Let tg be the time that a user needs to figure out the web site enough to accomplish the primary goal of the web site. Let tf be the time it takes for a user to become frustrated with the web site enough to be out of there and looking for another site. If tg < tf, then the web site is usable.

Obviously, what you want is some sort of overall Tg and Tf, possibly to calculate a usability quotient Tg/Tf. This is pretty easy to get by testing. Do two tests of user time, one where they have the option of going to another site and one where they don't. The former gives samples of tf, and the latter gives tg. You could also correlate the quotient by the number of times people actually did switch in the one experiment where they had the option, a handy statistical check.


The truth may be out there, but lies are inside your head.--Terry Pratchett


How do you find tf? (none / 0) (#60)
by sgp on Wed Apr 17, 2002 at 12:12:35 PM EST

If you're using testers, they will stay around after tf since they're being paid to. I guess you could get them to say "I've had enough, I'd be off elsewhere now in real life"

And you'll never know about the real users who disappear, unless you have a "stuff this I'm off" page for people to leave your site.

There are 10 types of people in the world:
Those who understand binary, and those who don't.

[ Parent ]

Nah, it's easy (none / 0) (#61)
by epepke on Wed Apr 17, 2002 at 01:06:48 PM EST

You find tf by giving them the option of switching to another site and seeing how long it takes them to choose that option. This requires a sort of "real world" simulation, where you have a goal and a list of sites. You don't force them to stay on the same site; this would defeat the purpose. Instead, you have a separate test where you force them to stay on the same site, which gives you tg


The truth may be out there, but lies are inside your head.--Terry Pratchett


[ Parent ]
Where G=Grudgingly, right?! (none / 0) (#63)
by sgp on Fri Apr 19, 2002 at 07:16:18 PM EST


There are 10 types of people in the world:
Those who understand binary, and those who don't.

[ Parent ]

Give up (4.00 / 2) (#57)
by selkirk on Wed Apr 17, 2002 at 01:33:02 AM EST

Numerical Metrics for web usability.

Look at the controversy over numerical metrics for measuring learning in schools and give up.

Better yet, try numerically defining pornography instead.

Usable websites: I know them when I see them.

Try GQM first... (5.00 / 1) (#59)
by Maniac on Wed Apr 17, 2002 at 10:31:24 AM EST

You can improve the metrics considerably if you take some time to determine...
  • the goals (user can succeed, quickly)
  • the questions to be answered to determine the goals have been met
  • the metrics to answer those questions
This series - goals, questions, metrics (or GQM) is a good framework to apply to many problems. If I read through the list of metrics you have proposed, I do not see a clear relationship to "usability".

Let me use an example a web site I have to use at work daily for filling in a time card. The steps I go through include (as action / response)...

  • Select time card URL from bookmarks. (or back from the window & used yesterday). System redirects me to one of two servers and asks me to sign in.
  • I enter my id / password, get a choice of time card, expense report, purchases, change password, and logout.
  • I select time card and get a page to select this week's time card or a previous one.
  • I select this week's card and get a form w/ the current week's card.
  • On Monday I have to type in or select values for several fields.
  • Every day I enter the hours (usually 8) and then select "validate". The system [usually] indicates the information is OK.
  • On Friday, I submit the time card. Every other day I save the time card. I get a confirmation that the data was saved (or submitted).
  • I select logout. The system responds with a you are logged out screen.
To improve the usability, I might have goals for accuracy (higher is better) and time to enter (lower is better). I can list many more but will keep the list short.

The accuracy questions I might ask are like - What is the % of invalid charge numbers used? What is the % of changes required the next week?

The time to enter questions I might ask are - How long does it take to complete each step? How long does the total task take?

With those questions, I would have metrics based on use of the real application to do the real task. For example, why do I care about the # of links as a metric? It would be better FOR ME as a user to change the first URL slightly and go directly to the time card form after the sign in and skip the two intermediate pages. Remember, I use this site EVERY DAY for my time card and have never used it for expense reports nor purchases.

To improve the accuracy, I might change the form on Monday to fill in last week's values (w/o hours). That way the recurring data gets entered once and is (almost always) error free after that.

These are a couple examples of using metrics to improve the quality of an application. The metrics you propose only address the real issue in a superficial manner.

Developing a Basic Numerical Metric for Web Usability | 63 comments (56 topical, 7 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!