What's wrong with Bobby?
by Flash Bristow, Website Accessibility ConsultantWhat is Bobby?
Bobby is a tool aimed at evaluating web pages for accessibility. There are two versions of Bobby, the free online version at http://bobby.watchfire.com/ which tests one page at a time, and the full version for Windows which is priced at $299. Most people will be using the free online version, and this is the one which I have tested.Bobby asks the user to input a URL and to choose whether to judge that page against the US accessibility regulations, Section 508, or the W3 Consortium's Web Content Accessibility Guidelines (WCAG) 1.0 standard, produced as part of the Web Accessibility Initiative (WAI). In my tests, I have been using the WCAG criteria, which can be viewed on this page.
Bobby then produces output which shows the audited page, littered with question marks where it considers there may be accessibility issues, followed by a report which details any contraventions of the regulations, along with any manual checks required by the user to ensure compliance.
[Bobby has now been superceded by WebXACT - however, this still falls down in many of the same ways.]
What is Bobby good at?
There are some accessibility checks which can be automated, for example ensuring that every page has a language declaration, every table has a summary, and that every image has an alt tag. Bobby is able to check this and give a straightforward success or failure. This reduces the need to go over every page with a toothcomb, and compensates for human error, such as if a tag is missed accidentally.Automated checks can be applied to approximately one third of the WCAG accessibility criteria.
What's wrong with Bobby?
If automated checks can be applied to around a third of the WCAG criteria, then it follows that the remaining two thirds will need some degree of manual checking. Bobby attempts to deal with this by listing issues which may need manual qualification, and then saying "If the Priority 1, 2 and 3 issues listed below do not apply to your page, then it qualifies as Bobby AAA Approved".In my opinion, having to apply manual checks rather negates the point of using an automated tool! In addition, many of the issues raised are "not triggered by any specific feature on your page, but are still important for accessibility" - so the user may end up checking for something which does not apply. Alternatively they may end up misinterpreting - or even ignoring - the criteria requiring user checks.
As well as producing false negatives in the manual check category, Bobby can also miss important issues. For example, I was asked to evaluate a site which relied entirely on Macromedia Flash. I tested in a text browser, found that there was no accessible content, and reported back accordingly. Another person had tested the same site with Bobby, which reported only one failure at the AA level (no doctype), and one at the AAA level (no language declaration). The fact that when the script was removed only a blank page remained was accidentally overlooked, and my acquaintance replied saying how well the page scored for accessibility! A similar problem was discussed in WaSP on BBC Accessibility which reports that Bobby found "One instance of missing alternative text on a form: Yes this is a problem, and requires a one minute fix. (However, there is a bigger problem that has been missed - the form in question is dependent on JavaScript)."
So it's fair to say that Bobby Approved sites are not able to boast compatibility with all the WCAG standards - rather that they have complied with "the bits of WCAG which can be tested by an automated tool". Unfortunately, many people seem to infer from the use of "AAA" in the name "Bobby AAA Approved" that it means "WCAG 1.0 WAI-AAA Approved" - this is clearly not the case.
However, if we accept the limitations of this being an automated tool, we need also to consider the value of the automated checks which are performed. The WCAG guidelines are to some extent open to interpretation - for instance, what comprises using navigation mechanisms in a "consistent" manner? The individual interpretations of each criteria by the tool designer can reflect on its overall efficiency and usefulness. This also applies to the manual checks, where our own interpretation may vary. For example, WCAG checkpoint 4.2 states: "Specify the expansion of each abbreviation or acronym in a document where it first occurs." Their recommendation is to use the title attribute of the ABBR or ACRONYM elements, but I can imagine that some web designers would think it adequate to merely use the expansion in the text, followed by the abbreviation in brackets, in the same manner as when we are introduced to an abbreviation in a book or magazine.
In July 2004 I ran a survey to find out how people perceived the use of various logos, such as the W3C WCAG conformance icons, membership of the Guild of Accessible Web Designers (GAWDS) and also Bobby AAA Approved.
The results showed that the only logo which performed badly was that of Bobby. Only 27% of respondents would be more likely to use a business displaying this icon, and 10% would actually be less likely to do so. This leaves a huge 63% who either didn't know what the logo meant or were ambivalent to its use. Following the results of my survey, I have stopped using the Bobby icon on my websites.
Bobby now has a successor, WebXACT, which checks for security and content features as well as accessibility. However, it actually requires the use of Javascript to run! When I access it in a text browser, I receive the warning "At this time WebXACT requires JavaScript support in order to operate. If you have a script-capable browser, please enable scripting before you continue."
There is a heated debate on the Accessify Forum on this subject, but it can be assumed that the issues surrounding Bobby - and its successor - will continue to be discussed for some time to come.
What should we use instead?
There are many alternative tools, although the considered wisdom seems to be "you get what you pay for". I would still favour meticulous manual application of the WCAG guidelines against the code of a website over use of a robot.Much of the commercially-available software is aimed at the US market and so uses Section 508 as its criteria, rather than WCAG checkpoints, so this is something to watch out for.
Here is a list of alternative tools, although I make no claims as to their functionality as I have not tested them all very thoroughly.
Ask Alice - http://askalice.ssbtechnologies.com:8080/ssb/aa/anon/index.jsp
- Free accessibility report using Section 508 or WCAG criteria.
- Provided by SSB, who may follow up via email.
Cynthia Says - http://www.contentquality.com/
- Free online check with a choice of Section 508 or WCAG criteria.
- Simple, effective report - my preferred free tool.
A-prompt - http://aprompt.snow.utoronto.ca/
- Free tool which checks against WCAG criteria and attempts to repair violations.
- Site must be downloaded to the local computer which can be awkward.
aDesigner - http://www.alphaworks.ibm.com/tech/adesigner
- Free, but you need to have a login on IBM's website
- Shows how your site appears to the visually impaired
LIFT - http://www.usablenet.com/
- Uses Section 508 criteria.
- Various versions, from LIFT for FrontPage ($299) to LIFT Online ($999).
- Free demonstration available from their website.
InFocus - http://www.ssbtechnologies.com/products/InFocus.php
- Checks against Section 508 or WCAG and attempts to repair violations
- From SSB, cost £1,029.07
- Free demonstration download on website
AccVerify / AccRepair - http://www.hisoftware.com/access/
- AccVerify checks against Section 508 or WCAG criteria
- AccRepair checks and uses a wizard to prompt for repairs
- AccVerify: £283.78, AccRepair: £570.43
There are other tools to help you deal with specific aspects of accessibility, such as Juicy Studio's Readability Test, which helps to judge how easy it is to read your content.
I should also mention my own accessibility audit, which judges sites against WCAG criteria. It's available for only £105, which is refundable against any remedial work needed - a bargain!
Flash Bristow, Website Accessibility Consultant
Web Design & Mastery
http://www.webdesignandmastery.com
07939 579090
Further reading:
- Accessify Forum discussion on WebXACT
- Notes on some tools for checking and improving Web page accessibility
- WaSP on BBC Accessibility
©2004 by Flash Bristow
"We are really delighted with our new website. Flash was extremely fast, friendly and helpful all the way through the process and went to great lengths to get the website just as we wanted. As complete novices to the world of website design, we were grateful to have a person who made the task run so smoothly.
Thank you very much!"
[Hackney Family Backup]
[Hackney Family Backup]
"Thank you so much for all your hard work on my behalf; you very quickly grasped the image I wanted to portray. I am really pleased with the website that was delivered earlier than I expected and on budget. Your extensive knowledge of websites and the legislation that affects them is impressive and led to a level of detail that I was not expecting. I literally could not have done it without you. It has been a real pleasure working with Web Design & Mastery."
[Kate Fishpool Associates]
[Kate Fishpool Associates]