of accessibility
Clearly this can be extended to mobile phone and PDA handsets - indeed, the
Symbian 60 phones and most other products from Nokia conform to a common
look-and-feel interface.
In the web environment, although much can be automated, experience shows that
real user testing or at least a clear methodology carried out by experts is still the
most accurate mechanism for establishing the accessibility level of a website
and/or software.
Next generation mobile devices will become more reliant on mobile web-based or
wireless services - the interaction between phone software, network based services
and web-based (external) systems will become more seamless - accessibility and
usability testing in all of these environments will become crucial if the end user
experience is to be a pleasant and productive one. A joined up holistic approach to
accessibility testing is therefore crucial.
6.2.2 Current trends in accessibility evaluation
Web accessibility evaluation has provided an exemplar of the widespread
resources available which developers can use to develop accessible web sites.
However, for the evaluation of other products and services, accessibility evaluation
is still very much in its infancy with little resource available to assist. To determine
the extent of the problem it was crucial to establish what the current trends in
industry were by:
• Investigating which organisations evaluate for accessibility
• Investigating how much industry really knows about evaluating for
accessibility
• Where they exist, establishing which evaluation methodologies are being
used
• Establishing which areas of industry are being evaluated.
To fulfil the objectives a two stage questionnaire process was carried out:
• Stage one identified how many of the most likely and what types of
European organisations there were, how many knew about accessibility
evaluation and how many evaluated next generation network services and
terminals for accessibility
• Stage two identified that of those that stated that they did know about
accessibility testing, which methodologies organisations were using and
which domains were being evaluated.
252
6. The role of evaluation
of accessibility
Two hundred European organisations that tested products or systems received the
stage one questionnaire. Of these, only twenty-six responded. Tw e n t y - f o u r
organisations informed COST 219ter that they conducted research into the
accessibility of next generation network services and terminals for disabled and
elderly people. The results showed that of the twenty-six respondents, nineteen
performed evaluations in usability and accessibility, with four evaluating for
accessibility only and three evaluating for usability only. Finally twenty-three
organisations agreed to take part in stage two of the process. A list of
organisations that test/evaluate accessibility can be found on the COST 219ter
website (http://www.cost219.org).
The second stage questionnaire was distributed to the twenty-three organisations
that agreed to take part. Twelve of these responded. The results showed that
popular methods used included questionnaires, interviews, observations, focus
groups, expert groups and panels, task analysis, user groups and user panels,
scenarios, prototyping and finally heuristic evaluations. This information links well
with the literature, which cites all these methods as popular [Stanton et al, 2005].
Popular areas of industry that were found to be evaluated included; mobile phones
and services, telecommunication services, s o f t wa r e, c o m p u t e r s, P D A’s and
handsets, and finally, smart homes.
The results of the questionnaires suggested that there was only a limited amount
of work being carried out into the evaluation of next generation network services
and terminals for accessibility with fewer than 15% responding to the first
questionnaire. Interestingly, the results from the first questionnaire showed that
most of the organisations evaluated for accessibility and usability with only a
minority stating they only tested for accessibility or usability. Further information
about the findings of the questionnaires can be obtained from the report written
by Anna-Liisa Salminen on the COST 219ter website (http://www.cost219.org).
The results also showed that a variety of methodologies were used covering
various domains for evaluating next generation of network services and terminals
for disabled and elderly people.
Finally, the results of both questionnaires evidently make it clear that the
evaluation of next generation network services and terminals is only being
conducted to a limited degree. The research has demonstrated that although a
number of evaluation methodologies exist and are being used, it is not obviously
apparent that any are thorough and specific enough to evaluate mobile phones for
accessibility and ease of use across various disabled user groups and the elderly
population in any depth.
253
6. The role of evaluation
of accessibility
6.2.3 Methodologies used by COST 219ter members
As part of the process of looking at current trends in industry for evaluating for
accessibility of next generation of network services and terminals, it was important
also to investigate what members of COST219ter were currently using. The Royal
National Institute of the Blind (RNIB), a member of COST 219ter, had indicated that
it carried out extensive evaluations on products and systems, including mobile
phones. COST 219ter felt that the methodology used by the RNIB should be
reviewed to understand it in greater detail and determine if anything could be
learnt from this evaluation methodology. The aim of this work was to determine if
a methodology exists which is suitable for recommendation for implementation
across industry by COST 219ter.
This visit identified that RNIB carried out extensive user evaluations on products to
ensure that they were accessible for blind and partially sighted people. From the
knowledge and expertise gained from these evaluations, RNIB created a simple set
of guidelines as a means of assessing products initially without involving end users.
However, these guidelines had some shortfalls. Firstly, they were not specifically
aimed at next generation mobile phones and as such were not detailed enough to
cover the exact requirements needed to fully evaluate a device such as a mobile
phone. This included the physical aspects unique to a mobile phone, the operating
system and the services that users can access from their mobile phone. Secondly,
these guidelines only considered the needs of blind and partially sighted people
and finally, the guidelines were only written at a very top level without any detail.
Further information about the findings of this work can be obtained from the
report written by Kristian Kristiansson on the COST 219ter website
(http://www.cost219.org).
After further analysis, it was decided that the evaluation method used by RNIB
showed potential for being used by COST 219ter for evaluating next generation
mobile phones. In order to fulfil the objectives of the Action it was recommended
that these guidelines be adapted to address the above issues.
Analysis showed that the specialist organisations that carry out evaluations of any
description in the blindness sector across Europe did not have as robust a
mechanism as used by RNIB and it was clear that certainly within the sector in
Europe, its techniques were verifiable, consistent and replicable in terms of
methodology and result.
254
6. The role of evaluation
of accessibility
6.2.4 Suitability of these methodologies for implementation
across industry
The results of the COST219ter research showed that at present there is no
evaluation methodology that addresses the needs of disabled and elderly users for
ensuring that next generation mobile phones are designed to be inclusive.
Therefore a toolkit is needed to address this gap to ensure that mobile phones are
developed to be as inclusive as possible.
Whilst this toolkit needs to evaluate accurately and effectively the accessibility of
a mobile phone, it also needs to be usable by the person or persons using it.
Without the usability of the toolkit being accounted for, there would be a danger
of having a solution which would not be accepted by the telecommunications
industry due to its own complexity. Therefore any toolkit should not assume prior
expertise or knowledge, yet it should be easy to understand and complete.
The research has shown that a toolkit could be proposed using RNIB’s evaluation
guidelines as its foundation. Such a toolkit would need to encompass the
accessibility needs of various disabled user groups and also need to be developed
so that it is aimed specifically at next generation mobile phones.
The example of automatic evaluation has been advantageous in web site
accessibility. This example deserves attention to determine if similar tools can be
produced to evaluate the accessibility of mobile phones particularly for evaluating
the accessibility of mobile web pages. This would satisfy manufacturers who do not
wish to extend design and manufacturing times whilst satisfying the needs of users
that are currently being excluded from using mobile phones.
6.3 Mobile phone evaluation toolkit
Edward Chandler, Elizabeth Dixon and Steve Tyler
The toolkit is a simple methodology which is capable of evaluating the accessibility
of a mobile phone. The toolkit comprises three sets of heuristics which cover the
basic accessibility features of the physical part of the phone, the operating system
and the services.
Each set has approximately 14 heuristic principles (rules of thumb). For each
principle, checkpoints are given to objectively measure the accessibility/usability of
the device against a set pass/fail criteria. Accompanying each set of heuristics is a
255
6. The role of evaluation
of accessibility
completion form which includes a fail criterion for each checkpoint. It also shows
whether or not the checkpoint is a high or low priority and provides a comments
s e c t i o n . F i n a l l y, it includes a section for any additional information and
recommendations for future work on that phone. The heuristic principles are as
follows:
Hardware principles
• Promote easy identification and orientation of the product
• Provide clear print (visual information)
• Provide clear visual information (non print)
• Promote the location of the buttons, switches, dials, and joysticks by sight
• Promote the location of the buttons, switches, dials, and joysticks by touch
• Promote the location of the connectors, sockets, battery, battery
compartment/back cover, SIM card holder, memory card and memory card
holders by sight
• Promote the location of the connectors, sockets, battery, battery
compartment/back cover, SIM card holder, memory cards, memory card slot
by touch
• Provide clear tactile information (texture/size/differences)
• Promote clear audible information
• Enable the battery, SIM card and memory card to be used easily
• Enable the product to be physically easy to use
• Promote compatibility with other devices
• Provide clear Instructions
• Provide easy to use packaging.
Software principles
• Allow choice over the input and output method
• Optimise the input method
256
6. The role of evaluation
of accessibility
• Optimise the output method
• Allow personalisation of the input
• Allow personalisation of the output
• Allow personalisation of the alerts
• Promote flexibility
• Promote consistency
• Provide easy to understand menus
• Provide intuitive prompts and informative feedback
• Promote easy to use soft keys
• Aid recovery and prevent errors
• Promote easy connectivity and compatibility with other devices and/or
software
• Aid task completion.
Web services principles
• Allow flexible access to the service
• Enable user style sheets
• Optimise colour and contrast
• Use text effectively to improve readability
• Consistent page layout
• Provide content that can easily be scanned
• Easy to use navigation
• Provide meaningful links
• Provide accessible forms
• Use JavaScript carefully
• Promote accessible multimedia
257
6. The role of evaluation
of accessibility
• Provide informative feedback and prompts
• Aid recovery and prevent errors
• Aid task completion.
Further information about these heuristics and their checkpoints can be obtained
from: www.cost219.org/toolkit.
It is important to remember that these heuristics may be applied at any time during
the design phase or at later stages of the development process or once a product
has been released to market. However, it is recommended that these guidelines
should be applied as soon as an initial mock up or paper prototype of the device
becomes available to minimise design costs.
This toolkit should never be used in isolation as a total evaluation process for
evaluating mobile phones. It is intended to highlight basic accessibility issues
which currently inhibit disabled users from using a mobile phone.
6.3.1 Development
When the toolkit was in development, it was clear from previous work that the
scope of the toolkit needed to be closely specified, it should be a simple
methodology capable of evaluating the accessibility of a mobile phone and finally
it would have to be validated against end user requirements.
Prior to commencing work on the toolkit, it was important to carry out a scoping
exercise defining the parameters of the toolkit. Evaluating a mobile device could
potentially include a vast array of possibilities. Therefore as part of this process the
following questions were addressed:
• What would be the main focus of the toolkit?
• What is the common core functionality found across different mobile
devices?
• How would a toolkit cater for the growing and varying array of functions
across devices?
• How can the toolkit be designed to be future proof (what are the key
functions that are likely to persist in the future?) Would there be an
increase of multimodal access to mobile phones?)
• Could the toolkit cover all user needs yet remain usable?
258
6. The role of evaluation
of accessibility
To take all these points into consideration, adapting and creating a toolkit to
address every functionality/service would be impractical. Therefore rather than
adapting the toolkit to cover every possibility, the scope of the mobile phone
evaluation toolkit was limited to cover:
• The physical part of the mobile phone (hardware)
• The core functionality (including the operating system and the software
installed)
• Mobile web based services.
Additional guidelines would be needed in the future to cover other functionalities
such as GPS software. These have not been included at this stage as it is likely that
these would require separate guidelines specific to technologies and deployed
services. Another area likely to require guidelines is for the creation of accessibility
advice for SMS services. However, it is vital to focus on the core functionality of the
phone first so that disabled and elderly users can use the phone.
The toolkit includes a list of guidelines for creating accessible mobile web services
that are provided over the mobile internet. These guidelines build on the
foundations already in place for web accessibility for larger screens.
In order to develop a simple methodology it was decided that detailed checkpoints
should be provided under each heuristic principle to guide the eva l u a t o r
completing the evaluation with the type of issues to check for, as well as providing
a completion form alongside the heuristics for ease of completion. During the
development of the toolkit the heuristic principles were adapted to accommodate
their new checkpoints for each principle.
Several types of mobile phones were analysed during the development of the
heuristics and their accompanying checkpoints so that a range of different phone
features could be accommodated by the methodology.
As part of this work the RNIB worked alongside other parties close to disabled end
users. These included the Swedish Handicap Institute, The Royal National Institute
for Deaf people (RNID) and Special education and Rehabilitation department of
faculty of Human Kinetics (FMH), Lisbon. The Swedish Handicap Institute drafted
their own set of heuristics for people with learning difficulties. RNID and FMH,
Lisbon provided recommendations to consider the needs of deaf and hard of
hearing users, people with limited dexterity and people with learning difficulties.
Each organisation provided recommendations in the form of checkpoints which
they felt needed to be added. These were inserted under the relevant heuristic
principles.
259
6. The role of evaluation
of accessibility
It was vital that the guidelines accurately reflected the accessibility issues that most
users frequently found so these guidelines could be used to accurately evaluate a
mobile phone. This would enable evaluators with practically no prior knowledge of
accessibility issues of mobile phones to be able to use these guidelines to identify
and resolve frequently found accessibility issues.
Once a full set of heuristics was available, it was essential that the heuristics were
validated to determine whether they accurately evaluated the accessibility of a
mobile phone. This validation involved comparing the results of the heuristic
evaluation against end user requirements. The results of the comparison would
show whether the heuristics and their checkpoints were successful in identifying
accessibility shortfalls in a mobile phone.
6.3.2 End user evaluation and toolkit validation
The aim of the validation was to identify whether or not the toolkit was successful
at identifying the accessibility issues which users found during the evaluation.
In order to achieve the validation of the toolkit, a project was organised which
would perform an evaluation of a mobile phone using the toolkit as well as an in-
depth end user evaluation. This study encompassed four user groups:
• Blind and partially sighted people
• Deaf and hard of hearing people
• People with limited dexterity
• People with learning difficulties.
These users were recruited by Intercollege in Cyprus, the Special education and
Rehabilitation department of faculty of Human Kinetics (FMH) in Portugal, RNID
and RNIB in the UK. The results were gathered using focus groups, interviews and
questionnaires for each group of end users. The results of the user trials in the UK,
Cyprus and Portugal were amalgamated and then compared with the results of an
expert evaluation using the heuristic principles. Further information about the
findings of this work can be obtained from the report written by Edward Chandler,
Elizabeth Dixon, Leonor Moniz Pereira and Cristina Espadinha on the COST 219ter
website (http://www.cost219.org).
During the evaluation, there were a number of unexpected results. A summary of
these has been highlighted below as well as a summary of the validation study as
a whole.
260
6. The role of evaluation
of accessibility
In general, the blind and partially sighted participants found they could dismantle
and assemble the phone although most had not done this before. These
participants stated that they would get a friend/relation or shop assistant to
dismantle and reassemble the phone due to the perceived complexity of the task.
As well as this, most of the UK based blind and partially sighted people had
avoided predictive text before the evaluation as they found it impossible to use.
However, the majority of the blind and partially sighted group thought that using
predictive text was fairly easy to do and would consider using it more. These results
were mimicked by the participants with learning difficulties too.
Other unlikely results were linked to menu customisation. The results showed that
most users wanted to be able to move menu items as required. This showed that
these users liked to make the phone easier to use for them in terms of menu
structure. Linked to this, there was almost a 50/50 split between whether to display
the menu as a grid format or as a list format. These results highlight the need to
make the operating system of the phone highly customisable.
Finally, the results showed that nearly all of the participants liked to change
profiles on their mobile phone. There was a strong demand from end users to be
able to alter how the mobile phone responds in different environments. This
highlighted the importance of user profiles and the benefits of user profiling.
The comparison of the results between the toolkit evaluation and the end user
evaluation confirmed whether or not the toolkit provided guidance for the type of
problems that users typically found. It showed where the toolkit matched typical
problems that users found, as well as highlighting any areas where the results did
not match. Checkpoints which did not match were highlighted for review.
The validation study has shown that generally the toolkit match what users with
disabilities require to determine if mobile phones are accessible and easy to use.
The study also identified that additional work was needed to refine some elements<