www.makingspeechestalk.com Open in urlscan Pro
146.185.146.39  Public Scan

Form analysis
0 forms found in the DOM

Text Content

USABILITY PILOTS - EGOVERNMENT BENCHMARK

home | Statistics | login
       



USABILITY INDICATORS - 29/09


This project aims to identify new usability indicators for the European
eGovernment benchmark. The benchmark is perhaps the most important EU digital
government policy instrument since the European Union has no direct competencies
in digital government. However, the eGovernment benchmark framework is under
continuous development, and in particular, the usability indicators have shown
limitations in terms of validity and comprehensiveness. Thus, this project aims
to improve the eGovernment benchmark, with specific regard to measuring
usability. The challenge is to identify feasible and effective indicators that
can be integrated into the current data collection methodology.

This is the current list of candidate indicators for updating the usability
perspective in the European eGovernment benchmark. The consortium produced a
long list of indicators with inputs from interviews, expert workshops, desk
research, literature review and stakeholder interactions. This list is
constantly revised and can change according to the information gathered
throughout the study.

Three perspectives need to be in mind when analysing this list. First,
indicators must comply with usability theory, addressing important usability and
user experience dimensions. Second, be aligned with policy priorities by
complementing principles laid out in the Tallinn or Berlin declaration of
eGovernment. Finally, indicators need to be technically feasible, indicators
that cannot be measured by mystery shoppers, evaluated by visiting the website
directly, or need inputs by asking for data from public administrations are
excluded from the list.
Index
Accuracy
Binary task completion
Accessibility
Speed and Performance
Search Engine Optimisation
Consistency and ease of use
Testing and reporting
Time
Support and help


ACCURACY

Accuracy measures the number of errors users make either during the process of
completing tasks or in the solution to the tasks. According to research done on
the topic of user usability, measuring accuracy permits the design of services
to be more efficient. In general, accuracy and completeness are essential to
quantify how users achieve specified goals.

Error messages: For any page requiring user input, if an input error is
detected, is the input identified as an error and described as text to notify
the user?sentence permalink

+


BINARY TASK COMPLETION

This section complements the accuracy measures. Binary task completion refers to
measures of whether users complete tasks or not. Task completion includes the
number of correct tasks and the number of tasks that users failed to finish
within a set time. It is an essential metric to evaluate if the services are
efficient in what they are proposing to do. To gather data for some of the
indicators, it would be necessary to access the logs and databases where the
digital public services are stored. This is the reason why some of them were
excluded during the assessment process.

Ability to change inputs immediately: Is it possible for the user to make
changes in their inputs while/after the service has been executed?sentence
permalink

+

Online service completion: Can the service be completed entirely online?sentence
permalink

+

Stop mid-transaction: Is it possible to stop the process and come back to
continue it without having to restart from scratch?sentence permalink

+

Proprietary technology: Does the service need proprietary technology to be
accessed? Does the service require proprietary technology to be completed? These
could be an eID reader or a paid software.sentence permalink

+


ACCESSIBILITY

Accessibility is an important aspect of usability as it can make the web pages
easier to navigate and enable users with impairments to access the service when,
if some measures were not taken, it would not be possible. Accessibility is
already included in the monitoring exercise in the context of the accessibility
directive. Some indicators are included here, in particular concerning less
technical aspects of accessibility, to complement the ones already measured in
the most recent data collection process of the eGovernment benchmark (2020).
However, most of them were not scored.

Clear language: Is the language clear and easy to understand for a user with no
prior knowledge as the accessibility standards of the World Wide Web Consortium
(WCAG 2.0 SC 3.1.5)?sentence permalink

+

Translatability: Is the language correctly translated to English by machine
learning tools such as Google Translate?sentence permalink

+


SPEED AND PERFORMANCE

While it is complicated to measure how much users like a site's design with web
APIs, measuring its speed and responsiveness is not. These indicators are
imperative to understand the quality of the online services and how efficient
they deliver excellent user usability. Knowing these variables will allow
countries to optimise for quality of UX, which is key to the long-term success
of any site on the web.

Input rate (clicks): How many clicks are necessary to finalise the
service?sentence permalink

+

Cumulative Layout Shift (CLS): CLS measures the largest burst of layout shift
scores for every unexpected layout shift that occurs during the entire lifespan
of a page. A layout shift occurs any time a visible element changes its position
from one rendered frame to the next. How long does it take for unexpected
elements to stop moving before the page fully loads?sentence permalink

+

First Contentful Paint (FCP): First Contentful Paint marks the time at which the
first text or image is painted/loaded. How long does it take for the first
content to appear on the page?sentence permalink

+

First Input Delay (FID): An important user-centric metric for measuring load
responsiveness because it quantifies the experience users feel when trying to
interact with unresponsive pages. How long does it take for the webpage to start
processing the first user interaction?sentence permalink

+

Largest Contentful Paint (LCP): Measures the perceived load speed because it
marks the point in the page load timeline when the page's main content has
likely loaded. How long does it take for the largest content on the page to
load?sentence permalink

+

Page requests: How many requests (scripts) are there on the webpage?sentence
permalink

+

Page size: How large (kb) is the page size?sentence permalink

+

Speed index: Speed Index measures how quickly content is visually displayed
during page load. It measures how quickly the contents of a page are visibly
populated. What is the speed index of the page (in seconds)?sentence permalink

+

Time to Interactive (TTI): Time to interactive is the amount of time it takes
for the page to become fully interactive. What is the TTI of the webpage (in
seconds)?sentence permalink

+

Total blocking time: Sum of all time periods between FCP and Time to
Interactive. What is the total blocking time of the page (in
milliseconds)?sentence permalink

+


SEARCH ENGINE OPTIMISATION

High SEO ranking is important for both findability and usability of the service.
Pages with good performance, accessibility and accuracy will have a higher
chance of scoring well and appearing first on search engines. Thus, it is also
vital to keep track of SEO metrics to assess which aspects the online services
can improve. It is also true that the findability of the service is an important
part of the overall user experience online.

Average position: Average position on search engines for a specific keyword to
find the service online. In this case, the keyword will be the name of the
service. What is the average position of the webpage?sentence permalink

+

H1 tag: Each web page should have a title using a h1 tag that describes its
topic or purpose. Does the webpage possess a single h1 tag depicting the title
or purpose of the page?sentence permalink

+

Text on images: The text on the website can be readable by software and search
engines. Is there any text embedded in images?sentence permalink

+

Visibility: This metric is based on CTR and shows a domain's progress. It scores
the percentage of clicks received based on the organic rankings positions across
all of the keywords tracked in a campaign.sentence permalink

+

URL structure: The user can easily identify the structure of the web page with
an organised URL structure. This also helps Google to index the page easily. Is
the URL structured in a categorised format?sentence permalink

+


CONSISTENCY AND EASE OF USE

Consistency and ease of use are founding concepts of good design. Ease of use is
a fundamental notion that explains how easily users can use a service, and
consistent user experience enhances ease of use and reduces the learning curve.
However, measuring them objectively and comparatively can be complex. Some
member states have made a substantial effort in providing a simple and
consistent user experience, for instance by consolidating their online presence
and creating guidelines, toolkits and templates with different levels of
enforcement. The indicators in this section aim to detect to what extent the
user experience is consistent, and member states have published instruments and
tools to achieve it.

Service design template/handbooks: Does the central administration provide
guidelines, standards, toolkits or design templates for other administrations to
structure their own services or websites?sentence permalink

+

Consistent design across websites: Do the websites of different services have
consistent branding and design elements?sentence permalink

+

Transparency of the process: Show the steps that are necessary to be executed
until the service is completed. Does the web page depict each step of the
process that the citizen needs to go through throughout the service
execution?sentence permalink

+

Visibility of system status and visual aids: Does the service have loading bars,
breadcrumbs, number of search results and other visual cues?sentence permalink

+


TESTING AND REPORTING

Metrics on usage are an important enabler of usability and a sign of its
priority. If countries make the data on the adoption of their digital services
publicly available, it is possible to measures many different KPIs within the
user usability perspective and identify potential gaps to be corrected.
Publicising data about the adoption of the digital service can be a great
incentive to enhance usability and improve the user experience.

Reporting: Are there performance reports with data of the service publicly
available?sentence permalink

+


TIME

Completion time can be an indicator of how well the online service is
structured. This is a standard metric in user usability, as the shorter the
processing time, the better, and the better the user experience, although
comparability remains a challenge. It is also recommended to set the right
expectations and let the citizen know how long the execution of the service will
take before starting the process, allowing the person to prepare more
efficiently.

Task completion time: The time users take to complete a task as a whole. How
long does it take to complete the service?sentence permalink

+

Time notification: Does the page inform how long it will take to complete the
service before starting?sentence permalink

+


SUPPORT AND HELP

Options for feedback and help are also long-standing metrics in user usability
and are also present in the Berlin and Tallinn declaration on eGovernment and in
the existing benchmarking methodology. Some new indicators were included to
complement the ones from the last eGovernment benchmark.

Chatbot: Is there an automated chatbot to reply to questions from the
user?sentence permalink

+

Existence of hints: Are there ways to help the user complete the service, such
as examples, common mistakes to avoid, tips?sentence permalink

+

Making Speeches Talk - There are 0 comments - Usability Pilots - eGovernment
benchmark



Privacy Policy - Cookies Policy

The comments are published under license CC-BY 4.0.



in caricamento