Code Monkey home page Code Monkey logo

alfa's Introduction

Alfa

♿ Suite of open and standards-based tools for performing reliable accessibility conformance testing at scale

Alfa is an open and standards-based accessibility conformance testing engine. It is used for testing websites built using HTML, CSS, and JavaScript against accessibility standards such as the Web Content Accessibility Guidelines (WCAG). Alfa is the result of distilling the best parts of Siteimprove's proprietary accessibility conformance testing engine, which Alfa has replaced, and implementing them on top of the open Accessibility Conformance Testing (ACT) Rules Format. It also brings several improvements that make it possible to implement and execute advanced rules without relying on Siteimprove infrastructure.

Contents

Goals

  1. Alfa sets out to strike a balance between false positives and negatives with the goal of having result sets reach a high F1 score. If a false positive is encountered, it is therefore just as important to avoid introducing a potential false negative as it is fixing the false positive.

  2. Alfa is committed to complete transparency on how test results came to be. Every line of code that has the potential to influence test results will therefore always reside within Alfa itself and never with a third-party. However, this does not mean that Alfa does not rely on third-party dependencies, only that there are limitations to what third-party dependencies may be used for.

  3. Alfa wants to foster a thriving ecosystem with people from many different backgrounds building on top of the core capabilities of Alfa. To this end, high-quality documentation is paramount to success. Picking up and putting any one of the many subsystems within Alfa to use should be a straightforward experience with every subsystem clearly stating its purpose and structure. This goal is currently far from met and will be prioritised.

Usage

Alfa is distributed through GitHub Packages as a set of separate packages that can be installed via your favourite npm-compatible package manager:

$ npm install @siteimprove/alfa-<package-name>

⚠️ Make sure to instruct your client to pull packages belonging to the @siteimprove scope from GitHub by adding the line @siteimprove:registry=https://npm.pkg.github.com/siteimprove to your .npmrc file. See Installing a package from Github registry for details.

On their own, each of these packages do very little, but when put together they provide a full suite of tools for performing accessibility comformance testing across all stages of the content development and publication workflow. If you are looking for an easy way to started using Alfa, check out the section on integrations; we might already have a ready-made solution for you!

At a high level, Alfa consumes implementations of rules specified in the Accessibility Conformance Testing (ACT) Rules Format and produces audit results in the Evaluation and Report Language (EARL) Schema encoded as JSON-LD.

Examples

A list of curated examples of usage can be found in the Alfa examples repository.

Here is an overview of some basic Alfa usage. Please note that these are TypeScript snippets that needs to be built into JavaScript code before being run.

Your first interaction with Alfa will likely be similar to this:

import { Audit, Rule } from "@siteimprove/alfa-act";

const input: I;
const rules: Iterable<Rule<I, T, Q>>;

const outcomes = await Audit.of(input, rules).evaluate();

Alfa is completely pluggable in regard to rules and only prescribes the implementation format. As such, there is nothing to configure when it comes to rules; simply pass in the rules you wish to run and results will be provided for those rules. To get you started, Alfa ships with a solid set of rules based on the Web Content Accessibility Guidelines (WCAG). Check the Alfa hub for details of these rules. They can simply be used as:

import { Audit } from "@siteimprove/alfa-act";

import rules from "@siteimprove/alfa-rules";

const input: I;

const outcomes = await Audit.of(input, rules).evaluate();

The last piece we are missing is input. Which specific input that needs to be supplied when running an audit will depend on the rules that are part of the audit as each rule specifies the input it requires. For the default WCAG rule set, the input will be a web page. To get you started, Alfa ships with a scraper that given a URL will fetch a representation of the page that can be used as input to the default rules:

import { Audit } from "@siteimprove/alfa-act";
import { Scraper } from "@siteimprove/alfa-scraper";

import rules from "@siteimprove/alfa-rules";

Scraper.with(async (scraper) => {
  for (const input of await scraper.scrape("http://example.com")) {
    const outcomes = await Audit.of(input, rules).evaluate();
  }
});

If you need to audit multiple pages across a site, but don't necessarily know the URL of each page beforehand, Alfa also ships with a crawler that builds on the scraper to discover and scrape linked pages:

import { Audit } from "@siteimprove/alfa-act";
import { Frontier } from "@siteimprove/alfa-frontier";
import { Crawler } from "@siteimprove/alfa-crawler";

import rules from "@siteimprove/alfa-rules";

const frontier = Frontier.of("http://example.com");

Crawler.with(async (crawler) => {
  for await (const result of crawler.crawl(frontier)) {
    for (const input of result) {
      const outcomes = await Audit.of(input, rules).evaluate();
    }
  }
});

For more complex use cases, please check the Alfa examples repository. It shows how to use Alfa to test components or full web pages; how to filter outcomes based on conformance level, WCAG version, or more; how to interact with pages (e.g. open a menu) before running an audit; how to answer questions asked by Alfa (cantTell outcomes); or how to add custom rules to the default rule set.

Command Line Interface

Alfa ships with a Command Line Interface, making it easy to audit a single page. The CLI lives in the Alfa integrations repository.

Integrations

Alfa ships with several ready-made integrations to various tools, making it easy and simple to integrate accessibility conformance testing as part of your development workflow. If you have suggestions for additional integerations, feel free to open an issue! We are always looking for new places where Alfa can be put to good use. Integrations live in the Alfa integrations repository.

⚠️ The integrations are still experimental and subject to change.

Requirements

Alfa will run in any ECMAScript 2020 compatible JavaScript environment including, but not limited to, recent versions of Node.js, Chrome, Firefox, Safari, and Edge. While it should be possible to build Alfa from source targeting older environments, we do not explicitly provide support for doing so as Alfa is reliant on data structures introduced in newer versions of ECMAScript.

Building

In order to build Alfa, a recent version (>= 14) of Node.js is required in addition to the Yarn package manager. Once Node.js and Yarn are installed, go ahead and install the Alfa development dependencies:

$ yarn install

When done, you can start a watcher that watches source files for changes and kicks off the associated build steps when they change:

$ yarn watch

Note that when the watcher is started it will also perform a build, which can take a bit of time if the project has not been built before.

As new code is pulled from the repository, changes to dependencies and code may require you to run the installation again or, if only code has changed, a build:

$ yarn build

If you want to run tests and make sure everything is working, use:

$ yarn test

When working on a specific package, you can run only these tests:

$ yarn test packages/alfa-<package-name>

If you would like to contribute to Alfa, make sure to check out the contribution guidelines. If you have any questions, you are also welcome to open an issue.

Experimenting

The special scratches directory is reserved for scratch files, which are useful for quickly running small code examples or even larger experiments. The directory is primed with a TypeScript configuration that sets up the needed project references and compiler options. All files within the directory, with the exception of the TypeScript configuration, are ignored by version control and so any files may be added without risk of them being checked into version control.

To create a new scratch file, add a new TypeScript file anywhere in the scratches directory, such as scratches/foo.ts, and do:

$ yarn build scratches

You can then run the built output of your scratch file in any supported JavaScript runtime, such as Node.js:

$ node scratches/foo.js

The scratch files are built with the rest of Alfa as per the previous section. When switching between branches, you may experience one or the other scratch file not compiling. To build just the Alfa packages and not the scratch files, you can do:

$ yarn build packages

The scratches configuration includes the JSX parser from @siteimprove/alfa-dom. So you can create a .tsx file and use JSX syntax directly to create Alfa DOM objects.

Architecture

At its core, Alfa is built around a tree structure that mirrors a subset of the Document Object Model (DOM) and CSS Object Model (CSSOM) interfaces. This tree structure can be created statically from an HTML document and associated CSS style sheets, or it can be extracted from within a browser to also provide executing of JavaScript. Anything else that a browser would typically provide, such as querying elements or computing styles, Alfa implements according to the corresponding specifications.

By implementing browser aspects, such as a style system and the accessibility tree, directly within Alfa, we gain the ability to do some things that would otherwise not be possible had we relied on only the APIs provided by the browser. The most radical difference is perhaps the computation of CSS properties, where Alfa can follow declared CSS properties up through cascade, inheritance, and absolutisation.

At the code level, Alfa is structured as a monolithic repository consisting of several packages that each have their own area of responsibility. You can find more information on the overall architecture of Alfa in the architecture documentation. We also write and maintain architecture decision reports if you want to get the complete picture of how Alfa came to be.

Guides

A list of guides on how to develop with Alfa can be found under guides.

Funding

European emblem

Alfa is part of a project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement Nº780057. We would like to give thanks to the European Commission for their grant, as well as all European citizens, who have indirectly contributed to making Alfa possible. You rock! 🙌

License

Copyright © Siteimprove A/S. Released under the terms of the MIT license.

alfa's People

Contributors

bugvi-benjamin-m avatar dan-tripp-siteimprove avatar dependabot[bot] avatar elenamongelli avatar jym77 avatar kasperisager avatar niclashedam avatar rcj-siteimprove avatar renovate[bot] avatar singingknight avatar siteimprove-builduser avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

alfa's Issues

Truncation of coverage suggestion output

The test output should be formatted such that large blocks are truncated. In addition, if one of the suggested uncovered blocks is really large, the number of blocks outputted should be cut down.

Investigate TypeScript builder program API

Our current use of the TypeScript language service API in which individual source files are diagnosed and emitted has at least the following drawbacks:

  • While diagnostics for source files that are not our own are actually collected, we never show these anywhere. Only recently have we introduced support for traversing source file dependencies (7c94bb0) such that we could improve upon this, but the next point will outline the problems with this as well.

  • While we can traverse source file dependencies by resolving imports, the TypeScript module resolver resolves imports from packages to their entry point defined in package.json, which is how things should work. What we're interested in, however, is which actual source files a given source file depends on and not necessarily which source files it imports.

In effect, directly running the TypeScript compiler can yield different results than when we compile the project using the language service API.

By using the builder program API introduced in TypeScript 2.7, we should be able to fix the drawbacks of our current approach. Specifically, the builder program API includes functionally for getting all dependencies of a file (https://github.com/Microsoft/TypeScript/blob/b7d7d5f7b39a5b9619c77590e5fe7f434ed68f1e/src/compiler/builder.ts#L651) in addition to providing incremental diagnostics for an entire program (https://github.com/Microsoft/TypeScript/blob/b7d7d5f7b39a5b9619c77590e5fe7f434ed68f1e/src/compiler/builder.ts#L660). How far these APIs will take us remains to be seen though.

Parallelize package builds across available cores

With the changes introduced in ff80be7, packages no longer use a shared workspace when performing full builds. As such, we now have the option to parallelize package builds across the available cores of the host system. By using the cluster module, we can create a number of process forks, each of which will handle the compilation of packages handed to them by the master process.

Do be aware of dependencies between packages! Hint: https://cs.stackexchange.com/questions/2524/getting-parallel-items-in-dependency-resolution/2525#2525

Implement tests for SVG aria mappings

Implement a few tests for the SVG aria mappings. There are very few that need this, as most of them are static. Use a black box approach and follow the documentation.

Implement remaining HTML grammar insertion modes

Fill in the blanks:

/**
* @see https://www.w3.org/TR/html/syntax.html#the-before-head-insertion-mode
*/
const beforeHead: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-head-insertion-mode
*/
// const inHead: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-head-noscript-insertion-mode
*/
// const inHeadNoscript: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-after-head-insertion-mode
*/
// const afterHead: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-body-insertion-mode
*/
// const inBody: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#sec-the-text-insertion-mode
*/
// const text: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-table-insertion-mode
*/
// const inTable: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-table-text-insertion-mode
*/
// const inTableText: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-caption-insertion-mode
*/
// const inCaption: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-column-group-insertion-mode
*/
// const inColumnGroup: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-table-body-insertion-mode
*/
// const inTableBody: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-row-insertion-mode
*/
// const inRow: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-cell-insertion-mode
*/
// const inCell: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-select-insertion-mode
*/
// const inSelect: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-select-in-table-insertion-mode
*/
// const inSelectInTable: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-template-insertion-mode
*/
// const inTemplate: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-after-body-insertion-mode
*/
// const afterBody: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-in-frameset-insertion-mode
*/
// const inFrameset: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-after-frameset-insertion-mode
*/
// const afterFrameset: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-after-after-body-insertion-mode
*/
// const afterAfterBody: InsertionMode = () => {};
/**
* @see https://www.w3.org/TR/html/syntax.html#the-after-after-frameset-insertion-mode
*/
// const afterAfterFrameset: InsertionMode = () => {};

Follow the recommended community standards

GitHub published a set of recommendations for a healthy open source project. Of the recommendations, we are lacking three;

  • Description
  • README
  • Code of conduct
  • Contributing
  • License
  • Issue templates
  • Pull request template

I think it makes good sense to write these soon, as it helps both contributors and maintainers in making a good and efficient contribution.

Implement selector matching for child index selectors

With support for An+B microsyntax introduced in #131, child index selectors such as :nth-child() will also have to be supported during selector matching. Look into existing browser implementations for hints on optimizations, if any are employed.

Replace W3C references with WHATWG references where needed

Per the Memorandum of Understanding between the W3C and the WHATWG (https://www.w3.org/2019/04/WHATWG-W3C-MOU.html), specifications such as HTML and DOM will now only be published at whatwg.org and redirected to from w3.org. As Alfa contains many references to both HTML and DOM, all of these references will need to point to whatwg.org rather than w3.org to avoid inconsistencies when additional references are added in the future and these necessarily point to whatwg.org.

Residency of alfa-bench code

Alfa-bench is as of today a separate package, which I think is questionable. The intent of the benchmark tool is for developers and contributors of alfa to benchmark parts of the codebase. For that reason, I think it makes more sense to let the benchmark toolset reside in a separate folder in the root or scripts.

One gain of detaching alfa-bench is that it will then be exempt from all coverage requirements.

Audit of Rule 2 takes decent amount of time and causes a timeout randomly

Running some new audits I stumbled upon a timeout, running audit on the same adress severall times made the audits run successfully and falsy at random.

I timed the audits and for rule 2 they take up to 100ms for auditing 5 applicables, 700ms for 13 applicables.

For "https://www.accessibility.nl/wai-tools/validation-test-sites/amazon.com/
" the audit times out allways

For "https://www.accessibility.nl/wai-tools/validation-test-sites/hp-a3-multifunction-printer-and-copier-hpr-official-site/" timed out once a while. outcome was passed when the audit did not timeout.

Let me know if more info is needed.

Implement XPath utilities

At the very least, we need a package implementing the following based on https://www.w3.org/TR/xpath-31/:

type XPath = ...

function parse(input: string): XPath | null

function evaluate(scope: Node, context: Node, expression: string | XPath): Iterable<Node>

Sanshikan rules are not seen as testable

Currently, Sanshikan rules are not determined to be testable, as they export a constant as an object with multiple arrow functions. Constants and arrow functions are by themselves not seen as testable, as constants are, well, constant and arrow functions can also be constant in some contexts (see fx. packages/alfa-aria/src/features/body.ts). Since Sanshikan rules should be tested, the isTestable heuristic needs to be extended.

Set up Continous Integration

I think it would make good sense to set up automatic testing for alfa. As the project is open source, we could set up travis-ci for free to run npm test or foreman build/test.ts.

alfa-scrape; Add a way to deal with redirect responses

When running the scraper on an url that redirects to another page, it throws an Error:

UnhandledPromiseRejectionWarning: 
    Error: Response body is unavailable for redirect responses at NetworkManager._handleRequestRedirect

It would be nice to have a way to handle a redirect with either passing an onRedirect-handler to the scraper oprions or handle it internally inside the scraper by starting a scrape on the redirected page.

Offcourse it also will make sense to be able to abort the scraping for certain unsafe redirects.

Figure out way to present coverage data

The @siteimprove/alfa-test package currently collects coverage data using V8's builtin coverage profiler. This proved to be the most performant and unobtrusive way of collecting coverage data compared to e.g. nyc. What we need now is a way of presenting this coverage data in a way that ideally integrates well with our existing workflow. Let's start out small and see if we can find a way of visualising what is missing similar to how we only report on failing tests. Creative solutions are welcome!

Implement support for An+B microsyntax

Pseudo-class selectors currently only support values that are either selectors or lists of selectors:

readonly value: Selector | Array<Selector> | null;

However, some functional pseudo-class selectors make use of the An+B microsyntax (https://www.w3.org/TR/css-syntax/#anb), for example :nth-child(2n+1). To support these kinds of selectors, we first need to implement support for this syntax within the selector parser. The lexer can already lex them:

test("Can lex an+b values", t => {
css(t, "2n+4", [
{
type: TokenType.Dimension,
value: 2,
integer: true,
unit: "n"
},
{
type: TokenType.Number,
value: 4,
integer: true
}
]);
});

Implement incremental builds

In continuation of #77 (comment), it would be great to have yarn prepare perform incremental builds either by default or based on a flag. We should by now have the necessary tools in place for implementing incremental builds and my plan for doing it is loosely as follows:

  • Use Graph and Project#resolveImports() for incrementally constructing a dependency graph.
  • Use Cache to store the version number of every TypeScript source file.
    • If the version does not change between builds and no dependencies of the file have changed, skip the file.
    • Otherwise, build the file, mark it as changed, update its dependencies in the Graph, and store its version in the Cache.

Implement typed JSON-LD model and associated APIs

Specifically, we need support for the following operations:

⚠️ Compaction and flattening have been de-scoped for now.

We might be able to make use of jsonld.js, though a few things about that project have me concerned. First and foremost, all their APIs are asynchronous with no synchronous alternative. As we will mostly concern ourselves with outgoing JSON-LD, i.e. data generated by Alfa, we will also operate mostly on local contexts. For this purpose, synchronous APIs are preferred as everything else is already synchronous.

Coverage warnings are missing for completely untested files

When running yarn test, the build system notifies the builder about any files with a coverage lower than 90%. Unfortunately, it completely ignores files that are entirely untested, as it never invokes the coverage suggestion tool. The aforementioned gives an incorrect overview of the coverage and should be corrected.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.