Visual Inspection: Transfer of GitHub Workflows & Enhancement of Examinations

Visual inspection is the evaluation of the visual aspect of a user interface. Screenshots of pages are captured to establish a “baseline,” or the current expectation of how each page should be displayed. Proposed modifications are then compared against this baseline. Any snapshots that deviate from the baseline are marked for further examination.

For instance, consider these two illustrations:

To an untrained observer, these images may appear quite similar. You might notice that they are different, but not pinpoint exactly how or what code alterations might have triggered that change.

Let’s now examine how Percy (a visual testing service by Browserstack) interprets this comparison:

The discrepancies between the snapshots are highlighted in vivid red. As we scroll down the page, we observe that the text and horizontal lines in the two images drift further apart vertically. This indicates a potential issue with content spacing, possibly in element height, margin, or padding. Visual testing aids in identifying and preventing visual regressions, empowering maintainers to incorporate visual changes in a controlled manner.

Our Use Scenario

Canonical’s Web team manages Vanilla Framework, a component library utilized by web assets across Canonical’s entire web presence. Alterations made to Vanilla impact all our downstream projects. Visual testing is crucial as it helps us avert visual regressions before they impact our users, guaranteeing that our applications appear and function precisely as intended.

Vanilla’s website showcases numerous usage examples, serving to illustrate our design and provide developers with confidence to construct websites using our examples as templates. We deploy Percy to assess each of these examples with every pull request. This enables us to monitor visual alterations to our design system with certainty.

The Challenge

Initially, Vanilla employed CircleCI for visual testing. CircleCI was chosen for its capability to execute workflows for fork pull requests with access to secret tokens, a capability not supported by GitHub actions. CircleCI retrieved the source code from each pull request, merged it into the source to ensure testing of the latest changes, and built, executed Vanilla, and conducted visual testing – all within a single workflow.

In our previous workflow, every pull request (irrespective of file changes or readiness state) was tested with every commit. Pull requests are frequently updated several times during the review process, leading to a cascade of tests with each synchronization event. We recursively captured every Vanilla component example markup file in our project, resulting in an accumulation of screenshots. Each test entailed around 1,400 screenshots, translating to thousands of screenshots for every PR, often pushing us beyond our screenshot quota.

Enhancement Objectives

• Transition from CircleCI to GitHub Actions for uniformity with our other CI workflows
• Reduction of the total number of tests conducted
• Reduction of the total number of screenshots captured per test

Our Resolution

Migration to GitHub Actions

In alignment with a strategy endorsed by the GitHub security team, we began employing a two-workflow approach to capture Percy snapshots and mitigate security hazards from executing workflows on forked pull requests.

Prepare Workflow

A “prepare” workflow triggers on every pull request push. This workflow uploads styling, example templates, and pull request metadata into a GitHub Actions artifact, which is subsequently picked up by the subsequent workflow. By solely utilizing the template and styling from the pull request, we curtail the quantity of pull request code utilized by the snapshot workflow.

“`yaml
name: “Prepare Percy build”
# Workflow steps and configurations detailed below in code snippet
“`

Snapshot Workflow

A “snapshot” workflow runs once the prepare workflow is completed, utilizing GitHub’s trigger for workflow runs. This workflow checks out the current version of main alongside the styling, templates, and PR metadata from the prepare workflow artifact. It replaces the styling and templates from main with the pull request code, builds, executes, and tests Vanilla using Percy.

To maintain Percy’s baseline, we devised a composite action to execute Vanilla’s build, run, and visual testing uniformly. The decision to create a composite action was to allow the building/testing code to be invoked as a step by other workflows without code duplication.

“`yaml
name: Percy Snapshot – composite/reusable action
# Workflow details and configurations detailed below in code snippet
“`

Subsequently, we established a workflow running tests on proposed modifications from pull requests utilizing the artifact from the pull request prepare workflow.

“`yaml
# Workflow details and configurations for snapshot action detailed below in code snippet
“`

Baseline Workflow

Visual testing hinges on comparing proposed changes against an approved baseline test outcome. We designate the main branch as the endorsed version of our source. Consequently, every push to main triggers a Percy test. Subsequently, the results of this test shape the new baseline until the subsequent push to main.

“`yaml
name: Update Percy Baseline
# Workflow details and configurations for baseline update detailed below in code snippet
“`

Minimizing Percy Usage

Enhancing Test Selectivity

To curtail the total number of tests conducted, we commenced running our visual tests more discerningly. We pinpointed the subsequent scenarios as a fundamental collection of scenarios necessitating visual testing on pull requests:

• Changes in styling and/or template files in non-draft pull requests to main
• Changes in any pull request to main with the “Review: Percy needed” label. This label is applied to manually request Percy testing, regardless of whether a PR modifies styling or template files.
• Inclusion of the “Review: Percy needed” label on a pull request to main

The paths specifier is employed to filter pull requests not modifying styling or template files. Nonetheless, when merging directory filtering with label filtering, we encountered a quandary. The paths filter acts akin to a logical AND at the workflow level, hence necessitating two distinct workflows: a prepare-label workflow and a prepare-push workflow. Both workflows summon a reusable workflow executing the test preparation logic, with different filters dictating job initiation.

The label workflow addresses pull requests specifically earmarked for Percy testing.

“`yaml
# Workflow details and configurations for label-based workflow detailed below in code snippet
“`

The push workflow caters to pull requests altering files pertinent for visual testing while ignoring draft pull requests.

“`yaml
# Workflow details and configurations for push-based workflow detailed below in code snippet
“`

Decreasing Snapshot Volume

Capturing each example as a distinct view led to a colossal number of screenshots per test, even when a component was as modest as 200x100px. To lessen the number of screenshots captured per test, we consolidated new “combined” examples of each component, exhibiting the contents of every component variant within a singular view.

Below is a snapshot of the base button example before the snapshot reduction initiative. This page was captured twice (once on mobile and once on desktop), similarly, the remaining button variant examples were also captured twice each, resulting in numerous screenshots just for button examples.

All button variant examples were merged into a single view, aiding in capturing all button variants in a single view, eluding dozens of screenshots for buttons alone.

By aggregating these examples into a single test instance, we can evaluate significantly more examples per screenshot, diminishing our overall screenshot consumption.

Result

We substantially reduced our total Percy usage and expedited our visual tests. We conduct visual testing towards the conclusion of a pull request’s scrutiny process, contrary to once per PR push. By combining our examples into broader views, we tangibly decreased our screenshots per test from ~1,400 to ~800 and decreased Percy’s build duration from ~4 minutes to ~2 minutes.

We could have further reduced our screenshot usage, however, the substantial decline in screenshots per test enabled us to broaden our test coverage by capturing every example in each color scheme, a task that would have demanded an unsustainable number of screenshots per test previously. By running Percy more selectively and combining our example test cases into larger views, we managed to reduce our test consumption while broadening our test coverage.

Future Strategies

• Slash the screenshot usage of our baseline tests by only updating the baseline when files altering Vanilla’s visual appearance are modified. This necessitates meticulous consideration of the files triggering a test, aiming to prevent discrepancies in Vanilla’s visual appearance from our main branch.
• Transfer our React Components library’s visual testing integration from CircleCI to GitHub Actions.

Source:
https://ubuntu.com//blog/visual-testing-github-actions-migration-test-optimisation