Using checkbox-mir to validate your snap graphical environment¶

This document explains how to use checkbox-mir to validate the graphical environment (drivers, userspace, snap setup) on your device.

The graphics-coreXX interface¶

To support graphics on Linux on a given piece of hardware, usually a kernel and userspace driver is required. When using snaps, the kernel driver is managed by the operating system, but the userspace driver needs to be available to your application.

The recommended approach for this is using a content snap, through the graphics-coreXX interface. The current iteration is graphics-core22, see here for more information on that setup: The graphics-core22 interface.

Checkbox¶

Checkbox is a test automation software we’re using extensively at Canonical. We chose it as a framework for providing these tests so that it’s easy to run without specific knowledge.

You can find out more about Checkbox on its GitHub page: Checkbox.

checkbox-mir¶

checkbox-mir is a suite of tests maintained by the Mir team that range from validating the DRM setup through to performance testing of Mir servers. It’s provided as a snap package, so on most Linux distributions it’s enough to:

$ sudo snap install checkbox-mir --devmode
checkbox-mir 0+git.7970755 from Canonical Certification Team (ce-certification-qa) installed

The --devmode flag is required for Checkbox to be able to perform the testing.

You can find out more about it provides through the snap info command:

$ snap info checkbox-mir
name:      checkbox-mir
summary:   Checkbox tests for the Mir project
publisher: Canonical Certification Team (ce-certification-qa)
store-url: https://snapcraft.io/checkbox-mir
license:   unset
description: |
  Collection of tests to be run on devices that are part of the mir project
commands:
  - checkbox-mir.checkbox-cli
  - checkbox-mir.graphics
  - checkbox-mir.mir
  - checkbox-mir.shell
  - checkbox-mir.snaps
  - checkbox-mir.test-runner
services:
  checkbox-mir.service: simple, enabled, active
snap-id:      zPeO6SzpA2i39SNFjpm8qAkMPTOMvMxh
tracking:     latest/stable
refresh-date: today at 12:04 CEST
channels:
  latest/stable:    0+git.7970755 2023-07-20 (49) 17MB -
  latest/candidate: ↑                                  
  latest/beta:      ↑                                  
  latest/edge:      0+git.7970755 2023-07-19 (49) 17MB -
installed:          0+git.7970755            (49) 17MB devmode

Prerequisites¶

Depending on the test being ran, there are different requirements for it to start, and the test framework will try and determine if it’s possible to run the test, rather than fail it.

Here’s a list of the different requirements:

graphics-test-tools installed,
mir-test-tools installed,
a graphics-coreXX provider snap installed (by default that’s mesa-core20 or mesa-core22, depending on which track of the above you install), and the above snaps, and checkbox-mir itself connected to it,
to run as a non-privileged user and confirm Mir interactions with the display manager, a local user session needs to be active,

`checkbox-mir.checkbox-cli`¶

checkbox-mir.checkbox-cli is the interactive entry point, in which you can manually select the test plan and tests to run, as well as control some aspects of the run:

 Select test plan
┌───────────────────────────────────────────────────────────────────────┐
│                                                                       │
│    ( ) Mir - smoke and performance testing of Mir itself              │
│    ( ) Snap - verify that the snaps work fine                         │
│    ( ) graphics-core - verify the graphics-core enablement            │
│                                                                       │
│                                                                       │
│                                                                       │
│                                                                       │
│                                                                       │
│                                                                       │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘
 Press <Enter> to continue                                      (H) Help 

See Checkbox documentation for more information on the framework itself and how to interact with it.

The graphics-core test plan¶

The graphics-coreXX test plan attempts to verify the graphics-coreXX enablement by running a handful of utilities verifying a number of aspects of graphics support:

DRM
KMS
eglinfo across GBM, Wayland and X
libVA
VDPAU
Vulkan

 Choose tests to run on your system:
┌───────────────────────────────────────────────────────────────────────┐
│[X] - Uncategorised                                                    │
│[X]    Collect information about connections                           │
│[X]    Enumerate available system executables                          │
│[X]    Dump DRM information for card 1: drm-pci-0000_00_02_0           │
│[X]    Run kmscube to verify GBM-KMS setup for card 1:                 │
│       drm-pci-0000_00_02_0                                            │
│[X]    Run eglinfo to verify EGL setup for the GBM platform            │
│[X]    Run eglinfo to verify EGL setup with a Wayland server           │
│[X]    Run eglinfo to verify EGL setup with a X server                 │
│[X]    Run vainfo to verify libVA setup                                │
│[X]    Run vdpauinfo to verify VDPAU setup                             │
│[X]    Run vkcube to verify Vulkan setup                               │
│                                                                       │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘
 Press (T) to start Testing                                     (H) Help 

The log results from each of these tests should provide insight into the status of the aspect in question. It’s out of scope for this document to discuss the results beyond their success / failure state.

The Mir test plan¶

This test plan runs the smoke and performance tests as found in the mir-test-tools snap. This is what we use to verify updated snaps continue to work across a number of hardware and scenarios.

 Choose tests to run on your system:
┌───────────────────────────────────────────────────────────────────────┐
│[X] - Uncategorised                                                    │
│[X]    Collect information about connections                           │
│[X]    Hardware Manifest                                               │
│[X]    Provide information about logind sessions                       │
│[X]    Run Mir performance test case:                                  │
│       CompositorPerformance.regression_test_1563287                   │
│[X]    Collect the test report from                                    │
│       mir/performance_CompositorPerformance.regression_test_1563287   │
│[X]    Run Mir performance test case: GLMark2Wayland.fullscreen        │
│[X]    Collect the test report from                                    │
│       mir/performance_GLMark2Wayland.fullscreen                       │
│[X]    Run Mir performance test case: GLMark2Wayland.windowed          │
│[X]    Collect the test report from                                    │
│       mir/performance_GLMark2Wayland.windowed                         │
│[X]    Run Mir performance test case: GLMark2Xwayland.fullscreen       │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘
 Press (T) to start Testing                                     (H) Help 

After selecting the tests, there’s another step that we use to skip tests we know fail on a given piece of hardware:

System Manifest:
┌───────────────────────────────────────────────────────────────────────┐
│ Does this machine have this piece of hardware?                        │
│   Run Hosted tests                   ( ) Yes   ( ) No                 │
│   Run Wayland tests                  ( ) Yes   ( ) No                 │
│   Run Xwayland tests                 ( ) Yes   ( ) No                 │
│                                                                       │
│                                                                       │
│                                                                       │
│                                                                       │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘
 Press (T) to start Testing                               Shortcuts: y/n

The Snap test plan¶

This test plan is primarily intended for the Mir team, allowing us to test updates to the snaps we maintain:

 Choose tests to run on your system:
┌───────────────────────────────────────────────────────────────────────┐
│[X] - Uncategorised                                                    │
│[X]    snaps/confined-shell-edge                                       │
│[X]    snaps/egmde-beta                                                │
│[X]    snaps/mir-kiosk-kodi-edge                                       │
│[X]    snaps/mir-kiosk-neverputt-edge                                  │
│[X]    snaps/mir-kiosk-scummvm-edge                                    │
│[X]    snaps/mir-test-tools-20-beta                                    │
│[X]    snaps/mir-test-tools-22-beta                                    │
│[X]    snaps/mircade-beta                                              │
│[X]    snaps/miriway-beta                                              │
│[X]    snaps/ubuntu-frame-osk-20-beta                                  │
│[X]    snaps/ubuntu-frame-osk-22-beta                                  │
│                                                                       │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘
 Press (T) to start Testing                                     (H) Help

Launchers¶

We provide a couple of launchers for automated runs:

checkbox-mir.graphics: runs the graphics-core test plan
checkbox-mir.mir: runs the Mir test plan

If you want to integrate checkbox-mir into your CI, these are likely what you want to run after having set your device up. You may also create a custom launcher, see the Checkbox launchers tutorial.

checkbox-mir.snaps preselects the Snap test plan, but lets you choose which snaps to test.

Running remotely¶

The recommended way to run Checkbox tests is over the network - and using Checkbox itself, rather than a SSH connection (see Checkbox Remote for more information).

To do this, install checkbox-mir and dependencies on the target device as you would normally, but rather than running the tests there, install Checkbox on your host and issue checkbox remote <ip.address>, pointing at the device you want to test.

You’ll see the same user interface, be able to run the tests as you would locally - but from a distance, without impacting the test results by the environment you run them from etc. The test results will also be brought into your host at the end.

We’d love your feedback, please come to https://github.com/MirServer/checkbox-mir and file issues, pull and enhancement requests. This test suite will grow for sure, and we’d love to know if you think things could be better.