Testing

This repository provides Robot Framework tests for the images, and a Robot Framework library which helps to write environment independent tests.

The core interfaces of the test library are:

  • Image: Abstraction for building images.
  • Power: Abstraction for power state management.
  • Comm: Abstraction for image interaction.

Image

The Image class provides a generic interface for building images. To implement this, it makes use of an image_interface in the background. The image_interface provides implementations of the Build and Clear keywords for a specific build framework like our taskfiles.

A example using the Image class is:

Clear    ${path}    task mrproper
${image}=    Build    ${path}

The first line Clear the build folder, i.e. removes previous build artifacts to ensure a clean build. It makes use of a specific command, task mrproper, instead of relying on the defaults. The task mrproper also deletes all local caches, to simulate a clean workspace and ensures that also the download of all used packages works.

The Build keyword runs the image build, relying on defaults. The path parameter gives the subfolder containing the image specification. The Image class also provides an init parameter to set a base path for all images.

Power

The Power class provides a generic interface for running images. To implement this, it makes use of an power_interface in the background. The power_interface provides implementations of the Power On and Power Off keywords. The power_qemu implementation implements these keywords for running images in QEMU.

A example using the Power class to run an image:

${process}=    Power On    ${image}

The Power On keyword gets a path to an image file, and runs this image using QEMU. To do this, it makes use of the task run_qemu command. This command is executed in a fresh shell, and the shell is the returned process. This process can be used afterwards with Comm to interact with the running image.

Hardware interfaces may use setup specific commands to power on an attached hardware, and should provide a process wrapping a serial connection session.

The Power Off keyword shall turn of the hardware. In case of QEMU, it kills the process. In case of a real hardware, it should do a power cut.

Comm

The Comm class provides a generic interface for interacting with the running image, and is the base for easy transferable and hardware independent tests. In the background, it makes use of the ProcIO class to send commands and collect the outputs. The Connect keyword attaches to the process providing the serial interface, and the Disconnect keyword detaches from the process and stops it. The test keywords provided by Comm are:

  • Login To Vm: This keyword automates the login to a getty session. It’s typically used after Power On and Connect.
  • Execute: This keyword runs the given command and provides the output generated by the command. It detects the end of the output by using an echo command, so it requires a shell session and a shell environment which has the echo command.
  • Wait For Line: This keyword reads the lines appearing at the serial terminal, and search for the given substring in the lines. If a match is found, all lines until the match, including the matching line, is returned. It can be used to wait for a given log signalling an expected system state.
  • Wait for Regex: This keyword is like Wait For Line, but searches using a regular expression. This may be necessary if the searched log is more complex.

These keywords build on low level keywords which should not be needed for tests. These low level keywords are:

  • Send Message: Sends a message to the serial terminal. A return (“\n”) is automatically added.
  • Send Key: Sends a key to the serial terminal.
  • Read Line: Reads the next output line from the serial terminal.

Util

The Util class provides useful more complex keywords implemented in Python. These keywords are:

  • Filter Lines Containing: This keyword filters lines containing a given string from a string containing multiple lines. It’s used to filter the startup and init manager logs for known lines which would be detected as issue.

Performance

See Performance.

Other classes

The test library contains some more helper classes. These classes are not intended for usage in tests and the interface may change over time.

  • ProcIO: Helper class for interfacing with a running subprocess. It runs threads to read and tag the lines generated by the process, and provides methods for sending messages to the process.
  • Fakeroot: Fakeroot offers methods to run commands on the test host environment.