Fedora has leveraged the benefits of a well-established, inclusive, and sturdy openQA system for conducting release validation tests and testing updates. A recent initiative expanded Fedora’s openQA infrastructure by integrating containers and cloud resources. By deploying openQA on the cloud, Fedora now safeguards against physical hardware failures and streamlines the process of adding resources to cater to future scalability needs. Moreover, transitioning to a cloud-based deployment model has made it more convenient for individuals to operate, test, and explore Fedora’s openQA platform. Those familiar with openQA can explore cloud deployment samples at: https://pagure.io/ansible-openqa-cloud.
Fedora’s openQA versus the original openQA project
Developers interested in extending, debugging, or experimenting with Fedora’s openQA should first comprehend how Fedora’s deployment aligns with the upstream project. The primary upstream repositories include openQA, which manages the web user interface, websockets, livehandler, scheduler, a PostgreSQL database, and various background tasks, and os-autoinst, the backend responsible for executing tests and reporting the outcomes.
Fedora depends on these upstream repositories and integrates them into the openqa and os-autoinst packages, respectively. Upon installing these packages, users can locate the upstream libraries at: /usr/share/openqa/lib/OpenQA and the upstream scripts at: /usr/share/openqa/script. The new cloud deployment grants access to these libraries and scripts within the openqa-webserver container.
Similarly, the backend code of os-autoinst can be found inside any instance of the openqa-worker container at: /usr/lib/os-autoinst. This allows users to delve into how os-autoinst launches QEMU virtual machines for test runs.
The newly-integrated openqa-database container permits access to the PostgreSQL database. Users can leverage the psql command to conveniently inspect and modify their local database instances.
Specific tests for Fedora
What sets Fedora, and all distributions utilizing openQA, apart is their repository of tests designed for running on operating system images. Fedora’s tests are stored in the os-autoinst-distri-fedora repository. While the test content is unique, openQA mandates that tests reside in a designated “tests” directory: /var/lib/openqa/share/tests/fedora.
Both the new openqa-webserver and openqa-worker containers contain these test repositories. In the cloud deployment, a small service named openqa-test-update runs every six hours to fetch any modifications to the tests. Users have the option to test their own fork of os-autoinst-distri-fedora by halting the test update service and pulling alterations from a forked repository instead.
Acquiring and Scheduling images for testing
Fedora’s openQA deployment offers a unique method for scheduling operating system images for testing. At a basic level, users can manually add images to the directory where openQA and the job settings anticipate finding them: either /var/lib/openqa/share/factory/iso or /var/lib/openqa/share/factory/hdd.
After downloading an ISO and placing it correctly, users can schedule a test using the openqa-cli tool. This automated approach simplifies the testing process and eliminates the need for manual scheduling for each image.
For more intricate scenarios like testing updates on existing OS images, Fedora utilizes a customized application called createhdds. This tool generates base disk images using the host’s kernel image and applies incoming updates for testing purposes.
The cloud deployment project faced challenges with cloud instances lacking nested virtualization capabilities, hindering the efficient operation of openqa-worker and createhdds. Ensuring access to /dev/kvm on cloud instances is crucial for optimal performance of openQA components.
In conclusion, the transition to a cloud-based deployment model has unlocked new possibilities for Fedora’s openQA platform, enabling greater flexibility and scalability. Future plans include migrating official Fedora openQA instances to this deployment methodology, with the ultimate goal of phasing out dedicated openQA hardware clusters. Special thanks to Meta for sponsoring this initiative and to Collabora for its implementation efforts.
