Red Hat Enterprise Linux and Secure Boot in a cloud environment

Share it

Secure Startup technology is a component of the Unified Extensible Firmware Interface (UEFI) specification. It serves as an effective and robust tool that can heighten the boot time security of an operating system by permitting only verified code to execute on that system. While this technology has been part of the UEFI specification since version 2.0 (2006) and is widely utilized by x86 hardware vendors, its integration into cloud environments has been a more recent development:

Google introduced Shielded VMs to the public in April 2019
Microsoft announced the general availability of Trusted Launch in November 2021
AWS added support for SecureBoot in May 2022
Red Hat Enterprise Linux has backed the Secure Startup technology since the RHEL 7 release in June 2014

This article delves into the process of activating and configuring Secure Startup for various types of Red Fedora Enterprise Linux (RHEL) images in AWS, Google Cloud, and Microsoft Azure.

Advantages of Utilizing Secure Startup

Enabling Secure Startup technology enhances the boot time security of an operating system. It ensures that the bootloader and kernel of the operating system are endorsed by a trusted vendor. Additionally, the RHEL kernel operates in a more secure “lockdown” mode by verifying that all subsequently loaded kernel modules are also validated by a trusted vendor.

Secure Startup also plays a pivotal role in the implementation of Confidential Virtual Machines (CVM). The proper implementation of CVM mandates that only trustworthy components are present in the boot chain. Typically, a combination of Secure Startup and Trusted Boot technology is employed for this purpose.

Comprehending the Secure Startup Process in RHEL

Secure Startup technology relies on public key cryptography and comprises the following core components:

Platform Key (PK): A self-signed key that establishes trust between your platform and the firmware of the platform
Key Exchange Key (KEK): Utilized to build trust between your operating system and the platform’s firmware. These keys are employed to update the Allowed Signature database (‘db’) and Forbidden Signature database (‘dbx’)
Allowed Signature database (‘db’): A compilation of certificates or binary hashes that determine if a particular binary is allowed to boot on the system. Furthermore, all certificates from ‘db’ are imported into the kernel “.platform” keyring, enabling the use of signed third-party kernel modules in the kernel lockdown mode
Forbidden Signature database (‘dbx’): A catalog of ‘bad’ certificates or binary hashes that are prohibited from booting on the system

When RHEL boots on a cloud instance with Secure Startup enabled, the following sequence of events transpires:

Upon VM initiation, it commences booting system firmware provided by the cloud provider
The firmware initializes UEFI variables from a variable store. During the initial boot of the VM, the store is initialized from the default values of the VM image
When booting, the firmware endeavors to load the operating system boot loader. In the case of an RHEL VM in an x86 UEFI environment, the initial bootloader is ‘shim’
The Shim binary in RHEL is authenticated by Microsoft’s “Microsoft Corporation UEFI CA 2011,” enabling RHEL to boot with Secure Startup enabled across various hardware and virtualized platforms where the Allowed Signature database (‘db’) is pre-populated with default Microsoft certificates
Shim supplements the trusted certificates list with Red Fedora Secure Startup CA, which can be located in the ‘redhat-sb-certs’ package in the Red Fedora CodeReady Linux Builder repository
Optionally, Shim can incorporate Machine Owner Key (‘MOK’) into the trusted certificates list. This option is commonly employed with bare-metal platforms where modifying Secure Startup variables necessitates collaboration with the OEM vendor

Following this, the boot path differs based on whether the traditional or Unified Kernel Image (UKI) layout is employed. In the case of booting the traditional RHEL kernel:

Shim loads the GRUB bootloader binary signed by the relevant certificate (e.g., ‘Red Fedora Secure Startup Signing 502’ on the x86_64 architecture, found in the ‘redhat-sb-certs’ package). This certificate, signed by Red Fedora Secure Startup CA, successfully verifies the check
GRUB loads the RHEL kernel signed by the corresponding certificate (e.g., ‘Red Fedora Secure Startup Signing 501’ on the x86_64 architecture, found in the ‘redhat-sb-certs’ package). This certificate, endorsed by Red Fedora Secure Startup CA, successfully verifies the check

Alternatively, in the case of RHEL UKI:

Shim directly loads RHEL UKI (‘kernel-uki-virt’ package). The UKI is validated by the appropriate certificate (e.g., ‘Red Fedora Secure Startup Signing 504’ on the x86_64 architecture, found in the ‘redhat-sb-certs’ package) endorsed by Red Fedora Secure Startup CA
If UKI command line extensions are present, their signatures are authenticated against the Allowed Signature database (‘db’) and Machine Owner Key (‘MOK’) before their usage. This provision allows for the use of UKI extensions endorsed by the operating system vendor (Red Fedora) and the end-user

When the RHEL kernel boots in Secure Startup mode, it enters the ‘lockdown’ mode, implying that all kernel modules must be authenticated by a trusted key. The modules shipped with the RHEL kernel (‘kernel-modules-core’, ‘kernel-modules’, ‘kernel-modules-extra’) are signed with an ephemeral key generated with each kernel build. Subsequently, the private facet of the key is discarded, ensuring that no other module can be authenticated with this key. Upon booting, the RHEL kernel supplements the kernel keyring with this ephemeral key, in addition to a few other keys employed for endorsing third-party modules. The certificates from the Secure Startup Allowed Signature database (‘db’) and Machine Owner Key (‘MOK’) are also considered trustworthy and can be utilized for module authentication.

AWS

At present, AWS supports Secure Startup for Nitro-based instances. The feature can be employed for both Marketplace Amazon Machine Images (AMIs) and custom RHEL AMIs.

Marketplace RHEL Images

With the releases of 9.3 and 8.9, the boot settings of RHEL Marketplace AMIs were modified to ‘UEFI Preferred,’ enabling the activation of Secure Startup and Nitro vTPM features. The configuration of a RHEL 9.4 AMI in ‘us-east-2’ would appear as follows:

$ aws ec2 describe-images --image-id ami-099f85fc24d27c2a7 --region us-east-2 | grep -E '"ImageId"|"Name"|"BootMode"'
 
 "ImageId": "ami-099f85fc24d27c2a7",
 "Name": "RHEL-9.4.0_HVM-20240423-x86_64-62-Hourly2-GP3",
 "BootMode": "uefi-preferred",

Upon creating an instance using a RHEL Marketplace AMI, it boots into setup mode, facilitating the update of Secure Startup UEFI variables from within the instance:

# mokutil --sb-state
SecureBoot disabled
Platform is in Setup Mode

To activate Secure Startup and add a custom signing certificate to ‘db,’ it is advisable not to generate and store private keys on the target instance. Keeping Secure Startup secrets on the same instance where they are utilized could expose the secrets to intruders, jeopardizing data security. Initially, the PK and KEK keys need to be generated using tools like ‘openssl’ and ‘python3-virt-firmware’:

$ uuidgen --random > GUID.txt
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform key/" -outform DER -out PK.cer
$ virt-fw-sigdb --add-cert "$(< GUID.txt)" PK.cer -o PK.esl
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -outform DER -out KEK.cer
$ virt-fw-sigdb --add-cert "$(< GUID.txt)" KEK.cer -o KEK.esl

A custom certificate is then generated, and Microsoft’s “Microsoft Corporation UEFI CA 2011” certificate is added to the Allowed Signature database (‘db’):

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform DER -out custom_db.cer
$ virt-fw-sigdb --add-cert "$(< GUID.txt)" custom_db.cer -o custom_db.esl
$ virt-fw-sigdb --add-cert 77fa9abd-0359-4d32-bd60-28f4e78f784b MicCorUEFCA2011_2011-06-27.crt -o ms_db.esl
$ cat custom_db.esl ms_db.esl > db.esl

The current list of forbidden signatures (‘dbx’) can be obtained from the UEFI Forum. Subsequently, PK.esl, KEK.esl, db.esl, and x64_DBXUpdate.bin are transferred to the target instance, and the variables are configured using the ‘efivar’ package:

# efivar -w -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-PK -f PK.esl
# efivar -w -n 8be4df61-93ca-11d2-aa0d-00e098032b8c-KEK -f KEK.esl
# efivar -w -n d719b2cb-3d3a-4596-a3bc-dad00e67656f-db -f db.esl
# efivar -w -n d719b2cb-3d3a-4596-a3bc-dad00e67656f-dbx -f x64_DBXUpdate.bin

Following a reboot of the instance, Secure Startup can be confirmed to be enabled:

# mokutil --sb-state
SecureBoot enabled

The integration of the custom certificate into the kernel keyring can be verified using the ‘keyctl’ tool from the ‘keyutils’ package:

# keyctl list %:.platform
4 keys in keyring:
136539426: ---lswrv     0     0 asymmetric: Signature Database key: f064979641c24e1b935e402bdbc3d5c4672a1acc
231557747: ---lswrv     0     0 asymmetric: Red Fedora Secure Startup CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
939747660: ---lswrv     0     0 asymmetric: Red Fedora Secure Startup CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
1040805986: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

Custom RHEL Images

Upon registration of a custom RHEL AMI, pre-filled UEFI Secure Startup variables can be configured. This approach ensures that all instances launched from the AMI will have Secure Startup enabled with preset variables even during the initial boot. Like the previous example, new PK/KEK keys need to be generated:

$ uuidgen --random > GUID.txt
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform key/" -outform DER -out PK.cer
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -outform DER -out KEK.cer

Microsoft’s ‘Microsoft Corporation UEFI CA 2011’ certificate and a custom certificate are utilized to populate the Allowed Signature database (‘db’):

$ openssl req -quiet -newkey rsa:4096 -nodes -keyout custom_db.key -new -x509 -sha256 -days 3650 -subj "/CN=Signature Database key/" --outform DER -out custom_db.cer

By leveraging the ‘virt-fw-vars’ tool from the ‘python3-virt-firmware’ package, the necessary blob can be generated:

$ virt-fw-vars --output-aws aws_blob.bin --set-pk "$(< GUID.txt)" PK.cer --add-kek "$(< GUID.txt)" KEK.cer --add-db "$(< GUID.txt)" custom_db.cer --add-db 77fa9abd-0359-4d32-bd60-28f4e78f784b MicCorUEFCA2011_2011-06-27.crt --set-dbx x64_DBXUpdate.bin

The AMI can then be registered with the desired Secure Startup variables using the ‘awscli2’ package. This is achieved by creating the AMI from a disk snapshot:

$ aws ec2 register-image --name rhel9-test-ami --architecture x86_64 --virtualization-type hvm --root-device-name "/dev/sda1" --block-device-mappings "{"DeviceName": "/dev/sda1","Ebs": {"SnapshotId": "snap-fffffffffffffffff"}}" --ena-support --boot-mode uefi --region eu-central-1 --uefi-data $(cat aws_blob.bin)

Instances created from this AMI will boot with Secure Startup enabled and the expected Secure Startup certificates propagated to the kernel keyring:

# mokutil --sb-state
SecureBoot enabled
# keyctl list %:.platform
4 keys in keyring:
907254483: ---lswrv     0     0 asymmetric: Signature Database key: f064979641c24e1b935e402bdbc3d5c4672a1acc
203975646: ---lswrv     0     0 asymmetric: Red Fedora Secure Startup CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
890038886: ---lswrv     0     0 asymmetric: Red Fedora Secure Startup CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
990786053: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4

Google Cloud

Google Cloud endorses Secure Startup with Shielded VMs. When creating an instance through the UI, the ‘Secure Boot’ setting can be pinpointed under ‘Security’:

Publicly available RHEL images can be initiated in a Secure Startup enabled state. By default, this incorporates the Microsoft certificates into the Allowed Signature database (‘db’):

# mokutil --sb-state
SecureBoot enabled
# keyctl list %:.platform
4 keys in keyring:
312311590: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
700176698: ---lswrv     0     0 asymmetric: Red Fedora Secure Startup CA 8: e1c6c580aa1e21d585aad9bf20f3929e5ec1f08b
 65600894: ---lswrv     0     0 asymmetric: Red Fedora Secure Startup CA 5: cc6fa5e72868ba494e939bbd680b9144769a9f8f
509209133: ---lswrv     0     0 asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53

During the registration of a new image version, custom Secure Startup variables can be specified. Similar to the AWS example, new PK/KEK keys must be produced:

$ uuidgen --random > GUID.txt
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 -subj "/CN=Platform key/" -outform DER -out PK.cer
$ openssl req -quiet -newkey rsa:4096 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 -subj "/CN=Key Exchange Key/" -outform DER -out KEK.cer

Microsoft’s ‘Microsoft Corporation UEFI CA 2011