Skip to content

GitLab

Last edited by Renaud Pacalet
Page history

secure boot

Secure Boot with SecBus

In order to guarantee that nobody can tamper with the external memories of a SoC it is not sufficient to be able to protect a running OS and its applications. The complete boot sequence, from the very first executed instruction, must also be protected. In this page we study the boot sequence of a Linux kernel on the Xilinx [Zynq] cores and especially on the ZedBoard prototyping board, our favourite demonstration platform. Zynq cores embed a ARM processor and all its usual peripherals (USB, Ethernet, flash...) including external memory controllers. This part forms what is named Processing System (PS) in Xilinx terminology. Zynq cores also embed an FPGA matrix, the Programmable Logic (PL). The ARM processor can interact with the PL using two dedicated 1GB address spaces and the PL can access the external memories.

The AXI simple bridge is the perfect prototype to experiment about the secure boot. Of course, as it does not implement any kind of protection, it cannot really secure anything. But as it can easily be replaced by the SecBus Hardware Security Module (HSM), it is good enough. Our goal is to avoid all accesses to the external memory, up to the point where the HSM and its control data structures could have been properly and securely initialized and the first accesses could be protected.

In the following we assume that you have a complete SecBus distribution installed and that you also have a working Xilinx SDK and Vivado suite (only version 2014.4 has been tested). You will also need a clone of the Xilinx git repositories of U-Boot, the Linux kernel and the device tree sources, plus a copy of the Xilinx ramdisk image. In the following we assume that they are installed in /opt/xlnx and that U-Boot and the Linux kernel have been built in /opt/xlnx/u-boot-xlnx/build and /opt/xlnx/linux-xlnx/build, respectively:

mkdir -p /opt/xlnx
cd /opt/xlnx
git clone http://github.com/Xilinx/u-boot-xlnx.git
git clone http://github.com/Xilinx/linux-xlnx.git
git clone http://github.com/Xilinx/device-tree-xlnx.git
wget http://www.wiki.xilinx.com/file/view/arm_ramdisk.image.gz
export CROSS_COMPILE=arm-xilinx-linux-gnueabi-
cd u-boot-xlnx
make O=build zynq_zed_defconfig
make O=build menuconfig (see the note below)
make O=build
cd ../linux-xlnx
make ARCH=arm O=build xilinx_zynq_defconfig
make ARCH=arm O=build menuconfig
make ARCH=arm O=build zImage

Note: when building U-Boot you may get error messages about missing headers. This can be caused by the missing RSA library. A workaround consists in configuring U-Boot such that it does not use it. When editing the configuration menu disable the signature verification of FIT uImages:

Boot images  --->
  Enable signature verification of FIT uImages --->
    No

and then, disable the use of the RSA library:

Library routines  --->
  Use RSA Library --->
    No

BootROM code

The first code executed by the Zynq PS is stored in an internal BootROM and runs entirely in an On-Chip Memory (OCM). It is thus protected in the SecBus threat model. Its role is to initialize the platform, identify the boot device (SD card, flash, JTAG...) defined by a set of on-board jumpers, find a boot image on the boot device, extract from it the First Stage Boot Loader (FSBL), load it in OCM and branch to the FSBL. The boot image typically also contains a PL bitstream and some user code like a Second Stage Boot loader (SSBL) or a user application.

In our experiments the boot image is stored in a SD card but it could come from any other source, this would make no difference in terms of security, except for the JTAG boot mode where all security features are disabled. Our boot image contains a FSBL, a bitstream to configure the PL with the AXI bridge and a U-Boot binary that plays the role of SSBL.

The components of the boot image can be encrypted (AES 256) and can carry a cryptographic header for authentication (RSA 2048 and SHA-256). The root of trust can be either the on-chip e-Fuses or the Battery-Backed RAM or a combination of the two. It is thus possible to detect that a secure boot image has been tampered with and to completely prevent its use. It is also possible, thanks to the embedded e-Fuses, to lock a device to a particular encrypted and authenticated boot image, either definitively or with a secure update mechanism. This very complete set of security-related features is documented in the "Zynq-7000 AP SoC Technical Reference Manual" (TRM) by Xilinx.

The First Stage Boot Loader (FSBL)

First change your working directory to the AXI simple bridge and synthesize the bridge:

cd secbus/vhdl/hsm/src/axi_simple_bridge
make axi_simple_bridge.vsyn

Wait until the synthesis completes. The results are stored in the axi_bridge_simple.vv-syn sub-directory. Create the FSBL:

make fsbl

The axi_simple_bridge.vv-syn/top.sdk/fsbl directory should have been created, populated with the default FSBL source code, and the FSBL should have been compiled. The resulting fsbl.elf ELF can be found in axi_simple_bridge.vv-syn/top.sdk/fsbl/executable.elf (and a copy named fsbl.elf in the AXI bridge directory) but this one does not yet have the behaviour we need for the secure boot.

The FSBL, after the optional authentication and decryption by the BootROM code, is stored in the OCM and executed. In a Secbus-equipped system the FSBL would first initialize the HSM, create some Security Policies (SP) and Page Security Parameters Entries (PSPE) and start using the DDR only once it is properly protected in confidentiality and/or integrity by the HSM. In the context of the AXI bridge experiments this is replaced by the simple fact that the FSBL only accesses the DDR through the AAS, never through the RAS.

In order to do this we must instruct the FSBL to load the SSBL in the AAS instead of the default RAS. By default the load address and the entry point of the SSBL are both 0x0400_0000 (in the RAS). This is easily changed by editing the boot.bif file used by the bootgen tool:

image:
{
  [bootloader]fsbl.elf
  bitstream.bit
  [load=0x84000000,startup=0x84000000]u-boot.elf
}

This is not sufficient for two reasons:

  • The default FSBL checks the load address and raises an error if it does not fall in the [DDR_START_ADDR,DDR_END_ADDR] range (see the LoadBootImage function in the image_mover.c of the FSBL).
  • The default FSBL first configures the PL with the bistream contained in the boot image but it sets the PS-PL level shifters and enables the PS-PL AXI interfaces only after loading the SSBL. Without a modification it would thus freeze when trying to load the SSBL in the AAS because of the disabled PS-PL interface.

Note: before patching the FSBL let us first add the -DFSBL_DEBUG_INFO flag to the CFLAGS make variable so that the FSBL prints detailed debugging information during its execution. Edit axi_simple_bridge.vv-syn/top.sdk/fsbl/Makefile and change:

CFLAGS :

to:

CFLAGS := -DFSBL_DEBUG_INFO

A working patch for the first problem consists in changing the following code in axi_simple_bridge.vv-syn/top.sdk/fsbl/image_mover.c:

if (PSPartitionFlag && (PartitionLoadAddr > DDR_END_ADDR)) {
  fsbl_printf(DEBUG_GENERAL,
      "INVALID_LOAD_ADDRESS_FAIL\r\n");
  OutputStatus(INVALID_LOAD_ADDRESS_FAIL);
  FsblFallback();
}

to:

u32 PatchedPartitionLoadAddr = PartitionLoadAddr;
if (BitstreamFlag) {
  PatchedPartitionLoadAddr &= ~0x80000000;
}
if (PSPartitionFlag && (PatchedPartitionLoadAddr > DDR_END_ADDR)) {
  fsbl_printf(DEBUG_GENERAL,
      "INVALID_LOAD_ADDRESS_FAIL\r\n");
  OutputStatus(INVALID_LOAD_ADDRESS_FAIL);
  FsblFallback();
}

The second problem (late activation of the PS-PL interface) can be solved with the FsblHookAfterBitstreamDload function (in axi_simple_bridge.vv-syn/top.sdk/fsbl/fsbl_hooks.c). As its name says it is called after the bistream has been used to configure the PL and before anything else. By default it simply prints a debug message. The code that sets the PS-PL level shifters and enables the PS-PL AXI interfaces can be found in the FsblHandoff function of axi_simple_bridge.vv-syn/top.sdk/fsbl/main.c. Copying it unmodified in the FsblHookAfterBitstreamDload function definition of axi_simple_bridge.vv-syn/top.sdk/fsbl/fsbl_hooks.c is sufficient. Change:

u32 FsblHookAfterBitstreamDload(void)
{
  u32 Status;

  Status = XST_SUCCESS;

  /*
   * User logic to be added here.
   * Errors to be stored in the status variable and returned
   */
  fsbl_printf(DEBUG_INFO, "In FsblHookAfterBitstreamDload function \r\n");

  return (Status);
}

to:

u32 FsblHookAfterBitstreamDload(void)
{
  u32 Status;

  Status = XST_SUCCESS;

  /*
   * User logic to be added here.
   * Errors to be stored in the status variable and returned
   */
  fsbl_printf(DEBUG_INFO, "In FsblHookAfterBitstreamDload function \r\n");

  extern u8 BitstreamFlag;
  /*
   * Enable level shifter
   */
  if(BitstreamFlag) {
    /*
     * FSBL will not enable the level shifters for a NON PS instantiated
     * Bitstream
     * CR# 671028
     * This flag can be set during compilation for a NON PS instantiated
     * bitstream
     */
#ifndef NON_PS_INSTANTIATED_BITSTREAM
#ifdef PS7_POST_CONFIG
    ps7_post_config();
    /*
     * Unlock SLCR for SLCR register write
     */
    SlcrUnlock();
#else
  /*
   * Set Level Shifters DT618760
   */
  Xil_Out32(PS_LVL_SHFTR_EN, LVL_PL_PS);
  fsbl_printf(DEBUG_INFO,"Enabling Level Shifters PL to PS "
      "Address = 0x%x Value = 0x%x \n\r",
      PS_LVL_SHFTR_EN, Xil_In32(PS_LVL_SHFTR_EN));

  /*
   * Enable AXI interface
   */
  Xil_Out32(FPGA_RESET_REG, 0);
  fsbl_printf(DEBUG_INFO,"AXI Interface enabled \n\r");
  fsbl_printf(DEBUG_INFO, "FPGA Reset Register "
      "Address = 0x%x , Value = 0x%x \r\n",
      FPGA_RESET_REG ,Xil_In32(FPGA_RESET_REG));
#endif
#endif
  }

  return (Status);
}

The equivalent code in the FsblHandoff function of axi_simple_bridge.vv-syn/top.sdk/fsbl/main.c can be disabled by enclosing it between a

#if 0
...
#endif

pair, but apparently this is not absolutely necessary and running it twice does not seem to harm. So far so good, if the AXI bridge was replaced by the SecBus HSM, if the FSBL was configuring it properly before loading the SSBL, and if the FSBL was authenticating the loaded SSBL image before jumping into it, we could claim that the boot sequence is secure, up to this point.

The main difficulties for the next steps are to configure the software components (U-Boot, Linux kernel) such that they use only the AAS, never the RAS. This is more tricky than it seems and the very first thing to do is understanding the memory mapping in a Zynq core.

Memory mapping

The simplest way to run a software stack with all memory accesses routed through the PL is to configure, compile and link it such that it uses the [0x8000_0000, 0x0xa000_0000[ address range (AAS) instead of [0x0000_0000, 0x4000_0000[ (RAS). And the simplest way to achieve this is by shifting all memory addresses by 0x8000_0000. If the original software would have accessed a memory location at address A in the RAS, the modified one accesses A+0x8000_0000 in AAS instead. The access is thus routed to the PL through the AXI_GP1 port. The AXI bridge in the PL shifts the address back to the RAS and forwards the access to the AXI_HP0 port that performs the access to DDR on behalf of the CPU and routes the responses back to the AXI_GP1 port. If the original software was running fine in the RAS, the modified one should thus run fine in the AAS. In a perfect world accesses that originally fell in the OCM would also fall in the OCM after modification and the same would hold for the DDR. The only difference should be a slow down due to the lower clock frequency of the PL and the lower performance of the AXI_GP1 port compared to the original direct interfaces between the CPU and the DDR controller. Let us name this desirable property the "perfect one-to-one mapping".

Unfortunately, things are a bit more complex because of the Zynq's sophisticated, configurable, memory mapping. The memory mapping depends on several configuration registers of the Zynq core (see chapter 4 "System Addresses" of the TRM for more information):

  • The 256 kB OCM can be mapped in the low or high address range by sections of 64 kB. The four Least Significant Bits (LSBs) of the slcr.OCM_CFG register (address 0xf800_0910) specify where the corresponding quarter of the OCM is mapped. If slcr.OCM_CFG[0], for instance, is set, the first 64 kB of the OCM are mapped high ([0xfffc_0000, 0xfffd_0000[), else they are mapped low ([0x0, 0x0001_0000[). Quarters mapped low hide the corresponding portion of DDR that becomes inaccessible.
  • The mapping of several memory areas depends on whether they are filtered by the Snoop Control Unit (SCU) or not. They can be mapped to the DDR, to the OCM or unmapped. The two LSBs of the SCU_CONTROL_REGISTER (address 0xf8f0_0000) are used to enable / disable the SCU and the address filtering. The Filtering_Start_Address_Register (address 0xf8f0_0040) and the Filtering_End_Address_Register (address 0xf8f0_0044) define the filtered address range, with a 1 MB granularity.
  • These mappings are not necessarily the same for the CPU/ACP, the AXI_HP ports or the other masters.

This last point is the most important because a memory location in RAS that was accessed by the original software (directly from CPU) could fall in OCM or in DDR but, from the AXI_HP0 port, it can fall in a different type of memory (DDR instead of OCM and conversely) or even be unmapped.

The default configuration (both after U-Boot booted and after the Linux kernel booted) is the following:

  • The complete OCM is mapped high ([0xfffc_0000, 0xffffffff]),
  • the SCU in enabled,
  • the filtering by the SCU is enabled,
  • and the filtered area is [0x0000_0000, 0xffe0_0000].

In this configuration the [0x0000_0000, 0x4000_0000[ range has the same mapping from the CPU/ACP and AXI_HP viewpoints... except for the [0x0000_0000, 0x0008_0000[ range which is mapped in the DDR for the CPU/ACP while it is unmapped for the AXI_HP. The above mentioned "perfect one-to-one mapping" property does not hold. This can be tested easily from U-Boot (to avoid interferences from the processor caches it is better to reset between two experiments). The following U-Boot log shows that accessing the RAS at address 0x0 works fine while accessing the same memory location in the AAS through the PL (address 0x8000_0000) raise a data abort error, probably because the AXI_HP0 port responds with a SLVERR or DECERR:

zynq-uboot> md.l 0x0 1
00000000: 62d59c45                               E..b
zynq-uboot> md.l 0x80000000 1
80000000:data abort
pc : [<1ff806a8>]          lr : [<1ff80688>]
reloc pc : [<0403f6a8>]    lr : [<0403f688>]
sp : 1f320d00  ip : 80000000     fp : 00000008
r10: 80000000  r9 : 1f320ef8     r8 : 80000000
r7 : 00000001  r6 : 00000001     r5 : 00000004  r4 : 00000004
r3 : 00000000  r2 : 00000802     r1 : 1f320d14  r0 : 00000009
Flags: nZCv  IRQs off  FIQs off  Mode SVC_32
Resetting CPU ...

resetting ...

Changing the SCU filtering start address solves the issue because the [0x0000_0000, 0x0008_0000[ range becomes unmapped for both masters and also because U-Boot does not use it (except if asked explicitly with a memory access command):

zynq-uboot> mw.l 0xf8f00040 0x00100000
zynq-uboot> md.l 0x0 1
00000000:data abort
pc : [<1ff816c0>]          lr : [<1ff56090>]
reloc pc : [<040406c0>]    lr : [<04015090>]
sp : 1f320448  ip : 00000000     fp : 00000008
r10: 00000000  r9 : 1f320ef8     r8 : 00000000
r7 : 1ff96f7d  r6 : 00000001     r5 : 1f3204c8  r4 : 00000001
r3 : 00000000  r2 : 1f320cf4     r1 : 1ff96f7d  r0 : 1f3204c8
Flags: nzCv  IRQs off  FIQs off  Mode SVC_32
Resetting CPU ...

resetting ...

Note: the SCU filtering start address must be a multiple of 1 MB, reason why we set it to 0x0010_0000 instead of 0x0008_0000.

Note: footnote (3) of table 4.1 ("System-Level Address Map") in the TRM indicates that, in this configuration, the [0x000c_0000, 0x0010_0000[ range is an alias of the OCM for the CPU/ACP but not for the other masters. This means that address 0x000c_0000 is the same as address 0xfffc_0000 but not as address 0x800c_0000:

zynq-uboot> md.l 0xf8f00040 2
f8f00040: 00000000 ffe00000                      ........
zynq-uboot> md.l 0x000c0000 8
000c0000: 22d59c44 36002208 024001a0 49824c59    D..".".6..@.YL.I
000c0010: 8cc31450 04a82100 1c082209 471b5146    P....!..."..FQ.G
zynq-uboot> md.l 0xfffc0000 8
fffc0000: ea00003d ea000025 ea000028 ea000035    =...%...(...5...
fffc0010: ea00002f e320f000 ea000000 ea00000f    /..... .........
zynq-uboot> mw.l 0xf8f00040 0x00100000
zynq-uboot> md.l 0x000c0000 8
000c0000: ea00003d ea000025 ea000028 ea000035    =...%...(...5...
000c0010: ea00002f e320f000 ea000000 ea00000f    /..... .........
zynq-uboot> md.l 0x800c0000 8
800c0000: 22d59c44 36002208 024001a0 49824c59    D..".".6..@.YL.I
800c0010: 8cc31450 04a82100 1c082209 471b5146    P....!..."..FQ.G

This could lead to problems in case the original software uses this [0x000c_0000, 0x0010_0000[ aliased address range because the aliasing would not work any more after the address shift by 0x8000_0000. When transforming the original software we should completely forbid the use of the lowest MB of memory ([0x0000_0000, 0x0010_0000[ in RAS, [0x80000000, 0x80100000[ in AAS).

Same experiments with the Linux kernel:

  1. Before changing the SCU filtering start address:

    zynq> devmem 0x0 32
0xE59F0000
zynq> devmem 0x80000000 32
Unhandled fault: external abort on non-linefetch (0x1018) at 0xb6fe9000
pgd = d5b4c000
[b6fe9000] *pgd=9e9db831, *pte=80000783, *ppte=80000e33
Bus error

  2. Changing the SCU filtering start address:

    zynq> devmem 0xf8f00040 32 0x00100000
zynq> devmem 0x0 32
Unhandled fault: external abort on non-linefetch (0x1018) at 0xb6f58000
pgd = d6058000
[b6f58000] *pgd=9ea22831, *pte=00000783, *ppte=00000e33
Bus error

This works because the Linux kernel and the device tree blob provided with the AXI bridge SDCard archive are configured such that the Linux kernel does not make use of the now forbidden first MB.

In fact, when the Linux kernel discovers its memory layout, it assumes that its usable memory is aligned on a 128 MB boundary. Changing this is possible but out of scope of this secure boot study. In the provided SDCard archive and in the following, we will thus restrict the usable memory to the [0x8800_0000, 0xa000_0000[ portion of the AAS (384 MB only), skipping the first 128 MB of the 512 MB ZedBord DDR.

U-Boot

To be done.

Linux Kernel

Note: things is sometimes easier when Linux runs with the L1 and L2 caches disabled. If you which to do so, please see the Disabling Zynq Caches section.

Booting linux in the AAS first requires modification of the load address of the Linux kernel U-Boot image (uImage). When building the Linux kernel image for U-Boot with the mkimage U-Boot tool, use 0x8800_8000 as load address and entry point, instead of the classical 0x0000_8000:

make ARCH=arm UIMAGE_LOADADDR=0x88008000 uImage

The Makefiles and TCL scripts provided in the SecBus distribution can do this for you automatically:

cd secbus/vhdl/hsm/src/axi_simple_bridge
make UILA=0x88008000 uimage

Next, the device tree must also be adapted. Usable memory must be moved above 0x8800_0000. This can be done by changing the device tree source. First create the device tree sources:

cd secbus/vhdl/hsm/src/axi_simple_bridge
make dts

The axi_simple_bridge.vv-syn/top.sdk/dts directory should have been created, populated with the default device tree sources. The memory entry must be set as follows in axi_simple_bridge.vv-syn/top.sdk/dts/system.dts:

memory {
		device_type = "memory";
		linux,usable-memory = <0x88000000 0x18000000>;
	};

Then, compile the device tree:

/opt/xlnx/linux-xlnx/build/scripts/dtc/dtc -I dts -O dtb -o devicetree.dtb \
axi_simple_bridge.vv-syn/top.sdk/dts/system.dts

The provided Makefiles and TCL scripts can do all this automatically for you:

make KNBA=0x88000000 KNMS=0x18000000 dtb

Instead of modifying the original device tree source file, this creates a copy named axi_simple_bridge.vv-syn/top.sdk/dts/system.dts.edited, modifies it and compiles it with the Linux device tree compiler. The resulting device tree blob is secbus/vhdl/hsm/src/axi_bridge/devicetree.dtb.

Finally, the parameters U-Boot passes to the kernel must be adapted. The U-Boot environment variable fdt_high must be set to 0xffff_ffff to prevent the device tree from being copied to another location during boot. Same with initrd_high for the ramdisk image:

zynq-uboot> setenv fdt_high 0xffffffff
zynq-uboot> setenv initrd_high 0xffffffff

The Linux kernel, ramdisk, and device tree blob must be loaded in the AAS and U-Boot must jump into the kernel using the AAS addresses:

zynq-uboot> fatload mmc 0 0x8b000000 uImage
zynq-uboot> fatload mmc 0 0x8c000000 uramdisk.image.gz
zynq-uboot> fatload mmc 0 0x8aa00000 devicetree.dtb
zynq-uboot> bootm 0x8b000000 0x8a000000 0x8aa00000

Again, the uEnv.txt file provided with the SDCard archive does this for you by setting automatically several U-Boot environment variables.