In the first part of this series, we talked about the risks involved with sharing a DMA-capable device between multiple VMs on an embedded system. Then we configured our FPGA design to preserve AXI stream IDs to differentiate transactions intended for different PL devices. In this part, we will configure seL4 so that it makes use of those stream IDs to passthrough devices to the correct VMs. While this blog is focused on seL4, the basic modifications should be the same for any hypervisor that uses device trees to provide system and virtual machine device information (e.g. Xen).
First, replace the system device tree with the updated device tree from the last part. For seL4, this can be done by replacing the kernel’s built-in platform device tree at tools/dts/zynqmp.dts. (For Xen, this can be done by simply loading the new device tree binary with u-boot. However additional changes are needed to the device tree to disable devices to be passed through and indicate to the hypervisor that the device will be used for passthrough by adding the “xen,passthrough” property.) This will make the hypervisor aware of the additional PL devices and ensures that they have the proper SMMU configuration.
Next, decide on what PL devices you would like each VM to have access to. There are several pieces of information that we need from the device tree for each of those devices – its address, its interrupts, and its stream IDs. For example, this Ethernet device uses interrupt lines 0x59, 0x5a, 0x5b, and 0x5c and is located at the address 0xa0000000:
axi_ethernet_0: ethernet@a0000000 {
axistream-connected = < 0x18 >;
axistream-control-connected = < 0x18 >;
clock-frequency = < 0x5f5e100 >;
clock-names = "s_axi_lite_clk", "axis_clk", "gtx_clk", "ref_clk";
clocks = < 0x03 0x47 >, < 0x03 0x47 >, < 0x19 >, < 0x1a >;
compatible = "xlnx,axi-ethernet-7.2", "xlnx,axi-ethernet-1.00.a";
device_type = "network";
interrupt-names = "mac_irq", "interrupt", "mm2s_introut", "s2mm_introut";
interrupt-parent = < 0x03 >;
interrupts = < 0x00 0x59 0x01 0x00 0x5a 0x04 0x00 0x5b 0x04 0x00 0x5c 0x04 >;
local-mac-address = [00 0a 35 00 00 00];
phy-mode = "rgmii";
reg = < 0x00 0xa0000000 0x00 0x40000 >;
xlnx = < 0x00 >;
xlnx,axiliteclkrate = < 0x00 >;
xlnx,axisclkrate = < 0x00 >;
xlnx,channel-ids = < 0x01 >;
xlnx,clockselection = < 0x00 >;
xlnx,enableasyncsgmii = < 0x00 >;
xlnx,gt-type = < 0x00 >;
xlnx,gtinex = < 0x00 >;
xlnx,gtlocation = < 0x00 >;
xlnx,gtrefclksrc = < 0x00 >;
xlnx,include-dre;
xlnx,instantiatebitslice0 = < 0x00 >;
xlnx,num-queues = /bits/ 16 < 0x01 >;
xlnx,phy-type = < 0x03 >;
xlnx,phyaddr = < 0x01 >;
xlnx,phyrst-board-interface-dummy-port = < 0x00 >;
xlnx,rable = < 0x00 >;
xlnx,rxcsum = < 0x00 >;
xlnx,rxlane0-placement = < 0x00 >;
xlnx,rxlane1-placement = < 0x00 >;
xlnx,rxmem = < 0x1000 >;
xlnx,rxnibblebitslice0used = < 0x00 >;
xlnx,tx-in-upper-nibble = < 0x01 >;
xlnx,txcsum = < 0x00 >;
xlnx,txlane0-placement = < 0x00 >;
xlnx,txlane1-placement = < 0x00 >;
axi_ethernet_0_mdio: mdio {
#address-cells = < 0x01 >;
#size-cells = < 0x00 >;
};
};
Once you have this information for each device, the final thing to do is to pass the devices through to the VM. For seL4, this requires modifications to the CAmkES application. First add the DMA and Ethernet device to the DTB for the VM that should have access to the Ethernet device:
vm0.dtb = dtb([
{"path": "/dma@a0040000"},
{"path": "/ethernet@a0000000"}
]);
Then, pass through the IRQs for the Ethernet device. To get the correct values, take the IRQ lines from the device tree and add 32 to those values to account for the SPI offset:
vm0.dtb_irqs = [121, 122, 123, 124];
Using this information, the seL4 build system will create a partial device tree for the guest with the passed through device information. (With Xen this partial device tree will need to be manually constructed and provided as part of the VM configuration file.)
Finally, add an SMMU map with the Stream IDs we added to the device tree:
vm0.smmu_map = [
{"sid": 0xE80, "cb": 0},
{"sid": 0xE81, "cb": 1},
{"sid": 0xE82, "cb": 2}
];
And that’s it! VM0 will now be able to access the AXI Ethernet device. Since the EthernetFMC design has three additional Ethernet ports, you could pass through each port to a different VM. This would enable you to use the same AXI interconnect with multiple VMs, while also maintaining security of the transactions such that different VMs cannot sniff out transactions being made by the other, corrupt other VM’s memory space, or interact with other VM’s Ethernet devices.
The full source code for this example consists of a VM with port 0 on the EthernetFMC passed through to it.
Providing secure separation between system components can be tricky when custom logic is involved, but if your secure embedded system utilizes programmable logic, it is essential that the interconnect and operating environment be properly configured to prevent accidental or malicious threats. If you need any help securing your SoC-based embedded system, schedule a meeting with our team and turn your product ideas into reality!
