Here at DornerWorks, we have been working on ways to help our customers quickly and easily leverage the open source Xen hypervisor for their embedded products. As a quick recap, a hypervisor is software that’s used to create virtual machines, allowing entire software stacks, including operating systems, to run on the same hardware. One of the problems with having a number of virtual machines is that you usually don’t have enough peripheral I/O devices, like UARTs, to go around. Luckily, Xen can provide a virtual console to each of those guests running on a VM so the user can interact with them. For ARM based platforms this means that guest running on the VM has to be para-virtualized, or modified, in order to use Xen’s virtual console. DornerWorks has developed a library that allows a bare metal application, one running without an operating system, or FreeRTOS applications to use Xen’s virtual console, providing serial connections to any number of virtual machines.
The diagrams below show how Xen’s virtual console works. When a Xen wants to send a character to a virtual machine, such as when an operator enters a command while the guest is “in focus,” the hypervisor puts the character at the end of its outgoing, which is the guest’s incoming, ring buffer, and raises an event to the guest. I’m not going to get into the details of Xen’s event framework, but suffice to say an event is multiplexed on top of a real interrupt, which for ARM is IRQ-31. The guest running in the virtual machine checks for the event, clears flags in shares with Xen for coordinating the event, and if the event ID indicates the virtual console, takes the character out of the buffer and makes it available to be used by other parts of the guest software.
When the guest wants to send a character to the Xen to be printed to the screen for the operator to see, the guest first adds the character to its outgoing, which is Xen’s incoming, buffer. It then makes a hypercall with registers set to request a virtual console write operation. The hypercall is like a service call, only handled by the hypervisor, running at exception level (EL) 2, instead of an operating system running at EL 1. When Xen handles a hypercall, it looks to see what operation was requested, and if it was for a console write operation, takes the characters in the shared ring buffer and handles them appropriately, which would be printing them to the screen using the real serial connection if the guest was currently “in focus”, via the xl console command.
That’s it; simple, right?
Well, in order to make use of the virtual console, a guest does have to do a few things to make it work. This is an example of para-virtualizing the guest, or modifying it to work with virtualizing, which is a pretty common approach with Xen on ARM. This is also why the Xen Virtual Console is sometimes referred to as the para-virtualized console, or PV console.
The DornerWorks library to enable guest virtual console takes care of most of those things for you. The first thing the library does is get the event channel ID of the console operation. This ID is needed for the guest to distinguish the console event from the other possible events when Xen raises an event interrupt to the guest. The guest requests this ID through a hypercall to Xen.
The second thing our library does for the guest is to set up a shared page of memory with Xen. This is needed as part of the Xen event interface. This shared page allows mutual access of various flags which coordinates event handling between the guest and the hypervisor. By sharing this page of memory, Xen and the guest can use special ARM commands to ensure atomic accesses to those flags, preventing any issues with a loss of data coherency. The guest picks a 4KB page to share with Xen, figures out the page number for the intermediate physical address (IPA) for that page, and passes that page number to Xen by means of another hypercall. Xen, knowing how the IPA maps to a physical address (PA), then sets up the necessary MMU table entries so it can also access the same underlying location in memory.
Once the event framework has been handled, the guest, through our library, obtains the memory address of the console ring buffer. This is another page of shared memory, but this time Xen will tell the guest what page to use. The guest obtains the IPA page number from Xen via another hypercall. Once the page is obtained, if the guest is using virtual address translation, an entry has to be added to the MMU to set up a virtual address that the guest can use to access that page. Our library expects the guest to be using using the XZD Bare Metal Container (BMC), which does take advantage of virtual address translation, so our library takes care of mapping that page for the guest.
At this point, the guest is ready to send characters to Xen. The guest does this by simply populating the guest’s outgoing, which is Xen’s incoming, ring buffer with one or more characters and then raising the console write event by means of the appropriate hypercall. This step is handled by DornerWorks’ virtual console library via the printk function.
Finally, to receive characters from Xen, the guest needs to set up an interrupt handler for the event interrupt, which is IRQ31 on the ARM version of Xen. This interrupt handler is where the flags from the shared memory set up in step 2 are checked and cleared. It bears repeating that access to these flags has to be done using special atomic operations to avoid data coherency issues. As mentioned previously, Xen will populate its outgoing, which is the guest’s incoming, ring buffer with characters, and then raise the interrupt to the guest. The guest’s interrupt handler will check for the event ID, and if for the console event, will then proceed empty the ring buffer into a local string buffer. The guest can register a function with the library that will be called with the local string buffer when the guest’s incoming ring buffer has been fully emptied, allowing the guest to handle the incoming data as it sees fit. By default, if the guest does not register a callback function, the library will echo the characters back to the Xen console, via printk, and can be configured to echo to a passed through UART via printf.
Our virtual console library provides the handler for IRQ31, but does not connect it up to the interrupt handling framework because that tends to vary based on whether or not you are using FreeRTOS or Xilinx’s “Standalone” library for board support. We do provide examples of how to setup the FreeRTOS in our Virtuosity distribution, and examples of how to setup the interrupt using the “Standalone” board support library can be found in the comments of “virt_console.h” of our library.
Once you have the virtual console support enabled in your guest, you can make your guest to be “in focus” by starting it up with the ‘-c’ flag, or `xl create –c <your_guest>.cfg`, or by giving it focus from dom0 by issuing `xl console <your_guest>`. To return to dom0 enter CTRL-J.
As always, you can download the Virtuosity for free, here.
DOWNLOAD VIRTUOSITY
Happy virtualizing!
