Dom0less + Argo enablement

Commit Message

Stefano Stabellini Sept. 13, 2021, 10:41 p.m. UTC

Hi all,

This email is for anybody interested in using Argo with Dom0less setups
for domain-to-domain communications.

Argo is a secure VM-to-VM communication mechanism based on hypercalls
[1]. It is a good fit for Dom0less setups because Argo is lightweight
and doesn't make use of PV frontends, backends, and xenstore. Thus, it
is easier to enable in simple Dom0less setups where VMs are booting in
parallel, backends cannot be assumed to be already up and running, and
the goal is just to get two VMs to talk to each others.

Argo makes use of event channels for notifications. Dom0less VMs don't
have event channels support yet, although it is work-in-progress. The
attached patch for Linux (not ready for upstreaming) enables the
necessary event channels initialization.

In addition, you also need to add the following to the DomU device tree
(see xen/arch/arm/domain_build.c:prepare_dtb_domU):

event-channel {
        compatible = "xen,xen-4.15", "xen,xen";
        interrupts = <0x01 0x0f 0xf08>;
        interrupt-parent = <0xfde8>;
};


The final pieces to enable Argo are:
- CONFIG_ARGO in the Xen build
- the Argo Linux kernel module
  https://github.com/OpenXT/linux-xen-argo
- the Yocto recipe was based on
  https://github.com/dozylynx/meta-argo-linux
- pass dom0less_domU to the domU kernel command line


Then you are good to go for using Argo in your Dom0less VMs, e.g.:

From receiver domain: streamTest -domid 5 -port 8022 -file test.txt -receive 
From sender domain: streamTest -domid 4 -port 8022 -file test.txt -send -connect

Many thanks to Alec Kwapis from DornerWorks for all the information and
the patch!

Cheers,

Stefano


[1] https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen

Comments

Christopher Clark Sept. 21, 2021, 8:42 a.m. UTC | #1

On Mon, Sep 13, 2021 at 3:41 PM Stefano Stabellini <[e-mail]>
wrote:

> Hi all,
>
> This email is for anybody interested in using Argo with Dom0less setups
> for domain-to-domain communications.
>
> Argo is a secure VM-to-VM communication mechanism based on hypercalls
> [1]. It is a good fit for Dom0less setups because Argo is lightweight
> and doesn't make use of PV frontends, backends, and xenstore. Thus, it
> is easier to enable in simple Dom0less setups where VMs are booting in
> parallel, backends cannot be assumed to be already up and running, and
> the goal is just to get two VMs to talk to each others.
>

Excellent - I am very happy to see interest in Argo for this


>
> Argo makes use of event channels for notifications. Dom0less VMs don't
> have event channels support yet, although it is work-in-progress. The
> attached patch for Linux (not ready for upstreaming) enables the
> necessary event channels initialization.
>
> In addition, you also need to add the following to the DomU device tree
> (see xen/arch/arm/domain_build.c:prepare_dtb_domU):
>
> event-channel {
>         compatible = "xen,xen-4.15", "xen,xen";
>         interrupts = <0x01 0x0f 0xf08>;
>         interrupt-parent = <0xfde8>;
> };
>
>
> The final pieces to enable Argo are:
> - CONFIG_ARGO in the Xen build
> - the Argo Linux kernel module
>   https://github.com/OpenXT/linux-xen-argo


This is an important pointer. The argo-linux directory in the repo contains
the source for the current best Linux Argo driver, which has been in use
and actively maintained in OpenXT for an extended period. It is derived
from the original XenClient v4v driver -- Nicolas Tsirakis did great work
porting all the OpenXT v4v software over to this Argo driver and the
libargo userspace component when we migrated. Jason Andryuk has recently
just completed some strong modernization work that advances this Argo
driver's use of Linux synchronization primitives.

For the curious, an alternative Linux Argo driver written by Eric Chanudet
is in also the same repository in the vsock-argo directory: it explores a
different approach for Linux to use the Argo primitives under the Vsock
address family to allow socket communication between Xen domains. A third
code base that could be of interest for developing a new Linux driver for
Argo would be a port of the HP uxen hypervisor's v4v driver, which has a
good modular structure, to Argo on Xen. The code for the starting point
(for a fairly adventurous project) is here:
https://github.com/uxen-virt/uxen/tree/ascara/vm-support/linux


> - the Yocto recipe was based on
>   https://github.com/dozylynx/meta-argo-linux


Ack - this Yocto layer shows recipes for the several components involved
with building an Argo-enabled system.


>
> - pass dom0less_domU to the domU kernel command line


>
> Then you are good to go for using Argo in your Dom0less VMs, e.g.:
>
> From receiver domain: streamTest -domid 5 -port 8022 -file test.txt
> -receive
> From sender domain: streamTest -domid 4 -port 8022 -file test.txt -send
> -connect
>
> Many thanks to Alec Kwapis from DornerWorks for all the information and
> the patch!
>

Thanks, Alec, for exploring Argo with dom0less and for developing the patch
-- excellent work! I am certain that it will be helpful.

Christopher



>
> Cheers,
>
> Stefano
>
>
> [1]
> https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen

Patch

From 6486a0d73d3d422f54f8b579a3550160098fcc93 Mon Sep 17 00:00:00 2001
From: Alec Kwapis <[e-mail]>
Date: Wed, 8 Sep 2021 09:01:46 -0400
Subject: [PATCH] Perform Xen Initialization in Dom0less

This patch allows some Xen initialization to occur during the boot process
of the Linux kernel. This already occurs for guests in a Dom0-managed
configuration, however, the xen_domain() API will evaluate to false
for guests in a Dom0less configuration.

Therefore, a Linux kernel command line argument was added ("dom0less_domU")
which signifies that Linux is being booted as a guest in Dom0less. This
allows certain initialization to occur, such as event channels and the shared
info page. This does not include grant table initialization.

This patch is necessary in getting Argo to work in Dom0less, which requires
both Xen event channels for the Argo virtual interrupt, and the shared info page.

Signed-off-by: Alec Kwapis <[e-mail]>
---
 arch/arm/xen/enlighten.c | 25 +++++++++++++++----------
 init/main.c              | 18 ++++++++++++++++++
 2 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c
index dd6804a64f1a..0c216b1ff279 100644
--- a/arch/arm/xen/enlighten.c
+++ b/arch/arm/xen/enlighten.c
@@ -36,6 +36,8 @@ 
 
 #include <linux/mm.h>
 
+extern bool dom0less_domU;
+
 struct start_info _xen_start_info;
 struct start_info *xen_start_info = &_xen_start_info;
 EXPORT_SYMBOL(xen_start_info);
@@ -311,7 +313,7 @@  static int __init xen_guest_init(void)
 	struct shared_info *shared_info_page = NULL;
 	int cpu;
 
-	if (!xen_domain())
+	if (!xen_domain() && !dom0less_domU)
 		return 0;
 
 	if (!acpi_disabled)
@@ -362,16 +364,19 @@  static int __init xen_guest_init(void)
 	for_each_possible_cpu(cpu)
 		per_cpu(xen_vcpu_id, cpu) = cpu;
 
-	xen_auto_xlat_grant_frames.count = gnttab_max_grant_frames();
-	if (xen_xlate_map_ballooned_pages(&xen_auto_xlat_grant_frames.pfn,
-					  &xen_auto_xlat_grant_frames.vaddr,
-					  xen_auto_xlat_grant_frames.count)) {
-		free_percpu(xen_vcpu_info);
-		return -ENOMEM;
+	if (xen_domain() && !dom0less_domU)
+	{
+		xen_auto_xlat_grant_frames.count = gnttab_max_grant_frames();
+		if (xen_xlate_map_ballooned_pages(&xen_auto_xlat_grant_frames.pfn,
+						  &xen_auto_xlat_grant_frames.vaddr,
+						  xen_auto_xlat_grant_frames.count)) {
+			free_percpu(xen_vcpu_info);
+			return -ENOMEM;
+		}
+		gnttab_init();
+		if (!xen_initial_domain())
+			xenbus_probe(NULL);
 	}
-	gnttab_init();
-	if (!xen_initial_domain())
-		xenbus_probe(NULL);
 
 	/*
 	 * Making sure board specific code will not set up ops for
diff --git a/init/main.c b/init/main.c
index 91f6ebb30ef0..917cd60a8997 100644
--- a/init/main.c
+++ b/init/main.c
@@ -149,6 +149,24 @@  static char *ramdisk_execute_command;
 bool static_key_initialized __read_mostly;
 EXPORT_SYMBOL_GPL(static_key_initialized);
 
+/*
+ * If set, this is an indication that Linux will be booted in a Xen dom0less
+ * configuration. This is necessary because the current xen_domain() API evaluates
+ * to false for dom0less domains on ARM, but the Linux kernel still needs to know
+ * that it is being booted in Xen to setup Xen features such as event channels and
+ * the shared info page.
+ */
+bool dom0less_domU = false;
+EXPORT_SYMBOL(dom0less_domU);
+
+static int __init set_dom0less_domU(char *str)
+{
+	dom0less_domU = true;
+	return 1;
+}
+
+__setup("dom0less_domU", set_dom0less_domU);
+
 /*
  * If set, this is an indication to the drivers that reset the underlying
  * device before going ahead with the initialization otherwise driver might
-- 
2.25.1