[v5,05/14] spapr: nested: Document Nested PAPR API

Commit Message

Harsh Prateek Bora March 8, 2024, 11:19 a.m. UTC

Adding initial documentation about Nested PAPR API to describe the set
of APIs and its usage. Also talks about the Guest State Buffer elements
and it's format which is used between L0/L1 to communicate L2 state.

Signed-off-by: Michael Neuling <[e-mail]>
Signed-off-by: Harsh Prateek Bora <[e-mail]>
---
 docs/devel/nested-papr.txt | 119 +++++++++++++++++++++++++++++++++++++
 1 file changed, 119 insertions(+)
 create mode 100644 docs/devel/nested-papr.txt

Comments

Nicholas Piggin March 12, 2024, 11:30 a.m. UTC | #1

On Fri Mar 8, 2024 at 9:19 PM AEST, Harsh Prateek Bora wrote:
> Adding initial documentation about Nested PAPR API to describe the set
> of APIs and its usage. Also talks about the Guest State Buffer elements
> and it's format which is used between L0/L1 to communicate L2 state.
>
> Signed-off-by: Michael Neuling <[e-mail]>
> Signed-off-by: Harsh Prateek Bora <[e-mail]>

Reviewed-by: Nicholas Piggin <[e-mail]>

> ---
>  docs/devel/nested-papr.txt | 119 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 119 insertions(+)
>  create mode 100644 docs/devel/nested-papr.txt
>
> diff --git a/docs/devel/nested-papr.txt b/docs/devel/nested-papr.txt
> new file mode 100644
> index 0000000000..90943650db
> --- /dev/null
> +++ b/docs/devel/nested-papr.txt
> @@ -0,0 +1,119 @@
> +Nested PAPR API (aka KVM on PowerVM)
> +====================================
> +
> +This API aims at providing support to enable nested virtualization with
> +KVM on PowerVM. While the existing support for nested KVM on PowerNV was
> +introduced with cap-nested-hv option, however, with a slight design change,
> +to enable this on papr/pseries, a new cap-nested-papr option is added. eg:
> +
> +  qemu-system-ppc64 -cpu POWER10 -machine pseries,cap-nested-papr=true ...
> +
> +Work by:
> +    Michael Neuling <[e-mail]>
> +    Vaibhav Jain <[e-mail]>
> +    Jordan Niethe <[e-mail]>
> +    Harsh Prateek Bora <[e-mail]>
> +    Shivaprasad G Bhat <[e-mail]>
> +    Kautuk Consul <[e-mail]>
> +
> +Below taken from the kernel documentation:
> +
> +Introduction
> +============
> +
> +This document explains how a guest operating system can act as a
> +hypervisor and run nested guests through the use of hypercalls, if the
> +hypervisor has implemented them. The terms L0, L1, and L2 are used to
> +refer to different software entities. L0 is the hypervisor mode entity
> +that would normally be called the "host" or "hypervisor". L1 is a
> +guest virtual machine that is directly run under L0 and is initiated
> +and controlled by L0. L2 is a guest virtual machine that is initiated
> +and controlled by L1 acting as a hypervisor. A significant design change
> +wrt existing API is that now the entire L2 state is maintained within L0.
> +
> +Existing Nested-HV API
> +======================
> +
> +Linux/KVM has had support for Nesting as an L0 or L1 since 2018
> +
> +The L0 code was added::
> +
> +   commit 8e3f5fc1045dc49fd175b978c5457f5f51e7a2ce
> +   Author: Paul Mackerras <[e-mail]>
> +   Date:   Mon Oct 8 16:31:03 2018 +1100
> +   KVM: PPC: Book3S HV: Framework and hcall stubs for nested virtualization
> +
> +The L1 code was added::
> +
> +   commit 360cae313702cdd0b90f82c261a8302fecef030a
> +   Author: Paul Mackerras <[e-mail]>
> +   Date:   Mon Oct 8 16:31:04 2018 +1100
> +   KVM: PPC: Book3S HV: Nested guest entry via hypercall
> +
> +This API works primarily using a signal hcall h_enter_nested(). This
> +call made by the L1 to tell the L0 to start an L2 vCPU with the given
> +state. The L0 then starts this L2 and runs until an L2 exit condition
> +is reached. Once the L2 exits, the state of the L2 is given back to
> +the L1 by the L0. The full L2 vCPU state is always transferred from
> +and to L1 when the L2 is run. The L0 doesn't keep any state on the L2
> +vCPU (except in the short sequence in the L0 on L1 -> L2 entry and L2
> +-> L1 exit).
> +
> +The only state kept by the L0 is the partition table. The L1 registers
> +it's partition table using the h_set_partition_table() hcall. All
> +other state held by the L0 about the L2s is cached state (such as
> +shadow page tables).
> +
> +The L1 may run any L2 or vCPU without first informing the L0. It
> +simply starts the vCPU using h_enter_nested(). The creation of L2s and
> +vCPUs is done implicitly whenever h_enter_nested() is called.
> +
> +In this document, we call this existing API the v1 API.
> +
> +New PAPR API
> +===============
> +
> +The new PAPR API changes from the v1 API such that the creating L2 and
> +associated vCPUs is explicit. In this document, we call this the v2
> +API.
> +
> +h_enter_nested() is replaced with H_GUEST_VCPU_RUN().  Before this can
> +be called the L1 must explicitly create the L2 using h_guest_create()
> +and any associated vCPUs() created with h_guest_create_vCPU(). Getting
> +and setting vCPU state can also be performed using h_guest_{g|s}et
> +hcall.
> +
> +The basic execution flow is for an L1 to create an L2, run it, and
> +delete it is:
> +
> +- L1 and L0 negotiate capabilities with H_GUEST_{G,S}ET_CAPABILITIES()
> +  (normally at L1 boot time).
> +
> +- L1 requests the L0 to create an L2 with H_GUEST_CREATE() and receives a token
> +
> +- L1 requests the L0 to create an L2 vCPU with H_GUEST_CREATE_VCPU()
> +
> +- L1 and L0 communicate the vCPU state using the H_GUEST_{G,S}ET() hcall
> +
> +- L1 requests the L0 to run the vCPU using H_GUEST_RUN_VCPU() hcall
> +
> +- L1 deletes L2 with H_GUEST_DELETE()
> +
> +For more details, please refer:
> +
> +[1] Linux Kernel documentation (upstream documentation commit):
> +
> +commit 476652297f94a2e5e5ef29e734b0da37ade94110
> +Author: Michael Neuling <[e-mail]>
> +Date:   Thu Sep 14 13:06:00 2023 +1000
> +
> +    docs: powerpc: Document nested KVM on POWER
> +
> +    Document support for nested KVM on POWER using the existing API as well
> +    as the new PAPR API. This includes the new HCALL interface and how it
> +    used by KVM.
> +
> +    Signed-off-by: Michael Neuling <[e-mail]>
> +    Signed-off-by: Jordan Niethe <[e-mail]>
> +    Signed-off-by: Michael Ellerman <[e-mail]>
> +    Link: https://msgid.link/[e-mail]

Patch

diff --git a/docs/devel/nested-papr.txt b/docs/devel/nested-papr.txt
new file mode 100644
index 0000000000..90943650db
--- /dev/null
+++ b/docs/devel/nested-papr.txt
@@ -0,0 +1,119 @@ 
+Nested PAPR API (aka KVM on PowerVM)
+====================================
+
+This API aims at providing support to enable nested virtualization with
+KVM on PowerVM. While the existing support for nested KVM on PowerNV was
+introduced with cap-nested-hv option, however, with a slight design change,
+to enable this on papr/pseries, a new cap-nested-papr option is added. eg:
+
+  qemu-system-ppc64 -cpu POWER10 -machine pseries,cap-nested-papr=true ...
+
+Work by:
+    Michael Neuling <[e-mail]>
+    Vaibhav Jain <[e-mail]>
+    Jordan Niethe <[e-mail]>
+    Harsh Prateek Bora <[e-mail]>
+    Shivaprasad G Bhat <[e-mail]>
+    Kautuk Consul <[e-mail]>
+
+Below taken from the kernel documentation:
+
+Introduction
+============
+
+This document explains how a guest operating system can act as a
+hypervisor and run nested guests through the use of hypercalls, if the
+hypervisor has implemented them. The terms L0, L1, and L2 are used to
+refer to different software entities. L0 is the hypervisor mode entity
+that would normally be called the "host" or "hypervisor". L1 is a
+guest virtual machine that is directly run under L0 and is initiated
+and controlled by L0. L2 is a guest virtual machine that is initiated
+and controlled by L1 acting as a hypervisor. A significant design change
+wrt existing API is that now the entire L2 state is maintained within L0.
+
+Existing Nested-HV API
+======================
+
+Linux/KVM has had support for Nesting as an L0 or L1 since 2018
+
+The L0 code was added::
+
+   commit 8e3f5fc1045dc49fd175b978c5457f5f51e7a2ce
+   Author: Paul Mackerras <[e-mail]>
+   Date:   Mon Oct 8 16:31:03 2018 +1100
+   KVM: PPC: Book3S HV: Framework and hcall stubs for nested virtualization
+
+The L1 code was added::
+
+   commit 360cae313702cdd0b90f82c261a8302fecef030a
+   Author: Paul Mackerras <[e-mail]>
+   Date:   Mon Oct 8 16:31:04 2018 +1100
+   KVM: PPC: Book3S HV: Nested guest entry via hypercall
+
+This API works primarily using a signal hcall h_enter_nested(). This
+call made by the L1 to tell the L0 to start an L2 vCPU with the given
+state. The L0 then starts this L2 and runs until an L2 exit condition
+is reached. Once the L2 exits, the state of the L2 is given back to
+the L1 by the L0. The full L2 vCPU state is always transferred from
+and to L1 when the L2 is run. The L0 doesn't keep any state on the L2
+vCPU (except in the short sequence in the L0 on L1 -> L2 entry and L2
+-> L1 exit).
+
+The only state kept by the L0 is the partition table. The L1 registers
+it's partition table using the h_set_partition_table() hcall. All
+other state held by the L0 about the L2s is cached state (such as
+shadow page tables).
+
+The L1 may run any L2 or vCPU without first informing the L0. It
+simply starts the vCPU using h_enter_nested(). The creation of L2s and
+vCPUs is done implicitly whenever h_enter_nested() is called.
+
+In this document, we call this existing API the v1 API.
+
+New PAPR API
+===============
+
+The new PAPR API changes from the v1 API such that the creating L2 and
+associated vCPUs is explicit. In this document, we call this the v2
+API.
+
+h_enter_nested() is replaced with H_GUEST_VCPU_RUN().  Before this can
+be called the L1 must explicitly create the L2 using h_guest_create()
+and any associated vCPUs() created with h_guest_create_vCPU(). Getting
+and setting vCPU state can also be performed using h_guest_{g|s}et
+hcall.
+
+The basic execution flow is for an L1 to create an L2, run it, and
+delete it is:
+
+- L1 and L0 negotiate capabilities with H_GUEST_{G,S}ET_CAPABILITIES()
+  (normally at L1 boot time).
+
+- L1 requests the L0 to create an L2 with H_GUEST_CREATE() and receives a token
+
+- L1 requests the L0 to create an L2 vCPU with H_GUEST_CREATE_VCPU()
+
+- L1 and L0 communicate the vCPU state using the H_GUEST_{G,S}ET() hcall
+
+- L1 requests the L0 to run the vCPU using H_GUEST_RUN_VCPU() hcall
+
+- L1 deletes L2 with H_GUEST_DELETE()
+
+For more details, please refer:
+
+[1] Linux Kernel documentation (upstream documentation commit):
+
+commit 476652297f94a2e5e5ef29e734b0da37ade94110
+Author: Michael Neuling <[e-mail]>
+Date:   Thu Sep 14 13:06:00 2023 +1000
+
+    docs: powerpc: Document nested KVM on POWER
+
+    Document support for nested KVM on POWER using the existing API as well
+    as the new PAPR API. This includes the new HCALL interface and how it
+    used by KVM.
+
+    Signed-off-by: Michael Neuling <[e-mail]>
+    Signed-off-by: Jordan Niethe <[e-mail]>
+    Signed-off-by: Michael Ellerman <[e-mail]>
+    Link: https://msgid.link/[e-mail]