| Message ID | 20220928195633.2348848-2-quic_eberman@quicinc.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Drivers for gunyah hypervisor | expand |
On Wed, Sep 28, 2022 at 12:56:20PM -0700, Elliot Berman wrote: > diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst > new file mode 100644 > index 000000000000..959f451caccd > --- /dev/null > +++ b/Documentation/virt/gunyah/index.rst > @@ -0,0 +1,114 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +================= > +Gunyah Hypervisor > +================= > + > +.. toctree:: > + :maxdepth: 1 > + > + message-queue > + > +Gunyah is a Type-1 hypervisor which is independent of any OS kernel, and runs in > +a higher CPU privilege level. It does not depend on any lower-privileged operating system > +for its core functionality. This increases its security and can support a much smaller > +trusted computing base than a Type-2 hypervisor. > + > +Gunyah is an open source hypervisor. The source repo is available at > +https://github.com/quic/gunyah-hypervisor. > + > +Gunyah provides these following features. > + > +- Scheduling: > + > + A scheduler for virtual CPUs (vCPUs) on physical CPUs and enables time-sharing > + of the CPUs. Gunyah supports two models of scheduling: > + > + 1. "Behind the back" scheduling in which Gunyah hypervisor schedules vCPUS on its own > + 2. "Proxy" scheduling in which a delegated VM can donate part of one of its vCPU slice > + to another VM's vCPU via a hypercall. > + > +- Memory Management: > + > + APIs handling memory, abstracted as objects, limiting direct use of physical > + addresses. Memory ownership and usage tracking of all memory under its control. > + Memory partitioning between VMs is a fundamental security feature. > + > +- Interrupt Virtualization: > + > + Uses CPU hardware interrupt virtualization capabilities. Interrupts are handled > + in the hypervisor and routed to the assigned VM. > + > +- Inter-VM Communication: > + > + There are several different mechanisms provided for communicating between VMs. > + > +- Virtual platform: > + > + Architectural devices such as interrupt controllers and CPU timers are directly provided > + by the hypervisor as well as core virtual platform devices and system APIs such as ARM PSCI. > + > +- Device Virtualization: > + > + Para-virtualization of devices is supported using inter-VM communication. > + > +Architectures supported > +======================= > +AArch64 with a GIC > + > +Resources and Capabilities > +========================== > + > +Some services or resources provided by the Gunyah hypervisor are described to a virtual machine by > +capability IDs. For instance, inter-VM communication is performed with doorbells and message queues. > +Gunyah allows access to manipulate that doorbell via the capability ID. These devices are described > +in Linux as a struct gunyah_resource. > + > +High level management of these resources is performed by the resource manager VM. RM informs a > +guest VM about resources it can access through either the device tree or via guest-initiated RPC. > + > +For each virtual machine, Gunyah maintains a table of resources which can be accessed by that VM. > +An entry in this table is called a "capability" and VMs can only access resources via this > +capability table. Hence, virtual Gunyah devices are referenced by a "capability IDs" and not a > +"resource IDs". A VM can have multiple capability IDs mapping to the same resource. If 2 VMs have > +access to the same resource, they may not be using the same capability ID to access that resource > +since the tables are independent per VM. > + > +Resource Manager > +================ > + > +The resource manager (RM) is a privileged application VM supporting the Gunyah Hypervisor. > +It provides policy enforcement aspects of the virtualization system. The resource manager can > +be treated as an extension of the Hypervisor but is separated to its own partition to ensure > +that the hypervisor layer itself remains small and secure and to maintain a separation of policy > +and mechanism in the platform. On arm64, RM runs at NS-EL1 similar to other virtual machines. > + > +Communication with the resource manager from each guest VM happens with message-queue.rst. Details > +about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c > + > +:: > + > + +-------+ +--------+ +--------+ > + | RM | | VM_A | | VM_B | > + +-.-.-.-+ +---.----+ +---.----+ > + | | | | > + +-.-.-----------.------------.----+ > + | | \==========/ | | > + | \========================/ | > + | Gunyah | > + +---------------------------------+ > + > +The source for the resource manager is available at https://github.com/quic/gunyah-resource-manager. > + > +The resource manager provides the following features: > + > +- VM lifecycle management: allocating a VM, starting VMs, destruction of VMs > +- VM access control policy, including memory sharing and lending > +- Interrupt routing configuration > +- Forwarding of system-level events (e.g. VM shutdown) to owner VM > + > +When booting a virtual machine which uses a devicetree, resource manager overlays a > +/hypervisor node. This node can let Linux know it is running as a Gunyah guest VM, > +how to communicate with resource manager, and basic description and capabilities of > +this VM. See Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml for a description > +of this node. The documentation LGTM. > diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst > new file mode 100644 > index 000000000000..e130f124ed52 > --- /dev/null > +++ b/Documentation/virt/gunyah/message-queue.rst > <snipped>... > +The diagram below shows how message queue works. A typical configuration involves > +2 message queues. Message queue 1 allows VM_A to send messages to VM_B. Message > +queue 2 allows VM_B to send messages to VM_A. > + > +1. VM_A sends a message of up to 1024 bytes in length. It raises a hypercall > + with the message to inform the hypervisor to add the message to > + message queue 1's queue. > +2. Gunyah raises the corresponding interrupt for VM_B when any of these happens: > + a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case. > + b. Explicility with gh_msgq_push command from VM_A. > + c. Message queue has reached a threshold depth. > +3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer. > + The nested list above should be separated with blank lines to be rendered properly: ---- >8 ---- diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst index e130f124ed525a..afaad99db215e6 100644 --- a/Documentation/virt/gunyah/message-queue.rst +++ b/Documentation/virt/gunyah/message-queue.rst @@ -20,9 +20,11 @@ queue 2 allows VM_B to send messages to VM_A. with the message to inform the hypervisor to add the message to message queue 1's queue. 2. Gunyah raises the corresponding interrupt for VM_B when any of these happens: + a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case. b. Explicility with gh_msgq_push command from VM_A. c. Message queue has reached a threshold depth. + 3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer. For VM_B to send a message to VM_A, the process is identical, except that hypercalls Thanks.
On 9/28/2022 8:43 PM, Bagas Sanjaya wrote: > On Wed, Sep 28, 2022 at 12:56:20PM -0700, Elliot Berman wrote: >> diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst >> new file mode 100644 >> index 000000000000..959f451caccd >> --- /dev/null >> +++ b/Documentation/virt/gunyah/index.rst >> @@ -0,0 +1,114 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +================= >> +Gunyah Hypervisor >> +================= >> + >> +.. toctree:: >> + :maxdepth: 1 >> + >> + message-queue >> + >> +Gunyah is a Type-1 hypervisor which is independent of any OS kernel, and runs in >> +a higher CPU privilege level. It does not depend on any lower-privileged operating system >> +for its core functionality. This increases its security and can support a much smaller >> +trusted computing base than a Type-2 hypervisor. >> + >> +Gunyah is an open source hypervisor. The source repo is available at >> +https://github.com/quic/gunyah-hypervisor. >> + >> +Gunyah provides these following features. >> + >> +- Scheduling: >> + >> + A scheduler for virtual CPUs (vCPUs) on physical CPUs and enables time-sharing >> + of the CPUs. Gunyah supports two models of scheduling: >> + >> + 1. "Behind the back" scheduling in which Gunyah hypervisor schedules vCPUS on its own >> + 2. "Proxy" scheduling in which a delegated VM can donate part of one of its vCPU slice >> + to another VM's vCPU via a hypercall. >> + >> +- Memory Management: >> + >> + APIs handling memory, abstracted as objects, limiting direct use of physical >> + addresses. Memory ownership and usage tracking of all memory under its control. >> + Memory partitioning between VMs is a fundamental security feature. >> + >> +- Interrupt Virtualization: >> + >> + Uses CPU hardware interrupt virtualization capabilities. Interrupts are handled >> + in the hypervisor and routed to the assigned VM. >> + >> +- Inter-VM Communication: >> + >> + There are several different mechanisms provided for communicating between VMs. >> + >> +- Virtual platform: >> + >> + Architectural devices such as interrupt controllers and CPU timers are directly provided >> + by the hypervisor as well as core virtual platform devices and system APIs such as ARM PSCI. >> + >> +- Device Virtualization: >> + >> + Para-virtualization of devices is supported using inter-VM communication. >> + >> +Architectures supported >> +======================= >> +AArch64 with a GIC >> + >> +Resources and Capabilities >> +========================== >> + >> +Some services or resources provided by the Gunyah hypervisor are described to a virtual machine by >> +capability IDs. For instance, inter-VM communication is performed with doorbells and message queues. >> +Gunyah allows access to manipulate that doorbell via the capability ID. These devices are described >> +in Linux as a struct gunyah_resource. >> + >> +High level management of these resources is performed by the resource manager VM. RM informs a >> +guest VM about resources it can access through either the device tree or via guest-initiated RPC. >> + >> +For each virtual machine, Gunyah maintains a table of resources which can be accessed by that VM. >> +An entry in this table is called a "capability" and VMs can only access resources via this >> +capability table. Hence, virtual Gunyah devices are referenced by a "capability IDs" and not a >> +"resource IDs". A VM can have multiple capability IDs mapping to the same resource. If 2 VMs have >> +access to the same resource, they may not be using the same capability ID to access that resource >> +since the tables are independent per VM. >> + >> +Resource Manager >> +================ >> + >> +The resource manager (RM) is a privileged application VM supporting the Gunyah Hypervisor. >> +It provides policy enforcement aspects of the virtualization system. The resource manager can >> +be treated as an extension of the Hypervisor but is separated to its own partition to ensure >> +that the hypervisor layer itself remains small and secure and to maintain a separation of policy >> +and mechanism in the platform. On arm64, RM runs at NS-EL1 similar to other virtual machines. >> + >> +Communication with the resource manager from each guest VM happens with message-queue.rst. Details >> +about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c >> + >> +:: >> + >> + +-------+ +--------+ +--------+ >> + | RM | | VM_A | | VM_B | >> + +-.-.-.-+ +---.----+ +---.----+ >> + | | | | >> + +-.-.-----------.------------.----+ >> + | | \==========/ | | >> + | \========================/ | >> + | Gunyah | >> + +---------------------------------+ >> + >> +The source for the resource manager is available at https://github.com/quic/gunyah-resource-manager. >> + >> +The resource manager provides the following features: >> + >> +- VM lifecycle management: allocating a VM, starting VMs, destruction of VMs >> +- VM access control policy, including memory sharing and lending >> +- Interrupt routing configuration >> +- Forwarding of system-level events (e.g. VM shutdown) to owner VM >> + >> +When booting a virtual machine which uses a devicetree, resource manager overlays a >> +/hypervisor node. This node can let Linux know it is running as a Gunyah guest VM, >> +how to communicate with resource manager, and basic description and capabilities of >> +this VM. See Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml for a description >> +of this node. > > The documentation LGTM. > >> diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst >> new file mode 100644 >> index 000000000000..e130f124ed52 >> --- /dev/null >> +++ b/Documentation/virt/gunyah/message-queue.rst >> <snipped>... >> +The diagram below shows how message queue works. A typical configuration involves >> +2 message queues. Message queue 1 allows VM_A to send messages to VM_B. Message >> +queue 2 allows VM_B to send messages to VM_A. >> + >> +1. VM_A sends a message of up to 1024 bytes in length. It raises a hypercall >> + with the message to inform the hypervisor to add the message to >> + message queue 1's queue. >> +2. Gunyah raises the corresponding interrupt for VM_B when any of these happens: >> + a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case. >> + b. Explicility with gh_msgq_push command from VM_A. >> + c. Message queue has reached a threshold depth. >> +3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer. >> + > > The nested list above should be separated with blank lines to be > rendered properly: > > ---- >8 ---- > > diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst > index e130f124ed525a..afaad99db215e6 100644 > --- a/Documentation/virt/gunyah/message-queue.rst > +++ b/Documentation/virt/gunyah/message-queue.rst > @@ -20,9 +20,11 @@ queue 2 allows VM_B to send messages to VM_A. > with the message to inform the hypervisor to add the message to > message queue 1's queue. > 2. Gunyah raises the corresponding interrupt for VM_B when any of these happens: > + > a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case. > b. Explicility with gh_msgq_push command from VM_A. > c. Message queue has reached a threshold depth. > + > 3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer. > > For VM_B to send a message to VM_A, the process is identical, except that hypercalls > > Thanks. > Thanks! Applied for next version.
diff --git a/Documentation/virt/gunyah/index.rst b/Documentation/virt/gunyah/index.rst new file mode 100644 index 000000000000..959f451caccd --- /dev/null +++ b/Documentation/virt/gunyah/index.rst @@ -0,0 +1,114 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +Gunyah Hypervisor +================= + +.. toctree:: + :maxdepth: 1 + + message-queue + +Gunyah is a Type-1 hypervisor which is independent of any OS kernel, and runs in +a higher CPU privilege level. It does not depend on any lower-privileged operating system +for its core functionality. This increases its security and can support a much smaller +trusted computing base than a Type-2 hypervisor. + +Gunyah is an open source hypervisor. The source repo is available at +https://github.com/quic/gunyah-hypervisor. + +Gunyah provides these following features. + +- Scheduling: + + A scheduler for virtual CPUs (vCPUs) on physical CPUs and enables time-sharing + of the CPUs. Gunyah supports two models of scheduling: + + 1. "Behind the back" scheduling in which Gunyah hypervisor schedules vCPUS on its own + 2. "Proxy" scheduling in which a delegated VM can donate part of one of its vCPU slice + to another VM's vCPU via a hypercall. + +- Memory Management: + + APIs handling memory, abstracted as objects, limiting direct use of physical + addresses. Memory ownership and usage tracking of all memory under its control. + Memory partitioning between VMs is a fundamental security feature. + +- Interrupt Virtualization: + + Uses CPU hardware interrupt virtualization capabilities. Interrupts are handled + in the hypervisor and routed to the assigned VM. + +- Inter-VM Communication: + + There are several different mechanisms provided for communicating between VMs. + +- Virtual platform: + + Architectural devices such as interrupt controllers and CPU timers are directly provided + by the hypervisor as well as core virtual platform devices and system APIs such as ARM PSCI. + +- Device Virtualization: + + Para-virtualization of devices is supported using inter-VM communication. + +Architectures supported +======================= +AArch64 with a GIC + +Resources and Capabilities +========================== + +Some services or resources provided by the Gunyah hypervisor are described to a virtual machine by +capability IDs. For instance, inter-VM communication is performed with doorbells and message queues. +Gunyah allows access to manipulate that doorbell via the capability ID. These devices are described +in Linux as a struct gunyah_resource. + +High level management of these resources is performed by the resource manager VM. RM informs a +guest VM about resources it can access through either the device tree or via guest-initiated RPC. + +For each virtual machine, Gunyah maintains a table of resources which can be accessed by that VM. +An entry in this table is called a "capability" and VMs can only access resources via this +capability table. Hence, virtual Gunyah devices are referenced by a "capability IDs" and not a +"resource IDs". A VM can have multiple capability IDs mapping to the same resource. If 2 VMs have +access to the same resource, they may not be using the same capability ID to access that resource +since the tables are independent per VM. + +Resource Manager +================ + +The resource manager (RM) is a privileged application VM supporting the Gunyah Hypervisor. +It provides policy enforcement aspects of the virtualization system. The resource manager can +be treated as an extension of the Hypervisor but is separated to its own partition to ensure +that the hypervisor layer itself remains small and secure and to maintain a separation of policy +and mechanism in the platform. On arm64, RM runs at NS-EL1 similar to other virtual machines. + +Communication with the resource manager from each guest VM happens with message-queue.rst. Details +about the specific messages can be found in drivers/virt/gunyah/rsc_mgr.c + +:: + + +-------+ +--------+ +--------+ + | RM | | VM_A | | VM_B | + +-.-.-.-+ +---.----+ +---.----+ + | | | | + +-.-.-----------.------------.----+ + | | \==========/ | | + | \========================/ | + | Gunyah | + +---------------------------------+ + +The source for the resource manager is available at https://github.com/quic/gunyah-resource-manager. + +The resource manager provides the following features: + +- VM lifecycle management: allocating a VM, starting VMs, destruction of VMs +- VM access control policy, including memory sharing and lending +- Interrupt routing configuration +- Forwarding of system-level events (e.g. VM shutdown) to owner VM + +When booting a virtual machine which uses a devicetree, resource manager overlays a +/hypervisor node. This node can let Linux know it is running as a Gunyah guest VM, +how to communicate with resource manager, and basic description and capabilities of +this VM. See Documentation/devicetree/bindings/firmware/gunyah-hypervisor.yaml for a description +of this node. diff --git a/Documentation/virt/gunyah/message-queue.rst b/Documentation/virt/gunyah/message-queue.rst new file mode 100644 index 000000000000..e130f124ed52 --- /dev/null +++ b/Documentation/virt/gunyah/message-queue.rst @@ -0,0 +1,52 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Message Queues +============== +Message queue is a simple low-capacity IPC channel between two VMs. It is +intended for sending small control and configuration messages. Each message +queue object is unidirectional, so a full-duplex IPC channel requires a pair of +objects. + +Messages can be up to 1024 bytes in length. Longer messages require a further +protocol on top of the message queue messages themselves. For instance, communication +with the resource manager adds a header field for sending longer messages via multiple +message fragments. + +The diagram below shows how message queue works. A typical configuration involves +2 message queues. Message queue 1 allows VM_A to send messages to VM_B. Message +queue 2 allows VM_B to send messages to VM_A. + +1. VM_A sends a message of up to 1024 bytes in length. It raises a hypercall + with the message to inform the hypervisor to add the message to + message queue 1's queue. +2. Gunyah raises the corresponding interrupt for VM_B when any of these happens: + a. gh_msgq_send has PUSH flag. Queue is immediately flushed. This is the typical case. + b. Explicility with gh_msgq_push command from VM_A. + c. Message queue has reached a threshold depth. +3. VM_B calls gh_msgq_recv and Gunyah copies message to requested buffer. + +For VM_B to send a message to VM_A, the process is identical, except that hypercalls +reference message queue 2's capability ID. + +:: + + +---------------+ +-----------------+ +---------------+ + | VM_A | |Gunyah hypervisor| | VM_B | + | | | | | | + | | | | | | + | | Tx | | | | + | |-------->| | Rx vIRQ | | + |gh_msgq_send() | Tx vIRQ |Message queue 1 |-------->|gh_msgq_recv() | + | |<------- | | | | + | | | | | | + | Message Queue | | | | Message Queue | + | driver | | | | driver | + | | | | | | + | | | | | | + | | | | Tx | | + | | Rx vIRQ | |<--------| | + |gh_msgq_recv() |<--------|Message queue 2 | Tx vIRQ |gh_msgq_send() | + | | | |-------->| | + | | | | | | + | | | | | | + +---------------+ +-----------------+ +---------------+ diff --git a/Documentation/virt/index.rst b/Documentation/virt/index.rst index 2f1cffa87b1b..418d540f5484 100644 --- a/Documentation/virt/index.rst +++ b/Documentation/virt/index.rst @@ -15,6 +15,7 @@ Linux Virtualization Support acrn/index coco/sev-guest hyperv/index + gunyah/index .. only:: html and subproject diff --git a/MAINTAINERS b/MAINTAINERS index f5ca4aefd184..e88ebb7cbcb8 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -8880,6 +8880,13 @@ L: linux-efi@vger.kernel.org S: Maintained F: block/partitions/efi.* +GUNYAH HYPERVISOR DRIVER +M: Elliot Berman <quic_eberman@quicinc.com> +M: Murali Nalajala <quic_mnalajal@quicinc.com> +L: linux-arm-msm@vger.kernel.org +S: Supported +F: Documentation/virt/gunyah/ + HABANALABS PCI DRIVER M: Oded Gabbay <ogabbay@kernel.org> S: Supported
Gunyah is an open-source Type-1 hypervisor developed by Qualcomm. It does not depend on any lower-privileged OS/kernel code for its core functionality. This increases its security and can support a smaller trusted computing based when compared to Type-2 hypervisors. Add documentation describing the Gunyah hypervisor and the main components of the Gunyah hypervisor which are of interest to Linux virtualization development. Signed-off-by: Elliot Berman <quic_eberman@quicinc.com> --- Documentation/virt/gunyah/index.rst | 114 ++++++++++++++++++++ Documentation/virt/gunyah/message-queue.rst | 52 +++++++++ Documentation/virt/index.rst | 1 + MAINTAINERS | 7 ++ 4 files changed, 174 insertions(+) create mode 100644 Documentation/virt/gunyah/index.rst create mode 100644 Documentation/virt/gunyah/message-queue.rst