[RFC,v4,49/49] multi-process: add configure and usage information

Commit Message

Jag Raman Oct. 24, 2019, 9:09 a.m. UTC

From: Elena Ufimtseva <[e-mail]>

Signed-off-by: Elena Ufimtseva <[e-mail]>
Signed-off-by: Jagannathan Raman <[e-mail]>
Signed-off-by: John G Johnson <[e-mail]>
---
 docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100644 docs/qemu-multiprocess.txt

Comments

Stefan Hajnoczi Nov. 7, 2019, 2:02 p.m. UTC | #1

On Thu, Oct 24, 2019 at 05:09:30AM -0400, Jagannathan Raman wrote:
> From: Elena Ufimtseva <[e-mail]>
> 
> Signed-off-by: Elena Ufimtseva <[e-mail]>
> Signed-off-by: Jagannathan Raman <[e-mail]>
> Signed-off-by: John G Johnson <[e-mail]>
> ---
>  docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 86 insertions(+)
>  create mode 100644 docs/qemu-multiprocess.txt
> 
> diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt
> new file mode 100644
> index 0000000..c29f4df
> --- /dev/null
> +++ b/docs/qemu-multiprocess.txt
> @@ -0,0 +1,86 @@
> +Multi-process QEMU
> +==================
> +
> +This document describes how to configure and use multi-process qemu.
> +For the design document refer to docs/devel/qemu-multiprocess.
> +
> +1) Configuration
> +----------------
> +
> +To enable support for multi-process add --enable-mpqemu
> +to the list of options for the "configure" script.
> +
> +
> +2) Usage
> +--------
> +
> +To start qemu with devices intended to run in a separate emulation
> +process without libvirtd support, the following should be used on QEMU
> +command line. As of now, we only support the emulation of lsi53c895a
> +in a separate process
> +
> +* Since parts of the RAM are shared between QEMU & remote process, a
> +  memory-backend-file is required to facilitate this, as follows:
> +
> +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
> +
> +* The devices to be emulated in the separate process are defined as
> +  before with addition of "rid" suboption that serves as a remote group
> +  identificator.
> +
> +  -device <device options>,rid="remote process id"
> +
> +  For exmaple, for non multi-process qemu:

s/exmaple/example/

> +    -device lsi53c895a,id=scsi0 device
> +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0
> +    -drive id=drive0,file=data-disk.img
> +
> +  and for multi-process qemu and no libvirt
> +  support (i.e. QEMU forks child processes):
> +    -device lsi53c895a,id=scsi0,rid=0
> +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid="0"
> +
> +* The command-line options for the remote process is added to the "command"

s/is added/are added/

> +  suboption of the newly added "-remote" option. 
> +
> +   -remote [socket],rid=,command="..."
> +
> +  The drives to be emulated by the remote process are specified as part of
> +  this command sub-option. The device to be used to connect to the monitor
> +  is also specified as part of this suboption.
> +
> +  For example, the following option adds a drive and monitor to the remote
> +  process:
> +  -remote rid=0,command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> +
> +  Note: There's an issue with this "command" subtion which we are in the

s/subtion/sub-option/

> +  process of fixing. To work around this issue, it requires additional
> +  "comma" characters as illustrated above, and in the example below.
> +
> +* Example QEMU command-line to launch lsi53c895a in a remote process
> +
> +  #/bin/sh
> +  qemu-system-x86_64 \
> +  -name "OL7.4" \
> +  -machine q35,accel=kvm \
> +  -smp sockets=1,cores=1,threads=1 \
> +  -cpu host \
> +  -m 2048 \
> +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \
> +  -numa node,memdev=mem \
> +  -device virtio-scsi-pci,id=virtio_scsi_pci0 \
> +  -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \
> +  -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \
> +  -boot d \
> +  -monitor stdio \
> +  -vnc :0 \
> +  -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \
> +  -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\
> +  -remote rid=8,command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> +
> +  We could connect to the monitor using the following command:
> +  socat /home/qmp-sock stdio
> +
> +  After hotplugging disks to the remote process, please execute the
> +  following command in the guest to refresh the list of storage devices:
> +  rescan_scsi_bus.sh -a

This documentation suggests that QEMU spawns the remote processes.  How
do this work with unprivileged QEMU?  Is there an additional step where
QEMU drops privileges after having spawned remote processes?

Remote processes require accesses to resources that the main QEMU
process does not need access to, so I'm wondering how this process model
ensures that each process has only the privileges it needs.

Stefan

Michael S. Tsirkin Nov. 7, 2019, 2:33 p.m. UTC | #2

On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
> This documentation suggests that QEMU spawns the remote processes.  How
> do this work with unprivileged QEMU?  Is there an additional step where
> QEMU drops privileges after having spawned remote processes?
> 
> Remote processes require accesses to resources that the main QEMU
> process does not need access to, so I'm wondering how this process model
> ensures that each process has only the privileges it needs.

I guess you have something like capabilities in mind?

When using something like selinux, priviledges are per binary
so the order of startup doesn't matter.

Daniel P. Berrangé Nov. 7, 2019, 2:39 p.m. UTC | #3

On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
> On Thu, Oct 24, 2019 at 05:09:30AM -0400, Jagannathan Raman wrote:
> > From: Elena Ufimtseva <[e-mail]>
> > 
> > Signed-off-by: Elena Ufimtseva <[e-mail]>
> > Signed-off-by: Jagannathan Raman <[e-mail]>
> > Signed-off-by: John G Johnson <[e-mail]>
> > ---
> >  docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 86 insertions(+)
> >  create mode 100644 docs/qemu-multiprocess.txt
> > 
> > diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt
> > new file mode 100644
> > index 0000000..c29f4df
> > --- /dev/null
> > +++ b/docs/qemu-multiprocess.txt
> > @@ -0,0 +1,86 @@
> > +Multi-process QEMU
> > +==================
> > +
> > +This document describes how to configure and use multi-process qemu.
> > +For the design document refer to docs/devel/qemu-multiprocess.
> > +
> > +1) Configuration
> > +----------------
> > +
> > +To enable support for multi-process add --enable-mpqemu
> > +to the list of options for the "configure" script.
> > +
> > +
> > +2) Usage
> > +--------
> > +
> > +To start qemu with devices intended to run in a separate emulation
> > +process without libvirtd support, the following should be used on QEMU
> > +command line. As of now, we only support the emulation of lsi53c895a
> > +in a separate process
> > +
> > +* Since parts of the RAM are shared between QEMU & remote process, a
> > +  memory-backend-file is required to facilitate this, as follows:
> > +
> > +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
> > +
> > +* The devices to be emulated in the separate process are defined as
> > +  before with addition of "rid" suboption that serves as a remote group
> > +  identificator.
> > +
> > +  -device <device options>,rid="remote process id"
> > +
> > +  For exmaple, for non multi-process qemu:
> 
> s/exmaple/example/
> 
> > +    -device lsi53c895a,id=scsi0 device
> > +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0
> > +    -drive id=drive0,file=data-disk.img
> > +
> > +  and for multi-process qemu and no libvirt
> > +  support (i.e. QEMU forks child processes):
> > +    -device lsi53c895a,id=scsi0,rid=0
> > +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid="0"
> > +
> > +* The command-line options for the remote process is added to the "command"
> 
> s/is added/are added/
> 
> > +  suboption of the newly added "-remote" option. 
> > +
> > +   -remote [socket],rid=,command="..."
> > +
> > +  The drives to be emulated by the remote process are specified as part of
> > +  this command sub-option. The device to be used to connect to the monitor
> > +  is also specified as part of this suboption.
> > +
> > +  For example, the following option adds a drive and monitor to the remote
> > +  process:
> > +  -remote rid=0,command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> > +
> > +  Note: There's an issue with this "command" subtion which we are in the
> 
> s/subtion/sub-option/
> 
> > +  process of fixing. To work around this issue, it requires additional
> > +  "comma" characters as illustrated above, and in the example below.
> > +
> > +* Example QEMU command-line to launch lsi53c895a in a remote process
> > +
> > +  #/bin/sh
> > +  qemu-system-x86_64 \
> > +  -name "OL7.4" \
> > +  -machine q35,accel=kvm \
> > +  -smp sockets=1,cores=1,threads=1 \
> > +  -cpu host \
> > +  -m 2048 \
> > +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \
> > +  -numa node,memdev=mem \
> > +  -device virtio-scsi-pci,id=virtio_scsi_pci0 \
> > +  -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \
> > +  -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \
> > +  -boot d \
> > +  -monitor stdio \
> > +  -vnc :0 \
> > +  -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \
> > +  -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\
> > +  -remote rid=8,command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> > +
> > +  We could connect to the monitor using the following command:
> > +  socat /home/qmp-sock stdio
> > +
> > +  After hotplugging disks to the remote process, please execute the
> > +  following command in the guest to refresh the list of storage devices:
> > +  rescan_scsi_bus.sh -a
> 
> This documentation suggests that QEMU spawns the remote processes.  How
> do this work with unprivileged QEMU?  Is there an additional step where
> QEMU drops privileges after having spawned remote processes?

This syntax is for the simple case without privilege separation.
If differing privilege levels are needed, then whatever spawns QEMU
should spawn the remote helper process ahead of time, and then just
pass the UNIX socket path to the -remote arg, instead of using
the 'command' parameter.

Regards,
Daniel

Jag Raman Nov. 7, 2019, 3:53 p.m. UTC | #4

On 11/7/2019 9:39 AM, Daniel P. Berrangé wrote:
> On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
>> On Thu, Oct 24, 2019 at 05:09:30AM -0400, Jagannathan Raman wrote:
>>> From: Elena Ufimtseva <[e-mail]>
>>>
>>> Signed-off-by: Elena Ufimtseva <[e-mail]>
>>> Signed-off-by: Jagannathan Raman <[e-mail]>
>>> Signed-off-by: John G Johnson <[e-mail]>
>>> ---
>>>   docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++
>>>   1 file changed, 86 insertions(+)
>>>   create mode 100644 docs/qemu-multiprocess.txt
>>>
>>> diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt
>>> new file mode 100644
>>> index 0000000..c29f4df
>>> --- /dev/null
>>> +++ b/docs/qemu-multiprocess.txt
>>> @@ -0,0 +1,86 @@
>>> +Multi-process QEMU
>>> +==================
>>> +
>>> +This document describes how to configure and use multi-process qemu.
>>> +For the design document refer to docs/devel/qemu-multiprocess.
>>> +
>>> +1) Configuration
>>> +----------------
>>> +
>>> +To enable support for multi-process add --enable-mpqemu
>>> +to the list of options for the "configure" script.
>>> +
>>> +
>>> +2) Usage
>>> +--------
>>> +
>>> +To start qemu with devices intended to run in a separate emulation
>>> +process without libvirtd support, the following should be used on QEMU
>>> +command line. As of now, we only support the emulation of lsi53c895a
>>> +in a separate process
>>> +
>>> +* Since parts of the RAM are shared between QEMU & remote process, a
>>> +  memory-backend-file is required to facilitate this, as follows:
>>> +
>>> +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
>>> +
>>> +* The devices to be emulated in the separate process are defined as
>>> +  before with addition of "rid" suboption that serves as a remote group
>>> +  identificator.
>>> +
>>> +  -device <device options>,rid="remote process id"
>>> +
>>> +  For exmaple, for non multi-process qemu:
>>
>> s/exmaple/example/
>>
>>> +    -device lsi53c895a,id=scsi0 device
>>> +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0
>>> +    -drive id=drive0,file=data-disk.img
>>> +
>>> +  and for multi-process qemu and no libvirt
>>> +  support (i.e. QEMU forks child processes):
>>> +    -device lsi53c895a,id=scsi0,rid=0
>>> +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid="0"
>>> +
>>> +* The command-line options for the remote process is added to the "command"
>>
>> s/is added/are added/
>>
>>> +  suboption of the newly added "-remote" option.
>>> +
>>> +   -remote [socket],rid=,command="..."
>>> +
>>> +  The drives to be emulated by the remote process are specified as part of
>>> +  this command sub-option. The device to be used to connect to the monitor
>>> +  is also specified as part of this suboption.
>>> +
>>> +  For example, the following option adds a drive and monitor to the remote
>>> +  process:
>>> +  -remote rid=0,command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
>>> +
>>> +  Note: There's an issue with this "command" subtion which we are in the
>>
>> s/subtion/sub-option/
>>
>>> +  process of fixing. To work around this issue, it requires additional
>>> +  "comma" characters as illustrated above, and in the example below.
>>> +
>>> +* Example QEMU command-line to launch lsi53c895a in a remote process
>>> +
>>> +  #/bin/sh
>>> +  qemu-system-x86_64 \
>>> +  -name "OL7.4" \
>>> +  -machine q35,accel=kvm \
>>> +  -smp sockets=1,cores=1,threads=1 \
>>> +  -cpu host \
>>> +  -m 2048 \
>>> +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \
>>> +  -numa node,memdev=mem \
>>> +  -device virtio-scsi-pci,id=virtio_scsi_pci0 \
>>> +  -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \
>>> +  -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \
>>> +  -boot d \
>>> +  -monitor stdio \
>>> +  -vnc :0 \
>>> +  -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \
>>> +  -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\
>>> +  -remote rid=8,command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
>>> +
>>> +  We could connect to the monitor using the following command:
>>> +  socat /home/qmp-sock stdio
>>> +
>>> +  After hotplugging disks to the remote process, please execute the
>>> +  following command in the guest to refresh the list of storage devices:
>>> +  rescan_scsi_bus.sh -a
>>
>> This documentation suggests that QEMU spawns the remote processes.  How
>> do this work with unprivileged QEMU?  Is there an additional step where
>> QEMU drops privileges after having spawned remote processes?
> 
> This syntax is for the simple case without privilege separation.
> If differing privilege levels are needed, then whatever spawns QEMU
> should spawn the remote helper process ahead of time, and then just
> pass the UNIX socket path to the -remote arg, instead of using
> the 'command' parameter.
> 
> Regards,
> Daniel

Thank You, Stefan, Michael & Daniel, for your comments. I had a chance
to sit down with my teammates to understand the feedback you gave at the
KVM Forum. Thank you for that, as well.

We currently support two ways of launching the remote process - one is
self-launch through QEMU, as outlined in this patch series. The other
approach is using an Orchestrator like libvirt (we haven't had the
chance to submit those patches for review yet).

In the case where libvirt is involved, it would assume the
responsibility of spawning the remote process first and pass in the info
required to connect to the remote process via command-line arguments to
QEMU. This support in QEMU is available in the current series. We
haven't sent the libvirt side of patches out for review yet. It would be
easier to upstream libvirt once the QEMU side of things is firmed up.

In the case of self-launch, our understanding is that QEMU has the
privilege to fork() the remote process until the "-sandbox" argument is
processed. However, if an Orchestrator prohibits QEMU from spawning
other processes from the get-go, then the Orchestrator would assume the
responsibility of spawning the remote process as well - like Daniel just
pointed out.

In both cases, we intend to apply the security policies required to
confine the remote process externally - probably through SELinux. We
haven't had the chance to upstream the SELinux policies yet, but we
previously sent a sample of the policies for your comments. Like Michael
pointed out earlier, the SELinux policies are per binary.

Thank you very much!
--
Jag

>

Stefan Hajnoczi Nov. 8, 2019, 11:14 a.m. UTC | #5

On Thu, Nov 07, 2019 at 10:53:27AM -0500, Jag Raman wrote:
> 
> 
> On 11/7/2019 9:39 AM, Daniel P. Berrangé wrote:
> > On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
> > > On Thu, Oct 24, 2019 at 05:09:30AM -0400, Jagannathan Raman wrote:
> > > > From: Elena Ufimtseva <[e-mail]>
> > > > 
> > > > Signed-off-by: Elena Ufimtseva <[e-mail]>
> > > > Signed-off-by: Jagannathan Raman <[e-mail]>
> > > > Signed-off-by: John G Johnson <[e-mail]>
> > > > ---
> > > >   docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++
> > > >   1 file changed, 86 insertions(+)
> > > >   create mode 100644 docs/qemu-multiprocess.txt
> > > > 
> > > > diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt
> > > > new file mode 100644
> > > > index 0000000..c29f4df
> > > > --- /dev/null
> > > > +++ b/docs/qemu-multiprocess.txt
> > > > @@ -0,0 +1,86 @@
> > > > +Multi-process QEMU
> > > > +==================
> > > > +
> > > > +This document describes how to configure and use multi-process qemu.
> > > > +For the design document refer to docs/devel/qemu-multiprocess.
> > > > +
> > > > +1) Configuration
> > > > +----------------
> > > > +
> > > > +To enable support for multi-process add --enable-mpqemu
> > > > +to the list of options for the "configure" script.
> > > > +
> > > > +
> > > > +2) Usage
> > > > +--------
> > > > +
> > > > +To start qemu with devices intended to run in a separate emulation
> > > > +process without libvirtd support, the following should be used on QEMU
> > > > +command line. As of now, we only support the emulation of lsi53c895a
> > > > +in a separate process
> > > > +
> > > > +* Since parts of the RAM are shared between QEMU & remote process, a
> > > > +  memory-backend-file is required to facilitate this, as follows:
> > > > +
> > > > +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
> > > > +
> > > > +* The devices to be emulated in the separate process are defined as
> > > > +  before with addition of "rid" suboption that serves as a remote group
> > > > +  identificator.
> > > > +
> > > > +  -device <device options>,rid="remote process id"
> > > > +
> > > > +  For exmaple, for non multi-process qemu:
> > > 
> > > s/exmaple/example/
> > > 
> > > > +    -device lsi53c895a,id=scsi0 device
> > > > +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0
> > > > +    -drive id=drive0,file=data-disk.img
> > > > +
> > > > +  and for multi-process qemu and no libvirt
> > > > +  support (i.e. QEMU forks child processes):
> > > > +    -device lsi53c895a,id=scsi0,rid=0
> > > > +    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid="0"
> > > > +
> > > > +* The command-line options for the remote process is added to the "command"
> > > 
> > > s/is added/are added/
> > > 
> > > > +  suboption of the newly added "-remote" option.
> > > > +
> > > > +   -remote [socket],rid=,command="..."
> > > > +
> > > > +  The drives to be emulated by the remote process are specified as part of
> > > > +  this command sub-option. The device to be used to connect to the monitor
> > > > +  is also specified as part of this suboption.
> > > > +
> > > > +  For example, the following option adds a drive and monitor to the remote
> > > > +  process:
> > > > +  -remote rid=0,command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> > > > +
> > > > +  Note: There's an issue with this "command" subtion which we are in the
> > > 
> > > s/subtion/sub-option/
> > > 
> > > > +  process of fixing. To work around this issue, it requires additional
> > > > +  "comma" characters as illustrated above, and in the example below.
> > > > +
> > > > +* Example QEMU command-line to launch lsi53c895a in a remote process
> > > > +
> > > > +  #/bin/sh
> > > > +  qemu-system-x86_64 \
> > > > +  -name "OL7.4" \
> > > > +  -machine q35,accel=kvm \
> > > > +  -smp sockets=1,cores=1,threads=1 \
> > > > +  -cpu host \
> > > > +  -m 2048 \
> > > > +  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \
> > > > +  -numa node,memdev=mem \
> > > > +  -device virtio-scsi-pci,id=virtio_scsi_pci0 \
> > > > +  -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \
> > > > +  -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \
> > > > +  -boot d \
> > > > +  -monitor stdio \
> > > > +  -vnc :0 \
> > > > +  -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \
> > > > +  -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\
> > > > +  -remote rid=8,command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
> > > > +
> > > > +  We could connect to the monitor using the following command:
> > > > +  socat /home/qmp-sock stdio
> > > > +
> > > > +  After hotplugging disks to the remote process, please execute the
> > > > +  following command in the guest to refresh the list of storage devices:
> > > > +  rescan_scsi_bus.sh -a
> > > 
> > > This documentation suggests that QEMU spawns the remote processes.  How
> > > do this work with unprivileged QEMU?  Is there an additional step where
> > > QEMU drops privileges after having spawned remote processes?
> > 
> > This syntax is for the simple case without privilege separation.
> > If differing privilege levels are needed, then whatever spawns QEMU
> > should spawn the remote helper process ahead of time, and then just
> > pass the UNIX socket path to the -remote arg, instead of using
> > the 'command' parameter.
> > 
> > Regards,
> > Daniel
> 
> Thank You, Stefan, Michael & Daniel, for your comments. I had a chance
> to sit down with my teammates to understand the feedback you gave at the
> KVM Forum. Thank you for that, as well.
> 
> We currently support two ways of launching the remote process - one is
> self-launch through QEMU, as outlined in this patch series. The other
> approach is using an Orchestrator like libvirt (we haven't had the
> chance to submit those patches for review yet).
> 
> In the case where libvirt is involved, it would assume the
> responsibility of spawning the remote process first and pass in the info
> required to connect to the remote process via command-line arguments to
> QEMU. This support in QEMU is available in the current series. We
> haven't sent the libvirt side of patches out for review yet. It would be
> easier to upstream libvirt once the QEMU side of things is firmed up.
> 
> In the case of self-launch, our understanding is that QEMU has the
> privilege to fork() the remote process until the "-sandbox" argument is
> processed. However, if an Orchestrator prohibits QEMU from spawning
> other processes from the get-go, then the Orchestrator would assume the
> responsibility of spawning the remote process as well - like Daniel just
> pointed out.
> 
> In both cases, we intend to apply the security policies required to
> confine the remote process externally - probably through SELinux. We
> haven't had the chance to upstream the SELinux policies yet, but we
> previously sent a sample of the policies for your comments. Like Michael
> pointed out earlier, the SELinux policies are per binary.

Sounds good, please document -remote socket= as an alternative to
-remote command= so it's clear that both approaches are supported.

Stefan

Stefan Hajnoczi Nov. 8, 2019, 11:17 a.m. UTC | #6

On Thu, Nov 07, 2019 at 09:33:45AM -0500, Michael S. Tsirkin wrote:
> On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
> > This documentation suggests that QEMU spawns the remote processes.  How
> > do this work with unprivileged QEMU?  Is there an additional step where
> > QEMU drops privileges after having spawned remote processes?
> > 
> > Remote processes require accesses to resources that the main QEMU
> > process does not need access to, so I'm wondering how this process model
> > ensures that each process has only the privileges it needs.
> 
> I guess you have something like capabilities in mind?

Or namespaces (unshare(2)).

> When using something like selinux, priviledges are per binary
> so the order of startup doesn't matter.

For static SELinux policies that make sense, thanks for explaining.

Does libvirt also perform dynamic (i.e. per-instance) SELinux
configuration?  I guess that cannot be associated with a specific binary
because multiple QEMU instances launch the same binary yet need to be
differentiated.

Stefan

Daniel P. Berrangé Nov. 8, 2019, 11:32 a.m. UTC | #7

On Fri, Nov 08, 2019 at 12:17:41PM +0100, Stefan Hajnoczi wrote:
> On Thu, Nov 07, 2019 at 09:33:45AM -0500, Michael S. Tsirkin wrote:
> > On Thu, Nov 07, 2019 at 03:02:20PM +0100, Stefan Hajnoczi wrote:
> > > This documentation suggests that QEMU spawns the remote processes.  How
> > > do this work with unprivileged QEMU?  Is there an additional step where
> > > QEMU drops privileges after having spawned remote processes?
> > > 
> > > Remote processes require accesses to resources that the main QEMU
> > > process does not need access to, so I'm wondering how this process model
> > > ensures that each process has only the privileges it needs.
> > 
> > I guess you have something like capabilities in mind?
> 
> Or namespaces (unshare(2)).
> 
> > When using something like selinux, priviledges are per binary
> > so the order of startup doesn't matter.
> 
> For static SELinux policies that make sense, thanks for explaining.
> 
> Does libvirt also perform dynamic (i.e. per-instance) SELinux
> configuration?  I guess that cannot be associated with a specific binary
> because multiple QEMU instances launch the same binary yet need to be
> differentiated.

In a traditional SELinux approach, the SELinux context used for any
process is determined by a combination of the label on the binary
and a transition rule.

eg if the qemu-system-x86_64 file is labelled qemu_exec_t, and
there's a context qemu_t for the QEMU process, a transition
rule is defined  "virtd_t + qemu_exec_t ->  qemu_t". This says
that when a process with context "vird_t" execs a binary labelled
qemu_exec_t, the new process gets qemu_t.

We sVirt, however, we can't rely on automatic transitions, because
we need to assign a unique MCS tag for each VM. Thus libvird will
explicitly tell SELinux what label to apply.

In the case of multiprocess QEMU, if using sVirt from libvirt, then
we'll need to continue setting the explicit labels as we'll still
need the MCS tags for each helper process.

If not using libvirt and sVirt, and wanting automatic SELinux
transitions for QEMU helper processes, then each helper would
need to be a separate binary on disk so that each helper can
be given a distinct file label, which in turns lets you define
a set of transitions for each helper according to its expected
access needs.

Having said all that I don't think its worth worrying about
this. Anyone who cares about SELinux with QEMU will want to
be using sVirt  or an equivalent approach to assign unique
MCS per VM. And thus automatic transitions are not possible
even if we had distinct binaries for each helper.

Regards,
Daniel

Patch

diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt
new file mode 100644
index 0000000..c29f4df
--- /dev/null
+++ b/docs/qemu-multiprocess.txt
@@ -0,0 +1,86 @@ 
+Multi-process QEMU
+==================
+
+This document describes how to configure and use multi-process qemu.
+For the design document refer to docs/devel/qemu-multiprocess.
+
+1) Configuration
+----------------
+
+To enable support for multi-process add --enable-mpqemu
+to the list of options for the "configure" script.
+
+
+2) Usage
+--------
+
+To start qemu with devices intended to run in a separate emulation
+process without libvirtd support, the following should be used on QEMU
+command line. As of now, we only support the emulation of lsi53c895a
+in a separate process
+
+* Since parts of the RAM are shared between QEMU & remote process, a
+  memory-backend-file is required to facilitate this, as follows:
+
+  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on
+
+* The devices to be emulated in the separate process are defined as
+  before with addition of "rid" suboption that serves as a remote group
+  identificator.
+
+  -device <device options>,rid="remote process id"
+
+  For exmaple, for non multi-process qemu:
+    -device lsi53c895a,id=scsi0 device
+    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0
+    -drive id=drive0,file=data-disk.img
+
+  and for multi-process qemu and no libvirt
+  support (i.e. QEMU forks child processes):
+    -device lsi53c895a,id=scsi0,rid=0
+    -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid="0"
+
+* The command-line options for the remote process is added to the "command"
+  suboption of the newly added "-remote" option. 
+
+   -remote [socket],rid=,command="..."
+
+  The drives to be emulated by the remote process are specified as part of
+  this command sub-option. The device to be used to connect to the monitor
+  is also specified as part of this suboption.
+
+  For example, the following option adds a drive and monitor to the remote
+  process:
+  -remote rid=0,command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
+
+  Note: There's an issue with this "command" subtion which we are in the
+  process of fixing. To work around this issue, it requires additional
+  "comma" characters as illustrated above, and in the example below.
+
+* Example QEMU command-line to launch lsi53c895a in a remote process
+
+  #/bin/sh
+  qemu-system-x86_64 \
+  -name "OL7.4" \
+  -machine q35,accel=kvm \
+  -smp sockets=1,cores=1,threads=1 \
+  -cpu host \
+  -m 2048 \
+  -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \
+  -numa node,memdev=mem \
+  -device virtio-scsi-pci,id=virtio_scsi_pci0 \
+  -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \
+  -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \
+  -boot d \
+  -monitor stdio \
+  -vnc :0 \
+  -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \
+  -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\
+  -remote rid=8,command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait"
+
+  We could connect to the monitor using the following command:
+  socat /home/qmp-sock stdio
+
+  After hotplugging disks to the remote process, please execute the
+  following command in the guest to refresh the list of storage devices:
+  rescan_scsi_bus.sh -a