| Message ID | 20190924124433.96810-8-slp@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Introduce the microvm machine type | expand |
On 24/09/19 14:44, Sergio Lopez wrote: > +Microvm is a machine type inspired by both NEMU and Firecracker, and > +constructed after the machine model implemented by the latter. I would say it's inspired by Firecracker only. The NEMU virt machine had virtio-pci and ACPI. > +It's main purpose is providing users a minimalist machine type free > +from the burden of legacy compatibility, I think this is too strong, especially if you keep the PIC and PIT. :) Maybe just "It's a minimalist machine type without PCI support designed for short-lived guests". > +serving as a stepping stone > +for future projects aiming at improving boot times, reducing the > +attack surface and slimming down QEMU's footprint. "Microvm also establishes a baseline for benchmarking QEMU and operating systems, since it is optimized for both boot time and footprint". > +The microvm machine type supports the following devices: > + > + - ISA bus > + - i8259 PIC > + - LAPIC (implicit if using KVM) > + - IOAPIC (defaults to kernel_irqchip_split = true) > + - i8254 PIT Do we need the PIT? And perhaps the PIC even? Paolo > + - MC146818 RTC (optional) > + - kvmclock (if using KVM) > + - fw_cfg > + - One ISA serial port (optional) > + - Up to eight virtio-mmio devices (configured by the user) > +
Hi, > +microvm.kernel-cmdline=bool (Set off to disable adding virtio-mmio devices to the kernel cmdline) Hmm, is that the long-term plan? IMO the virtio-mmio devices should be discoverable somehow. ACPI, or device-tree, or fw_cfg, or ... > +As no current FW is able to boot from a block device using virtio-mmio > +as its transport, To fix that the firmware must be able to find the virtio-mmio devices. cheers, Gerd
Paolo Bonzini <pbonzini@redhat.com> writes: > On 24/09/19 14:44, Sergio Lopez wrote: >> +Microvm is a machine type inspired by both NEMU and Firecracker, and >> +constructed after the machine model implemented by the latter. > > I would say it's inspired by Firecracker only. The NEMU virt machine > had virtio-pci and ACPI. Actually, the NEMU reference comes from the fact that, originally, microvm.c code was based on virt.c, but on v4 all that is already gone, so it makes sense to remove the reference. >> +It's main purpose is providing users a minimalist machine type free >> +from the burden of legacy compatibility, > > I think this is too strong, especially if you keep the PIC and PIT. :) > Maybe just "It's a minimalist machine type without PCI support designed > for short-lived guests". OK. >> +serving as a stepping stone >> +for future projects aiming at improving boot times, reducing the >> +attack surface and slimming down QEMU's footprint. > > "Microvm also establishes a baseline for benchmarking QEMU and operating > systems, since it is optimized for both boot time and footprint". Well, I prefer my paragraph, but I'm good with either. >> +The microvm machine type supports the following devices: >> + >> + - ISA bus >> + - i8259 PIC >> + - LAPIC (implicit if using KVM) >> + - IOAPIC (defaults to kernel_irqchip_split = true) >> + - i8254 PIT > > Do we need the PIT? And perhaps the PIC even? We need the PIT for non-KVM accel (if present with KVM and kernel_irqchip_split = off, it basically becomes a placeholder), and the PIC for both the PIT and the ISA serial port. Thanks, Sergio. >> + - MC146818 RTC (optional) >> + - kvmclock (if using KVM) >> + - fw_cfg >> + - One ISA serial port (optional) >> + - Up to eight virtio-mmio devices (configured by the user) >> +
Gerd Hoffmann <kraxel@redhat.com> writes: > Hi, > >> +microvm.kernel-cmdline=bool (Set off to disable adding virtio-mmio devices to the kernel cmdline) > > Hmm, is that the long-term plan? IMO the virtio-mmio devices should be > discoverable somehow. ACPI, or device-tree, or fw_cfg, or ... I'd say that depends on the machine type. ARM's virt and vexpress do support virtio-mmio devices, adding them to a generated DTB. For microvm that's simply not worth it. Fiddling with the command line achieves the same result without any significant drawbacks, with less code and less consumed cycles on both sides. >> +As no current FW is able to boot from a block device using virtio-mmio >> +as its transport, > > To fix that the firmware must be able to find the virtio-mmio devices. No FW supports modern virtio-mmio transports anyway. And, from microvm's perspective, there's little incentive to change this situation, given that it's main use cases (serverless computing and VM-isolated containers) will run with an external kernel. Thanks, Sergio.
On 25/09/19 07:49, Sergio Lopez wrote: >>> +serving as a stepping stone >>> +for future projects aiming at improving boot times, reducing the >>> +attack surface and slimming down QEMU's footprint. >> >> "Microvm also establishes a baseline for benchmarking QEMU and operating >> systems, since it is optimized for both boot time and footprint". > > Well, I prefer my paragraph, but I'm good with either. You're right my version sort of missed the point. What about s/benchmarking/benchmarking and optimizing/? >>> +The microvm machine type supports the following devices: >>> + >>> + - ISA bus >>> + - i8259 PIC >>> + - LAPIC (implicit if using KVM) >>> + - IOAPIC (defaults to kernel_irqchip_split = true) >>> + - i8254 PIT >> >> Do we need the PIT? And perhaps the PIC even? > > We need the PIT for non-KVM accel (if present with KVM and > kernel_irqchip_split = off, it basically becomes a placeholder) Why? > and the > PIC for both the PIT and the ISA serial port. Can't the ISA serial port work with the IOAPIC? Paolo
Paolo Bonzini <pbonzini@redhat.com> writes: > On 25/09/19 07:49, Sergio Lopez wrote: >>>> +serving as a stepping stone >>>> +for future projects aiming at improving boot times, reducing the >>>> +attack surface and slimming down QEMU's footprint. >>> >>> "Microvm also establishes a baseline for benchmarking QEMU and operating >>> systems, since it is optimized for both boot time and footprint". >> >> Well, I prefer my paragraph, but I'm good with either. > > You're right my version sort of missed the point. What about > s/benchmarking/benchmarking and optimizing/? > >>>> +The microvm machine type supports the following devices: >>>> + >>>> + - ISA bus >>>> + - i8259 PIC >>>> + - LAPIC (implicit if using KVM) >>>> + - IOAPIC (defaults to kernel_irqchip_split = true) >>>> + - i8254 PIT >>> >>> Do we need the PIT? And perhaps the PIC even? >> >> We need the PIT for non-KVM accel (if present with KVM and >> kernel_irqchip_split = off, it basically becomes a placeholder) > > Why? Perhaps I'm missing something. Is some other device supposed to be acting as a HW timer while running with TCG acceleration? >> and the >> PIC for both the PIT and the ISA serial port. > > Can't the ISA serial port work with the IOAPIC? Hm... I'm not sure. I wanted to give it a try, but then noticed that multiple places in the code (like hw/intc/apic.c:560) do expect to have an ISA PIC present through the isa_pic global variable. I guess we should be able to work around this, but I'm not sure if it's really worth it. What do you think? Thanks, Sergio.
Hi, > For microvm that's simply not worth it. Fiddling with the command line > achieves the same result without any significant drawbacks, Assuming you actually can fiddle with the command line, which is only the case with direct kernel boot. > > To fix that the firmware must be able to find the virtio-mmio devices. > > No FW supports modern virtio-mmio transports anyway. Well, we change that if we want ... > And, from microvm's perspective, there's little incentive to change > this situation, given that it's main use cases (serverless computing > and VM-isolated containers) will run with an external kernel. If direct kernel boot is the only use case microvm ever wants support, then there is little reason to go the extra mile for optional seabios support. cheers, Gerd
On 25/09/19 10:40, Sergio Lopez wrote: >>> We need the PIT for non-KVM accel (if present with KVM and >>> kernel_irqchip_split = off, it basically becomes a placeholder) >> Why? > > Perhaps I'm missing something. Is some other device supposed to be > acting as a HW timer while running with TCG acceleration? Sure, the LAPIC timer. I wonder if Linux, however, wants to use the PIT in order to calibrate the LAPIC timer if TSC deadline mode is unavailable. >>> and the PIC for both the PIT and the ISA serial port. >> >> Can't the ISA serial port work with the IOAPIC? > > Hm... I'm not sure. I wanted to give it a try, but then noticed that > multiple places in the code (like hw/intc/apic.c:560) do expect to have > an ISA PIC present through the isa_pic global variable. > > I guess we should be able to work around this, but I'm not sure if it's > really worth it. What do you think? You can add a paragraph saying that in the future the list could be reduced further. I think that the direction we want to go is to only leave the IOAPIC around (the ISA devices in this respect are no different from the virtio-mmio devices). But you're right about isa_pic. I wonder if it's as easy as this: diff --git a/hw/intc/apic.c b/hw/intc/apic.c index bce89911dc..5d03e48a19 100644 --- a/hw/intc/apic.c +++ b/hw/intc/apic.c @@ -610,7 +610,7 @@ int apic_accept_pic_intr(DeviceState *dev) if ((s->apicbase & MSR_IA32_APICBASE_ENABLE) == 0 || (lvt0 & APIC_LVT_MASKED) == 0) - return 1; + return isa_pic != NULL; return 0; } Thanks, Paolo
Paolo Bonzini <pbonzini@redhat.com> writes: > On 25/09/19 10:40, Sergio Lopez wrote: >>>> We need the PIT for non-KVM accel (if present with KVM and >>>> kernel_irqchip_split = off, it basically becomes a placeholder) >>> Why? >> >> Perhaps I'm missing something. Is some other device supposed to be >> acting as a HW timer while running with TCG acceleration? > > Sure, the LAPIC timer. I wonder if Linux, however, wants to use the PIT > in order to calibrate the LAPIC timer if TSC deadline mode is unavailable. Ah, yes. I was so confused by the nomenclature that I assumed we didn't have a userspace implementation of it. On the other hand, as you suspect, without the PIT Linux does hang in TSC calibration with TCG accel. A simple option could be adding it only if we're running without KVM. >>>> and the PIC for both the PIT and the ISA serial port. >>> >>> Can't the ISA serial port work with the IOAPIC? >> >> Hm... I'm not sure. I wanted to give it a try, but then noticed that >> multiple places in the code (like hw/intc/apic.c:560) do expect to have >> an ISA PIC present through the isa_pic global variable. >> >> I guess we should be able to work around this, but I'm not sure if it's >> really worth it. What do you think? > > You can add a paragraph saying that in the future the list could be > reduced further. I think that the direction we want to go is to only > leave the IOAPIC around (the ISA devices in this respect are no > different from the virtio-mmio devices). > > But you're right about isa_pic. I wonder if it's as easy as this: > > diff --git a/hw/intc/apic.c b/hw/intc/apic.c > index bce89911dc..5d03e48a19 100644 > --- a/hw/intc/apic.c > +++ b/hw/intc/apic.c > @@ -610,7 +610,7 @@ int apic_accept_pic_intr(DeviceState *dev) > > if ((s->apicbase & MSR_IA32_APICBASE_ENABLE) == 0 || > (lvt0 & APIC_LVT_MASKED) == 0) > - return 1; > + return isa_pic != NULL; > > return 0; > } Yes, that would do the trick. There's another use of it at hw/intc/ioapic.c:78, but we should be safe as, at least in the case of Linux, DM_EXTINT is only used in check_timer(), which is only called if it detects a i8259 PIC. We should probably add an assertion with an informative message, just in case. Thanks, Sergio.
On 25/09/19 13:04, Sergio Lopez wrote: > Yes, that would do the trick. There's another use of it at > hw/intc/ioapic.c:78, but we should be safe as, at least in the case of > Linux, DM_EXTINT is only used in check_timer(), which is only called if > it detects a i8259 PIC. Even there it is actually LVT0's DM_EXTINT, not the IOAPIC's. I think pic_read_irq would have returned 7 (spurious IRQ on master i8259) until commit 29bb5317cb ("i8259: QOM cleanups", 2013-04-29), so we should fix it. Paolo > We should probably add an assertion with an informative message, just in > case.
Paolo Bonzini <pbonzini@redhat.com> writes: > On 24/09/19 14:44, Sergio Lopez wrote: >> +Microvm is a machine type inspired by both NEMU and Firecracker, and >> +constructed after the machine model implemented by the latter. > > I would say it's inspired by Firecracker only. The NEMU virt machine > had virtio-pci and ACPI. > >> +It's main purpose is providing users a minimalist machine type free >> +from the burden of legacy compatibility, > > I think this is too strong, especially if you keep the PIC and PIT. :) > Maybe just "It's a minimalist machine type without PCI support designed > for short-lived guests". > >> +serving as a stepping stone >> +for future projects aiming at improving boot times, reducing the >> +attack surface and slimming down QEMU's footprint. > > "Microvm also establishes a baseline for benchmarking QEMU and operating > systems, since it is optimized for both boot time and footprint". > >> +The microvm machine type supports the following devices: >> + >> + - ISA bus >> + - i8259 PIC >> + - LAPIC (implicit if using KVM) >> + - IOAPIC (defaults to kernel_irqchip_split = true) >> + - i8254 PIT > > Do we need the PIT? And perhaps the PIC even? > I'm going back to this level of the thread, because after your suggestion I took a deeper look at how things work around the PIC, and discovered I was completely wrong about my assumptions. For virtio-mmio devices, given that we don't have the ability to configure vectors (as it's done in the PCI case) we're stuck with the ones provided by the platform PIC, which in the x86 case is the i8259 (at least from Linux's perspective). So we can get rid of the IOAPIC, but we need to keep the i8259 (we have both a userspace and a kernel implementation too, so it should be fine). As for the PIT, we can omit it if we're running with KVM acceleration, as kvmclock will be used to calculate loops per jiffie and avoid the calibration, leaving it enabled otherwise. Thanks, Sergio.
On 25/09/19 17:04, Sergio Lopez wrote: > I'm going back to this level of the thread, because after your > suggestion I took a deeper look at how things work around the PIC, and > discovered I was completely wrong about my assumptions. > > For virtio-mmio devices, given that we don't have the ability to > configure vectors (as it's done in the PCI case) we're stuck with the > ones provided by the platform PIC, which in the x86 case is the i8259 > (at least from Linux's perspective). > > So we can get rid of the IOAPIC, but we need to keep the i8259 (we have > both a userspace and a kernel implementation too, so it should be fine). Hmm... I would have thought the vectors are just GSIs, which will be configured to the IOAPIC if it is present. Maybe something is causing Linux to ignore the IOAPIC? > As for the PIT, we can omit it if we're running with KVM acceleration, > as kvmclock will be used to calculate loops per jiffie and avoid the > calibration, leaving it enabled otherwise. Can you make it an OnOffAuto property, and default to on iff !KVM? Paolo
Paolo Bonzini <pbonzini@redhat.com> writes: > On 25/09/19 17:04, Sergio Lopez wrote: >> I'm going back to this level of the thread, because after your >> suggestion I took a deeper look at how things work around the PIC, and >> discovered I was completely wrong about my assumptions. >> >> For virtio-mmio devices, given that we don't have the ability to >> configure vectors (as it's done in the PCI case) we're stuck with the >> ones provided by the platform PIC, which in the x86 case is the i8259 >> (at least from Linux's perspective). >> >> So we can get rid of the IOAPIC, but we need to keep the i8259 (we have >> both a userspace and a kernel implementation too, so it should be fine). > > Hmm... I would have thought the vectors are just GSIs, which will be > configured to the IOAPIC if it is present. Maybe something is causing > Linux to ignore the IOAPIC? Turns out it was a bug in microvm. I was writing 0 to FW_CFG_NB_CPUS (because I was using x86ms->boot_cpus instead of ms->smp.cpus), which led to a broken MP table, causing Linux to ignore it and, as a side effect to disable IOAPIC symmetric I/O mode. After fixing it we can, indeed, boot without the i8259 \o/ : / # dmesg | grep legacy [ 0.074144] Using NULL legacy PIC / # cat /pr[ 12.116930] random: fast init done / # cat /proc/interrupts CPU0 CPU1 4: 0 278 IO-APIC 4-edge ttyS0 12: 48 0 IO-APIC 12-edge virtio0 NMI: 0 0 Non-maskable interrupts LOC: 124 98 Local timer interrupts SPU: 0 0 Spurious interrupts PMI: 0 0 Performance monitoring interrupts IWI: 0 0 IRQ work interrupts RTR: 0 0 APIC ICR read retries RES: 476 535 Rescheduling interrupts CAL: 0 76 Function call interrupts TLB: 0 0 TLB shootdowns HYP: 0 0 Hypervisor callback interrupts ERR: 0 MIS: 0 PIN: 0 0 Posted-interrupt notification event NPI: 0 0 Nested posted-interrupt event PIW: 0 0 Posted-interrupt wakeup event There's still one problem. If the Guest doesn't have TSC_DEADLINE_TIME, Linux hangs on APIC timer calibration. I'm looking for a way to work around this. Worst case scenario, we can check for that feature and add both PIC and PIT if is missing. >> As for the PIT, we can omit it if we're running with KVM acceleration, >> as kvmclock will be used to calculate loops per jiffie and avoid the >> calibration, leaving it enabled otherwise. > > Can you make it an OnOffAuto property, and default to on iff !KVM? Sure. Thanks, Sergio.
On 26/09/19 08:23, Sergio Lopez wrote: > > There's still one problem. If the Guest doesn't have TSC_DEADLINE_TIME, > Linux hangs on APIC timer calibration. I'm looking for a way to work > around this. Worst case scenario, we can check for that feature and add > both PIC and PIT if is missing. > Huh, that's a silly thing that Linux is doing! If KVM is in use, the LAPIC timer frequency is known to be 1 GHz. arch/x86/kernel/kvm.c can just set lapic_timer_period = 1000000000 / HZ; and that should disabled LAPIC calibration if TSC deadline is absent. Paolo
Paolo Bonzini <pbonzini@redhat.com> writes: > On 26/09/19 08:23, Sergio Lopez wrote: >> >> There's still one problem. If the Guest doesn't have TSC_DEADLINE_TIME, >> Linux hangs on APIC timer calibration. I'm looking for a way to work >> around this. Worst case scenario, we can check for that feature and add >> both PIC and PIT if is missing. >> > > Huh, that's a silly thing that Linux is doing! If KVM is in use, the > LAPIC timer frequency is known to be 1 GHz. > > arch/x86/kernel/kvm.c can just set > > lapic_timer_period = 1000000000 / HZ; > > and that should disabled LAPIC calibration if TSC deadline is absent. Given that they can only be omitted when an specific set of conditions is met, I think I'm going to make them optional but enabled by default. I'll also point to this in the documentation. Thanks, Sergio
On 26/09/19 12:16, Sergio Lopez wrote: >> If KVM is in use, the >> LAPIC timer frequency is known to be 1 GHz. >> >> arch/x86/kernel/kvm.c can just set >> >> lapic_timer_period = 1000000000 / HZ; >> >> and that should disabled LAPIC calibration if TSC deadline is absent. > Given that they can only be omitted when an specific set of conditions > is met, I think I'm going to make them optional but enabled by default. Please do introduce the infrastructure to make them OnOffAuto, and for now make Auto the same as On. We have time to review that since microvm is not versioned. Thanks, Paolo > I'll also point to this in the documentation.
Paolo Bonzini <pbonzini@redhat.com> writes: > On 26/09/19 12:16, Sergio Lopez wrote: >>> If KVM is in use, the >>> LAPIC timer frequency is known to be 1 GHz. >>> >>> arch/x86/kernel/kvm.c can just set >>> >>> lapic_timer_period = 1000000000 / HZ; >>> >>> and that should disabled LAPIC calibration if TSC deadline is absent. >> Given that they can only be omitted when an specific set of conditions >> is met, I think I'm going to make them optional but enabled by default. > > Please do introduce the infrastructure to make them OnOffAuto, and for > now make Auto the same as On. We have time to review that since microvm > is not versioned. OK, sounds like a good idea to me. Thanks, Sergio.
diff --git a/docs/microvm.txt b/docs/microvm.txt new file mode 100644 index 0000000000..0241226b2a --- /dev/null +++ b/docs/microvm.txt @@ -0,0 +1,78 @@ +Microvm is a machine type inspired by both NEMU and Firecracker, and +constructed after the machine model implemented by the latter. + +It's main purpose is providing users a minimalist machine type free +from the burden of legacy compatibility, serving as a stepping stone +for future projects aiming at improving boot times, reducing the +attack surface and slimming down QEMU's footprint. + +The microvm machine type supports the following devices: + + - ISA bus + - i8259 PIC + - LAPIC (implicit if using KVM) + - IOAPIC (defaults to kernel_irqchip_split = true) + - i8254 PIT + - MC146818 RTC (optional) + - kvmclock (if using KVM) + - fw_cfg + - One ISA serial port (optional) + - Up to eight virtio-mmio devices (configured by the user) + +It supports the following machine-specific options: + +microvm.option-roms=bool (Set off to disable loading option ROMs) +microvm.isa-serial=bool (Set off to disable the instantiation an ISA serial port) +microvm.rtc=bool (Set off to disable the instantiation of an MC146818 RTC) +microvm.kernel-cmdline=bool (Set off to disable adding virtio-mmio devices to the kernel cmdline) + +By default, microvm uses qboot as its BIOS, to obtain better boot +times, but it's also compatible with SeaBIOS. + +As no current FW is able to boot from a block device using virtio-mmio +as its transport, a microvm-based VM needs to be run using a host-side +kernel and, optionally, an initrd image. + +This is an example of instantiating a microvm VM with a virtio-mmio +based console: + +qemu-system-x86_64 -M microvm + -enable-kvm -cpu host -m 512m -smp 2 \ + -kernel vmlinux -append "console=hvc0 root=/dev/vda" \ + -nodefaults -no-user-config -nographic \ + -chardev stdio,id=virtiocon0,server \ + -device virtio-serial-device \ + -device virtconsole,chardev=virtiocon0 \ + -drive id=test,file=test.img,format=raw,if=none \ + -device virtio-blk-device,drive=test \ + -netdev tap,id=tap0,script=no,downscript=no \ + -device virtio-net-device,netdev=tap0 + +This is another example, this time using an ISA serial port, useful +for debugging purposes: + +qemu-system-x86_64 -M microvm \ + -enable-kvm -cpu host -m 512m -smp 2 \ + -kernel vmlinux -append "earlyprintk=ttyS0 console=ttyS0 root=/dev/vda" \ + -nodefaults -no-user-config -nographic \ + -serial stdio \ + -drive id=test,file=test.img,format=raw,if=none \ + -device virtio-blk-device,drive=test \ + -netdev tap,id=tap0,script=no,downscript=no \ + -device virtio-net-device,netdev=tap0 + +Finally, in this example a microvm VM is instantiated without RTC, +without an ISA serial port and without loading the option ROMs, +obtaining the smallest configuration: + +qemu-system-x86_64 -M microvm,rtc=off,isa-serial=off,option-roms=off \ + -enable-kvm -cpu host -m 512m -smp 2 \ + -kernel vmlinux -append "console=hvc0 root=/dev/vda" \ + -nodefaults -no-user-config -nographic \ + -chardev stdio,id=virtiocon0,server \ + -device virtio-serial-device \ + -device virtconsole,chardev=virtiocon0 \ + -drive id=test,file=test.img,format=raw,if=none \ + -device virtio-blk-device,drive=test \ + -netdev tap,id=tap0,script=no,downscript=no \ + -device virtio-net-device,netdev=tap0
Document the new microvm machine type. Signed-off-by: Sergio Lopez <slp@redhat.com> --- docs/microvm.txt | 78 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/microvm.txt