From patchwork Mon Jan 13 14:01:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Kotov X-Patchwork-Id: 11330273 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F16EC6C1 for ; Mon, 13 Jan 2020 14:04:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 674D420678 for ; Mon, 13 Jan 2020 14:04:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=yandex-team.ru header.i=@yandex-team.ru header.b="Iab57YVx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 674D420678 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=yandex-team.ru Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:50878 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ir0KA-0006jj-Hc for patchwork-qemu-devel@patchwork.kernel.org; Mon, 13 Jan 2020 09:04:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58447) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ir0Hl-0001up-W3 for qemu-devel@nongnu.org; Mon, 13 Jan 2020 09:01:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ir0Hh-0005jM-Qg for qemu-devel@nongnu.org; Mon, 13 Jan 2020 09:01:39 -0500 Received: from forwardcorp1p.mail.yandex.net ([2a02:6b8:0:1472:2741:0:8b6:217]:37462) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ir0Hh-0005cq-3L for qemu-devel@nongnu.org; Mon, 13 Jan 2020 09:01:37 -0500 Received: from mxbackcorp2j.mail.yandex.net (mxbackcorp2j.mail.yandex.net [IPv6:2a02:6b8:0:1619::119]) by forwardcorp1p.mail.yandex.net (Yandex) with ESMTP id E6CFB2E14CB; Mon, 13 Jan 2020 17:01:31 +0300 (MSK) Received: from vla1-5a8b76e65344.qloud-c.yandex.net (vla1-5a8b76e65344.qloud-c.yandex.net [2a02:6b8:c0d:3183:0:640:5a8b:76e6]) by mxbackcorp2j.mail.yandex.net (mxbackcorp/Yandex) with ESMTP id 550MBlBHoJ-1VLah1EM; Mon, 13 Jan 2020 17:01:31 +0300 Precedence: bulk DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1578924091; bh=rRxTyrByTSj+qunoOKpt1PLG0YLSoHmiXfDV8UvB+z4=; h=Message-Id:Date:Subject:To:From:Cc; b=Iab57YVxaJK1/5mbi75lZOA1SUldTPikFQgj+h++sRoPfP66VGZLtEyxY6OU4sIiH OXi6YxPJTFvyESJsbBtpnXoAAq7I2ZgFDey4X0px3H+Xg6mSsxOwZmic0p5vu8s/wH V2N6x1I7QDT5LXV4f91s3rT/DAZ+oSqIeYjqhOrg= Authentication-Results: mxbackcorp2j.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Received: from dynamic-red.dhcp.yndx.net (dynamic-red.dhcp.yndx.net [2a02:6b8:0:408:c8ef:4094:aec7:9c0f]) by vla1-5a8b76e65344.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id B0RjkbXcyT-1VV0qsew; Mon, 13 Jan 2020 17:01:31 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) From: Yury Kotov To: qemu-devel@nongnu.org Subject: [RFC PATCH] pcie: Defer hot unplug until migration is complete Date: Mon, 13 Jan 2020 17:01:22 +0300 Message-Id: <20200113140122.29486-1-yury-kotov@yandex-team.ru> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a02:6b8:0:1472:2741:0:8b6:217 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yc-core@yandex-team.ru, Juan Quintela , "Dr. David Alan Gilbert" , "Michael S. Tsirkin" Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Devices hot-plug during migration is not allowed and disabled in corresponding QMP-commands (device_add, device_del). But guest still can unplug device by powering it off (Example: echo 0 > /sys/bus/pci/slots/XXX/power). Fix it by deferring unplugging until the migration is complete. Signed-off-by: Yury Kotov --- hw/pci-bridge/gen_pcie_root_port.c | 7 ++++ hw/pci-bridge/ioh3420.c | 7 ++++ hw/pci-bridge/xio3130_downstream.c | 7 ++++ hw/pci/pcie.c | 54 +++++++++++++++++++++++------- hw/pci/pcie_port.c | 47 ++++++++++++++++++++++++++ include/hw/pci/pcie.h | 1 + include/hw/pci/pcie_port.h | 20 +++++++++++ 7 files changed, 130 insertions(+), 13 deletions(-) diff --git a/hw/pci-bridge/gen_pcie_root_port.c b/hw/pci-bridge/gen_pcie_root_port.c index 9eaefebca8..5b3c202341 100644 --- a/hw/pci-bridge/gen_pcie_root_port.c +++ b/hw/pci-bridge/gen_pcie_root_port.c @@ -100,6 +100,9 @@ static void gen_rp_realize(DeviceState *dev, Error **errp) } } +static const VMStateDescription vmstate_rp_deffered_unplug = + VMSTATE_DEFFERED_UNPLUG("pcie-root-port"); + static const VMStateDescription vmstate_rp_dev = { .name = "pcie-root-port", .priority = MIG_PRI_PCI_BUS, @@ -114,6 +117,10 @@ static const VMStateDescription vmstate_rp_dev = { GenPCIERootPort, gen_rp_test_migrate_msix), VMSTATE_END_OF_LIST() + }, + .subsections = (const VMStateDescription * []) { + &vmstate_rp_deffered_unplug, + NULL } }; diff --git a/hw/pci-bridge/ioh3420.c b/hw/pci-bridge/ioh3420.c index f1e16135a3..2399a9a87f 100644 --- a/hw/pci-bridge/ioh3420.c +++ b/hw/pci-bridge/ioh3420.c @@ -82,6 +82,9 @@ static void ioh3420_interrupts_uninit(PCIDevice *d) msi_uninit(d); } +static const VMStateDescription vmstate_ioh3420_deffered_unplug = + VMSTATE_DEFFERED_UNPLUG("ioh-3240-express-root-port"); + static const VMStateDescription vmstate_ioh3420 = { .name = "ioh-3240-express-root-port", .priority = MIG_PRI_PCI_BUS, @@ -93,6 +96,10 @@ static const VMStateDescription vmstate_ioh3420 = { VMSTATE_STRUCT(parent_obj.parent_obj.parent_obj.exp.aer_log, PCIESlot, 0, vmstate_pcie_aer_log, PCIEAERLog), VMSTATE_END_OF_LIST() + }, + .subsections = (const VMStateDescription * []) { + &vmstate_ioh3420_deffered_unplug, + NULL } }; diff --git a/hw/pci-bridge/xio3130_downstream.c b/hw/pci-bridge/xio3130_downstream.c index a9f084b863..a5b4fe08ee 100644 --- a/hw/pci-bridge/xio3130_downstream.c +++ b/hw/pci-bridge/xio3130_downstream.c @@ -139,6 +139,9 @@ static Property xio3130_downstream_props[] = { DEFINE_PROP_END_OF_LIST() }; +static const VMStateDescription vmstate_xio3130_downstream_deffered_unplug = + VMSTATE_DEFFERED_UNPLUG("xio3130-express-downstream-port"); + static const VMStateDescription vmstate_xio3130_downstream = { .name = "xio3130-express-downstream-port", .priority = MIG_PRI_PCI_BUS, @@ -150,6 +153,10 @@ static const VMStateDescription vmstate_xio3130_downstream = { VMSTATE_STRUCT(parent_obj.parent_obj.parent_obj.exp.aer_log, PCIESlot, 0, vmstate_pcie_aer_log, PCIEAERLog), VMSTATE_END_OF_LIST() + }, + .subsections = (const VMStateDescription * []) { + &vmstate_xio3130_downstream_deffered_unplug, + NULL } }; diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c index 08718188bb..29f0e5c05b 100644 --- a/hw/pci/pcie.c +++ b/hw/pci/pcie.c @@ -28,6 +28,8 @@ #include "hw/pci/pcie_regs.h" #include "hw/pci/pcie_port.h" #include "qemu/range.h" +#include "sysemu/sysemu.h" +#include "migration/misc.h" //#define DEBUG_PCIE #ifdef DEBUG_PCIE @@ -575,6 +577,7 @@ void pcie_cap_slot_reset(PCIDevice *dev) if (dev->cap_present & QEMU_PCIE_SLTCAP_PCP) { /* Downstream ports enforce device number 0. */ + PCIESlot *slot = PCIE_SLOT(dev); bool populated = pci_bridge_get_sec_bus(PCI_BRIDGE(dev))->devices[0]; uint16_t pic; @@ -588,6 +591,7 @@ void pcie_cap_slot_reset(PCIDevice *dev) pic = populated ? PCI_EXP_SLTCTL_PIC_ON : PCI_EXP_SLTCTL_PIC_OFF; pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTCTL, pic); + slot->unplug_is_deferred = false; } pci_word_test_and_clear_mask(exp_cap + PCI_EXP_SLTSTA, @@ -608,13 +612,42 @@ void pcie_cap_slot_get(PCIDevice *dev, uint16_t *slt_ctl, uint16_t *slt_sta) *slt_sta = pci_get_word(exp_cap + PCI_EXP_SLTSTA); } +static void pcie_cap_slot_unplug(PCIDevice *dev) +{ + uint32_t pos = dev->exp.exp_cap; + uint8_t *exp_cap = dev->config + pos; + PCIBus *sec_bus = pci_bridge_get_sec_bus(PCI_BRIDGE(dev)); + + pci_for_each_device(sec_bus, pci_bus_num(sec_bus), + pcie_unplug_device, NULL); + pci_word_test_and_clear_mask(exp_cap + PCI_EXP_SLTSTA, PCI_EXP_SLTSTA_PDS); + if (dev->cap_present & QEMU_PCIE_LNKSTA_DLLLA) { + pci_word_test_and_clear_mask(exp_cap + PCI_EXP_LNKSTA, + PCI_EXP_LNKSTA_DLLLA); + } + pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA, PCI_EXP_SLTSTA_PDC); + hotplug_event_notify(dev); +} + +void pcie_cap_slot_deferred_unplug(PCIDevice *dev) +{ + PCIESlot *slot = PCIE_SLOT(dev); + + if (migration_is_idle() && slot->unplug_is_deferred) { + pcie_cap_slot_unplug(dev); + slot->unplug_is_deferred = false; + } +} + void pcie_cap_slot_write_config(PCIDevice *dev, uint16_t old_slt_ctl, uint16_t old_slt_sta, uint32_t addr, uint32_t val, int len) { + PCIESlot *slot = PCIE_SLOT(dev); uint32_t pos = dev->exp.exp_cap; uint8_t *exp_cap = dev->config + pos; uint16_t sltsta = pci_get_word(exp_cap + PCI_EXP_SLTSTA); + bool may_unplug; if (ranges_overlap(addr, len, pos + PCI_EXP_SLTSTA, 2)) { /* @@ -660,22 +693,17 @@ void pcie_cap_slot_write_config(PCIDevice *dev, * this is a work around for guests that overwrite * control of powered off slots before powering them on. */ - if ((sltsta & PCI_EXP_SLTSTA_PDS) && (val & PCI_EXP_SLTCTL_PCC) && - (val & PCI_EXP_SLTCTL_PIC_OFF) == PCI_EXP_SLTCTL_PIC_OFF && + may_unplug = (val & PCI_EXP_SLTCTL_PCC) && + (val & PCI_EXP_SLTCTL_PIC_OFF) == PCI_EXP_SLTCTL_PIC_OFF; + if (may_unplug && (sltsta & PCI_EXP_SLTSTA_PDS) && (!(old_slt_ctl & PCI_EXP_SLTCTL_PCC) || (old_slt_ctl & PCI_EXP_SLTCTL_PIC_OFF) != PCI_EXP_SLTCTL_PIC_OFF)) { - PCIBus *sec_bus = pci_bridge_get_sec_bus(PCI_BRIDGE(dev)); - pci_for_each_device(sec_bus, pci_bus_num(sec_bus), - pcie_unplug_device, NULL); - - pci_word_test_and_clear_mask(exp_cap + PCI_EXP_SLTSTA, - PCI_EXP_SLTSTA_PDS); - if (dev->cap_present & QEMU_PCIE_LNKSTA_DLLLA) { - pci_word_test_and_clear_mask(exp_cap + PCI_EXP_LNKSTA, - PCI_EXP_LNKSTA_DLLLA); + slot->unplug_is_deferred = !migration_is_idle(); + if (!slot->unplug_is_deferred) { + pcie_cap_slot_unplug(dev); } - pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA, - PCI_EXP_SLTSTA_PDC); + } else if (!may_unplug) { + slot->unplug_is_deferred = false; } hotplug_event_notify(dev); diff --git a/hw/pci/pcie_port.c b/hw/pci/pcie_port.c index c19a9be592..bd5fbf6827 100644 --- a/hw/pci/pcie_port.c +++ b/hw/pci/pcie_port.c @@ -23,6 +23,9 @@ #include "hw/qdev-properties.h" #include "qemu/module.h" #include "hw/hotplug.h" +#include "sysemu/runstate.h" +#include "migration/migration.h" +#include "migration/misc.h" void pcie_port_init_reg(PCIDevice *d) { @@ -150,6 +153,48 @@ static Property pcie_slot_props[] = { DEFINE_PROP_END_OF_LIST() }; +bool vmstate_deffered_unplug_needed(void *opaque) +{ + PCIESlot *slot = opaque; + + return slot->unplug_is_deferred; +} + +static void pcie_slot_migration_notifier_cb(Notifier *notifier, void *data) +{ + PCIESlot *slot = container_of(notifier, PCIESlot, migration_notifier); + + pcie_cap_slot_deferred_unplug(PCI_DEVICE(slot)); +} + +static void pcie_slot_vm_state_change(void *opaque, int running, RunState state) +{ + PCIESlot *slot = opaque; + + pcie_cap_slot_deferred_unplug(PCI_DEVICE(slot)); +} + +static void pcie_slot_init(Object *obj) +{ + PCIESlot *slot = PCIE_SLOT(obj); + + slot->unplug_is_deferred = false; + slot->migration_notifier = (Notifier) { + .notify = pcie_slot_migration_notifier_cb + }; + add_migration_state_change_notifier(&slot->migration_notifier); + slot->vmstate_change = + qemu_add_vm_change_state_handler(pcie_slot_vm_state_change, slot); +} + +static void pcie_slot_finalize(Object *obj) +{ + PCIESlot *slot = PCIE_SLOT(obj); + + remove_migration_state_change_notifier(&slot->migration_notifier); + qemu_del_vm_change_state_handler(slot->vmstate_change); +} + static void pcie_slot_class_init(ObjectClass *oc, void *data) { DeviceClass *dc = DEVICE_CLASS(oc); @@ -166,6 +211,8 @@ static const TypeInfo pcie_slot_type_info = { .name = TYPE_PCIE_SLOT, .parent = TYPE_PCIE_PORT, .instance_size = sizeof(PCIESlot), + .instance_init = pcie_slot_init, + .instance_finalize = pcie_slot_finalize, .abstract = true, .class_init = pcie_slot_class_init, .interfaces = (InterfaceInfo[]) { diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h index 7064875835..128f26199e 100644 --- a/include/hw/pci/pcie.h +++ b/include/hw/pci/pcie.h @@ -110,6 +110,7 @@ void pcie_cap_slot_get(PCIDevice *dev, uint16_t *slt_ctl, uint16_t *slt_sta); void pcie_cap_slot_write_config(PCIDevice *dev, uint16_t old_slt_ctl, uint16_t old_slt_sta, uint32_t addr, uint32_t val, int len); +void pcie_cap_slot_deferred_unplug(PCIDevice *dev); int pcie_cap_slot_post_load(void *opaque, int version_id); void pcie_cap_slot_push_attention_button(PCIDevice *dev); diff --git a/include/hw/pci/pcie_port.h b/include/hw/pci/pcie_port.h index 7515430087..32e45f0c89 100644 --- a/include/hw/pci/pcie_port.h +++ b/include/hw/pci/pcie_port.h @@ -23,6 +23,9 @@ #include "hw/pci/pci_bridge.h" #include "hw/pci/pci_bus.h" +#include "qemu/notify.h" +#include "sysemu/sysemu.h" +#include "migration/vmstate.h" #define TYPE_PCIE_PORT "pcie-port" #define PCIE_PORT(obj) OBJECT_CHECK(PCIEPort, (obj), TYPE_PCIE_PORT) @@ -44,6 +47,10 @@ void pcie_port_init_reg(PCIDevice *d); struct PCIESlot { /*< private >*/ PCIEPort parent_obj; + bool unplug_is_deferred; + Notifier migration_notifier; + VMChangeStateEntry *vmstate_change; + /*< public >*/ /* pci express switch port with slot */ @@ -58,6 +65,19 @@ struct PCIESlot { QLIST_ENTRY(PCIESlot) next; }; +bool vmstate_deffered_unplug_needed(void *opaque); + +#define VMSTATE_DEFFERED_UNPLUG(parent_section_name) { \ + .name = parent_section_name "/deffered-unplug", \ + .version_id = 1, \ + .minimum_version_id = 1, \ + .needed = vmstate_deffered_unplug_needed, \ + .fields = (VMStateField[]) { \ + VMSTATE_BOOL(unplug_is_deferred, PCIESlot), \ + VMSTATE_END_OF_LIST() \ + } \ +} + void pcie_chassis_create(uint8_t chassis_number); PCIESlot *pcie_chassis_find_slot(uint8_t chassis, uint16_t slot); int pcie_chassis_add_slot(struct PCIESlot *slot);