From patchwork Wed Jun 8 23:06:53 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jianjun Duan X-Patchwork-Id: 9165899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DD4D060832 for ; Wed, 8 Jun 2016 23:08:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C64F4282DC for ; Wed, 8 Jun 2016 23:08:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA8EC2830C; Wed, 8 Jun 2016 23:08:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 15F18282DC for ; Wed, 8 Jun 2016 23:08:10 +0000 (UTC) Received: from localhost ([::1]:59877 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bAmZz-0004iu-6G for patchwork-qemu-devel@patchwork.kernel.org; Wed, 08 Jun 2016 19:08:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44364) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bAmZH-0004Y1-G8 for qemu-devel@nongnu.org; Wed, 08 Jun 2016 19:07:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bAmZC-0008Kn-KC for qemu-devel@nongnu.org; Wed, 08 Jun 2016 19:07:22 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:54498 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bAmZC-0008Kf-F3 for qemu-devel@nongnu.org; Wed, 08 Jun 2016 19:07:18 -0400 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.11/8.16.0.11) with SMTP id u58N3pSq074805 for ; Wed, 8 Jun 2016 19:07:18 -0400 Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.149]) by mx0b-001b2d01.pphosted.com with ESMTP id 23e9m44fcr-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 08 Jun 2016 19:07:18 -0400 Received: from localhost by e31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 8 Jun 2016 17:07:17 -0600 Received: from d03dlp01.boulder.ibm.com (9.17.202.177) by e31.co.us.ibm.com (192.168.1.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 8 Jun 2016 17:07:11 -0600 X-IBM-Helo: d03dlp01.boulder.ibm.com X-IBM-MailFrom: duanj@linux.vnet.ibm.com X-IBM-RcptTo: aurelien@aurel32.net; dmitry@daynix.com; david@gibson.dropbear.id.au; blauwirbel@gmail.com; veroniabahaa@gmail.com; mark.cave-ayland@ilande.co.uk; leon.alrae@imgtec.com; peter.maydell@linaro.org; qemu-devel@nongnu.org; qemu-ppc@nongnu.org; amit.shah@redhat.com; kraxel@redhat.com; kwolf@redhat.com; mreitz@redhat.com; mst@redhat.com; pbonzini@redhat.com; quintela@redhat.com; rth@twiddle.net Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by d03dlp01.boulder.ibm.com (Postfix) with ESMTP id 3D9C61FF001F; Wed, 8 Jun 2016 17:06:55 -0600 (MDT) Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u58N7ANP45220012; Wed, 8 Jun 2016 23:07:10 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8E10BAC041; Wed, 8 Jun 2016 19:07:10 -0400 (EDT) Received: from jianjun-w530.ibm.com (unknown [9.48.122.30]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP id 34821AC03F; Wed, 8 Jun 2016 19:07:07 -0400 (EDT) From: Jianjun Duan To: qemu-devel@nongnu.org Date: Wed, 8 Jun 2016 16:06:53 -0700 X-Mailer: git-send-email 1.9.1 In-Reply-To: <1465427217-5551-1-git-send-email-duanj@linux.vnet.ibm.com> References: <1465427217-5551-1-git-send-email-duanj@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16060823-8235-0000-0000-0000088B39F6 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16060823-8236-0000-0000-0000320C717E Message-Id: <1465427217-5551-3-git-send-email-duanj@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-06-08_08:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1604210000 definitions=main-1606080261 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 148.163.158.5 Subject: [Qemu-devel] [QEMU RFC PATCH v4 2/6] migration: defined VMStateDescription struct for spapr_drc X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: veroniabahaa@gmail.com, peter.maydell@linaro.org, duanj@linux.vnet.ibm.com, mst@redhat.com, quintela@redhat.com, mark.cave-ayland@ilande.co.uk, mdroth@linux.vnet.ibm.com, mreitz@redhat.com, blauwirbel@gmail.com, amit.shah@redhat.com, qemu-ppc@nongnu.org, kraxel@redhat.com, kwolf@redhat.com, dmitry@daynix.com, pbonzini@redhat.com, rth@twiddle.net, leon.alrae@imgtec.com, aurelien@aurel32.net, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP To manage hotplug/unplug of dynamic resources such as PCI cards, memory, and CPU on sPAPR guests, a firmware abstraction known as a Dynamic Resource Connector (DRC) is used to assign a particular dynamic resource to the guest, and provide an interface for the guest to manage configuration/removal of the resource associated with it. To migrate the hotplugged resources in migration, the associated DRC state need be migrated. To migrate the DRC state, we defined the VMStateDescription struct for spapr_drc to enable the transmission of spapr_drc state in migration. Not all the elements in the DRC state are migrated. Only those ones modifiable or needed by guest actions or device add/remove operation are migrated. From the perspective of device hotplugging, if we hotplug a device on the source, we need to "coldplug" it on the target. The states across two hosts for the same device are not the same. Ideally we want the states be same after migration so that the device would function as hotplugged on the target. For example we can unplug it. The minimum DRC state we need to transfer should cover all the pieces changed by hotplugging. Out of the elements of the DRC state, isolation_state, allocation_sate, and configured are involved in the DR state transition diagram from PAPR+ 2.7, 13.4. configured and signalled are needed in attaching and detaching devices. indicator_state provides users with hardware state information. These 6 elements are migrated. detach_cb in the DRC state is a function pointer that cannot be migrated. We set it right after DRC state is migrated so that a migrated hot-unplug event could finish its work. The instance_id is used to identify objects in migration. We set instance_id of DRC using the unique index so that it is the same across migration. Signed-off-by: Jianjun Duan --- hw/ppc/spapr_drc.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++ hw/ppc/spapr_pci.c | 22 +++++++++++++++ include/hw/ppc/spapr_drc.h | 9 ++++++ 3 files changed, 100 insertions(+) diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c index 94c875d..7e6161d 100644 --- a/hw/ppc/spapr_drc.c +++ b/hw/ppc/spapr_drc.c @@ -617,6 +617,71 @@ static void spapr_dr_connector_instance_init(Object *obj) NULL, NULL, NULL, NULL); } +static bool spapr_drc_needed(void *opaque) +{ + sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque; + sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + bool rc = false; + sPAPRDREntitySense value; + + drck->entity_sense(drc, &value); + /* If no dev is plugged in there is no need to migrate the DRC state */ + if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) { + return false; + } + /* + * If there is dev plugged in, we need to migrate the DRC state when + * it is different from cold-plugged state + */ + switch(drc->type) { + /* for PCI type */ + case SPAPR_DR_CONNECTOR_TYPE_PCI: + rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) && + (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) && + drc->configured && drc->signalled && !drc->awaiting_release); + break; + /* for LMB type */ + case SPAPR_DR_CONNECTOR_TYPE_LMB: + rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) && + (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) && + drc->configured && drc->signalled && !drc->awaiting_release); + break; + default: + ; + } + + return rc; +} + +/* detach_cb needs be set since it is not migrated */ +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc, + spapr_drc_detach_cb *detach_cb) +{ + drc->detach_cb = detach_cb; +} + +/* return the unique drc index as instance_id for qom interfaces*/ +static int get_instance_id(DeviceState *dev) +{ + return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev))); +} + +static const VMStateDescription vmstate_spapr_drc = { + .name = "spapr_drc", + .version_id = 1, + .minimum_version_id = 1, + .needed = spapr_drc_needed, + .fields = (VMStateField []) { + VMSTATE_UINT32(isolation_state, sPAPRDRConnector), + VMSTATE_UINT32(allocation_state, sPAPRDRConnector), + VMSTATE_UINT32(indicator_state, sPAPRDRConnector), + VMSTATE_BOOL(configured, sPAPRDRConnector), + VMSTATE_BOOL(awaiting_release, sPAPRDRConnector), + VMSTATE_BOOL(signalled, sPAPRDRConnector), + VMSTATE_END_OF_LIST() + } +}; + static void spapr_dr_connector_class_init(ObjectClass *k, void *data) { DeviceClass *dk = DEVICE_CLASS(k); @@ -625,6 +690,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data) dk->reset = reset; dk->realize = realize; dk->unrealize = unrealize; + dk->vmsd = &vmstate_spapr_drc; + dk->dev_get_instance_id = get_instance_id; drck->set_isolation_state = set_isolation_state; drck->set_indicator_state = set_indicator_state; drck->set_allocation_state = set_allocation_state; @@ -638,6 +705,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data) drck->detach = detach; drck->release_pending = release_pending; drck->set_signalled = set_signalled; + drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb; + /* * Reason: it crashes FIXME find and document the real reason */ diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index 68de523..e03cec8 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -1575,11 +1575,33 @@ static void spapr_pci_pre_save(void *opaque) } } +/* + * detach_cb in the DRC state is a function pointer that cannot be + * migrated. We set it right after migration so that a migrated + * hot-unplug event could finish its work. + */ +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev, + void *opaque) +{ + sPAPRPHBState *sphb = opaque; + sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev); + sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb); +} + static int spapr_pci_post_load(void *opaque, int version_id) { sPAPRPHBState *sphb = opaque; gpointer key, value; int i; + PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus; + unsigned int bus_no = 0; + + /* Set detach_cb for the drc unconditionally after migration */ + if (bus) { + pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb, + &bus_no); + } for (i = 0; i < sphb->msi_devs_num; ++i) { key = g_memdup(&sphb->msi_devs[i].key, diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h index fa21ba0..a68433b 100644 --- a/include/hw/ppc/spapr_drc.h +++ b/include/hw/ppc/spapr_drc.h @@ -190,6 +190,15 @@ typedef struct sPAPRDRConnectorClass { void *detach_cb_opaque, Error **errp); bool (*release_pending)(sPAPRDRConnector *drc); void (*set_signalled)(sPAPRDRConnector *drc); + + /* + * QEMU interface for setting detach_cb after migration. + * detach_cb in the DRC state is a function pointer that cannot be + * migrated. We set it right after migration so that a migrated + * hot-unplug event could finish its work. + */ + void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc, + spapr_drc_detach_cb *detach_cb); } sPAPRDRConnectorClass; sPAPRDRConnector *spapr_dr_connector_new(Object *owner,