From patchwork Wed Jun 1 08:57:43 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 9146755 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E123060777 for ; Wed, 1 Jun 2016 09:02:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D64B11FF65 for ; Wed, 1 Jun 2016 09:02:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CB06D25819; Wed, 1 Jun 2016 09:02:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, MSGID_FROM_MTA_HEADER,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1DD671FF65 for ; Wed, 1 Jun 2016 09:02:41 +0000 (UTC) Received: from localhost ([::1]:41084 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b822y-0008P8-SQ for patchwork-qemu-devel@patchwork.kernel.org; Wed, 01 Jun 2016 05:02:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49053) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b822E-0008HF-VN for qemu-devel@nongnu.org; Wed, 01 Jun 2016 05:02:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b822A-0001jO-OP for qemu-devel@nongnu.org; Wed, 01 Jun 2016 05:01:53 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:43166) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b822A-0001in-G0 for qemu-devel@nongnu.org; Wed, 01 Jun 2016 05:01:50 -0400 Received: from pps.filterd (m0075771.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.11/8.16.0.11) with SMTP id u518wwi5029365 for ; Wed, 1 Jun 2016 05:01:48 -0400 Message-Id: <201606010901.u518wwi5029365@mx0a-001b2d01.pphosted.com> Received: from e23smtp07.au.ibm.com (e23smtp07.au.ibm.com [202.81.31.140]) by mx0a-001b2d01.pphosted.com with ESMTP id 239svy5ajc-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 01 Jun 2016 05:01:48 -0400 Received: from localhost by e23smtp07.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 1 Jun 2016 19:01:45 +1000 Received: from d23dlp03.au.ibm.com (202.81.31.214) by e23smtp07.au.ibm.com (202.81.31.204) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 1 Jun 2016 19:01:43 +1000 X-IBM-Helo: d23dlp03.au.ibm.com X-IBM-MailFrom: aik@ozlabs.ru X-IBM-RcptTo: qemu-devel@nongnu.org;qemu-ppc@nongnu.org Received: from d23relay10.au.ibm.com (d23relay10.au.ibm.com [9.190.26.77]) by d23dlp03.au.ibm.com (Postfix) with ESMTP id E656F3578057; Wed, 1 Jun 2016 19:00:14 +1000 (EST) Received: from d23av06.au.ibm.com (d23av06.au.ibm.com [9.190.235.151]) by d23relay10.au.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u51900vj64094246; Wed, 1 Jun 2016 19:00:08 +1000 Received: from d23av06.au.ibm.com (localhost [127.0.0.1]) by d23av06.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u518xacS031157; Wed, 1 Jun 2016 18:59:36 +1000 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.192.253.14]) by d23av06.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u518xaRC030511; Wed, 1 Jun 2016 18:59:36 +1000 Received: from bran.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) by ozlabs.au.ibm.com (Postfix) with ESMTP id 9F46EA039A; Wed, 1 Jun 2016 18:57:49 +1000 (AEST) Received: from vpl2.ozlabs.ibm.com (vpl2.ozlabs.ibm.com [10.61.141.27]) by bran.ozlabs.ibm.com (Postfix) with ESMTP id 99D5BE3AE5; Wed, 1 Jun 2016 18:57:49 +1000 (AEST) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Wed, 1 Jun 2016 18:57:43 +1000 X-Mailer: git-send-email 2.5.0.rc3 In-Reply-To: <1464771463-37214-1-git-send-email-aik@ozlabs.ru> References: <1464771463-37214-1-git-send-email-aik@ozlabs.ru> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16060109-0044-0000-0000-000001AE9082 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16060109-0045-0000-0000-000004D85FF4 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-06-01_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=97 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1604210000 definitions=main-1606010107 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 148.163.156.1 Subject: [Qemu-devel] [PATCH qemu v17 12/12] spapr_iommu, vfio, memory: Notify IOMMU about starting/stopping listening X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kardashevskiy , Alex Williamson , qemu-ppc@nongnu.org, Alexander Graf , David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP The sPAPR TCE tables manage 2 copies when VFIO is using an IOMMU - a guest view of the table and a hardware TCE table. If there is no VFIO presense in the address space, then just the guest view is used, if this is the case, it is allocated in the KVM. However since there is no support yet for VFIO in KVM TCE hypercalls, when we start using VFIO, we need to move the guest view from KVM to the userspace; and we need to do this for every IOMMU on a bus with VFIO devices. This adds notify_started/notify_stopped callbacks in MemoryRegionIOMMUOps to notify IOMMU that listeners were set/removed. This allows IOMMU to take necessary steps before actual notifications happen and do proper cleanup when the last notifier is removed. This implements the callbacks for the sPAPR IOMMU - notify_started() reallocated the guest view to the user space, notify_stopped() does the opposite. This removes explicit spapr_tce_set_need_vfio() call from PCI hotplug path as the new callbacks do this better - they notify IOMMU at the exact moment when the configuration is changed, and this also includes the case of PCI hot unplug. This adds MemoryRegion* to memory_region_unregister_iommu_notifier() as we need iommu_ops to call notify_stopped() and Notifier* does not store the owner. Signed-off-by: Alexey Kardashevskiy Reviewed-by: David Gibson --- Changes: v17: * replaced IOMMU users counting with simple QLIST_EMPTY() * renamed the callbacks * removed requirement for region_del() to be called on memory_listener_unregister() v16: * added a use counter in VFIOAddressSpace->VFIOIOMMUMR v15: * s/need_vfio/vfio-Users/g --- hw/ppc/spapr_iommu.c | 12 ++++++++++++ hw/ppc/spapr_pci.c | 6 ------ hw/vfio/common.c | 5 +++-- include/exec/memory.h | 8 +++++++- memory.c | 10 +++++++++- 5 files changed, 31 insertions(+), 10 deletions(-) diff --git a/hw/ppc/spapr_iommu.c b/hw/ppc/spapr_iommu.c index 90a45c0..994a8a0 100644 --- a/hw/ppc/spapr_iommu.c +++ b/hw/ppc/spapr_iommu.c @@ -156,6 +156,16 @@ static uint64_t spapr_tce_get_page_sizes(MemoryRegion *iommu) return 1ULL << tcet->page_shift; } +static void spapr_tce_notify_started(MemoryRegion *iommu) +{ + spapr_tce_set_need_vfio(container_of(iommu, sPAPRTCETable, iommu), true); +} + +static void spapr_tce_notify_stopped(MemoryRegion *iommu) +{ + spapr_tce_set_need_vfio(container_of(iommu, sPAPRTCETable, iommu), false); +} + static int spapr_tce_table_post_load(void *opaque, int version_id) { sPAPRTCETable *tcet = SPAPR_TCE_TABLE(opaque); @@ -236,6 +246,8 @@ static const VMStateDescription vmstate_spapr_tce_table = { static MemoryRegionIOMMUOps spapr_iommu_ops = { .translate = spapr_tce_translate_iommu, .get_page_sizes = spapr_tce_get_page_sizes, + .notify_started = spapr_tce_notify_started, + .notify_stopped = spapr_tce_notify_stopped, }; static int spapr_tce_table_realize(DeviceState *dev) diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index bcf0360..06ce902 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -1089,12 +1089,6 @@ static void spapr_phb_add_pci_device(sPAPRDRConnector *drc, void *fdt = NULL; int fdt_start_offset = 0, fdt_size; - if (object_dynamic_cast(OBJECT(pdev), "vfio-pci")) { - sPAPRTCETable *tcet = spapr_tce_find_by_liobn(phb->dma_liobn[0]); - - spapr_tce_set_need_vfio(tcet, true); - } - fdt = create_device_tree(&fdt_size); fdt_start_offset = spapr_create_pci_child_dt(phb, pdev, fdt, 0); if (!fdt_start_offset) { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 7f55c26..356640e 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -522,7 +522,8 @@ static void vfio_listener_region_del(MemoryListener *listener, QLIST_FOREACH(giommu, &container->giommu_list, giommu_next) { if (giommu->iommu == section->mr) { - memory_region_unregister_iommu_notifier(&giommu->n); + memory_region_unregister_iommu_notifier(giommu->iommu, + &giommu->n); QLIST_REMOVE(giommu, giommu_next); g_free(giommu); break; @@ -1094,7 +1095,7 @@ static void vfio_disconnect_container(VFIOGroup *group) QLIST_REMOVE(container, next); QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) { - memory_region_unregister_iommu_notifier(&giommu->n); + memory_region_unregister_iommu_notifier(giommu->iommu, &giommu->n); QLIST_REMOVE(giommu, giommu_next); g_free(giommu); } diff --git a/include/exec/memory.h b/include/exec/memory.h index bd9625f..f08439b 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -151,6 +151,10 @@ struct MemoryRegionIOMMUOps { IOMMUTLBEntry (*translate)(MemoryRegion *iommu, hwaddr addr, bool is_write); /* Returns supported page sizes */ uint64_t (*get_page_sizes)(MemoryRegion *iommu); + /* Called when the first notifier is set */ + void (*notify_started)(MemoryRegion *iommu); + /* Called when the last notifier is removed */ + void (*notify_stopped)(MemoryRegion *iommu); }; typedef struct CoalescedMemoryRange CoalescedMemoryRange; @@ -619,9 +623,11 @@ void memory_region_iommu_replay(MemoryRegion *mr, Notifier *n, bool is_write); * memory_region_unregister_iommu_notifier: unregister a notifier for * changes to IOMMU translation entries. * + * @mr: the memory region which was observed and for which notity_stopped() + * needs to be called * @n: the notifier to be removed. */ -void memory_region_unregister_iommu_notifier(Notifier *n); +void memory_region_unregister_iommu_notifier(MemoryRegion *mr, Notifier *n); /** * memory_region_name: get a memory region's name diff --git a/memory.c b/memory.c index 761ae92..ee41649 100644 --- a/memory.c +++ b/memory.c @@ -1497,6 +1497,10 @@ bool memory_region_is_logging(MemoryRegion *mr, uint8_t client) void memory_region_register_iommu_notifier(MemoryRegion *mr, Notifier *n) { + if (mr->iommu_ops->notify_started && + QLIST_EMPTY(&mr->iommu_notify.notifiers)) { + mr->iommu_ops->notify_started(mr); + } notifier_list_add(&mr->iommu_notify, n); } @@ -1530,9 +1534,13 @@ void memory_region_iommu_replay(MemoryRegion *mr, Notifier *n, bool is_write) } } -void memory_region_unregister_iommu_notifier(Notifier *n) +void memory_region_unregister_iommu_notifier(MemoryRegion *mr, Notifier *n) { notifier_remove(n); + if (mr->iommu_ops->notify_stopped && + QLIST_EMPTY(&mr->iommu_notify.notifiers)) { + mr->iommu_ops->notify_stopped(mr); + } } void memory_region_notify_iommu(MemoryRegion *mr,