From patchwork Mon Mar 9 14:07:23 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 5967871 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 5102F9F2A9 for ; Mon, 9 Mar 2015 14:12:06 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 9738C201C0 for ; Mon, 9 Mar 2015 14:12:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 518E7202EB for ; Mon, 9 Mar 2015 14:11:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932164AbbCIOLl (ORCPT ); Mon, 9 Mar 2015 10:11:41 -0400 Received: from e23smtp06.au.ibm.com ([202.81.31.148]:39065 "EHLO e23smtp06.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932518AbbCIOJS (ORCPT ); Mon, 9 Mar 2015 10:09:18 -0400 Received: from /spool/local by e23smtp06.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 10 Mar 2015 00:09:16 +1000 Received: from d23dlp02.au.ibm.com (202.81.31.213) by e23smtp06.au.ibm.com (202.81.31.212) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 10 Mar 2015 00:09:14 +1000 Received: from d23relay07.au.ibm.com (d23relay07.au.ibm.com [9.190.26.37]) by d23dlp02.au.ibm.com (Postfix) with ESMTP id DCD482BB0051; Tue, 10 Mar 2015 01:09:13 +1100 (EST) Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139]) by d23relay07.au.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id t29E95UB12058834; Tue, 10 Mar 2015 01:09:13 +1100 Received: from d23av04.au.ibm.com (localhost [127.0.0.1]) by d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id t29E8dPc015221; Tue, 10 Mar 2015 01:08:40 +1100 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.192.253.14]) by d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id t29E8dqJ014772; Tue, 10 Mar 2015 01:08:39 +1100 Received: from bran.ozlabs.ibm.com (haven.au.ibm.com [9.192.253.15]) by ozlabs.au.ibm.com (Postfix) with ESMTP id 9DBF0A03FB; Tue, 10 Mar 2015 01:07:44 +1100 (AEDT) Received: from ka1.ozlabs.ibm.com (ka1.ozlabs.ibm.com [10.61.145.11]) by bran.ozlabs.ibm.com (Postfix) with ESMTP id D6A2516A9AE; Tue, 10 Mar 2015 01:07:43 +1100 (AEDT) From: Alexey Kardashevskiy To: linuxppc-dev@lists.ozlabs.org Cc: Alexey Kardashevskiy , Benjamin Herrenschmidt , Paul Mackerras , Alex Williamson , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 27/29] vfio: powerpc/spapr: powerpc/powernv/ioda2: Rework ownership Date: Tue, 10 Mar 2015 01:07:23 +1100 Message-Id: <1425910045-26167-28-git-send-email-aik@ozlabs.ru> X-Mailer: git-send-email 2.0.0 In-Reply-To: <1425910045-26167-1-git-send-email-aik@ozlabs.ru> References: <1425910045-26167-1-git-send-email-aik@ozlabs.ru> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15030914-0021-0000-0000-000000E354BF Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Before the IOMMU user would take control over the IOMMU table belonging to a specific IOMMU group. This approach did not allow sharing tables between IOMMU groups attached to the same container. This introduces a new IOMMU ownership flavour when the user can not just control the existing IOMMU table but remove/create tables on demand. If an IOMMU supports a set_ownership() callback, that lets the user have full control over the IOMMU group. When the ownership is taken, the platform code removes all the windows so the caller must create them. Before returning the ownership back to the platform code, the user has to unprogram and remove all the tables it created. Old-style ownership is still supported allowing VFIO to run on older P5IOC2 and IODA IO controllers. Signed-off-by: Alexey Kardashevskiy --- arch/powerpc/platforms/powernv/pci-ioda.c | 30 +++++++++++++++--- drivers/vfio/vfio_iommu_spapr_tce.c | 51 ++++++++++++++++++++++++------- 2 files changed, 66 insertions(+), 15 deletions(-) diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 07857c4..afb6906 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -1620,11 +1620,33 @@ static void pnv_ioda2_set_ownership(struct iommu_table_group *table_group, { struct pnv_ioda_pe *pe = container_of(table_group, struct pnv_ioda_pe, table_group); - if (enable) - iommu_take_ownership(table_group); - else - iommu_release_ownership(table_group); + if (enable) { + pnv_pci_ioda2_unset_window(&pe->table_group, 0); + pnv_pci_free_table(&pe->table_group.tables[0]); + } else { + struct iommu_table *tbl = &pe->table_group.tables[0]; + int64_t rc; + rc = pnv_pci_ioda2_create_table(&pe->table_group, 0, + IOMMU_PAGE_SHIFT_4K, + pe->phb->ioda.m32_pci_base, + POWERNV_IOMMU_DEFAULT_LEVELS, tbl); + if (rc) { + pe_err(pe, "Failed to create 32-bit TCE table, err %ld", + rc); + return; + } + + iommu_init_table(tbl, pe->phb->hose->node); + + rc = pnv_pci_ioda2_set_window(&pe->table_group, 0, tbl); + if (rc) { + pe_err(pe, "Failed to configure 32-bit TCE table, err %ld\n", + rc); + pnv_pci_free_table(tbl); + return; + } + } pnv_pci_ioda2_set_bypass(pe, !enable); } diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c index d665ddc..3bc0645 100644 --- a/drivers/vfio/vfio_iommu_spapr_tce.c +++ b/drivers/vfio/vfio_iommu_spapr_tce.c @@ -426,18 +426,11 @@ static int tce_iommu_clear(struct tce_container *container, static void tce_iommu_release(void *iommu_data) { struct tce_container *container = iommu_data; - struct iommu_table *tbl; - struct iommu_table_group *table_group; WARN_ON(container->grp); - if (container->grp) { - table_group = iommu_group_get_iommudata(container->grp); - tbl = &table_group->tables[0]; - tce_iommu_clear(container, tbl, tbl->it_offset, tbl->it_size); - + if (container->grp) tce_iommu_detach_group(iommu_data, container->grp); - } tce_mem_unregister_all(container); tce_iommu_disable(container); @@ -826,14 +819,24 @@ static int tce_iommu_attach_group(void *iommu_data, if (!table_group->ops || !table_group->ops->set_ownership) { ret = iommu_take_ownership(table_group); + } else if (!table_group->ops->create_table || + !table_group->ops->set_window) { + WARN_ON_ONCE(1); + ret = -EFAULT; } else { /* * Disable iommu bypass, otherwise the user can DMA to all of * our physical memory via the bypass window instead of just * the pages that has been explicitly mapped into the iommu */ + struct iommu_table tbltmp = { 0 }, *tbl = &tbltmp; + table_group->ops->set_ownership(table_group, true); - ret = 0; + ret = table_group->ops->create_table(table_group, 0, + IOMMU_PAGE_SHIFT_4K, + table_group->tce32_size, 1, tbl); + if (!ret) + ret = table_group->ops->set_window(table_group, 0, tbl); } if (ret) @@ -852,6 +855,7 @@ static void tce_iommu_detach_group(void *iommu_data, { struct tce_container *container = iommu_data; struct iommu_table_group *table_group; + long i; mutex_lock(&container->lock); if (iommu_group != container->grp) { @@ -875,10 +879,35 @@ static void tce_iommu_detach_group(void *iommu_data, BUG_ON(!table_group); /* Kernel owns the device now, we can restore bypass */ - if (!table_group->ops || !table_group->ops->set_ownership) + if (!table_group->ops || !table_group->ops->set_ownership) { + for (i = 0; i < IOMMU_TABLE_GROUP_MAX_TABLES; ++i) { + struct iommu_table *tbl = &table_group->tables[i]; + + if (!tbl->it_size) + continue; + + if (!tbl->it_ops) + goto unlock_exit; + tce_iommu_clear(container, tbl, + tbl->it_offset, tbl->it_size); + } iommu_release_ownership(table_group); - else + } else if (!table_group->ops->unset_window) { + WARN_ON_ONCE(1); + } else { + for (i = 0; i < IOMMU_TABLE_GROUP_MAX_TABLES; ++i) { + struct iommu_table *tbl = &table_group->tables[i]; + + table_group->ops->unset_window(table_group, i); + tce_iommu_clear(container, tbl, + tbl->it_offset, tbl->it_size); + + if (tbl->it_ops->free) + tbl->it_ops->free(tbl); + } + table_group->ops->set_ownership(table_group, false); + } unlock_exit: