From patchwork Wed Feb 13 09:29:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Knut Omang X-Patchwork-Id: 10809445 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B93F13BF for ; Wed, 13 Feb 2019 09:31:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49B2A2CA92 for ; Wed, 13 Feb 2019 09:31:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3DC6C2CAD0; Wed, 13 Feb 2019 09:31:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9EB4D2CACB for ; Wed, 13 Feb 2019 09:31:54 +0000 (UTC) Received: from localhost ([127.0.0.1]:53848 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gtqtV-0008GH-Vb for patchwork-qemu-devel@patchwork.kernel.org; Wed, 13 Feb 2019 04:31:54 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33265) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gtqsF-0007Ef-NR for qemu-devel@nongnu.org; Wed, 13 Feb 2019 04:30:37 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gtqsB-00021z-T1 for qemu-devel@nongnu.org; Wed, 13 Feb 2019 04:30:33 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:56834) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gtqs9-0001r0-T3 for qemu-devel@nongnu.org; Wed, 13 Feb 2019 04:30:31 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9OZ0b132168; Wed, 13 Feb 2019 09:30:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2018-07-02; bh=qN6KXV4/VaEUMeTlnFW0ABU3nM8MF9X8VJDb1Oing6Q=; b=cbIJbj8bZnXQjBqOfLwaLLJOkZaXThMcWVNDW7EUCP0+0SenNCfXgiW9NTCGs7nDzlxN mEG2t9hMuNXvKyIpu0d5o/vKrzgGK3NAlB8s+DPt6VnagxrzDuBxSxcS7pQRV25lx16b oz/FSes3icDwVdgUzNKiOaSzDItNv/Y84+B0M6+WIDxOmpnmtflppZZoq6QlUDgLrQDs T8pNETVkaJqg/oMgKQQGWDkszHnV5o3wSeWduGzDmceLnyARsJ5EDkVUGQync7z9P1Un OiskNE+lnml1pcg9asuNi6zRUNWO1ZBhb1jOo13GLSxm5fa5sxiXcJ9hJ3QfJggRIPyT Wg== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2qhrekgtkc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:30:21 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9UKgL006570 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:30:20 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1D9UJNH005214; Wed, 13 Feb 2019 09:30:19 GMT Received: from abi.no.oracle.com (/10.172.144.123) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 01:30:18 -0800 From: Knut Omang To: qemu-devel@nongnu.org Date: Wed, 13 Feb 2019 10:29:58 +0100 Message-Id: <3df33b2bd332259dc548751b3c3c5eaa297428b0.1550050121.git-series.knut.omang@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=973 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v4 1/2] pcie: Add a simple PCIe ACS (Access Control Services) helper function X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Michael S . Tsirkin" , Tal Attaly , Knut Omang , Alex Williamson , Elijah Shakkour , Stefan Hajnoczi Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Add a helper function to add PCIe capability for Access Control Services (ACS) ACS support in the associated root port is needed to pass through individual functions of a device to different VMs with VFIO without Alex Williamson's pcie_acs_override kernel patch or similar in the guest. Single function devices, or multifunction devices that all goes to the same VM works fine even without ACS, as VFIO will avoid putting the root port itself into the IOMMU group even without ACS support in the port. Multifunction endpoints may also implement an ACS capability, only on function 0, and with more limited features. Signed-off-by: Knut Omang --- hw/pci/pcie.c | 39 +++++++++++++++++++++++++++++++++++++++- include/hw/pci/pcie.h | 6 ++++++- include/hw/pci/pcie_regs.h | 4 ++++- 3 files changed, 49 insertions(+) diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c index 230478f..6e87994 100644 --- a/hw/pci/pcie.c +++ b/hw/pci/pcie.c @@ -906,3 +906,42 @@ void pcie_ats_init(PCIDevice *dev, uint16_t offset) pci_set_word(dev->wmask + dev->exp.ats_cap + PCI_ATS_CTRL, 0x800f); } + +/* ACS (Access Control Services) */ +void pcie_acs_init(PCIDevice *dev, uint16_t offset) +{ + bool is_pcie_slot = !!object_dynamic_cast(OBJECT(dev), TYPE_PCIE_SLOT); + uint16_t cap_bits = 0; + + /* + * For endpoints, only multifunction devices may have an + * ACS capability, and only on function 0: + */ + assert(is_pcie_slot || + ((dev->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) && + PCI_FUNC(dev->devfn))); + + pcie_add_capability(dev, PCI_EXT_CAP_ID_ACS, PCI_ACS_VER, offset, + PCI_ACS_SIZEOF); + dev->exp.acs_cap = offset; + + if (is_pcie_slot) { + /* + * Endpoints may also implement ACS, and optionally RR and CR, + * if they want to support p2p, but only slots may + * implement SV, TB or UF: + */ + cap_bits = + PCI_ACS_SV | PCI_ACS_TB | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF; + } + + pci_set_word(dev->config + offset + PCI_ACS_CAP, cap_bits); + pci_set_word(dev->wmask + offset + PCI_ACS_CTRL, cap_bits); +} + +void pcie_acs_reset(PCIDevice *dev) +{ + if (dev->exp.acs_cap) { + pci_set_word(dev->config + dev->exp.acs_cap + PCI_ACS_CTRL, 0); + } +} diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h index 5b82a0d..e30334d 100644 --- a/include/hw/pci/pcie.h +++ b/include/hw/pci/pcie.h @@ -79,6 +79,9 @@ struct PCIExpressDevice { /* Offset of ATS capability in config space */ uint16_t ats_cap; + + /* ACS */ + uint16_t acs_cap; }; #define COMPAT_PROP_PCP "power_controller_present" @@ -128,6 +131,9 @@ void pcie_add_capability(PCIDevice *dev, uint16_t offset, uint16_t size); void pcie_sync_bridge_lnk(PCIDevice *dev); +void pcie_acs_init(PCIDevice *dev, uint16_t offset); +void pcie_acs_reset(PCIDevice *dev); + void pcie_ari_init(PCIDevice *dev, uint16_t offset, uint16_t nextfn); void pcie_dev_ser_num_init(PCIDevice *dev, uint16_t offset, uint64_t ser_num); void pcie_ats_init(PCIDevice *dev, uint16_t offset); diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h index ad4e780..1db86b0 100644 --- a/include/hw/pci/pcie_regs.h +++ b/include/hw/pci/pcie_regs.h @@ -175,4 +175,8 @@ typedef enum PCIExpLinkWidth { PCI_ERR_COR_INTERNAL | \ PCI_ERR_COR_HL_OVERFLOW) +/* ACS */ +#define PCI_ACS_VER 0x1 +#define PCI_ACS_SIZEOF 8 + #endif /* QEMU_PCIE_REGS_H */ From patchwork Wed Feb 13 09:29:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Knut Omang X-Patchwork-Id: 10809441 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8F5613B4 for ; Wed, 13 Feb 2019 09:31:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D54F42CAB0 for ; Wed, 13 Feb 2019 09:31:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D35042CAC8; Wed, 13 Feb 2019 09:31:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4EE492CAB0 for ; Wed, 13 Feb 2019 09:31:54 +0000 (UTC) Received: from localhost ([127.0.0.1]:53840 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gtqtU-0007Ew-PL for patchwork-qemu-devel@patchwork.kernel.org; Wed, 13 Feb 2019 04:31:52 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33266) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gtqsF-0007Eg-NO for qemu-devel@nongnu.org; Wed, 13 Feb 2019 04:30:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gtqsB-000220-Su for qemu-devel@nongnu.org; Wed, 13 Feb 2019 04:30:33 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:56846) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gtqs9-0001uR-TF for qemu-devel@nongnu.org; Wed, 13 Feb 2019 04:30:31 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9OYc0132162; Wed, 13 Feb 2019 09:30:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2018-07-02; bh=t4s1/RHZIRpS4uHzuXWg9e8GZ3ck9F+y8m0hGkVuOzc=; b=DQRJV6yqUMSdnbNnq8nJ2IlGU+tksYmSKhTpGsxkIV/HoVwbLF7XCIdnBrgZEkZyt9Y2 PLj7LFcLVE9vFpNzbFv1d+nH4NAM30jS6O9jLy0TA6tKWWObWl5I9/3l55s9WiTuklSD nI9+auucRNcHpovyto+qWdXNUDRfps7zRJQLQEH054aieYtoszRFbVg/06ecbogieGit 73nWnoNuQyHJQ1I7F0gq24y33ENj5nnOv2bezAr0MLpxIKKTp6CJVi+WF/Tnldbj+4KZ Bzg2FxRgG0LwfC7HnFOwnfMmYmwVUq68xZVChw09bkH9m8o9YrGAEZxze7OEbrksg+GH yQ== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2qhrekgtkj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:30:22 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9ULve028523 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:30:21 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1D9ULhb011730; Wed, 13 Feb 2019 09:30:21 GMT Received: from abi.no.oracle.com (/10.172.144.123) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 01:30:21 -0800 From: Knut Omang To: qemu-devel@nongnu.org Date: Wed, 13 Feb 2019 10:29:59 +0100 Message-Id: <96507e5d139a2db338471b51991a10da4048c7bd.1550050121.git-series.knut.omang@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130069 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH v4 2/2] gen_pcie_root_port: Add ACS (Access Control Services) capability X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Michael S . Tsirkin" , Tal Attaly , Knut Omang , Alex Williamson , Elijah Shakkour , Stefan Hajnoczi Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Claim ACS support in the generic PCIe root port to allow passthrough of individual functions of a device to different guests (in a nested virt.setting) with VFIO. Without this patch, all functions of a device, such as all VFs of an SR/IOV device, will end up in the same IOMMU group. A similar situation occurs on Windows with Hyper-V. In the single function device case, it also has a small cosmetic benefit in that the root port itself is not grouped with the device. VFIO handles that situation in that binding rules only apply to endpoints, so it does not limit passthrough in those cases. Signed-off-by: Knut Omang Reviewed-by: Marcel Apfelbaum --- hw/pci-bridge/gen_pcie_root_port.c | 4 ++++ hw/pci-bridge/pcie_root_port.c | 4 ++++ include/hw/pci/pcie_port.h | 1 + 3 files changed, 9 insertions(+) diff --git a/hw/pci-bridge/gen_pcie_root_port.c b/hw/pci-bridge/gen_pcie_root_port.c index 9766edb..26bda73 100644 --- a/hw/pci-bridge/gen_pcie_root_port.c +++ b/hw/pci-bridge/gen_pcie_root_port.c @@ -20,6 +20,9 @@ OBJECT_CHECK(GenPCIERootPort, (obj), TYPE_GEN_PCIE_ROOT_PORT) #define GEN_PCIE_ROOT_PORT_AER_OFFSET 0x100 +#define GEN_PCIE_ROOT_PORT_ACS_OFFSET \ + (GEN_PCIE_ROOT_PORT_AER_OFFSET + PCI_ERR_SIZEOF) + #define GEN_PCIE_ROOT_PORT_MSIX_NR_VECTOR 1 typedef struct GenPCIERootPort { @@ -149,6 +152,7 @@ static void gen_rp_dev_class_init(ObjectClass *klass, void *data) rpc->interrupts_init = gen_rp_interrupts_init; rpc->interrupts_uninit = gen_rp_interrupts_uninit; rpc->aer_offset = GEN_PCIE_ROOT_PORT_AER_OFFSET; + rpc->acs_offset = GEN_PCIE_ROOT_PORT_ACS_OFFSET; } static const TypeInfo gen_rp_dev_info = { diff --git a/hw/pci-bridge/pcie_root_port.c b/hw/pci-bridge/pcie_root_port.c index 34ad767..e94d918 100644 --- a/hw/pci-bridge/pcie_root_port.c +++ b/hw/pci-bridge/pcie_root_port.c @@ -47,6 +47,7 @@ static void rp_reset(DeviceState *qdev) pcie_cap_deverr_reset(d); pcie_cap_slot_reset(d); pcie_cap_arifwd_reset(d); + pcie_acs_reset(d); pcie_aer_root_reset(d); pci_bridge_reset(qdev); pci_bridge_disable_base_limit(d); @@ -106,6 +107,9 @@ static void rp_realize(PCIDevice *d, Error **errp) pcie_aer_root_init(d); rp_aer_vector_update(d); + if (rpc->acs_offset) { + pcie_acs_init(d, rpc->acs_offset); + } return; err: diff --git a/include/hw/pci/pcie_port.h b/include/hw/pci/pcie_port.h index df242a0..09586f4 100644 --- a/include/hw/pci/pcie_port.h +++ b/include/hw/pci/pcie_port.h @@ -78,6 +78,7 @@ typedef struct PCIERootPortClass { int exp_offset; int aer_offset; int ssvid_offset; + int acs_offset; /* If nonzero, optional ACS capability offset */ int ssid; } PCIERootPortClass;