From patchwork Tue Jul 19 17:42:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CDAA1CCA485 for ; Tue, 19 Jul 2022 17:43:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370798.602687 (Exim 4.92) (envelope-from ) id 1oDrFa-0006OW-K1; Tue, 19 Jul 2022 17:43:14 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370798.602687; Tue, 19 Jul 2022 17:43:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFa-0006Na-Ev; Tue, 19 Jul 2022 17:43:14 +0000 Received: by outflank-mailman (input) for mailman id 370798; Tue, 19 Jul 2022 17:43:13 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFZ-00065X-Dp for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:13 +0000 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [2a00:1450:4864:20::636]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 417dca07-078a-11ed-924f-1f966e50362f; Tue, 19 Jul 2022 19:43:10 +0200 (CEST) Received: by mail-ej1-x636.google.com with SMTP id oy13so28511195ejb.1 for ; Tue, 19 Jul 2022 10:43:10 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:09 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 417dca07-078a-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tJNTg1Q/tJV0tMcGJ0EnKDCjisTQ8wg7Qbj1vC+6DIM=; b=JEBiRzNq8/rEVpY4Y1nCW+0agQpHA0wUkr+t5hIdFIBb9KlxybaeCAObAKObfaxV3K CQGvoMUAmYrEUj3Eo5BAlMm1g65MjO/b36aDofazXPb03Hc5ijiHyONtsOlEpRBj4/er xcYSLyjKsWm0shAZcoBRcEqb5sc+v0kcXmAbYRKjZp3+UHsV0gdOu3PdjgnxTitpfzNP DjrYSzpoJ9VALSlYMFm34nRMLO6WouZsnBSwdr7FySAwxXbgE7S3F8kdxceX3eXQqcFI nEjlv7B/84y37y1us2gVDx7znmZ12i3QplCweCTbqFsZ1oQVv6O2e+PEGBgWc3i01oIH jh/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tJNTg1Q/tJV0tMcGJ0EnKDCjisTQ8wg7Qbj1vC+6DIM=; b=hkSmE4kEyYL4XlwGrA97/Jv1+J66P9kGtdW2sVKjxH+xJ5A9eC46mAjlS/Pb6E2/0S 7roUbGRO4TraCLiPV/UHV2lD25tMq21UoVUHZaE+IzmPr54F9apOa5uE7DNZ60LabTr4 ds614Tc+SFUixeJ1RHVG7pUPvDTm7ZB21sauu8tDziB2TbY5IyHGxMb0JNWTkSa+lo6F LJpS0iN/fiPldQm0Cz4DNyJSeR8Jq6kG4N0un3Und6beurNjb0VURkKUJQTCZ5VPeb89 rnqLimbmkUqg9mpuEEIe+j7e5/QBlsgWW5zPPHGng1CQjM4P0xxGHT33ar59D23F5hJt 9f8g== X-Gm-Message-State: AJIora90aSc2p84Ckb4X6lT0YwbYHjKCXSyM6s+td/HpraVPN6rM49z/ tFAK47PMZTWuLI8WVxEGdZETEJ1W1/6IFw== X-Google-Smtp-Source: AGRyM1sWKSEJ1JhwSXANfNs9LH6TodgS9OnuY5ABFJc+y4wgMeLWpJHFKORNVHHvQJvIQj35lX12ow== X-Received: by 2002:a17:907:168d:b0:72b:5b67:83e6 with SMTP id hc13-20020a170907168d00b0072b5b6783e6mr32078995ejc.709.1658252589631; Tue, 19 Jul 2022 10:43:09 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , Stefano Stabellini , Julien Grall , Bertrand Marquis , Volodymyr Babchuk Subject: [PATCH V7 01/11] xen/pci: arm: add stub for is_memory_hole Date: Tue, 19 Jul 2022 20:42:43 +0300 Message-Id: <20220719174253.541965-2-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add a stub for is_memory_hole which is required for PCI passthrough on Arm. Signed-off-by: Oleksandr Andrushchenko --- OT: It looks like the discussion got stuck. As I understand this patch is not immediately needed in the context of current series as PCI passthrough is not enabled on Arm at the moment. So the patch could be added later on, but it is needed to allow PCI passthrough to be built on Arm for those who want to test it. Copy here some context provided by Julien: Here a summary of the discussion (+ some my follow-up thoughts): is_memory_hole() was recently introduced on x86 (see commit 75cc460a1b8c "xen/pci: detect when BARs are not suitably positioned") to check whether the BAR are positioned outside of a valid memory range. This was introduced to work-around quirky firmware. In theory, this could also happen on Arm. In practice, this may not happen but it sounds better to sanity check that the BAR contains "valid" I/O range. On x86, this is implemented by checking the region is not described is in the e820. IIUC, on Arm, the BARs have to be positioned in pre-defined ranges. So I think it would be possible to implement is_memory_hole() by going through the list of hostbridges and check the ranges. But first, I'd like to confirm my understanding with Rahul, and others. If we were going to go this route, I would also rename the function to be better match what it is doing (i.e. it checks the BAR is correctly placed). As a potentially optimization/hardening for Arm, we could pass the hostbridge so we don't have to walk all of them. --- xen/arch/arm/mm.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index 009b8cd9ef..bb34b97eb5 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -1708,6 +1708,12 @@ unsigned long get_upper_mfn_bound(void) return max_page - 1; } +bool is_memory_hole(mfn_t start, mfn_t end) +{ + /* TODO: this needs to be properly implemented. */ + return true; +} + /* * Local variables: * mode: C From patchwork Tue Jul 19 17:42:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 143A5CCA486 for ; Tue, 19 Jul 2022 17:43:37 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370797.602682 (Exim 4.92) (envelope-from ) id 1oDrFa-0006LU-C5; Tue, 19 Jul 2022 17:43:14 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370797.602682; Tue, 19 Jul 2022 17:43:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFa-0006LN-85; Tue, 19 Jul 2022 17:43:14 +0000 Received: by outflank-mailman (input) for mailman id 370797; Tue, 19 Jul 2022 17:43:13 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFY-00065R-QO for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:12 +0000 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [2a00:1450:4864:20::52c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 42540942-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:12 +0200 (CEST) Received: by mail-ed1-x52c.google.com with SMTP id y8so20648121eda.3 for ; Tue, 19 Jul 2022 10:43:12 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:10 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 42540942-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Clq/yFmOZ38glmpc8+TyB4sKPvTCJ7X1eyQxmF0coEI=; b=d5FPSK0K72d9Xe+i6ecKVccyRdhudBtSq7Gsg52mnFqT46RzZB1R2LF+jD9ndUfGaM /uPxrDP1hg7nUgQneTN2h0UYnFDfP3aswJ32KwNf8uQr4+EmToIUFBrmuRa4R7uPnVRL xzsUeROARqcaMcnfU6HQtED+vuNpnEynxFuwoh025WabBI6mYmFD8E19m+pEV0zKF7G6 +sAFvVejJ6236UnrN6QNFyeJanWDNpSBvBvyKtIHowxZf1TNfHjvtOVls/Ub8GA3mrb8 M0tOk64zMa6jovUmywas8nnjkkDWOxzAagsVH/NHiVNJEe6gPKXo/zlh7m3wUtWW9/z3 V8ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Clq/yFmOZ38glmpc8+TyB4sKPvTCJ7X1eyQxmF0coEI=; b=fkYnMy8Zpt8Lshhs9vHSk2I5U+5cA2cS3QGm5X9RkPVY+XbxCsBcp7XUYuzOIPC9+Z b5bjtEuZUZkmWhEnOttUc+ujJnHkmifOH9zG3F450WpSPMaa+6gaezIMQ5VFxhft1jFG IRa2uqHG1CnUgDspWa0qczXI49bKrZ6bS1l1rzz90vydpNs/k0cAP7ydmyxK9WRS/1Wa xmAHnVQYMTm8vadLeMzNKfNEdno7i/3TCmG6yore2M5A1cI/6dyIBMMrRhkq94D0IxDp R5HakVozA5B107ZAXfmMgEafCr367bO/1pizcBkrDBklgEegLNAphtKGRPaIlooBQ7WS WNPQ== X-Gm-Message-State: AJIora+0LelLZBAUPwoFN+RP/sS+zSi0NmA38JXsGS/aWTInZ4PxmjMB ySrh68n7aJifMCnDSnUqPXfhqZBcl4Fpxg== X-Google-Smtp-Source: AGRyM1sl4lvRtDpSZF0fInywXXwKy2gpEGVI470vMbcD0kTRR34dE4NikC9nXtxRM4smnaZPKlSnAQ== X-Received: by 2002:aa7:c250:0:b0:43b:25fb:4903 with SMTP id y16-20020aa7c250000000b0043b25fb4903mr36375785edo.253.1658252591074; Tue, 19 Jul 2022 10:43:11 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini , Wei Liu , Paul Durrant , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= Subject: [PATCH V7 02/11] vpci: add hooks for PCI device assign/de-assign Date: Tue, 19 Jul 2022 20:42:44 +0300 Message-Id: <20220719174253.541965-3-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - do not pass struct domain to vpci_{assign|deassign}_device as pdev->domain can be used - do not leave the device assigned (pdev->domain == new domain) in case vpci_assign_device fails: try to de-assign and if this also fails, then crash the domain - re-work according to the new locking scheme (ASSERTs) - OT: rebased Since v5: - do not split code into run_vpci_init - do not check for is_system_domain in vpci_{de}assign_device - do not use vpci_remove_device_handlers_locked and re-allocate pdev->vpci completely - make vpci_deassign_device void Since v4: - de-assign vPCI from the previous domain on device assignment - do not remove handlers in vpci_assign_device as those must not exist at that point Since v3: - remove toolstack roll-back description from the commit message as error are to be handled with proper cleanup in Xen itself - remove __must_check - remove redundant rc check while assigning devices - fix redundant CONFIG_HAS_VPCI check for CONFIG_HAS_VPCI_GUEST_SUPPORT - use REGISTER_VPCI_INIT machinery to run required steps on device init/assign: add run_vpci_init helper Since v2: - define CONFIG_HAS_VPCI_GUEST_SUPPORT so dead code is not compiled for x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/Kconfig | 4 ++++ xen/drivers/passthrough/pci.c | 11 +++++++++++ xen/drivers/vpci/vpci.c | 31 +++++++++++++++++++++++++++++++ xen/include/xen/vpci.h | 15 +++++++++++++++ 4 files changed, 61 insertions(+) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47..780490cf8e 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_VPCI + endmenu diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index f93922acc8..56af1dbd97 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -1019,6 +1019,8 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + vpci_deassign_device(pdev); + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1558,6 +1560,7 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) { const struct domain_iommu *hd = dom_iommu(d); struct pci_dev *pdev; + uint8_t old_devfn; int rc = 0; if ( !is_iommu_enabled(d) ) @@ -1577,6 +1580,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) if ( pdev->broken && d != hardware_domain && d != dom_io ) goto done; + vpci_deassign_device(pdev); + rc = pdev_msix_assign(d, pdev); if ( rc ) goto done; @@ -1594,6 +1599,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) pci_to_dev(pdev), flag)) ) goto done; + old_devfn = devfn; + for ( ; pdev->phantom_stride; rc = 0 ) { devfn += pdev->phantom_stride; @@ -1603,6 +1610,10 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) pci_to_dev(pdev), flag); } + rc = vpci_assign_device(pdev); + if ( rc && deassign_device(d, seg, bus, old_devfn) ) + domain_crash(d); + done: if ( rc ) printk(XENLOG_G_WARNING "%pd: assign (%pp) failed (%d)\n", diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 674c9b347d..d187901422 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -92,6 +92,37 @@ int vpci_add_handlers(struct pci_dev *pdev) return rc; } + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct pci_dev *pdev) +{ + int rc; + + ASSERT(pcidevs_write_locked()); + + if ( !has_vpci(pdev->domain) ) + return 0; + + rc = vpci_add_handlers(pdev); + if ( rc ) + vpci_deassign_device(pdev); + + return rc; +} + +/* Notify vPCI that device is de-assigned from guest. */ +void vpci_deassign_device(struct pci_dev *pdev) +{ + ASSERT(pcidevs_write_locked()); + + if ( !has_vpci(pdev->domain) ) + return; + + vpci_remove_device(pdev); +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 7ab39839ff..e5501b9207 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -254,6 +254,21 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) } #endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int vpci_assign_device(struct pci_dev *pdev); +void vpci_deassign_device(struct pci_dev *pdev); +#else +static inline int vpci_assign_device(struct pci_dev *pdev) +{ + return 0; +}; + +static inline void vpci_deassign_device(struct pci_dev *pdev) +{ +}; +#endif + #endif /* From patchwork Tue Jul 19 17:42:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 511F7C43334 for ; Tue, 19 Jul 2022 17:43:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370799.602704 (Exim 4.92) (envelope-from ) id 1oDrFb-0006qj-RP; Tue, 19 Jul 2022 17:43:15 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370799.602704; Tue, 19 Jul 2022 17:43:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFb-0006qa-OG; Tue, 19 Jul 2022 17:43:15 +0000 Received: by outflank-mailman (input) for mailman id 370799; Tue, 19 Jul 2022 17:43:14 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFa-00065R-4T for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:14 +0000 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [2a00:1450:4864:20::631]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 42fd6b47-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:13 +0200 (CEST) Received: by mail-ej1-x631.google.com with SMTP id sz17so28557006ejc.9 for ; Tue, 19 Jul 2022 10:43:13 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:11 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 42fd6b47-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FgW5CBHXdcnwWKRIlMpQopBORvtG8zr4OSYqoD8fHyU=; b=CXgYMn7KHHmlUqP2ylgTZ4SLg8AaLwROxv/SKbNwpq/p3FgltdZdTs6yeq+BDsJ7aV YNoGKCkkYI4h1wKk0+RIKQUNDiyQxnSWgcqG8fkYU+2pp6vzaMinRYeP6DBDRHWDB7RO XlRGcN7iLtGIFfRGPoGPYYcIy0rQi1Ldw8NoKvdrN7kUDXjQK6VRBAWdZm3xUdy60EIQ IN2j7d+00XPogxocVYsNGBfyN88FEtKUFwbtcMWb3ey5vYb60PUAOA3GGHgbAJ+/2z6R E/Icog1jEAu9L4AtdD1rDxNsLkkGDYRhMOYo91mPWJhMbeKft6Iz5rhsCOIF+a62OEYT A3sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FgW5CBHXdcnwWKRIlMpQopBORvtG8zr4OSYqoD8fHyU=; b=tBP421QWy7xNtktv2Z03NW7PB1jVDDgOY2q0d/qtaq+KYQKyXfJxfn63K9NZiogNdP r0SnRkiUBiuW5hXv1rjQOFaPx1yDoIxFSJMT9YxIsEHj8mmVILMWBNvC3SudTIiHMZ9a yoFUJ4DDqsosA+I5+mhwqr7VToVWWx33GZNkh5K8KFa4oR7HJhEBV29gPsH13uoOjjGa NYqPD8ZcNAKNIMRDNg6O8d1PLfpIc5Bk01saZfccxvSo6yitNbn1zwgIAEwC2rkbD4XR 6p3MPoPxZoyeiv6fRFL9hwq43OLXwCpBROg4Ts60PT+VwB4nksQiQSmqjxKYDQZVlvuh REMw== X-Gm-Message-State: AJIora8r977G3T3TKQyAMWUIj8yYcsZU4Vh7ctNY49V9qz0Hn2y9k69r Vww8VOjI1IELjTNcLVK0vzMUeV4DwnnA6A== X-Google-Smtp-Source: AGRyM1ubvpmCjkSWVW5k2cVxXUpISmssUtDd1KiuVlTJ0QCQMdV9iUR30c8eFT1SCyrdIoIl4TkXrw== X-Received: by 2002:a17:907:948e:b0:72d:3fd2:5da0 with SMTP id dm14-20020a170907948e00b0072d3fd25da0mr30077049ejc.225.1658252592168; Tue, 19 Jul 2022 10:43:12 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= Subject: [PATCH V7 03/11] vpci/header: implement guest BAR register handlers Date: Tue, 19 Jul 2022 20:42:45 +0300 Message-Id: <20220719174253.541965-4-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. All empty, IO and ROM BARs for guests are emulated by returning 0 on reads and ignoring writes: this BARs are special with this respect as their lower bits have special meaning, so returning default ~0 on read may confuse guest OS. Memory decoding is initially disabled when used by guests in order to prevent the BAR being placed on top of a RAM region. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - unify the writing of the PCI_COMMAND register on the error path into a label - do not introduce bar_ignore_access helper and open code - s/guest_bar_ignore_read/empty_bar_read - update error message in guest_bar_write - only setup empty_bar_read for IO if !x86 - OT: rebased - OT: add cf_check specifier to guest_bar_(write)read() and empty_bar_read() Since v5: - make sure that the guest set address has the same page offset as the physical address on the host - remove guest_rom_{read|write} as those just implement the default behaviour of the registers not being handled - adjusted comment for struct vpci.addr field - add guest handlers for BARs which are not handled and will otherwise return ~0 on read and ignore writes. The BARs are special with this respect as their lower bits have special meaning, so returning ~0 doesn't seem to be right Since v4: - updated commit message - s/guest_addr/guest_reg Since v3: - squashed two patches: dynamic add/remove handlers and guest BAR handler implementation - fix guest BAR read of the high part of a 64bit BAR (Roger) - add error handling to vpci_assign_device - s/dom%pd/%pd - blank line before return Since v2: - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code has been eliminated from being built on x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment --- xen/drivers/vpci/header.c | 151 +++++++++++++++++++++++++++++++------- xen/include/xen/vpci.h | 3 + 2 files changed, 126 insertions(+), 28 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index e0461b1139..9fbbdc3500 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -412,6 +412,71 @@ static void cf_check bar_write( pci_conf_write32(pdev->sbdf, reg, val); } +static void cf_check guest_bar_write( + const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) +{ + struct vpci_bar *bar = data; + bool hi = false; + uint64_t guest_reg = bar->guest_reg; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + guest_reg &= ~(0xffffffffull << (hi ? 32 : 0)); + guest_reg |= (uint64_t)val << (hi ? 32 : 0); + + guest_reg &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; + + /* + * Make sure that the guest set address has the same page offset + * as the physical address on the host or otherwise things won't work as + * expected. + */ + if ( (guest_reg & (~PAGE_MASK & PCI_BASE_ADDRESS_MEM_MASK)) != + (bar->addr & ~PAGE_MASK) ) + { + gprintk(XENLOG_WARNING, + "%pp: ignored BAR %zu write attempting to change page offset\n", + &pdev->sbdf, bar - pdev->vpci->header.bars + hi); + return; + } + + bar->guest_reg = guest_reg; +} + +static uint32_t cf_check guest_bar_read( + const struct pci_dev *pdev, unsigned int reg, void *data) +{ + const struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + + return bar->guest_reg >> (hi ? 32 : 0); +} + +static uint32_t cf_check empty_bar_read( + const struct pci_dev *pdev, unsigned int reg, void *data) +{ + return 0; +} + static void cf_check rom_write( const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -468,6 +533,7 @@ static int cf_check init_bars(struct pci_dev *pdev) struct vpci_header *header; struct vpci_bar *bars; int rc; + bool is_hwdom = is_hardware_domain(pdev->domain); ASSERT(pcidevs_write_locked()); @@ -512,13 +578,12 @@ static int cf_check init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } + goto fail; continue; } @@ -527,6 +592,17 @@ static int cf_check init_bars(struct pci_dev *pdev) if ( (val & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO ) { bars[i].type = VPCI_BAR_IO; + +#ifndef CONFIG_X86 + if ( !is_hwdom ) + { + rc = vpci_add_register(pdev->vpci, empty_bar_read, NULL, + reg, 4, &bars[i]); + if ( rc ) + goto fail; + } +#endif + continue; } if ( (val & PCI_BASE_ADDRESS_MEM_TYPE_MASK) == @@ -538,14 +614,20 @@ static int cf_check init_bars(struct pci_dev *pdev) rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size, (i == num_bars - 1) ? PCI_BAR_LAST : 0); if ( rc < 0 ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } + goto fail; if ( size == 0 ) { bars[i].type = VPCI_BAR_EMPTY; + + if ( !is_hwdom ) + { + rc = vpci_add_register(pdev->vpci, empty_bar_read, NULL, + reg, 4, &bars[i]); + if ( rc ) + goto fail; + } + continue; } @@ -553,34 +635,47 @@ static int cf_check init_bars(struct pci_dev *pdev) bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) - { - pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); - return rc; - } + goto fail; } - /* Check expansion ROM. */ - rc = pci_size_mem_bar(pdev->sbdf, rom_reg, &addr, &size, PCI_BAR_ROM); - if ( rc > 0 && size ) + /* Check expansion ROM: we do not handle ROM for guests. */ + if ( is_hwdom ) { - struct vpci_bar *rom = &header->bars[num_bars]; + rc = pci_size_mem_bar(pdev->sbdf, rom_reg, &addr, &size, PCI_BAR_ROM); + if ( rc > 0 && size ) + { + struct vpci_bar *rom = &header->bars[num_bars]; - rom->type = VPCI_BAR_ROM; - rom->size = size; - rom->addr = addr; - header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & - PCI_ROM_ADDRESS_ENABLE; + rom->type = VPCI_BAR_ROM; + rom->size = size; + rom->addr = addr; + header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & + PCI_ROM_ADDRESS_ENABLE; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, rom); + if ( rc ) + rom->type = VPCI_BAR_EMPTY; + } + } + else + { + rc = vpci_add_register(pdev->vpci, empty_bar_read, NULL, + rom_reg, 4, &header->bars[num_bars]); if ( rc ) - rom->type = VPCI_BAR_EMPTY; + goto fail; } return (cmd & PCI_COMMAND_MEMORY) ? modify_bars(pdev, cmd, false) : 0; + + fail: + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + return rc; } REGISTER_VPCI_INIT(init_bars, VPCI_PRIORITY_MIDDLE); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index e5501b9207..6e1d3b93cd 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -67,7 +67,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical (host) address. */ uint64_t addr; + /* Guest view of the BAR: address and lower bits. */ + uint64_t guest_reg; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Tue Jul 19 17:42:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CA366CCA47F for ; Tue, 19 Jul 2022 17:43:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370800.602715 (Exim 4.92) (envelope-from ) id 1oDrFd-00077o-5d; Tue, 19 Jul 2022 17:43:17 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370800.602715; Tue, 19 Jul 2022 17:43:17 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFd-00077Z-19; Tue, 19 Jul 2022 17:43:17 +0000 Received: by outflank-mailman (input) for mailman id 370800; Tue, 19 Jul 2022 17:43:15 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFb-00065X-2k for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:15 +0000 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [2a00:1450:4864:20::52b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 43cfb566-078a-11ed-924f-1f966e50362f; Tue, 19 Jul 2022 19:43:14 +0200 (CEST) Received: by mail-ed1-x52b.google.com with SMTP id m16so20604420edb.11 for ; Tue, 19 Jul 2022 10:43:14 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:12 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 43cfb566-078a-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cpVLKnhd6vjujxD0bO2KThhMDJ+iVwVoqh13SC2q0cs=; b=CaEKmCTRcR8/SMRC0bQf+dOv9ySs6y3MjxkY0nxL38jn9DyDTVsUkvy9ZAHNwAQLUb 9PsHRS/MaA6iyDNGA87wfELTHOAsXgeUi44dzyPCvik1wvaojXPc4Cx5sdfckg0fWebP cc219FKu25+5dlHkgUDbhuknkXK8lem6VSZwLGW/7cdxLwwGTYenTN6e3PjlMmAbN2W4 CS9Sm/Psi7omj8llD7gcATQhopxoKJIfBZvlhyRJbQ3upJpAY5JvVVFv3ikETm0FZci/ g/ZeKkUGDsrsiaPWABpUrGO/2OwrY9PmD99wwQoNmAYMJsvw3q7zfdJMDKbtNZ4Bw5LB /jFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cpVLKnhd6vjujxD0bO2KThhMDJ+iVwVoqh13SC2q0cs=; b=kkPEF8SOekhfihud6SJ3BwzPidYZGYxYnKb2p2IX7k9W2ctGEpqYYAzk1KcwOljoNd Y19bH9CEqbxsTfcHcCepXv7vL4L6TZ0Q3cziHZW1nFAkkDN+WbUaXeHKTF4+GT8ZXG0j pyHsl6fiznMiQaZskDYx6qDL6g3kc7zlYtzsXNazAQII6j+GzgPZ0GZNgWYnkYUHOYao fHF08rE5DhCNhyX1cnWEG3cr4lVKuJx/XoYjwE3MRIvw4vU4kHABa0DUkhqJ4yINO7Ay 1rl6pA31NxJy6xn/xsIp2A1ErE2b2EbH7bDLoEQ/8ThsAHMRDPuvQI/RZkzzuLcCjEfB IjNw== X-Gm-Message-State: AJIora9RIbrwqi+ihS7TYXQXxLExEoIA70owytqJl6sCJ7HEawplO1L7 8ckZ33c7V8VKM8/FDOyInwRwGX0T73VtgQ== X-Google-Smtp-Source: AGRyM1tJIw9jRhYJ3lV/7WT5PUgwsKPZGJm0z6/cKEw7ZfGIW7GlhYz1AxpnWJFoCsR9lgCnsRsy1Q== X-Received: by 2002:a05:6402:5245:b0:43a:a024:82cc with SMTP id t5-20020a056402524500b0043aa02482ccmr45478822edd.56.1658252593657; Tue, 19 Jul 2022 10:43:13 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini , Wei Liu Subject: [PATCH V7 04/11] rangeset: add RANGESETF_no_print flag Date: Tue, 19 Jul 2022 20:42:46 +0300 Message-Id: <20220719174253.541965-5-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are range sets which should not be printed, so introduce a flag which allows marking those as such. Implement relevant logic to skip such entries while printing. While at it also simplify the definition of the flags by directly defining those without helpers. Suggested-by: Jan Beulich Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Jan Beulich Reviewed-by: Rahul Singh --- Since v5: - comment indentation (Jan) Since v1: - update BUG_ON with new flag - simplify the definition of the flags --- xen/common/rangeset.c | 5 ++++- xen/include/xen/rangeset.h | 5 +++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/xen/common/rangeset.c b/xen/common/rangeset.c index a6ef264046..f8b909d016 100644 --- a/xen/common/rangeset.c +++ b/xen/common/rangeset.c @@ -433,7 +433,7 @@ struct rangeset *rangeset_new( INIT_LIST_HEAD(&r->range_list); r->nr_ranges = -1; - BUG_ON(flags & ~RANGESETF_prettyprint_hex); + BUG_ON(flags & ~(RANGESETF_prettyprint_hex | RANGESETF_no_print)); r->flags = flags; safe_strcpy(r->name, name ?: "(no name)"); @@ -575,6 +575,9 @@ void rangeset_domain_printk( list_for_each_entry ( r, &d->rangesets, rangeset_list ) { + if ( r->flags & RANGESETF_no_print ) + continue; + printk(" "); rangeset_printk(r); printk("\n"); diff --git a/xen/include/xen/rangeset.h b/xen/include/xen/rangeset.h index 135f33f606..f7c69394d6 100644 --- a/xen/include/xen/rangeset.h +++ b/xen/include/xen/rangeset.h @@ -49,8 +49,9 @@ void rangeset_limit( /* Flags for passing to rangeset_new(). */ /* Pretty-print range limits in hexadecimal. */ -#define _RANGESETF_prettyprint_hex 0 -#define RANGESETF_prettyprint_hex (1U << _RANGESETF_prettyprint_hex) +#define RANGESETF_prettyprint_hex (1U << 0) + /* Do not print entries marked with this flag. */ +#define RANGESETF_no_print (1U << 1) bool_t __must_check rangeset_is_empty( const struct rangeset *r); From patchwork Tue Jul 19 17:42:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31F0ECCA487 for ; Tue, 19 Jul 2022 17:43:37 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370801.602726 (Exim 4.92) (envelope-from ) id 1oDrFe-0007Qk-M1; Tue, 19 Jul 2022 17:43:18 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370801.602726; Tue, 19 Jul 2022 17:43:18 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFe-0007QV-HS; Tue, 19 Jul 2022 17:43:18 +0000 Received: by outflank-mailman (input) for mailman id 370801; Tue, 19 Jul 2022 17:43:17 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFd-00065R-3p for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:17 +0000 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [2a00:1450:4864:20::634]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 44a5100f-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:15 +0200 (CEST) Received: by mail-ej1-x634.google.com with SMTP id bp15so28611028ejb.6 for ; Tue, 19 Jul 2022 10:43:15 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:14 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 44a5100f-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OieAisZo52BPPW28GQWLm26fgvpHr9TxOwcPehMBtUc=; b=pbAkZHs2Ovwil0XJtD3oLEGPLgaJs45DiqIUbhMdbla4V4G66V2YM4ikeg2qrQ4pQw 8/LSdMqbJ6E+mFQAt8XRlTJUPgMZN151f6EYkKYb0fAiuBrs1j3fhvSErbv02PkeQa/I HPftSqB2NfU1oH77y6mUGxYi+wQvlS8BJZ4adtdukSgrgJcKJqMEJcT0KYcsKrZkvjLI TfZL3Pdtbk9ttEqdqFJ+oxV/AEfCxldvFqHsYSm+GIFLEA/anf5hzZ7z5F0keoBc5AWa fEl1M9z9MtUXK+uoGvHnRR5q5mbbiskniNEKeZAuXAv112O9QKZTdiOcYjRihdjLy5aB u7mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OieAisZo52BPPW28GQWLm26fgvpHr9TxOwcPehMBtUc=; b=B5EX3oKnMGXGzT9mtd+mXQDh/W/wVN6WSl9D1fHZj8TFbXlFgbqQ74dkIvRWyInHlW 0LEyGFvvFlq2xp6dTZfRJJFEBaQ/swBl1YOncWUPI2oNnV81UX8o4VbGHvp97bVIgvKS /TYt/CUf20Wl7T+aIqwVJ84sZYnnqwFm93sdq46iNwJywdrxcBeTAhyY0YuSIE20RBYK rVrX7GWKtfV0LUQJJ415s3OuDwNIyRo/3J8HrqyMfD5rxIavnWSQsrLyK9NoxQuOiVWi 5ZkUNXfTfiBkItFvBQP/MDQG93Lumd4VTzZhOWPyI1sHO+eZDkK3NkbK85fjoncEr2Mt M0pw== X-Gm-Message-State: AJIora8rznq5QX6q6DFSt8cDkrQb1SAAQZ5xhnP2Xt7d365kGpz1GUI/ ih2nEPPYc736r2GFLm3WKiiU6ibv6qQu6Q== X-Google-Smtp-Source: AGRyM1tbRAV2e9n0Nj7kC5Lvvd3jfZNCpCrgvfb24h7ry3pXKn7AQqZZMwhIeAfOW5ccrheE5vPlUQ== X-Received: by 2002:a17:907:28d4:b0:72b:49c0:d04a with SMTP id en20-20020a17090728d400b0072b49c0d04amr32968382ejc.141.1658252594893; Tue, 19 Jul 2022 10:43:14 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= Subject: [PATCH V7 05/11] vpci/header: handle p2m range sets per BAR Date: Tue, 19 Jul 2022 20:42:47 +0300 Message-Id: <20220719174253.541965-6-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. As the range sets are now created when a PCI device is added and destroyed when it is removed so make them named and accounted. Note that rangesets were chosen here despite there being only up to 3 separate ranges in each set (typically just 1). But rangeset per BAR was chosen for the ease of implementation and existing code re-usability. This is in preparation of making non-identity mappings in p2m for the MMIOs. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - update according to the new locking scheme - remove odd fail label in modify_bars - OT: rebased Since v5: - fix comments - move rangeset allocation to init_bars and only allocate for MAPPABLE BARs - check for overlap with the already setup BAR ranges Since v4: - use named range sets for BARs (Jan) - changes required by the new locking scheme - updated commit message (Jan) Since v3: - re-work vpci_cancel_pending accordingly to the per-BAR handling - s/num_mem_ranges/map_pending and s/uint8_t/bool - ASSERT(bar->mem) in modify_bars - create and destroy the rangesets on add/remove --- xen/drivers/vpci/header.c | 241 +++++++++++++++++++++++++++----------- xen/drivers/vpci/vpci.c | 5 + xen/include/xen/vpci.h | 3 +- 3 files changed, 182 insertions(+), 67 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 9fbbdc3500..f14ff11882 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -131,64 +131,106 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + struct pci_dev *pdev = v->vpci.pdev; + + if ( !pdev ) + return false; + + pcidevs_read_lock(); + + if ( v->vpci.map_pending ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); - - if ( rc == -ERESTART ) - return true; - - pcidevs_read_lock(); - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); - pcidevs_read_unlock(); - - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; - if ( rc ) + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. - */ - pcidevs_write_lock(); - vpci_remove_device(v->vpci.pdev); - pcidevs_write_unlock(); + struct vpci_bar *bar = &header->bars[i]; + int rc; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + { + pcidevs_read_unlock(); + return true; + } + + spin_lock(&pdev->vpci->lock); + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(pdev, rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : + v->vpci.cmd, !rc && v->vpci.rom_only); + spin_unlock(&pdev->vpci->lock); + + if ( rc ) + { + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this + * is safe for Dom0, for DomUs the domain needs to be killed in + * order to avoid leaking stale p2m mappings on failure. + */ + v->vpci.map_pending = false; + pcidevs_read_unlock(); + + if ( is_hardware_domain(v->domain) ) + { + pcidevs_write_lock(); + vpci_remove_device(v->vpci.pdev); + pcidevs_write_unlock(); + } + else + domain_crash(v->domain); + + return false; + } } + + v->vpci.map_pending = false; } + pcidevs_read_unlock(); + return false; } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) + ASSERT(pcidevs_write_locked()); + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - /* - * It's safe to drop and re-acquire the lock in this context - * without risking pdev disappearing because devices cannot be - * removed until the initial domain has been started. - */ - pcidevs_write_unlock(); - process_pending_softirqs(); - pcidevs_write_lock(); + struct vpci_bar *bar = &header->bars[i]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + { + /* + * It's safe to drop and reacquire the lock in this context + * without risking pdev disappearing because devices cannot be + * removed until the initial domain has been started. + */ + pcidevs_write_unlock(); + process_pending_softirqs(); + pcidevs_write_lock(); + } } - rangeset_destroy(mem); if ( !rc ) modify_decoding(pdev, cmd, false); @@ -196,7 +238,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only) { struct vcpu *curr = current; @@ -207,7 +249,7 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; + curr->vpci.map_pending = true; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; /* @@ -221,43 +263,61 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; - - if ( !mem ) - return -ENOMEM; + bool map_pending; /* - * Create a rangeset that represents the current device BARs memory region - * and compare it against all the currently active BAR memory regions. If - * an overlap is found, subtract it from the region to be mapped/unmapped. + * Create a rangeset per BAR that represents the current device memory + * region and compare it against all the currently active BAR memory + * regions. If an overlap is found, subtract it from the region to be + * mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + if ( !bar->mem ) + continue; + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) ) continue; - rc = rangeset_add_range(mem, start, end); + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); return rc; } + + /* Check for overlap with the already setup BAR ranges. */ + for ( j = 0; j < i; j++ ) + { + struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove overlapping range [%lx, %lx]: %d\n", + start, end, rc); + return rc; + } + } } /* Remove any MSIX regions if present. */ @@ -267,14 +327,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + return rc; + } } } @@ -306,7 +373,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || + if ( !bar->enabled || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against other * BARs in the same device for overlaps, but not against the @@ -315,12 +383,11 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); return rc; } } @@ -339,10 +406,23 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + map_pending = false; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + if ( !rangeset_is_empty(header->bars[i].mem) ) + { + map_pending = true; + break; + } + + /* If there's no mapping work write the command register now. */ + if ( !map_pending ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only); return 0; } @@ -525,6 +605,19 @@ static void cf_check rom_write( rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static int bar_add_rangeset(struct pci_dev *pdev, struct vpci_bar *bar, int i) +{ + char str[32]; + + snprintf(str, sizeof(str), "%pp:BAR%d", &pdev->sbdf, i); + + bar->mem = rangeset_new(pdev->domain, str, RANGESETF_no_print); + if ( !bar->mem ) + return -ENOMEM; + + return 0; +} + static int cf_check init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -611,6 +704,13 @@ static int cf_check init_bars(struct pci_dev *pdev) else bars[i].type = VPCI_BAR_MEM32; + rc = bar_add_rangeset(pdev, &bars[i], i); + if ( rc ) + { + bars[i].type = VPCI_BAR_EMPTY; + goto fail; + } + rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size, (i == num_bars - 1) ? PCI_BAR_LAST : 0); if ( rc < 0 ) @@ -661,6 +761,15 @@ static int cf_check init_bars(struct pci_dev *pdev) rom_reg, 4, rom); if ( rc ) rom->type = VPCI_BAR_EMPTY; + else + { + rc = bar_add_rangeset(pdev, rom, i); + if ( rc ) + { + rom->type = VPCI_BAR_EMPTY; + goto fail; + } + } } } else diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index d187901422..f683346285 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -38,6 +38,8 @@ extern vpci_register_init_t *const __end_vpci_array[]; void vpci_remove_device(struct pci_dev *pdev) { + unsigned int i; + ASSERT(pcidevs_write_locked()); if ( !has_vpci(pdev->domain) || !pdev->vpci ) @@ -54,6 +56,9 @@ void vpci_remove_device(struct pci_dev *pdev) xfree(r); } spin_unlock(&pdev->vpci->lock); + + for ( i = 0; i < ARRAY_SIZE(pdev->vpci->header.bars); i++ ) + rangeset_destroy(pdev->vpci->header.bars[i].mem); if ( pdev->vpci->msix && pdev->vpci->msix->pba ) iounmap(pdev->vpci->msix->pba); xfree(pdev->vpci->msix); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 6e1d3b93cd..6332659c4a 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -72,6 +72,7 @@ struct vpci { /* Guest view of the BAR: address and lower bits. */ uint64_t guest_reg; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -146,9 +147,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + bool map_pending : 1; bool rom_only : 1; }; From patchwork Tue Jul 19 17:42:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 71CFFC433EF for ; Tue, 19 Jul 2022 17:43:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370803.602736 (Exim 4.92) (envelope-from ) id 1oDrFg-0007iH-0N; Tue, 19 Jul 2022 17:43:20 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370803.602736; Tue, 19 Jul 2022 17:43:19 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFf-0007he-Rz; Tue, 19 Jul 2022 17:43:19 +0000 Received: by outflank-mailman (input) for mailman id 370803; Tue, 19 Jul 2022 17:43:18 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFe-00065R-48 for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:18 +0000 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [2a00:1450:4864:20::62c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 4543bb89-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:16 +0200 (CEST) Received: by mail-ej1-x62c.google.com with SMTP id l23so28588728ejr.5 for ; Tue, 19 Jul 2022 10:43:16 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:15 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4543bb89-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=shxNV9joemK5imu4wT4+Id91nCPY01JMn0lJabYidsY=; b=eVHUjDx8fQyS+XcgwN0giiiOlBNaG1kuJPLiM5qEXzokJ/H8xzEiCobcMgiP1TD4tt g7YfLq9ADCL1ix35Dnb3kvv+QacuObxCwfEke6l9P5BfQ0mDfZEh4CvOy/CEQJI8qrGX M7bILaJys4OtfsRv0P/ZeJxyTKaYtkJlNlF7dcX767gA/TvEQQqNa9n5aEOd+lXRvum7 Yy7ktAlWEwWeaHCT02VSbtQEzIBZc58NMWMv388dOUBu5WHzs+REF6t65kW849DVZsWO xRudRUPW2uTys3nT4SHheP+DzE5NuwSQqSsogtNBT6t1THv0J9I5KVsZZLajiwprbck/ dEoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=shxNV9joemK5imu4wT4+Id91nCPY01JMn0lJabYidsY=; b=kIR2ldiTGTJesMPLh0onwX2pme+DEx/CVyY7m79r2ZTYPlXn4508GVE6KpDPhza4DD pPxwCTtXRcFN1UqCbJFIq8p3zxhWa3pLwDCDeN3X856zVtgzgHB9ns4aBKGkRC39FKvY MLL4+SxGutTJKORWvFoIKjSkUSruvDLjWyKQAkbp7jm5d4mi+dhewsDbKMo9heBFQ2Ob /FGby+yNE8+7XZtSP/TTrocgm5SoMNbSly21ccNiPfujko5jloxVS+m36p7/rBSUTxF3 IA5Daxxhc98CKbKAqmm6hLmL5EJf1NtZpcLwOBuKOGfkHWI1CxtbWFbW7jTntt/S6k43 03GQ== X-Gm-Message-State: AJIora87Yny0IrU7wMcQW5t6rmR8oGdMu+kNp1DffkVeFMTX9w+SDeLs V6tZKxUolAib6qB0cNmpY7O+IeAQ6p4Cbg== X-Google-Smtp-Source: AGRyM1tToApNxORVJ+5ePOsc4RwRSTArOU6bp+WD5v34DnOee+CQ52yyGCpFW054KCZ6s5dnBIPCjA== X-Received: by 2002:a17:907:3fa8:b0:72b:5895:f54f with SMTP id hr40-20020a1709073fa800b0072b5895f54fmr31647848ejc.464.1658252596053; Tue, 19 Jul 2022 10:43:16 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= Subject: [PATCH V7 06/11] vpci/header: program p2m with guest BAR view Date: Tue, 19 Jul 2022 20:42:48 +0300 Message-Id: <20220719174253.541965-7-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the PCI bus driver in the hardware domain. This way hardware domain sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - remove debug print in map_range callback - remove "identity" from the debug print Since v4: - moved start_{gfn|mfn} calculation into map_range - pass vpci_bar in the map_data instead of start_{gfn|mfn} - s/guest_addr/guest_reg Since v3: - updated comment (Roger) - removed gfn_add(map->start_gfn, rc); which is wrong - use v->domain instead of v->vpci.pdev->domain - removed odd e.g. in comment - s/d%d/%pd in altered code - use gdprintk for map/unmap logs Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index f14ff11882..4e6547a54d 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,7 @@ struct map_data { struct domain *d; + const struct vpci_bar *bar; bool map; }; @@ -41,8 +42,21 @@ static int cf_check map_range( for ( ; ; ) { + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn = _gfn(PFN_DOWN(is_hardware_domain(map->d) + ? map->bar->addr + : map->bar->guest_reg)); + /* Physical start address of the BAR. */ + mfn_t start_mfn = _mfn(PFN_DOWN(map->bar->addr)); unsigned long size = e - s + 1; + /* + * Ranges to be mapped don't always start at the BAR start address, as + * there can be holes or partially consumed ranges. Account for the + * offset of the current address from the BAR start. + */ + start_gfn = gfn_add(start_gfn, s - mfn_x(start_mfn)); + /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +66,8 @@ static int cf_check map_range( * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -62,8 +76,8 @@ static int cf_check map_range( if ( rc < 0 ) { printk(XENLOG_G_WARNING - "Failed to identity %smap [%lx, %lx] for d%d: %d\n", - map->map ? "" : "un", s, e, map->d->domain_id, rc); + "Failed to %smap [%lx, %lx] for %pd: %d\n", + map->map ? "" : "un", s, e, map->d, rc); break; } ASSERT(rc < size); @@ -155,6 +169,7 @@ bool vpci_process_pending(struct vcpu *v) if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -218,6 +233,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) { From patchwork Tue Jul 19 17:42:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 868B3CCA488 for ; Tue, 19 Jul 2022 17:43:37 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370804.602747 (Exim 4.92) (envelope-from ) id 1oDrFh-00080n-Ft; Tue, 19 Jul 2022 17:43:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370804.602747; Tue, 19 Jul 2022 17:43:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFh-000801-5Q; Tue, 19 Jul 2022 17:43:21 +0000 Received: by outflank-mailman (input) for mailman id 370804; Tue, 19 Jul 2022 17:43:19 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFf-00065R-4W for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:19 +0000 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [2a00:1450:4864:20::52c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 4614dfb9-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:18 +0200 (CEST) Received: by mail-ed1-x52c.google.com with SMTP id y8so20648513eda.3 for ; Tue, 19 Jul 2022 10:43:18 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:16 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4614dfb9-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NQVKGUXCWq6gwb2au9pJADkwFsbt3FsEYw9qYU1cR14=; b=D9S5tYQQeZHWtI9eesc3VZSqYMxyAotVHoWWiJ7yYXf2UlUbz9JY1/YTEO0g6yWeiY UCsW4olncR1OY0PPhWMKYCCGmJIGcnv1JXE7Ls4LWcidv9XU1RYDp0ThkNPxN8/GouGj all35/LcBjtqVPu2CWxYD1ydAvgOTHOjD9SQ5ekMPVm9o47Lqfp5kyt6YHAaH3u69y73 CFqNXyJpq+nWoR2SixcgsPVzGesYfFRXnDR9eoKSadi12IwNeJxpfiXd53HaLxUPfpK5 j/acfgMEh845vECyhcRq4LfK5ADBzdyGyzxBdVEDUlRYE9nKzPWo9lEfhy3HkENZgASF gIfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NQVKGUXCWq6gwb2au9pJADkwFsbt3FsEYw9qYU1cR14=; b=26VWz8Hg/8Iuy/yMBh8FnczxRtPIa7RyCGAE9kpMFc2sVBTaN1dPwJeFo2f1YsP31B 2gH/LGUt5OmTMcDC/KoF3LjoMMOqGMc2Q6ezUEbnpL94aiJA4Mkp/rXzwL1HKgg8LTeY hPlgOX8/prMuk6m97Wf8/oZlatZRFlaXE15lRGxNuxoPSyI8aUWBXfCLhV4Zcq+gZB4J XpRbnFWqSSjebpYyepXYXd0rQzvyGlV7vkcqFSCK/L79cn00c8HghX8ZQflMu5bUH/eu EWGUWzehNQv5zghIl1xh5Tp+ZG2Ob/EdLzMAZocDMiDkVwZaRUc25QGDac2GGeoeicxs r7fA== X-Gm-Message-State: AJIora8hBUr5mu7XH4+5ibYvVUpiIPQC7elJURy1EjmMhWmcB8bEGrTd zVpoYp9H6JRTERveotlgz3T3P9ViZR4+rg== X-Google-Smtp-Source: AGRyM1vJguGRMwMPFUWXrI9LSVBvRsLUYgXtF2tzmcgsvM+heKS5t0myo4bf5MxMeZPzYYVdvf+WsA== X-Received: by 2002:a05:6402:3881:b0:43a:691f:8e3b with SMTP id fd1-20020a056402388100b0043a691f8e3bmr45389885edb.217.1658252597458; Tue, 19 Jul 2022 10:43:17 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= Subject: [PATCH V7 07/11] vpci/header: emulate PCI_COMMAND register for guests Date: Tue, 19 Jul 2022 20:42:49 +0300 Message-Id: <20220719174253.541965-8-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Xen and/or Dom0 may have put values in PCI_COMMAND which they expect to remain unaltered. PCI_COMMAND_SERR bit is a good example: while the guest's view of this will want to be zero initially, the host having set it to 1 may not easily be overwritten with 0, or else we'd effectively imply giving the guest control of the bit. Thus, PCI_COMMAND register needs proper emulation in order to honor host's settings. There are examples of emulators [1], [2] which already deal with PCI_COMMAND register emulation and it seems that at most they care about is the only INTx bit (besides IO/memory enable and bus master which are write through). It could be because in order to properly emulate the PCI_COMMAND register we need to know about the whole PCI topology, e.g. if any setting in device's command register is aligned with the upstream port etc. This makes me think that because of this complexity others just ignore that. Neither I think this can easily be done in Xen case. According to "PCI LOCAL BUS SPECIFICATION, REV. 3.0", section "6.2.2 Device Control" the reset state of the command register is typically 0, so when assigning a PCI device use 0 as the initial state for the guest's view of the command register. For now our emulation only makes sure INTx is set according to the host requirements, i.e. depending on MSI/MSI-X enabled state. This implementation and the decision to only emulate INTx bit for now is based on the previous discussion at [3]. [1] https://github.com/qemu/qemu/blob/master/hw/xen/xen_pt_config_init.c#L310 [2] https://github.com/projectacrn/acrn-hypervisor/blob/master/hypervisor/hw/pci.c#L336 [3] https://patchwork.kernel.org/project/xen-devel/patch/20210903100831.177748-9-andr2000@gmail.com/ Signed-off-by: Oleksandr Andrushchenko --- Since v6: - fold guest's logic into cmd_write - implement cmd_read, so we can report emulated INTx state to guests - introduce header->guest_cmd to hold the emulated state of the PCI_COMMAND register for guests - OT: rebased - OT: add cf_check specifier to cmd_read() Since v5: - add additional check for MSI-X enabled while altering INTX bit - make sure INTx disabled while guests enable MSI/MSI-X Since v3: - gate more code on CONFIG_HAS_MSI - removed logic for the case when MSI/MSI-X not enabled --- xen/drivers/vpci/header.c | 38 +++++++++++++++++++++++++++++++++++++- xen/drivers/vpci/msi.c | 4 ++++ xen/drivers/vpci/msix.c | 4 ++++ xen/include/xen/vpci.h | 3 +++ 4 files changed, 48 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 4e6547a54d..2ce69a63a2 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -443,11 +443,27 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) return 0; } +/* TODO: Add proper emulation for all bits of the command register. */ static void cf_check cmd_write( const struct pci_dev *pdev, unsigned int reg, uint32_t cmd, void *data) { uint16_t current_cmd = pci_conf_read16(pdev->sbdf, reg); + if ( !is_hardware_domain(pdev->domain) ) + { + struct vpci_header *header = data; + + header->guest_cmd = cmd; +#ifdef CONFIG_HAS_PCI_MSI + if ( pdev->vpci->msi->enabled || pdev->vpci->msix->enabled ) + /* + * Guest wants to enable INTx, but it can't be enabled + * if MSI/MSI-X enabled. + */ + cmd |= PCI_COMMAND_INTX_DISABLE; +#endif + } + /* * Let Dom0 play with all the bits directly except for the memory * decoding one. @@ -464,6 +480,19 @@ static void cf_check cmd_write( pci_conf_write16(pdev->sbdf, reg, cmd); } +static uint32_t cf_check cmd_read( + const struct pci_dev *pdev, unsigned int reg, void *data) +{ + if ( !is_hardware_domain(pdev->domain) ) + { + struct vpci_header *header = data; + + return header->guest_cmd; + } + + return pci_conf_read16(pdev->sbdf, reg); +} + static void cf_check bar_write( const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -665,8 +694,15 @@ static int cf_check init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } + /* + * According to "PCI LOCAL BUS SPECIFICATION, REV. 3.0", section "6.2.2 + * Device Control" the reset state of the command register is + * typically all 0's, so this is used as initial value for the guests. + */ + ASSERT(header->guest_cmd == 0); + /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, + rc = vpci_add_register(pdev->vpci, cmd_read, cmd_write, PCI_COMMAND, 2, header); if ( rc ) return rc; diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index d864f740cf..c8c495e2d7 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -70,6 +70,10 @@ static void cf_check control_write( if ( vpci_msi_arch_enable(msi, pdev, vectors) ) return; + + /* Make sure guest doesn't enable INTx while enabling MSI. */ + if ( !is_hardware_domain(pdev->domain) ) + pci_intx(pdev, false); } else vpci_msi_arch_disable(msi, pdev); diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 35cc9280f4..06f84b8b02 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -92,6 +92,10 @@ static void cf_check control_write( for ( i = 0; i < msix->max_entries; i++ ) if ( !msix->entries[i].masked && msix->entries[i].updated ) update_entry(&msix->entries[i], pdev, i); + + /* Make sure guest doesn't enable INTx while enabling MSI-X. */ + if ( !is_hardware_domain(pdev->domain) ) + pci_intx(pdev, false); } else if ( !new_enabled && msix->enabled ) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 6332659c4a..1010f68c28 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -87,6 +87,9 @@ struct vpci { } bars[PCI_HEADER_NORMAL_NR_BARS + 1]; /* At most 6 BARS + 1 expansion ROM BAR. */ + /* Guest view of the PCI_COMMAND register. */ + uint16_t guest_cmd; + /* * Store whether the ROM enable bit is set (doesn't imply ROM BAR * is mapped into guest p2m) if there's a ROM BAR on the device. From patchwork Tue Jul 19 17:42:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5305C3F2D4 for ; Tue, 19 Jul 2022 17:43:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370806.602752 (Exim 4.92) (envelope-from ) id 1oDrFi-0008At-6s; Tue, 19 Jul 2022 17:43:22 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370806.602752; Tue, 19 Jul 2022 17:43:22 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFi-00089A-28; Tue, 19 Jul 2022 17:43:22 +0000 Received: by outflank-mailman (input) for mailman id 370806; Tue, 19 Jul 2022 17:43:20 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFg-00065R-DZ for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:20 +0000 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [2a00:1450:4864:20::632]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 46a6d59e-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:19 +0200 (CEST) Received: by mail-ej1-x632.google.com with SMTP id j22so28609572ejs.2 for ; Tue, 19 Jul 2022 10:43:19 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:17 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 46a6d59e-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QdkNpMhNG/rH2hDrjEqNlnDvarGoZ2Hnwipc//T8xoA=; b=kCtcd6y9IjewbRIotgLC7v/cJIoXJ/MN4xbaTvTJxILlca8FQtjx948FxXUmHlH9Nz u7PCFpRxM0iUCn/e7JteV+V+nQVNBckd9beuv9H/J1pMA1MyWT1jLP9rliJuApMopxEt E3Bc/5rvdaKAIsFW9B3ucUUa+3PPeuChPaBvKj3LzlB+cMDTrf9vfoFIJzE1bd4yAWt3 7BsZG9BYr5VjPQFcUblpnLgZE91KDhSkilg1V4smxIBkAgOp4gRWVzKJZM86l+J3vebi 8ve5+kyCH7QB0JDzVJyCdhqepz1TuVEuVkTJjOTw9zlA4zPbwG0Rzt7Rgxct++30Bmd5 3FKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QdkNpMhNG/rH2hDrjEqNlnDvarGoZ2Hnwipc//T8xoA=; b=vT3/jQIt44NR9n1nXzMupEyK0KVzhTuG5ytbiOxxyEfhjDI95GR4GRB+iYkp3ZkDXQ 6rKPTcLVCniIGbJx8PJCf+Quhpzvn1G7W4LkODX35KvTBmoTiIbjUXm9nTDmjmd38LMj jSkLup7ZbSyPpnM1QQUblifIct2Z3WLU2PzVkVEER8UTPnckaiMZPGyBqkSrwIGApv75 Rxcfg7/wUOnfaaRpmg4ldwRDqF5VdsFsH0DzXaCx81O8UnUqM5KVogd83grOPPyzobq/ MU/W6u1u/tw/GOWJISg8dgkbgwCe4yC+2otcLkx1FvCpUT3jYpJrKawoa6ZqHssV6+Kc p7Xg== X-Gm-Message-State: AJIora83D9rLmzAoRu33D4QxzEPjAJ3EKAAlOYu+JYZAclxRj/uhDLAc ODhI3mWbAPn+0OZX98yu5PQpiC6G+o0yCA== X-Google-Smtp-Source: AGRyM1vWyMAQazg5NUObXpt91XDu9miFPFNRbJJ25XFfOGHsIqfJSP8H9L3QnEklIiBZJi3ckOePmw== X-Received: by 2002:a17:907:94cb:b0:72f:1d8e:7305 with SMTP id dn11-20020a17090794cb00b0072f1d8e7305mr14904155ejc.625.1658252598420; Tue, 19 Jul 2022 10:43:18 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= Subject: [PATCH V7 08/11] vpci/header: reset the command register when adding devices Date: Tue, 19 Jul 2022 20:42:50 +0300 Message-Id: <20220719174253.541965-9-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Reset the command register when assigning a PCI device to a guest: according to the PCI spec the PCI_COMMAND register is typically all 0's after reset, but this might not be true for the guest as it needs to respect host's settings. For that reason, do not write 0 to the PCI_COMMAND register directly, but go through the corresponding emulation layer (cmd_write), which will take care about the actual bits written. Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Rahul Singh --- Since v6: - use cmd_write directly without introducing emulate_cmd_reg - update commit message with more description on all 0's in PCI_COMMAND Since v5: - updated commit message Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 2ce69a63a2..1be9775dda 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -701,6 +701,10 @@ static int cf_check init_bars(struct pci_dev *pdev) */ ASSERT(header->guest_cmd == 0); + /* Reset the command register for guests. */ + if ( !is_hwdom ) + cmd_write(pdev, PCI_COMMAND, 0, header); + /* Setup a handler for the command register. */ rc = vpci_add_register(pdev->vpci, cmd_read, cmd_write, PCI_COMMAND, 2, header); From patchwork Tue Jul 19 17:42:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 97557CCA48A for ; Tue, 19 Jul 2022 17:43:37 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370807.602761 (Exim 4.92) (envelope-from ) id 1oDrFj-0008Jh-57; Tue, 19 Jul 2022 17:43:23 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370807.602761; Tue, 19 Jul 2022 17:43:23 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFi-0008ID-Pb; Tue, 19 Jul 2022 17:43:22 +0000 Received: by outflank-mailman (input) for mailman id 370807; Tue, 19 Jul 2022 17:43:21 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFh-00065X-Em for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:21 +0000 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [2a00:1450:4864:20::52b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 478036d2-078a-11ed-924f-1f966e50362f; Tue, 19 Jul 2022 19:43:20 +0200 (CEST) Received: by mail-ed1-x52b.google.com with SMTP id m16so20604833edb.11 for ; Tue, 19 Jul 2022 10:43:20 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:19 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 478036d2-078a-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rMoBmnl3ZpBTv1UmtiTdiuto4NAuEX1ahKih6uEYEJw=; b=j+R/OvzbEZtIhpUEhIGWc3q1bbZHYp0Ul+K4Dz37jhZz1+6vh8PCxAdE6dTmaGXtW6 AVnPLavEd7Syclorws2JcUeO0Ym2q9+9XlivFihN3LffQaeuDEQZcQuLS3PhI8c3e0M8 H5d8biejz0cHioAeGCZa0kcBFxN6ksKnK8+XWSKEeS/l4Cqv9N4V7/f6qbhg629nMOnw rEOKgkL+i552j3dW2PRr0/3H69bdD2k/vTwnLDG49dNpkqLKW+ac3A6SNOchhs6cCtVK MXNTUs9RN9Ubp8B9PbOMfDTxeUqgEOhqONELTgmT+axiBQMnvZvssNO2OpyaF6kTEi/y r7Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rMoBmnl3ZpBTv1UmtiTdiuto4NAuEX1ahKih6uEYEJw=; b=5dyBaKsu0MYfAYICVh/9ljA9TidTryulC0zAlKibiN4Jtv/iZd2vtecXdUO1bZFb4s cLSWIjqUEo1AJzzhzzZrzJV5uIOnqX4sIrwTWuhpuMau7rinA9gSe4f8heCMZECpUnSo oeff1Zl1X3c+n0LilhRkBNU1mTtxg9WkGd96bwDqS2c6ASgDhg2j++zXIHc6g28buszI 1gF2e0S2EX2y7Bxn/ygVFDuGG7MW3VGpKxcLrCVjXwv/W7wVk8MkF0YOXk3/H9Z5rram fllaNCc49P42kmpOIxDYcaE9F8dhjMbPl1nfg0dVLi4mMWnFASxobpWgtRsncWmZkZ4i 9gsg== X-Gm-Message-State: AJIora8gcIF0U8o3ZWD8XBZ7AIqgPoi+XEI9jDImLyIPHwzp+CS5fTD1 w5UUc7/KbrcdSSKm5a3CTfXFDeF9ylUWtw== X-Google-Smtp-Source: AGRyM1vwWKS94qRtIYCQT74pZ5Yf4pcr8nEvoMGSizTcpXdveRMSImguFS8MGQrIxdv0ZUQ8RxYMAg== X-Received: by 2002:a05:6402:26c3:b0:43a:a846:b2c1 with SMTP id x3-20020a05640226c300b0043aa846b2c1mr45920791edd.133.1658252599786; Tue, 19 Jul 2022 10:43:19 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini , Wei Liu Subject: [PATCH V7 09/11] vpci: add initial support for virtual PCI bus topology Date: Tue, 19 Jul 2022 20:42:51 +0300 Message-Id: <20220719174253.541965-10-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Please note, that at the moment only function 0 of a multifunction device can be passed through. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - re-work wrt new locking scheme - OT: add ASSERT(pcidevs_write_locked()); to add_virtual_device() Since v5: - s/vpci_add_virtual_device/add_virtual_device and make it static - call add_virtual_device from vpci_assign_device and do not use REGISTER_VPCI_INIT machinery - add pcidevs_locked ASSERT - use DECLARE_BITMAP for vpci_dev_assigned_map Since v4: - moved and re-worked guest sbdf initializers - s/set_bit/__set_bit - s/clear_bit/__clear_bit - minor comment fix s/Virtual/Guest/ - added VPCI_MAX_VIRT_DEV constant (PCI_SLOT(~0) + 1) which will be used later for counting the number of MMIO handlers required for a guest (Julien) Since v3: - make use of VPCI_INIT - moved all new code to vpci.c which belongs to it - changed open-coded 31 to PCI_SLOT(~0) - added comments and code to reject multifunction devices with functions other than 0 - updated comment about vpci_dev_next and made it unsigned int - implement roll back in case of error while assigning/deassigning devices - s/dom%pd/%pd Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/drivers/vpci/vpci.c | 70 ++++++++++++++++++++++++++++++++++++++++- xen/include/xen/sched.h | 8 +++++ xen/include/xen/vpci.h | 11 +++++++ 3 files changed, 88 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index f683346285..d4601ecf9b 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -84,6 +84,9 @@ int vpci_add_handlers(struct pci_dev *pdev) INIT_LIST_HEAD(&pdev->vpci->handlers); spin_lock_init(&pdev->vpci->lock); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + pdev->vpci->guest_sbdf.sbdf = ~0; +#endif for ( i = 0; i < NUM_VPCI_INIT; i++ ) { @@ -99,6 +102,62 @@ int vpci_add_handlers(struct pci_dev *pdev) } #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +static int add_virtual_device(struct pci_dev *pdev) +{ + struct domain *d = pdev->domain; + pci_sbdf_t sbdf = { 0 }; + unsigned long new_dev_number; + + if ( is_hardware_domain(d) ) + return 0; + + ASSERT(pcidevs_write_locked()); + + /* + * Each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ + if ( pdev->info.is_extfn ) + { + gdprintk(XENLOG_ERR, "%pp: only function 0 passthrough supported\n", + &pdev->sbdf); + return -EOPNOTSUPP; + } + + new_dev_number = find_first_zero_bit(d->vpci_dev_assigned_map, + VPCI_MAX_VIRT_DEV); + if ( new_dev_number >= VPCI_MAX_VIRT_DEV ) + return -ENOSPC; + + __set_bit(new_dev_number, &d->vpci_dev_assigned_map); + + /* + * Both segment and bus number are 0: + * - we emulate a single host bridge for the guest, e.g. segment 0 + * - with bus 0 the virtual devices are seen as embedded + * endpoints behind the root complex + * + * TODO: add support for multi-function devices. + */ + sbdf.devfn = PCI_DEVFN(new_dev_number, 0); + pdev->vpci->guest_sbdf = sbdf; + + return 0; + +} + +static void vpci_remove_virtual_device(const struct pci_dev *pdev) +{ + ASSERT(pcidevs_write_locked()); + + if ( pdev->vpci ) + { + __clear_bit(pdev->vpci->guest_sbdf.dev, + &pdev->domain->vpci_dev_assigned_map); + pdev->vpci->guest_sbdf.sbdf = ~0; + } +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct pci_dev *pdev) { @@ -111,8 +170,16 @@ int vpci_assign_device(struct pci_dev *pdev) rc = vpci_add_handlers(pdev); if ( rc ) - vpci_deassign_device(pdev); + goto fail; + + rc = add_virtual_device(pdev); + if ( rc ) + goto fail; + + return 0; + fail: + vpci_deassign_device(pdev); return rc; } @@ -124,6 +191,7 @@ void vpci_deassign_device(struct pci_dev *pdev) if ( !has_vpci(pdev->domain) ) return; + vpci_remove_virtual_device(pdev); vpci_remove_device(pdev); } #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index b9515eb497..a2848a5740 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -457,6 +457,14 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * The bitmap which shows which device numbers are already used by the + * virtual PCI bus topology and is used to assign a unique SBDF to the + * next passed through virtual PCI device. + */ + DECLARE_BITMAP(vpci_dev_assigned_map, VPCI_MAX_VIRT_DEV); +#endif #endif #ifdef CONFIG_HAS_PASSTHROUGH diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 1010f68c28..cc14b0086d 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -21,6 +21,13 @@ typedef int vpci_register_init_t(struct pci_dev *dev); #define VPCI_ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12) +/* + * Maximum number of devices supported by the virtual bus topology: + * each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ +#define VPCI_MAX_VIRT_DEV (PCI_SLOT(~0) + 1) + #define REGISTER_VPCI_INIT(x, p) \ static vpci_register_init_t *const x##_entry \ __used_section(".data.vpci." p) = x @@ -145,6 +152,10 @@ struct vpci { struct vpci_arch_msix_entry arch; } entries[]; } *msix; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* Guest SBDF of the device. */ + pci_sbdf_t guest_sbdf; +#endif #endif }; From patchwork Tue Jul 19 17:42:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31383C43334 for ; Tue, 19 Jul 2022 17:43:39 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370809.602781 (Exim 4.92) (envelope-from ) id 1oDrFl-0000bb-V4; Tue, 19 Jul 2022 17:43:25 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370809.602781; Tue, 19 Jul 2022 17:43:25 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFl-0000a0-JR; Tue, 19 Jul 2022 17:43:25 +0000 Received: by outflank-mailman (input) for mailman id 370809; Tue, 19 Jul 2022 17:43:23 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFi-00065X-Q4 for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:22 +0000 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [2a00:1450:4864:20::630]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 4852fe6a-078a-11ed-924f-1f966e50362f; Tue, 19 Jul 2022 19:43:22 +0200 (CEST) Received: by mail-ej1-x630.google.com with SMTP id fy29so27438152ejc.12 for ; Tue, 19 Jul 2022 10:43:22 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:20 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4852fe6a-078a-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xeDSBefOUDefddwDFekw6Jxk+8IVVFXqS6nwlVZXVQU=; b=LFZN7CtylG3g24WT/N4AzwAJAclOs6Jx4biOe+TPkuFnIdk67Ggth4osh4UWA1JQqg 2gWZv+LO0nfgLBOAcPTUBBLpqIuToascsQZgquN3fv4dbRcVHpZC42GTPl/fvZBywLu0 y6PM9A0BjpL0q9G5eBCDlAYBBib8KInXOkC7/P7AXDSX68KgfxwcEjjgaaHJE+ZnevXy uWnEBnI+2f1rzfKJJZoO1tVwNRHLTCpeMi0g+0m2vcszSZjWYI/3aFQQWPZv6hrp6OxV JsvhmJGOcyVFchLGxCfKdJjbXXfHQtAy17DVsp4qdNoFdOyzzam7N2de7xowP84uRxJu 4GDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xeDSBefOUDefddwDFekw6Jxk+8IVVFXqS6nwlVZXVQU=; b=pPtsW0MyvprNACdJUvPTWvoa1ZzFzMR7UnjwJFkDUKPxahb96IqZNmq41wcz8l9/KN yt8ZEkuFwJJubNVmGcZ0pSro7oI960c6VvzWJ/ejRjx0ObTz++GY3/l4TS6JJDYfZEJ3 K3u6oRbbbX6DykbvzkT1CNm+RClhaVsWkBgBwOd2dnZhFDGtNBnq0PD3t9oXoho94bFP 1adQkTnbVimYXtf9ms1zN1PObUrN1eFJjD9usptpToEIbHNMSg49I0YNx9fhDlrM+QDU wNxMIZmaP/2XcSFXRwqbWbozJWWGvErFSRGCSFzg1So1CidxjA//Oar00ySTQvtydMmf qkYg== X-Gm-Message-State: AJIora+NJvarU5EznIjZaA2xL79YT/SwygenFaBd08B2l7pSk0YFfpEG BN3Vnyrtl4fvl8D4hq54N2SqzbjDQVFmmw== X-Google-Smtp-Source: AGRyM1ta7Q2IduHJgFwfDvZzHGMnbRGwVjNFMtsjyrVAgo3Vzo5xc2id5a7tTgOh4fNVI5H16aNGSQ== X-Received: by 2002:a17:907:7632:b0:72b:4d5d:ecce with SMTP id jy18-20020a170907763200b0072b4d5deccemr32856791ejc.11.1658252601136; Tue, 19 Jul 2022 10:43:21 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , Stefano Stabellini , Julien Grall , Bertrand Marquis , Volodymyr Babchuk , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= Subject: [PATCH V7 10/11] xen/arm: translate virtual PCI bus topology for guests Date: Tue, 19 Jul 2022 20:42:52 +0300 Message-Id: <20220719174253.541965-11-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- Since v6: - add pcidevs locking to vpci_translate_virtual_device - update wrt to the new locking scheme Since v5: - add vpci_translate_virtual_device for #ifndef CONFIG_HAS_VPCI_GUEST_SUPPORT case to simplify ifdefery - add ASSERT(!is_hardware_domain(d)); to vpci_translate_virtual_device - reset output register on failed virtual SBDF translation Since v4: - indentation fixes - constify struct domain - updated commit message - updates to the new locking scheme (pdev->vpci_lock) Since v3: - revisit locking - move code to vpci.c Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/vpci.c | 17 +++++++++++++++++ xen/drivers/vpci/vpci.c | 26 ++++++++++++++++++++++++++ xen/include/xen/vpci.h | 7 +++++++ 3 files changed, 50 insertions(+) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index a9fc5817f9..84b2b068a0 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -41,6 +41,16 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + { + *r = ~0ul; + return 1; + } + if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -59,6 +69,13 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, struct pci_host_bridge *bridge = p; pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + return 1; + return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); } diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index d4601ecf9b..fc2c51dc3e 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -158,6 +158,32 @@ static void vpci_remove_virtual_device(const struct pci_dev *pdev) } } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool vpci_translate_virtual_device(struct domain *d, pci_sbdf_t *sbdf) +{ + struct pci_dev *pdev; + + ASSERT(!is_hardware_domain(d)); + + pcidevs_read_lock(); + for_each_pdev( d, pdev ) + { + if ( pdev->vpci && (pdev->vpci->guest_sbdf.sbdf == sbdf->sbdf) ) + { + /* Replace guest SBDF with the physical one. */ + *sbdf = pdev->sbdf; + pcidevs_read_unlock(); + return true; + } + } + + pcidevs_read_unlock(); + return false; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct pci_dev *pdev) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index cc14b0086d..5749d8da78 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -276,6 +276,7 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) /* Notify vPCI that device is assigned/de-assigned to/from guest. */ int vpci_assign_device(struct pci_dev *pdev); void vpci_deassign_device(struct pci_dev *pdev); +bool vpci_translate_virtual_device(struct domain *d, pci_sbdf_t *sbdf); #else static inline int vpci_assign_device(struct pci_dev *pdev) { @@ -285,6 +286,12 @@ static inline int vpci_assign_device(struct pci_dev *pdev) static inline void vpci_deassign_device(struct pci_dev *pdev) { }; + +static inline bool vpci_translate_virtual_device(struct domain *d, + pci_sbdf_t *sbdf) +{ + return false; +} #endif #endif From patchwork Tue Jul 19 17:42:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12922850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5840FC433EF for ; Tue, 19 Jul 2022 17:43:38 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.370810.602785 (Exim 4.92) (envelope-from ) id 1oDrFm-0000fN-Jg; Tue, 19 Jul 2022 17:43:26 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 370810.602785; Tue, 19 Jul 2022 17:43:26 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFm-0000eF-3L; Tue, 19 Jul 2022 17:43:26 +0000 Received: by outflank-mailman (input) for mailman id 370810; Tue, 19 Jul 2022 17:43:24 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oDrFj-00065R-U2 for xen-devel@lists.xenproject.org; Tue, 19 Jul 2022 17:43:24 +0000 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [2a00:1450:4864:20::62e]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 490d4515-078a-11ed-bd2d-47488cf2e6aa; Tue, 19 Jul 2022 19:43:23 +0200 (CEST) Received: by mail-ej1-x62e.google.com with SMTP id mf4so28602384ejc.3 for ; Tue, 19 Jul 2022 10:43:23 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id wj18-20020a170907051200b0072af92fa086sm6903268ejb.32.2022.07.19.10.43.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 10:43:21 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 490d4515-078a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mjfe83MniR2Jpzqqr5fpFWKhsRCC+quCKfhPY3IFOKo=; b=OQ2AVbzzIBKQSpHMBmd5Jzm4KjCKHI2AAmLHJXEljMxRabf2UMa3xCil5cWsSzMrZs RwsTk3NLGqkmP3w3yS8GXO+hUMMADmmSeHB7jnhbmLwJNBhZUvWSmztZjBvxxSdZFBaL uV8IBGw0b60nevXgPI5yGzPaAA5lgK/7Pep3xuNqVhSy7sAWE1yG1LPFn4MaXmfrKx0h mwAAcP+CluaUkKNp7ZSNiw8seuvSI1GhqGZm3/kP2Y3/N+EH/aLXncaQ0b4x7CaY27LQ 3so8rkxBrQOJx3l2Jmtfzfb9Jua7272wcq92zN4mDgLqnYQhRmOicq39wagsTk+dDqvE 1DMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mjfe83MniR2Jpzqqr5fpFWKhsRCC+quCKfhPY3IFOKo=; b=wEFqAo1J8PP3GiSlADc3JploblLN137V0Oz75lzUmgBtBkMV4t0SheX1LYhJNMu/zW 5udMsGwNO2Sh4MsOuK9bwiw7OGXtlxaHwRu1/Eej6dyLRbkmxQ7io1BPYp7N67UTQ2G4 7jDf56Jws7ewZEHokg9GYhI9Hv+6pC3vNfWDvfp6pcLhoPvxbwbWX8ovgztXzY917bTb viXeC0N9d68iUv95CPUOPut4mfNZp3OCLYI6VIqZBC1Ss7yHB3MYF50ibL8+hHt25W/W W9HUo4fAGP/7YjvBDq6798P/KMKLh4rEpJNZ6QkwaMxMmNs9qIM7mUqTgGtWwHqCyhLE X5Zg== X-Gm-Message-State: AJIora+u8GK+0IJPwx5eJd6KLwowIqoexTgVkV4iY0+/ja/LV+zZFk/h SZ2QgcoqtYZ0EH/7Ppl5WjjMkBhsWp8yFg== X-Google-Smtp-Source: AGRyM1ulGFt/09wDa5Hg84aWzr2K3uZuAfWScHqhuYAENdA5DS01nR+/TJ+ZwDfTeJ2Lg/7e/X6S9Q== X-Received: by 2002:a17:907:75d9:b0:72b:1472:ba19 with SMTP id jl25-20020a17090775d900b0072b1472ba19mr30896306ejc.657.1658252602444; Tue, 19 Jul 2022 10:43:22 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Andrushchenko , Bertrand Marquis , Volodymyr Babchuk , Julien Grall , Julien Grall , Stefano Stabellini Subject: [PATCH V7 11/11] xen/arm: account IO handlers for emulated PCI MSI-X Date: Tue, 19 Jul 2022 20:42:53 +0300 Message-Id: <20220719174253.541965-12-olekstysh@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220719174253.541965-1-olekstysh@gmail.com> References: <20220719174253.541965-1-olekstysh@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko At the moment, we always allocate an extra 16 slots for IO handlers (see MAX_IO_HANDLER). So while adding IO trap handlers for the emulated MSI-X registers we need to explicitly tell that we have additional IO handlers, so those are accounted. Signed-off-by: Oleksandr Andrushchenko Acked-by: Julien Grall Reviewed-by: Rahul Singh --- Cc: Julien Grall Cc: Stefano Stabellini --- This actually moved here from the part 2 of the prep work for PCI passthrough on Arm as it seems to be the proper place for it. Since v5: - optimize with IS_ENABLED(CONFIG_HAS_PCI_MSI) since VPCI_MAX_VIRT_DEV is defined unconditionally New in v5 --- xen/arch/arm/vpci.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 84b2b068a0..c5902cb9d3 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -131,6 +131,8 @@ static int vpci_get_num_handlers_cb(struct domain *d, unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) { + unsigned int count; + if ( !has_vpci(d) ) return 0; @@ -151,7 +153,17 @@ unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) * For guests each host bridge requires one region to cover the * configuration space. At the moment, we only expose a single host bridge. */ - return 1; + count = 1; + + /* + * There's a single MSI-X MMIO handler that deals with both PBA + * and MSI-X tables per each PCI device being passed through. + * Maximum number of emulated virtual devices is VPCI_MAX_VIRT_DEV. + */ + if ( IS_ENABLED(CONFIG_HAS_PCI_MSI) ) + count += VPCI_MAX_VIRT_DEV; + + return count; } /*