From patchwork Fri Nov 5 06:56:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29CA3C433FE for ; Fri, 5 Nov 2021 06:56:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D6B1D61279 for ; Fri, 5 Nov 2021 06:56:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D6B1D61279 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222086.384111 (Exim 4.92) (envelope-from ) id 1mit9P-0008Hr-PJ; Fri, 05 Nov 2021 06:56:35 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222086.384111; Fri, 05 Nov 2021 06:56:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9P-0008Gf-It; Fri, 05 Nov 2021 06:56:35 +0000 Received: by outflank-mailman (input) for mailman id 222086; Fri, 05 Nov 2021 06:56:34 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9O-0008Eb-Oj for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:34 +0000 Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [2a00:1450:4864:20::532]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 834c34e1-3e05-11ec-9787-a32c541c8605; Fri, 05 Nov 2021 07:56:34 +0100 (CET) Received: by mail-ed1-x532.google.com with SMTP id w1so30026395edd.10 for ; Thu, 04 Nov 2021 23:56:34 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:33 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 834c34e1-3e05-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yH9Aud+KhB6bv78AcEs/8owAaYp6cfId4snOD7ig8jw=; b=EfwEGn9TETYI3BA8ZmMTaDRqGPxCqXvnVwC3dRPjriRMBXX3h/wixWYL8mkilXFism 4ROBY4RdKHXgMgbKWI8NO+s0nJQU9XgD9EA5O4lH5BDPOcIaiqUD4yE/rWlJf0zj0hzD 7iPduDZB2eOSWU3sVQQDxlKr62mlS28xpaWu1Oy8KxJzWnmK0/ajj3LKL3hQeynGu9gq tIS2yxE9jh/KHIlICHhSdzlmCyGORvS1MuI8ipwcmEYxbQaX/xs+3NNHfXuKpcEyLL9P H0wFfHiQQxHVtPrrjCTc5+tRUOkUnnMUMk2xmc/XYoSc99XeyLGbspcGP1hgKtePggK4 dbjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yH9Aud+KhB6bv78AcEs/8owAaYp6cfId4snOD7ig8jw=; b=WGgHvPIxk+fmqz2WLT0K3snTOipPGx0fJb+hJC7xr31Pg78S0tHYJeUtAf9iEGnRzI lCzXR0X40UqXnGzWplU0hHkJhrR/bZ22uFrlFAIi4+KrGTHVQXj3F9JTShvhm6fOTEZ2 ElM022M2VdjXprgv9vCjtVcwiGIGLQ/2JHShrDjjdxtUWyuq1BDfnrcvyauAqilK5GA6 bCFa1VUHevtG5kjcF6KNvqEdEc0kTj99aIg1EbDNq305y5uLBRf00+xJfRDqFB4gh4Sn MIfZLrwNDb64qvGE3hoJSiUsd5moNXk8fJowkYq1BXRpKKVnKhBoLTg+5EG4Z/8BbVr+ 00Yg== X-Gm-Message-State: AOAM533zlrCt0MT8AhX98tIBsFm2H8COVLCY7S2po19SaACHSdFVWqKD 3gGRY3nSwU/QkVA4mCppIcPfp7S99/bC4A== X-Google-Smtp-Source: ABdhPJyPoRRSpp2BwAz5XkpMz3lTivAT+uV0OLjngnMXEocJYUDT7N4y3Xq77SbrGFLNMo7gJZ1yZg== X-Received: by 2002:a17:906:3647:: with SMTP id r7mr22797776ejb.124.1636095393576; Thu, 04 Nov 2021 23:56:33 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 01/11] vpci: fix function attributes for vpci_process_pending Date: Fri, 5 Nov 2021 08:56:19 +0200 Message-Id: <20211105065629.940943-2-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko vpci_process_pending is defined with different attributes, e.g. with __must_check if CONFIG_HAS_VPCI enabled and not otherwise. Fix this by defining both of the definitions with __must_check. Fixes: 14583a590783 ("7fbb096bf345 kconfig: don't select VPCI if building a shim-only binary") Signed-off-by: Oleksandr Andrushchenko --- Cc: Roger Pau Monné New in v4 --- xen/include/xen/vpci.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 9ea66e033f11..3f32de9d7eb3 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -247,7 +247,7 @@ static inline void vpci_write(pci_sbdf_t sbdf, unsigned int reg, ASSERT_UNREACHABLE(); } -static inline bool vpci_process_pending(struct vcpu *v) +static inline bool __must_check vpci_process_pending(struct vcpu *v) { ASSERT_UNREACHABLE(); return false; From patchwork Fri Nov 5 06:56:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E503C43219 for ; Fri, 5 Nov 2021 06:56:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E03FF61283 for ; Fri, 5 Nov 2021 06:56:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E03FF61283 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222087.384128 (Exim 4.92) (envelope-from ) id 1mit9R-0000KL-VF; Fri, 05 Nov 2021 06:56:37 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222087.384128; Fri, 05 Nov 2021 06:56:37 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9R-0000KE-Re; Fri, 05 Nov 2021 06:56:37 +0000 Received: by outflank-mailman (input) for mailman id 222087; Fri, 05 Nov 2021 06:56:36 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9Q-0008NQ-Cl for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:36 +0000 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [2a00:1450:4864:20::52c]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 83fe1709-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:35 +0100 (CET) Received: by mail-ed1-x52c.google.com with SMTP id v11so27203983edc.9 for ; Thu, 04 Nov 2021 23:56:35 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:34 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 83fe1709-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6lI/emOWTuirK6y60vvPuIerMn7oLAhUBrzibAoZIaY=; b=WCrBBXgcHf+gNUzvLec/FGEN0I45bKwR7dhe/BEj3ewFeldoCRSnISCh/7BsOUw3EM z/AyEABFXCyhXDZli/d3jPsKiKxWW6KseG/9ZmImy7j7UX1PULDxn1xTTU7nDKSEclZm n3MUlfXe2cXlZiQ7upfmT647hQ2GqlsADdm0+Gq8+KY34i+ge+i5b8os1r9/KtsHphB0 yrcINSdO9pQwGCc+2Hy8esbXiBTVdEUTH75khVXCHVqaXYjtOWaV0VsYcs4Qt5JHa9lS b0qNKzb5OqGWm4kDfQS1EIO86SoYQRoSbXuU2n4/WZaw7QJr1C04crS1O3C9MPrJtCg0 EfOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6lI/emOWTuirK6y60vvPuIerMn7oLAhUBrzibAoZIaY=; b=hsRIVW3EdS+8RZ/5AD2PK5CuNEgzIqbh0M1hS76ZNswfP7bQrprFotz7SHGbMzQT9x SnyvWRSvdPH8QlJ77z9L7gQQiSUbzt9CH2XWxkkYOv5QHuYGhCLGTdsbTgaWex+eOrxg q3h2PgX327ByVcF3wUPo2XXrOJ2Ruo7O5wdXmFZ+hRgDk6HNwnCjCrBcU5kqwitAiWBS jeqvIw8+UFbr8Sed5i9lmGx9OfmHTLKi+Nwi6vhZlVG40QkXSL9gThn8pNhBwGgkS0SI RvniRrt+kFDNM2kUSNYM8CkbbbhohFwGkHyJuj37l0p3A3g9lOwCihLk43pWXzjcWtsb XIKg== X-Gm-Message-State: AOAM5303A3T+DT1bZq3VN/k+S2V7MhctbZEQKuFinyXoHNf0hSlXiFUB Zi+7/DZ8ryzAQTumtxiAtAmxb8/dTUJArA== X-Google-Smtp-Source: ABdhPJymCd9sLsIV7Rr3M2VzY0Yo2fETPycY1rmqoGvzy38X4VgIs53DsYMsZFc/OMd+LldvIKHN6A== X-Received: by 2002:a17:906:730c:: with SMTP id di12mr66471844ejc.184.1636095394698; Thu, 04 Nov 2021 23:56:34 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 02/11] vpci: cancel pending map/unmap on vpci removal Date: Fri, 5 Nov 2021 08:56:20 +0200 Message-Id: <20211105065629.940943-3-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a vPCI is removed for a PCI device it is possible that we have scheduled a delayed work for map/unmap operations for that device. For example, the following scenario can illustrate the problem: pci_physdev_op pci_add_device init_bars -> modify_bars -> defer_map -> raise_softirq(SCHEDULE_SOFTIRQ) iommu_add_device <- FAILS vpci_remove_device -> xfree(pdev->vpci) leave_hypervisor_to_guest vpci_process_pending: v->vpci.mem != NULL; v->vpci.pdev->vpci == NULL For the hardware domain we continue execution as the worse that could happen is that MMIO mappings are left in place when the device has been deassigned For unprivileged domains that get a failure in the middle of a vPCI {un}map operation we need to destroy them, as we don't know in which state the p2m is. This can only happen in vpci_process_pending for DomUs as they won't be allowed to call pci_add_device. Signed-off-by: Oleksandr Andrushchenko --- Cc: Roger Pau Monné New in v4 --- xen/drivers/vpci/header.c | 15 +++++++++++++-- xen/drivers/vpci/vpci.c | 2 ++ xen/include/xen/vpci.h | 6 ++++++ 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 40ff79c33f8f..ef538386e95d 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -149,8 +149,7 @@ bool vpci_process_pending(struct vcpu *v) !rc && v->vpci.rom_only); spin_unlock(&v->vpci.pdev->vpci->lock); - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; + vpci_cancel_pending(v->vpci.pdev); if ( rc ) /* * FIXME: in case of failure remove the device from the domain. @@ -165,6 +164,18 @@ bool vpci_process_pending(struct vcpu *v) return false; } +void vpci_cancel_pending(const struct pci_dev *pdev) +{ + struct vcpu *v = current; + + /* Cancel any pending work now. */ + if ( v->vpci.mem && v->vpci.pdev == pdev) + { + rangeset_destroy(v->vpci.mem); + v->vpci.mem = NULL; + } +} + static int __init apply_map(struct domain *d, const struct pci_dev *pdev, struct rangeset *mem, uint16_t cmd) { diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 657697fe3406..4e24956419aa 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -51,6 +51,8 @@ void vpci_remove_device(struct pci_dev *pdev) xfree(r); } spin_unlock(&pdev->vpci->lock); + + vpci_cancel_pending(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 3f32de9d7eb3..609d6383b252 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -56,6 +56,7 @@ uint32_t vpci_hw_read32(const struct pci_dev *pdev, unsigned int reg, * should not run. */ bool __must_check vpci_process_pending(struct vcpu *v); +void vpci_cancel_pending(const struct pci_dev *pdev); struct vpci { /* List of vPCI handlers for a device. */ @@ -252,6 +253,11 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) ASSERT_UNREACHABLE(); return false; } + +static inline void vpci_cancel_pending(const struct pci_dev *pdev) +{ + ASSERT_UNREACHABLE(); +} #endif #endif From patchwork Fri Nov 5 06:56:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E31DC4332F for ; Fri, 5 Nov 2021 06:56:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 02B126127B for ; Fri, 5 Nov 2021 06:56:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 02B126127B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222088.384139 (Exim 4.92) (envelope-from ) id 1mit9T-0000bd-7K; Fri, 05 Nov 2021 06:56:39 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222088.384139; Fri, 05 Nov 2021 06:56:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9T-0000bS-3M; Fri, 05 Nov 2021 06:56:39 +0000 Received: by outflank-mailman (input) for mailman id 222088; Fri, 05 Nov 2021 06:56:37 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9R-0008NQ-3q for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:37 +0000 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [2a00:1450:4864:20::536]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 849e8569-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:36 +0100 (CET) Received: by mail-ed1-x536.google.com with SMTP id g14so29235879edz.2 for ; Thu, 04 Nov 2021 23:56:36 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:35 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 849e8569-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ndViLhJiu5nJA5llxtFbdf7nx2zoDFBKma1YLn+Pglc=; b=O8CfU7uVVxraZ90rmP/W7ldSpEaLtcuqJl04xFonUzrW4ZHU+AieeyGl4xo8ymCEE2 F1yrTAkkKUGDv0srM0HLcueKvmd8iQMSZDxUnoY4SloB+2xXg4y06Okekq07O2rrWgSM w3BM1tXvBhJ7mmCBABL7jmR5FSxICrh5YweHft2v8ekLuB5aXstiGGYlN5/hpxePp72j gePT1kfSPwV9VluNAs2+iIYlVcC3b8gIV1f5ANzq9Ok7TfBNZwfpiihepf0KOzRnZgC8 iBAk9mgwqltzMIuz0nOyCdU+xsh30OYi9a/Jw+/d2WkSSthQfRd7YsZLv4mIhkrnDxKx U0Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ndViLhJiu5nJA5llxtFbdf7nx2zoDFBKma1YLn+Pglc=; b=cokzhkZ2VR2XLCc351NSnC+0fIOVhcYSmL/6qbMXHk/43eet7fjN6FJRmRTpgyJ5L9 fmTz+rYuelhu1KCW/GPrA6hyuu1Qcstih0Y6+NB189S//FiB7v1bT4VN9/RCOymMfNjN 10Rh2QDr5UQ6NOMv1q8GWMtg/BL8jjUaF2mpXxbQTBStP2GMs3QHylsvVAePNcSe8h1M dNGAiKarXCD+5muTj3snnqBTpjlmvxmYCXNS+/Xe82kEwABzBqvTKUqKPVdwAOChRjRI N0WgH7mCHSnpr6bxHpCePFgusgm5CnUn12v3+W+5a6rx4U8pf2oxrVwgdA2mJ9EhAfXj 2Ecw== X-Gm-Message-State: AOAM533zPdk2zXzsfIsQUJcB+f+a7CBdPsng3rnenZt0LhTgtCl6am6v CnzxmiDeR2cnq/9ZBKFaP+4yYue2Udi9Gw== X-Google-Smtp-Source: ABdhPJwvgUws2STx5uDR0C5x7eXgFsEjsvQ41PddklT8xFMzxt05VZNafwWyw+SPkIGRrx9enPTVmg== X-Received: by 2002:a17:906:270e:: with SMTP id z14mr69685944ejc.414.1636095395773; Thu, 04 Nov 2021 23:56:35 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 03/11] vpci: make vpci registers removal a dedicated function Date: Fri, 5 Nov 2021 08:56:21 +0200 Message-Id: <20211105065629.940943-4-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko This is in preparation for dynamic assignment of the vpci register handlers depending on the domain: hwdom or guest. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - remove all R-b's due to changes - s/vpci_remove_device_registers/vpci_remove_device_handlers - minor comment cleanup Since v1: - constify struct pci_dev where possible --- xen/drivers/vpci/vpci.c | 6 +++++- xen/include/xen/vpci.h | 2 ++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 4e24956419aa..d7f033a0811f 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -35,7 +35,7 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) -void vpci_remove_device(struct pci_dev *pdev) +void vpci_remove_device_handlers(const struct pci_dev *pdev) { if ( !has_vpci(pdev->domain) ) return; @@ -51,8 +51,12 @@ void vpci_remove_device(struct pci_dev *pdev) xfree(r); } spin_unlock(&pdev->vpci->lock); +} +void vpci_remove_device(struct pci_dev *pdev) +{ vpci_cancel_pending(pdev); + vpci_remove_device_handlers(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 609d6383b252..1883b9d08a70 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -30,6 +30,8 @@ int __must_check vpci_add_handlers(struct pci_dev *dev); /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); +/* Remove all handlers for the device. */ +void vpci_remove_device_handlers(const struct pci_dev *pdev); /* Add/remove a register handler. */ int __must_check vpci_add_register(struct vpci *vpci, From patchwork Fri Nov 5 06:56:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88603C4321E for ; Fri, 5 Nov 2021 06:56:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 394366124F for ; Fri, 5 Nov 2021 06:56:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 394366124F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222089.384143 (Exim 4.92) (envelope-from ) id 1mit9T-0000e5-JN; Fri, 05 Nov 2021 06:56:39 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222089.384143; Fri, 05 Nov 2021 06:56:39 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9T-0000da-CM; Fri, 05 Nov 2021 06:56:39 +0000 Received: by outflank-mailman (input) for mailman id 222089; Fri, 05 Nov 2021 06:56:38 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9S-0008Eb-Ei for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:38 +0000 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [2a00:1450:4864:20::529]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 855f26ec-3e05-11ec-9787-a32c541c8605; Fri, 05 Nov 2021 07:56:37 +0100 (CET) Received: by mail-ed1-x529.google.com with SMTP id b15so10471843edd.7 for ; Thu, 04 Nov 2021 23:56:37 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:36 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 855f26ec-3e05-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/BhVSzCyfrKMDozwQH/ExTubI3Dx6s3goXHMqPKDfjg=; b=K7U9vUsEU4wDAlfzmWQaQ3shYxNk5c+bFa7pzSjVc9szODwmj997M5/1FpSgD0eaVK 7WplqIySozk5AxYy4oMhaEPLGSxfkN7celQjwmaMeHS+9al9oeFLxn+hEpRWIJa0eRol dlHYax4HBRFmhwYC3bpinjWlU6KifLIXjkHPNDZCfYl72u7mFp94E98wGor0WhXKYgxH qQEwipLYMJzmuI3UVKAce6WXBmnz/4GF2u00pzkS12WokvNCrlTiduespTGbsV7HPQ7H nOb5v5gbIG0bG+S8nGlsSF0o31vuHYLCu5UMBL/Az9um0vy3VGFtqOD3W4GYs+hpk3ur x7Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/BhVSzCyfrKMDozwQH/ExTubI3Dx6s3goXHMqPKDfjg=; b=SQq3uEdo7fmz7TJxivUoW8bVukEoAv/vH6ieZUZdIei13N6BKsniNPyQqpHaKUsyP4 FmRz2rIWhX35FFw7Ls4I6HH0rrXGTTNg2h/gZmkHJw+zt0TIeoQnh+RtwPZV44Ziiy9P 6P3HCJZMgaFm1ncT/4p+5KGQwcyvm5OwBZA2Ck00WhmpEgLNct78YxnUU1p6t/hMezf7 j7MPwReGMLFEALfrm267j/Zs/DKargxbOWTl5Niz5ligCl9J873xbj4nLOMSJueTnXHP 6ePn3jvdR5Hfatc4KWQhIBMRSHAHJWy0k4CJ/WAtOCDr0UKgGypIErhVrYWLmuNZhlg7 poFA== X-Gm-Message-State: AOAM532sxEcSLL6kVmZhBTXlGBFYtaxadXAx7lOfnr1UFSiknYSKZyYa 54rCsZIYsfk0ZWj6a5B71cXEAVuWhbaraQ== X-Google-Smtp-Source: ABdhPJwH3nj2qgzWQi6uwb8KYRy2nn02Sm8YsU5o9cvWZgY9/c+lRZTPHGNkKX53KMbOHn9w8eTdEg== X-Received: by 2002:a17:906:3408:: with SMTP id c8mr14550874ejb.41.1636095397004; Thu, 04 Nov 2021 23:56:37 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 04/11] vpci: add hooks for PCI device assign/de-assign Date: Fri, 5 Nov 2021 08:56:22 +0200 Message-Id: <20211105065629.940943-5-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - remove toolstack roll-back description from the commit message as error are to be handled with proper cleanup in Xen itself - remove __must_check - remove redundant rc check while assigning devices - fix redundant CONFIG_HAS_VPCI check for CONFIG_HAS_VPCI_GUEST_SUPPORT - use REGISTER_VPCI_INIT machinery to run required steps on device init/assign: add run_vpci_init helper Since v2: - define CONFIG_HAS_VPCI_GUEST_SUPPORT so dead code is not compiled for x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/Kconfig | 4 +++ xen/drivers/passthrough/pci.c | 6 ++++ xen/drivers/vpci/vpci.c | 57 ++++++++++++++++++++++++++++++----- xen/include/xen/vpci.h | 16 ++++++++++ 4 files changed, 75 insertions(+), 8 deletions(-) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47a6..780490cf8e39 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_VPCI + endmenu diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index a9d31293ac09..529a4f50aa80 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -873,6 +873,10 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + ret = vpci_deassign_device(d, pdev); + if ( ret ) + goto out; + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1445,6 +1449,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) rc = hd->platform_ops->assign_device(d, devfn, pci_to_dev(pdev), flag); } + rc = vpci_assign_device(d, pdev); + done: if ( rc ) printk(XENLOG_G_WARNING "%pd: assign (%pp) failed (%d)\n", diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index d7f033a0811f..5f086398a98c 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -63,11 +63,25 @@ void vpci_remove_device(struct pci_dev *pdev) pdev->vpci = NULL; } -int vpci_add_handlers(struct pci_dev *pdev) +static int run_vpci_init(struct pci_dev *pdev) { unsigned int i; int rc = 0; + for ( i = 0; i < NUM_VPCI_INIT; i++ ) + { + rc = __start_vpci_array[i](pdev); + if ( rc ) + break; + } + + return rc; +} + +int vpci_add_handlers(struct pci_dev *pdev) +{ + int rc; + if ( !has_vpci(pdev->domain) ) return 0; @@ -81,18 +95,45 @@ int vpci_add_handlers(struct pci_dev *pdev) INIT_LIST_HEAD(&pdev->vpci->handlers); spin_lock_init(&pdev->vpci->lock); - for ( i = 0; i < NUM_VPCI_INIT; i++ ) - { - rc = __start_vpci_array[i](pdev); - if ( rc ) - break; - } - + rc = run_vpci_init(pdev); if ( rc ) vpci_remove_device(pdev); return rc; } + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct domain *d, struct pci_dev *pdev) +{ + int rc; + + /* It only makes sense to assign for hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + vpci_remove_device_handlers(pdev); + + rc = run_vpci_init(pdev); + if ( rc ) + vpci_deassign_device(d, pdev); + + return rc; +} + +/* Notify vPCI that device is de-assigned from guest. */ +int vpci_deassign_device(struct domain *d, struct pci_dev *pdev) +{ + /* It only makes sense to de-assign from hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + vpci_remove_device_handlers(pdev); + + return 0; +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 1883b9d08a70..a016b4197801 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -262,6 +262,22 @@ static inline void vpci_cancel_pending(const struct pci_dev *pdev) } #endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int vpci_assign_device(struct domain *d, struct pci_dev *pdev); +int vpci_deassign_device(struct domain *d, struct pci_dev *pdev); +#else +static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) +{ + return 0; +}; + +static inline int vpci_deassign_device(struct domain *d, struct pci_dev *pdev) +{ + return 0; +}; +#endif + #endif /* From patchwork Fri Nov 5 06:56:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3493AC433EF for ; Fri, 5 Nov 2021 06:56:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D6AF461252 for ; Fri, 5 Nov 2021 06:56:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D6AF461252 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222090.384161 (Exim 4.92) (envelope-from ) id 1mit9V-00018V-1v; Fri, 05 Nov 2021 06:56:41 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222090.384161; Fri, 05 Nov 2021 06:56:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9U-00018H-SN; Fri, 05 Nov 2021 06:56:40 +0000 Received: by outflank-mailman (input) for mailman id 222090; Fri, 05 Nov 2021 06:56:39 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9T-0008Eb-Mr for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:39 +0000 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [2a00:1450:4864:20::534]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 861e9070-3e05-11ec-9787-a32c541c8605; Fri, 05 Nov 2021 07:56:38 +0100 (CET) Received: by mail-ed1-x534.google.com with SMTP id f4so29495659edx.12 for ; Thu, 04 Nov 2021 23:56:38 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:37 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 861e9070-3e05-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ML1aAu3IHztXCkFRlcAEFUek/kmwkbVwdWAMwIa7IiY=; b=B+kLXAGSoP5K67q+UjBfKNTD07Hi8tXBKZLuFqxX1ybcLGn6BehgY/xDJFrLTcoL/P 2KTYZxueyCl84lqK9QCrmfVdk3xpuuNI5eQXRzEA9wjKSZKSd8K2jtyEf6k6qiB7yxCJ s6LRoVrhDoG9Ro1fEEaw8EvoPdbSVouXPqeT13NrKyyMibrHi2y7xgxi23UWK3elpIfu zZVJI3Yrl5M7E4D/DczQ8NxtOx6U+iMrgOGq+9TJiSGgRe/nN9Ir+XH/EhVO8jrJFpT1 7op+WMUShlFkxkQUnYlDzuzQyG7Yo+fVBW76d3tXmiImIkGYmMTWEtFPcdGbOM01AhWH iXhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ML1aAu3IHztXCkFRlcAEFUek/kmwkbVwdWAMwIa7IiY=; b=WKMa7fmMdYIGPeWdjEzMa6pmHm8/mswW16zy0o9zMogtw76+hBhCKlfHvjIt1dZTvE VelJL1KPJXwSMT8FZ1+Q8GzqJFeyIH+lgfOjgi9PV+IFPpL7WO78SH/KLweaeddvUGqx idOxgXwndzDgXikB1DEhFFes1uvD1Azzdb1wRSUFj6xuOZc5hifbYKkcZ5k/uJ5V6GKm 2RdViZzRDlsOSe/dmLFq7DOV4jLH6b+SN87ZxGH9++yzoeLDEyBo2Rn6u4bwE3busSRG beUp1OaQJj2gA5LdUV6yTNjg2JwgZAVidJeW0K++jzLBgNrA57ilfw6lLiGgvlugS2vf oQpQ== X-Gm-Message-State: AOAM532rBehjrUnER4975Muz97aRjxy7Kort4MQXgqrQVrhvUrqVLIYK v2o8R2Ox4BJvOaewl7xm9aJIzlOqMHX8cw== X-Google-Smtp-Source: ABdhPJxj//nL6Afs0JcLrsOFV1xEzm8zxoqIz8BOeE7e8dGiIwXDkh0VQ5NF5uzvT6GZpOM6XJun2g== X-Received: by 2002:a17:907:6e16:: with SMTP id sd22mr55367485ejc.542.1636095398279; Thu, 04 Nov 2021 23:56:38 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 05/11] vpci/header: implement guest BAR register handlers Date: Fri, 5 Nov 2021 08:56:23 +0200 Message-Id: <20211105065629.940943-6-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. ROM BAR is only handled for the hardware domain and for guest domains there is a stub: at the moment PCI expansion ROM is x86 only, so it might not be used by other architectures without emulating x86. Other use-cases may include using that expansion ROM before Xen boots, hence no emulation is needed in Xen itself. Or when a guest wants to use the ROM code which seems to be rare. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - squashed two patches: dynamic add/remove handlers and guest BAR handler implementation - fix guest BAR read of the high part of a 64bit BAR (Roger) - add error handling to vpci_assign_device - s/dom%pd/%pd - blank line before return Since v2: - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code has been eliminated from being built on x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment --- xen/drivers/vpci/header.c | 72 +++++++++++++++++++++++++++++++++++---- xen/include/xen/vpci.h | 3 ++ 2 files changed, 69 insertions(+), 6 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index ef538386e95d..1239051ee8ff 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -408,6 +408,48 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write32(pdev->sbdf, reg, val); } +static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ + struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + bar->guest_addr &= ~(0xffffffffull << (hi ? 32 : 0)); + bar->guest_addr |= (uint64_t)val << (hi ? 32 : 0); + + bar->guest_addr &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; +} + +static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + const struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + + return bar->guest_addr >> (hi ? 32 : 0); +} + static void rom_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -456,6 +498,17 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static void guest_rom_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ +} + +static uint32_t guest_rom_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + return 0xffffffff; +} + static int init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -464,6 +517,7 @@ static int init_bars(struct pci_dev *pdev) struct vpci_header *header = &pdev->vpci->header; struct vpci_bar *bars = header->bars; int rc; + bool is_hwdom = is_hardware_domain(pdev->domain); switch ( pci_conf_read8(pdev->sbdf, PCI_HEADER_TYPE) & 0x7f ) { @@ -503,8 +557,10 @@ static int init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -544,8 +600,10 @@ static int init_bars(struct pci_dev *pdev) bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -565,8 +623,10 @@ static int init_bars(struct pci_dev *pdev) header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & PCI_ROM_ADDRESS_ENABLE; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_rom_read, + is_hwdom ? rom_write : guest_rom_write, + rom_reg, 4, rom); if ( rc ) rom->type = VPCI_BAR_EMPTY; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index a016b4197801..3e7428da822c 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -70,7 +70,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical view of the BAR. */ uint64_t addr; + /* Guest view of the BAR. */ + uint64_t guest_addr; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Fri Nov 5 06:56:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD97FC433F5 for ; Fri, 5 Nov 2021 06:57:01 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 911C761279 for ; Fri, 5 Nov 2021 06:57:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 911C761279 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222091.384172 (Exim 4.92) (envelope-from ) id 1mit9W-0001RQ-IV; Fri, 05 Nov 2021 06:56:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222091.384172; Fri, 05 Nov 2021 06:56:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9W-0001R3-Ax; Fri, 05 Nov 2021 06:56:42 +0000 Received: by outflank-mailman (input) for mailman id 222091; Fri, 05 Nov 2021 06:56:41 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9V-0008Eb-9o for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:41 +0000 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [2a00:1450:4864:20::529]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 86e6829e-3e05-11ec-9787-a32c541c8605; Fri, 05 Nov 2021 07:56:40 +0100 (CET) Received: by mail-ed1-x529.google.com with SMTP id g14so29236259edz.2 for ; Thu, 04 Nov 2021 23:56:40 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:39 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 86e6829e-3e05-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ov4OeG0y0S9wF384aoMSS8nrG9C7KacWnjexXbSAJKA=; b=Ygg8Kw2PgYYWbUaPHVHr3ncCSlDkgVKitIY41ydr4W+kINTYeSzHNrli5PH5igfBJf 5KVg9L165M5E/gh2/9bgNC36oXtFOvRGx3cqiBCs3wiPvjVHKNA7zvt5y9kobuyNkawp s4GEGfD06lo+L5iqn8R6Dq85OFPEiXTbx8la/0M7+GtQ6loojNKAye8xb/uDZqVOLm9A vVLHZGGJBP4tsn9OVj73P9YyrWxSjfpwtOgTfc0Cp9b2325M1Fu+/08hztAJuh4D5n84 2KCSrX9+1wIu8YgMfzPG69sLy6aABux0EgwPN2Tlna/DhsNwP5bm3xOs/r+KdQPJfbIJ hleA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ov4OeG0y0S9wF384aoMSS8nrG9C7KacWnjexXbSAJKA=; b=1O+wZq8mEneiR4sITWrmGqz8+eo/ZMwJOXl3X/Pxptf2YAH+gATF1CP7aSYep4rvJA v14DSkIKGMggov8RwpTXgmkFRBiNH2Ao3M/fkRpthoAZtFV6Z1KWggQFEQtpIlXvhNFR CyEOqtkESmcm9ZgOsN3VthDPlcNK5AcEWSaX3Yaqcbkfhsi8Hv+1c1OenslamD/ifrTH uF0IbfQ2i1kkwGoHT2/MKNoHYKO8zoAR4ZfvI6HjunIcOSQFm8bhLYbxWm/BGskhHXXF 7JPHXcg3qgkHb29S5MW/0AnChnmZReToSZUllqXDDF15JGfdlka48D67QqW76Cxhl+GO AFVw== X-Gm-Message-State: AOAM532IM9bl5+zV26TVpWskzNTee1jTS9I8AbgeBCbp0e2K5MHobXGz F6/XHYMkp6v9p0LXu+PoMGgP48Rn2VBZ5Q== X-Google-Smtp-Source: ABdhPJzwW/asJb0bvmcPKQN0yvkDuDBQ2D84ldeYdNb020GJuam20MKFJNTL5jXaXCWobIQlP0aZZQ== X-Received: by 2002:a17:906:2887:: with SMTP id o7mr67588980ejd.425.1636095399489; Thu, 04 Nov 2021 23:56:39 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 06/11] vpci/header: handle p2m range sets per BAR Date: Fri, 5 Nov 2021 08:56:24 +0200 Message-Id: <20211105065629.940943-7-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. This is in preparation of making non-identity mappings in p2m for the MMIOs/ROM. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - re-work vpci_cancel_pending accordingly to the per-BAR handling - s/num_mem_ranges/map_pending and s/uint8_t/bool - ASSERT(bar->mem) in modify_bars - create and destroy the rangesets on add/remove --- xen/drivers/vpci/header.c | 178 ++++++++++++++++++++++++++------------ xen/drivers/vpci/vpci.c | 26 +++++- xen/include/xen/vpci.h | 3 +- 3 files changed, 150 insertions(+), 57 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 1239051ee8ff..5fc2dfbbc864 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -131,34 +131,50 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + if ( v->vpci.map_pending ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); + struct pci_dev *pdev = v->vpci.pdev; + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; - if ( rc == -ERESTART ) - return true; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + int rc; - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); + if ( rangeset_is_empty(bar->mem) ) + continue; - vpci_cancel_pending(v->vpci.pdev); - if ( rc ) - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. - */ - vpci_remove_device(v->vpci.pdev); + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + return true; + + spin_lock(&pdev->vpci->lock); + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(pdev, + rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, + !rc && v->vpci.rom_only); + spin_unlock(&pdev->vpci->lock); + + if ( rc ) + { + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this is + * safe for Dom0, for DomUs the domain will likely need to be + * killed in order to avoid leaking stale p2m mappings on + * failure. + */ + vpci_remove_device(pdev); + break; + } + } + v->vpci.map_pending = false; } return false; @@ -169,22 +185,48 @@ void vpci_cancel_pending(const struct pci_dev *pdev) struct vcpu *v = current; /* Cancel any pending work now. */ - if ( v->vpci.mem && v->vpci.pdev == pdev) + if ( v->vpci.map_pending && v->vpci.pdev == pdev) { - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + int rc; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, 0, ~0ULL); + if ( !rc ) + printk(XENLOG_ERR + "%pd %pp failed to remove range set for BAR: %d\n", + v->domain, &pdev->sbdf, rc); + } + v->vpci.map_pending = false; } } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + if ( rangeset_is_empty(bar->mem) ) + continue; - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) - process_pending_softirqs(); - rangeset_destroy(mem); + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + process_pending_softirqs(); + } if ( !rc ) modify_decoding(pdev, cmd, false); @@ -192,7 +234,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only) { struct vcpu *curr = current; @@ -203,9 +245,9 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; + curr->vpci.map_pending = true; /* * Raise a scheduler softirq in order to prevent the guest from resuming * execution with pending mapping operations, to trigger the invocation @@ -217,42 +259,40 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; - - if ( !mem ) - return -ENOMEM; + bool map_pending; /* - * Create a rangeset that represents the current device BARs memory region + * Create a rangeset per BAR that represents the current device memory region * and compare it against all the currently active BAR memory regions. If * an overlap is found, subtract it from the region to be mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with all the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + ASSERT(bar->mem); + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) ) continue; - rc = rangeset_add_range(mem, start, end); + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } @@ -263,14 +303,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + goto fail; + } } } @@ -302,7 +349,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || + if ( !bar->enabled || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against other * BARs in the same device for overlaps, but not against the @@ -311,13 +359,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } } @@ -335,12 +382,35 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + map_pending = false; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + if ( !rangeset_is_empty(header->bars[i].mem) ) + { + map_pending = true; + break; + } + + /* + * There are cases when PCI device, root port for example, has neither + * memory space nor IO. In this case PCI command register write is + * missed resulting in the underlying PCI device not functional, so: + * - if there are no regions write the command register now + * - if there are regions then defer work and write later on + */ + if ( !map_pending ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only); return 0; + +fail: + vpci_cancel_pending(pdev); + return rc; } static void cmd_write(const struct pci_dev *pdev, unsigned int reg, diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 5f086398a98c..45733300f00b 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -55,7 +55,12 @@ void vpci_remove_device_handlers(const struct pci_dev *pdev) void vpci_remove_device(struct pci_dev *pdev) { + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + vpci_cancel_pending(pdev); + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + rangeset_destroy(header->bars[i].mem); vpci_remove_device_handlers(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); @@ -80,6 +85,8 @@ static int run_vpci_init(struct pci_dev *pdev) int vpci_add_handlers(struct pci_dev *pdev) { + struct vpci_header *header; + unsigned int i; int rc; if ( !has_vpci(pdev->domain) ) @@ -95,10 +102,25 @@ int vpci_add_handlers(struct pci_dev *pdev) INIT_LIST_HEAD(&pdev->vpci->handlers); spin_lock_init(&pdev->vpci->lock); + header = &pdev->vpci->header; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + bar->mem = rangeset_new(NULL, NULL, 0); + if ( !bar->mem ) + { + rc = -ENOMEM; + goto fail; + } + } + rc = run_vpci_init(pdev); - if ( rc ) - vpci_remove_device(pdev); + if ( !rc ) + return 0; + fail: + vpci_remove_device(pdev); return rc; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 3e7428da822c..143f3166a730 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -75,6 +75,7 @@ struct vpci { /* Guest view of the BAR. */ uint64_t guest_addr; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -149,9 +150,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + bool map_pending : 1; bool rom_only : 1; }; From patchwork Fri Nov 5 06:56:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97C86C433FE for ; Fri, 5 Nov 2021 06:57:01 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 508316124F for ; Fri, 5 Nov 2021 06:57:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 508316124F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222092.384183 (Exim 4.92) (envelope-from ) id 1mit9Y-0001l3-1L; Fri, 05 Nov 2021 06:56:44 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222092.384183; Fri, 05 Nov 2021 06:56:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9X-0001jh-Oh; Fri, 05 Nov 2021 06:56:43 +0000 Received: by outflank-mailman (input) for mailman id 222092; Fri, 05 Nov 2021 06:56:42 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9W-0008NQ-0U for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:42 +0000 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [2a00:1450:4864:20::533]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 87859bdc-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:41 +0100 (CET) Received: by mail-ed1-x533.google.com with SMTP id f8so29948103edy.4 for ; Thu, 04 Nov 2021 23:56:41 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:40 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 87859bdc-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XyZSMZ7iKKLkWA3FJonAF/JNUpQ9LSX1yaRvNFIyvMA=; b=paJTQbNmX2422YdQxWAkilGdnCnfHcrX2/PzNL261GPmAbwsW/hW0kMVLU7BDmktI9 +JANFVQP1BD65WstErEcXciOyRkV+R1+rIcSIldz9iiZz9R1NqpO/uDu7NI80dpdkYTW kQ4y8h1qZ5/OeU6Sf8vPNnkB+orjbVrz53tjMfsRJz2MJJ6SFSVYtyV3I56I7Su5ANvx j2UqBdk0N++TAeevhZHkrGmZ93CtMT4iINrgTFs4mdGojt4F7ZCvfJ9eFIq1i7Llo6xl ebr68ogUy0sBwWO7Om6TPtN/Nc1yl6KavKrpJBrEn4vwqZ6tpdqoCAoOxFPcv3NpziqY cXSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XyZSMZ7iKKLkWA3FJonAF/JNUpQ9LSX1yaRvNFIyvMA=; b=N5OvLfxvUmZ/dspgu4KNY3YjfyEQ5HEpIEuw1WMneZLsH6aKWb2zkysaYlMl1gjMzN xpB3LRzeUchVAHBXfHZsvnt/Z/phr6GZ7W13MY/m8mCjVfGezItZkM6icH0t8OicW6Ie TFyUthVuh/sTk+TR+tsfUFPDIfUgsGafEJIZup6IsQIdaSQ4+MKZ9X83we4cISeb1Pyi flJtGuW7UQacgcNfqYAQahadkkO2ShzEMjtXIgwLgYindW5CvgMl8iZqZMIrkaMAM9pj e8hObMTym9gHXhJApZgeM7ee/divEWitMRP7j3ZCigd/tabRbp0DtWWktFlCEVH4g+sE A7Cg== X-Gm-Message-State: AOAM532d6KOXxM1zl1RVPRfvMmcaNl+B8Iuo/P2iNDndU67bLRW18Wc4 iuiMw/QeoKzYQ1wEeomg7Ea1w8kC92o7sQ== X-Google-Smtp-Source: ABdhPJw6edY8YQ99u9eqfsvwlkGtZAjVgFNAwPaaqGlEbF+LcSDWS6OORU0obPFktGIYzCLWO6v6Bw== X-Received: by 2002:a17:907:8a1b:: with SMTP id sc27mr35150537ejc.572.1636095400630; Thu, 04 Nov 2021 23:56:40 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 07/11] vpci/header: program p2m with guest BAR view Date: Fri, 5 Nov 2021 08:56:25 +0200 Message-Id: <20211105065629.940943-8-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the host bridge in the hardware domain. This way hardware doamin sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - updated comment (Roger) - removed gfn_add(map->start_gfn, rc); which is wrong - use v->domain instead of v->vpci.pdev->domain - removed odd e.g. in comment - s/d%d/%pd in altered code - use gdprintk for map/unmap logs Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 5fc2dfbbc864..34158da2d5f6 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,10 @@ struct map_data { struct domain *d; + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn; + /* Physical start address of the BAR. */ + mfn_t start_mfn; bool map; }; @@ -37,12 +41,24 @@ static int map_range(unsigned long s, unsigned long e, void *data, unsigned long *c) { const struct map_data *map = data; + gfn_t start_gfn; int rc; for ( ; ; ) { unsigned long size = e - s + 1; + /* + * Ranges to be mapped don't always start at the BAR start address, as + * there can be holes or partially consumed ranges. Account for the + * offset of the current address from the BAR start. + */ + start_gfn = gfn_add(map->start_gfn, s - mfn_x(map->start_mfn)); + + gdprintk(XENLOG_G_DEBUG, + "%smap [%lx, %lx] -> %#"PRI_gfn" for %pd\n", + map->map ? "" : "un", s, e, gfn_x(start_gfn), + map->d); /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +68,10 @@ static int map_range(unsigned long s, unsigned long e, void *data, * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, + size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, + size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -62,8 +80,8 @@ static int map_range(unsigned long s, unsigned long e, void *data, if ( rc < 0 ) { printk(XENLOG_G_WARNING - "Failed to identity %smap [%lx, %lx] for d%d: %d\n", - map->map ? "" : "un", s, e, map->d->domain_id, rc); + "Failed to identity %smap [%lx, %lx] for %pd: %d\n", + map->map ? "" : "un", s, e, map->d, rc); break; } ASSERT(rc < size); @@ -149,6 +167,10 @@ bool vpci_process_pending(struct vcpu *v) if ( rangeset_is_empty(bar->mem) ) continue; + data.start_gfn = + _gfn(PFN_DOWN(is_hardware_domain(v->domain) + ? bar->addr : bar->guest_addr)); + data.start_mfn = _mfn(PFN_DOWN(bar->addr)); rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -223,6 +245,9 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( rangeset_is_empty(bar->mem) ) continue; + data.start_gfn = _gfn(PFN_DOWN(is_hardware_domain(d) + ? bar->addr : bar->guest_addr)); + data.start_mfn = _mfn(PFN_DOWN(bar->addr)); while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) process_pending_softirqs(); From patchwork Fri Nov 5 06:56:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E635C4332F for ; Fri, 5 Nov 2021 06:57:02 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4BCEF61252 for ; Fri, 5 Nov 2021 06:57:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4BCEF61252 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222093.384187 (Exim 4.92) (envelope-from ) id 1mit9Y-0001uY-Sf; Fri, 05 Nov 2021 06:56:44 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222093.384187; Fri, 05 Nov 2021 06:56:44 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9Y-0001rV-J9; Fri, 05 Nov 2021 06:56:44 +0000 Received: by outflank-mailman (input) for mailman id 222093; Fri, 05 Nov 2021 06:56:43 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9X-0008NQ-0W for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:43 +0000 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [2a00:1450:4864:20::536]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 882d0480-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:42 +0100 (CET) Received: by mail-ed1-x536.google.com with SMTP id g14so29236504edz.2 for ; Thu, 04 Nov 2021 23:56:42 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:41 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 882d0480-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pQYQjf/W6J0SupymAgyjXFbgcoTAynkHvQvG8kaNKtE=; b=REKx4tJPrYDsJUNCJKpj14ih7omKhQQe5cRTJOAGrnflNXA0QYA6mXstHT5V0pVFf6 gNsdSf977poOoLNRmfBcW3sgmOZfOSLP0RSbsm0LCdFFSvfQYHnMDE3bUxk8m793XkIC /9K17kLCe9oGpNBb2DuvKwFkdh+gBs5zkdJx0zPY8ZVwqkL0ogMJob4pqdbPDiAVWOpc xRe7z9xHKBwMYi6LwPnCz6byiN8MmQR0wpZl9ZhyG6kNRYEFrBuUxdql6TGnv2ZD4+Y1 NP52JvIugUJUToEgMWtL5Ubo+ViZQLrWdfSd2QzrpbWzLU7/d+HxQ9I7lJfjPuONycLB ispQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pQYQjf/W6J0SupymAgyjXFbgcoTAynkHvQvG8kaNKtE=; b=1uwUcklCiIJFLSLhTgUMJ4jYQGoRN2o2TIyoFzMh+G3HEBjZMLTF4saD939ycfRMmo cIFWkgDdMhhmqWH2/I3VvODN+lMDaqxt44OxhLjQTWixhqBurJvotWjUs6Bf1RWF552c H/alaZPbCE2d9B3Vr7jG1Ze53VsN86WGfJu2JUj4AbDTqifqcTdlOMERHAYPy3NgIR93 F7XNGH832Omo4TPeBBt3dV0WWCOrpy7dXSmtDFsm271dPG0+roaFXvkEJH00Rt/r1cOB g/UtzEycbRa99K0OwzepFadmP/2ilMY5dHhWv7kJJprM5PPpGYyFY+1V5ZwLIdAln8QJ o/dw== X-Gm-Message-State: AOAM531beRvJ+f8EVpsuGorHvMJMof0K8agxqF6KEYs0KVSquF2wY8iB bpB2Qo3wNCaCfCXRhZaFCIiSZ0ucHLMHuA== X-Google-Smtp-Source: ABdhPJyfcQnz2s3OZb7zo0UDGNsehpeHVTWJcimzQRS/jBRT+UODYLMfQyENC816TfBAVXOfgiVs3Q== X-Received: by 2002:a17:907:d94:: with SMTP id go20mr8942339ejc.78.1636095401673; Thu, 04 Nov 2021 23:56:41 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 08/11] vpci/header: emulate PCI_COMMAND register for guests Date: Fri, 5 Nov 2021 08:56:26 +0200 Message-Id: <20211105065629.940943-9-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add basic emulation support for guests. At the moment only emulate PCI_COMMAND_INTX_DISABLE bit, the rest is not emulated yet and left as TODO. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - gate more code on CONFIG_HAS_MSI - removed logic for the case when MSI/MSI-X not enabled --- xen/drivers/vpci/header.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 34158da2d5f6..64cfc268c341 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -459,6 +459,22 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + /* TODO: Add proper emulation for all bits of the command register. */ + +#ifdef CONFIG_HAS_PCI_MSI + if ( pdev->vpci->msi->enabled ) + { + /* Guest wants to enable INTx. It can't be enabled if MSI/MSI-X enabled. */ + cmd |= PCI_COMMAND_INTX_DISABLE; + } +#endif + + cmd_write(pdev, reg, cmd, data); +} + static void bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -631,8 +647,9 @@ static int init_bars(struct pci_dev *pdev) } /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, + is_hwdom ? cmd_write : guest_cmd_write, + PCI_COMMAND, 2, header); if ( rc ) return rc; From patchwork Fri Nov 5 06:56:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D4FBC433EF for ; Fri, 5 Nov 2021 06:57:05 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1DD0360F70 for ; Fri, 5 Nov 2021 06:57:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1DD0360F70 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222094.384201 (Exim 4.92) (envelope-from ) id 1mit9a-0002Kt-Cb; Fri, 05 Nov 2021 06:56:46 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222094.384201; Fri, 05 Nov 2021 06:56:46 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9a-0002K5-6n; Fri, 05 Nov 2021 06:56:46 +0000 Received: by outflank-mailman (input) for mailman id 222094; Fri, 05 Nov 2021 06:56:44 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9Y-0008NQ-49 for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:44 +0000 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [2a00:1450:4864:20::52e]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 88d4f31b-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:43 +0100 (CET) Received: by mail-ed1-x52e.google.com with SMTP id b15so10472327edd.7 for ; Thu, 04 Nov 2021 23:56:43 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:42 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 88d4f31b-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f+1sZF7ROamhznPmSSkwy1b94iIH1O5NvYMBmRXBAH8=; b=A2TDKmeWGL2QFwTzv8KDdpoYv2eJE6NeCgA7AVx1l6oYPcqGOyG28ee0T4YGHbwTmY 5Lxfgyn7gO6AjCLLcWNAVOgS4y34NOACvl0IGfGnW2uW9wCbAUtLH+XkO/7wCiI2sM3Q nO+XzRkBs0mq7Lv4TuegSkQACn7rOI3Odu61EnNiQi/AfEoXf5yR70Lyx+XuBU8GTPVq lzJpEwS7erjnemiMGTjBJvBgE+ixpjo/VS+l6ngpTp2bfvwwIuQAqrrzwVlru23wi1t6 aclcSAHLHG1ZVsqPHVIvPDqM/SiXoHYrlo4jDJdn+x/fsG5tz/SteTSsjH8ErUJMgah7 iXgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f+1sZF7ROamhznPmSSkwy1b94iIH1O5NvYMBmRXBAH8=; b=lI1sRInMZoxFufHmtTXXuhK9LSv9cKDmV/nfEjo9BuUqqqN0r/y45HObNmDYa67yWS Gcl849dWENP1gNPTvb4xC20D+AgsXwfOaJxkIgHtAEj6EAN1j1crE9SAXgV4RSlDtGni bdUVhRSICDsf3eA5tuGCaO7DzXrxZWmAL9/eCSJgjRykttNUv9VaV2NFKTtpp5jvKQlZ iWO+IG48V3KXShnJlq5t03nnofxJe8nvdQUt4GU0dFvNkYO3qoTJLVLM+Z0VxxFIyoDl xGK1gwnMsKi1rxica2a2Uf3M061Zf/F2nqiS5Aw7eBvV2kNaOaTrohhD4G8pT2s1obOm jdfw== X-Gm-Message-State: AOAM530GMkr6DgvllKNPUUmYKCosyBpVW7eoxqeUPRFTDt6VIZw2CU1L 3B3pFcRHChQJY95cq/NaPrN9yNbL4f9y1A== X-Google-Smtp-Source: ABdhPJwSvyYC+lehQ6glU0T9q9cBKvQJaw1rh4jciNd34Vt0gDz8IkOSukIGAv03A6mROto6h+7HKA== X-Received: by 2002:aa7:c1c6:: with SMTP id d6mr35817680edp.203.1636095402813; Thu, 04 Nov 2021 23:56:42 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 09/11] vpci/header: reset the command register when adding devices Date: Fri, 5 Nov 2021 08:56:27 +0200 Message-Id: <20211105065629.940943-10-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Reset the command register when passing through a PCI device: it is possible that when passing through a PCI device its memory decoding bits in the command register are already set. Thus, a guest OS may not write to the command register to update memory decoding, so guest mappings (guest's view of the BARs) are left not updated. Signed-off-by: Oleksandr Andrushchenko --- Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 64cfc268c341..680319b3a63f 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -459,8 +459,7 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } -static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, - uint32_t cmd, void *data) +static uint32_t emulate_cmd_reg(const struct pci_dev *pdev, uint32_t cmd) { /* TODO: Add proper emulation for all bits of the command register. */ @@ -472,7 +471,13 @@ static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, } #endif - cmd_write(pdev, reg, cmd, data); + return cmd; +} + +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + cmd_write(pdev, reg, emulate_cmd_reg(pdev, cmd), data); } static void bar_write(const struct pci_dev *pdev, unsigned int reg, @@ -646,6 +651,10 @@ static int init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } + /* Reset the command register for the guest. */ + if ( !is_hwdom ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, emulate_cmd_reg(pdev, 0)); + /* Setup a handler for the command register. */ rc = vpci_add_register(pdev->vpci, vpci_hw_read16, is_hwdom ? cmd_write : guest_cmd_write, From patchwork Fri Nov 5 06:56:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A201C433F5 for ; Fri, 5 Nov 2021 06:57:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 31F3B60F70 for ; Fri, 5 Nov 2021 06:57:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 31F3B60F70 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222095.384213 (Exim 4.92) (envelope-from ) id 1mit9c-0002i0-3f; Fri, 05 Nov 2021 06:56:48 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222095.384213; Fri, 05 Nov 2021 06:56:48 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9b-0002fW-Py; Fri, 05 Nov 2021 06:56:47 +0000 Received: by outflank-mailman (input) for mailman id 222095; Fri, 05 Nov 2021 06:56:45 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9Z-0008NQ-EU for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:45 +0000 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [2a00:1450:4864:20::535]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 898bd0fe-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:44 +0100 (CET) Received: by mail-ed1-x535.google.com with SMTP id ee33so29969942edb.8 for ; Thu, 04 Nov 2021 23:56:44 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:43 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 898bd0fe-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AyhDGXJChmWgqH4NGZkz5hh623ehbMnVZq0mFeTc4N8=; b=S74F3KC5Ll7YUiFRMNzJmPU1QEjH37KMs0pYrg9xJLZn2V9McTPuQuSvBOMyE6IBK+ ZXL6caNB4Kay83nmnMI3A1oFymf6Cw5M25klv8kmgHbY1ztmnBiqWxYUHj4POot147SE 4DqfjshSdkR/0W39QQzvMjANhQnkY9Iw31TugmdSyBwgTot+HT+Nw/z/EiECBR9TZCq3 URK3xEVJPcu3gw2TEOiRoc8j2YkbdCNfljotoetQ5Xk31cn4AMf8+sDwZSs7/gpjWYL4 vDo42kC/mXAW1BtQlM7rZ+yA8N2+4waNT34QDVF5laHsNflg4G7RrBXyH1UgRUxeHo2Q URNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AyhDGXJChmWgqH4NGZkz5hh623ehbMnVZq0mFeTc4N8=; b=km3EKa1tvfgetkE3+2nhpqBtV6QaWSl13f1Noi/Bxx7JiIgFRB8RzUu6ubn20sK1lG sZZNxUx88act7ON9JUsRkZvZn5+ajdK+MENWKKRgKpceVZblUF9frK7/qam4cwF99pC8 2HexSECQeWZ9YGWCjx2uZGoeB8HNn+kr+FKJIa7d5XbTzyEV68tsZTplfsYiLckQv1UB V7PjtkTE7lPZxyhhCxgEylr/YQMFR4afzq9utLgSphdtgGypQ4ysNxgYqBpwbO1steKQ gydJlJYzkbx2MK5i9ku77JUS8Sl4jn2io/Qb2ecnka7Mo5l1du+DWm0YBoMP0VPYVybg 3AfA== X-Gm-Message-State: AOAM530vTxRyKO3kM9+fWgdQiuVTP9RtVzsdY/Tjmh3JdrV7ZFReJRTc xb9fMmQwnu80XehqLnnHvVlBYMsXFcOVtA== X-Google-Smtp-Source: ABdhPJwEvro3SUyelH8U7WRbrV+bXktuw9rqoGQ+lFYh6UrNug2qOPwBNRl25+SrND7hV+wShj6HEg== X-Received: by 2002:a05:6402:520b:: with SMTP id s11mr37432396edd.213.1636095404026; Thu, 04 Nov 2021 23:56:44 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 10/11] vpci: add initial support for virtual PCI bus topology Date: Fri, 5 Nov 2021 08:56:28 +0200 Message-Id: <20211105065629.940943-11-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Please note, that at the moment only function 0 of a multifunction device can be passed through. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - make use of VPCI_INIT - moved all new code to vpci.c which belongs to it - changed open-coded 31 to PCI_SLOT(~0) - revisited locking: add dedicated vdev list's lock - added comments and code to reject multifunction devices with functions other than 0 - updated comment about vpci_dev_next and made it unsigned int - implement roll back in case of error while assigning/deassigning devices - s/dom%pd/%pd Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/drivers/vpci/vpci.c | 52 +++++++++++++++++++++++++++++++++++++++++ xen/include/xen/sched.h | 8 +++++++ xen/include/xen/vpci.h | 4 ++++ 3 files changed, 64 insertions(+) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 45733300f00b..6657d236dc1a 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -101,6 +101,9 @@ int vpci_add_handlers(struct pci_dev *pdev) INIT_LIST_HEAD(&pdev->vpci->handlers); spin_lock_init(&pdev->vpci->lock); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + pdev->vpci->guest_sbdf.sbdf = ~0; +#endif header = &pdev->vpci->header; for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) @@ -125,6 +128,54 @@ int vpci_add_handlers(struct pci_dev *pdev) } #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +int vpci_add_virtual_device(struct pci_dev *pdev) +{ + struct domain *d = pdev->domain; + pci_sbdf_t sbdf; + unsigned long new_dev_number; + + /* + * Each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ + if ( pdev->info.is_extfn ) + { + gdprintk(XENLOG_ERR, "%pp: only function 0 passthrough supported\n", + &pdev->sbdf); + return -EOPNOTSUPP; + } + + new_dev_number = find_first_zero_bit(&d->vpci_dev_assigned_map, + PCI_SLOT(~0) + 1); + if ( new_dev_number > PCI_SLOT(~0) ) + return -ENOSPC; + + set_bit(new_dev_number, &d->vpci_dev_assigned_map); + + /* + * Both segment and bus number are 0: + * - we emulate a single host bridge for the guest, e.g. segment 0 + * - with bus 0 the virtual devices are seen as embedded + * endpoints behind the root complex + * + * TODO: add support for multi-function devices. + */ + sbdf.sbdf = 0; + sbdf.devfn = PCI_DEVFN(new_dev_number, 0); + pdev->vpci->guest_sbdf = sbdf; + + return 0; + +} +REGISTER_VPCI_INIT(vpci_add_virtual_device, VPCI_PRIORITY_MIDDLE); + +static void vpci_remove_virtual_device(struct domain *d, + const struct pci_dev *pdev) +{ + clear_bit(pdev->vpci->guest_sbdf.dev, &d->vpci_dev_assigned_map); + pdev->vpci->guest_sbdf.sbdf = ~0; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { @@ -150,6 +201,7 @@ int vpci_deassign_device(struct domain *d, struct pci_dev *pdev) if ( is_system_domain(d) || !has_vpci(d) ) return 0; + vpci_remove_virtual_device(d, pdev); vpci_remove_device_handlers(pdev); return 0; diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 28146ee404e6..10bff103317c 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -444,6 +444,14 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * The bitmap which shows which device numbers are already used by the + * virtual PCI bus topology and is used to assign a unique SBDF to the + * next passed through virtual PCI device. + */ + unsigned long vpci_dev_assigned_map; +#endif #endif #ifdef CONFIG_HAS_PASSTHROUGH diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 143f3166a730..9cc7071bc0af 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -145,6 +145,10 @@ struct vpci { struct vpci_arch_msix_entry arch; } entries[]; } *msix; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* Virtual SBDF of the device. */ + pci_sbdf_t guest_sbdf; +#endif #endif }; From patchwork Fri Nov 5 06:56:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12604303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29472C433EF for ; Fri, 5 Nov 2021 06:57:07 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D774D60F70 for ; Fri, 5 Nov 2021 06:57:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D774D60F70 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.222096.384221 (Exim 4.92) (envelope-from ) id 1mit9d-0002rM-Jx; Fri, 05 Nov 2021 06:56:49 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 222096.384221; Fri, 05 Nov 2021 06:56:49 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9c-0002o7-Dt; Fri, 05 Nov 2021 06:56:48 +0000 Received: by outflank-mailman (input) for mailman id 222096; Fri, 05 Nov 2021 06:56:47 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mit9a-0008NQ-SQ for xen-devel@lists.xenproject.org; Fri, 05 Nov 2021 06:56:46 +0000 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [2a00:1450:4864:20::52b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 8a5cd3be-3e05-11ec-a9d2-d9f7a1cc8784; Fri, 05 Nov 2021 07:56:46 +0100 (CET) Received: by mail-ed1-x52b.google.com with SMTP id w1so30027716edd.10 for ; Thu, 04 Nov 2021 23:56:46 -0700 (PDT) Received: from localhost.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id dt4sm3735150ejb.27.2021.11.04.23.56.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Nov 2021 23:56:44 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 8a5cd3be-3e05-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ku75VZX/TQ1wxjSevU0hc8PtkdlgtlMhhIG2xgl6Ofc=; b=NDxX5Vm/k7LYA5JCIm0+QlVCsPrIHbjm785HHBMfst0ab/k3PkoKp6vV7UtSKJnK1I zmZt3fjOD4AsivSHuRVhUlEVeRyNBWJw8H76QEe8QoI5hViHDly4GU7XY5tQSZI95MJB cJN5Kv+OCyQ4+UkATNtLw89S7+aKTnt3W69te4SFDQD9nKjzf5xPMX4v1PddbMdOqhfc XaHX7XRc4oXNQP+2PQG6UXw4nx3knp52pVfSoMlEryfTy7Ef6oJpGZ8Igh9RvhyDA3Eu YQlK6E2e2NHAIOEBbVJr1M2nfL2nMAFYEZnlZF2ATBU1ewkl5VQqD3zuTnUU8VoY0lHg B46A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ku75VZX/TQ1wxjSevU0hc8PtkdlgtlMhhIG2xgl6Ofc=; b=vmCpnMlFnLTnIWOytwcpTTMvQv5vAVOqyvDbarPp8cMmOGWA9GIzzSySj56OXBLv+b 7q7JEO1imZdZj/F0tLUEyDqf4UFctcfKsgPNPQiAavzIGi8M1abQ8P2RuiskK3HVofzk KP6+CKMR0TR02LO/llutcA5WZ9eDll1DK3W4+xtG9b2u325VGvwuoEeyah6COtzJSpVM qGmw2oGdDNhVR3qomUZCoSuJtOyrm0Mjq71oqesl07hErHhVEVsPdecVCg85QDlzVs6S bGOk7+AqBJNxGrDWkpzpdnwtB6P1pT8KsHW0MYw5+cK0gZ8s0yapzzEHMM7VtAIc8qMF xjUA== X-Gm-Message-State: AOAM530fUTyEbyx+aZivXH5r7Z0Z927PF0vOuD2koum2yGJJWOsGvR3E CFsjF7cSbX9+YMUtGAOvY2Ke3ZH7/CbHdw== X-Google-Smtp-Source: ABdhPJznZ1Bt/7VgyWdSOs5D8rXKBDaD9RskvSN9Sr6eFAexPm+adsmBAZZCGxHmQQPmpDaN31lv+A== X-Received: by 2002:a05:6402:5190:: with SMTP id q16mr37435741edd.12.1636095405353; Thu, 04 Nov 2021 23:56:45 -0700 (PDT) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v4 11/11] xen/arm: translate virtual PCI bus topology for guests Date: Fri, 5 Nov 2021 08:56:29 +0200 Message-Id: <20211105065629.940943-12-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211105065629.940943-1-andr2000@gmail.com> References: <20211105065629.940943-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - revisit locking - move code to vpci.c Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/vpci.c | 18 ++++++++++++++++++ xen/drivers/vpci/vpci.c | 30 ++++++++++++++++++++++++++++++ xen/include/xen/vpci.h | 1 + 3 files changed, 49 insertions(+) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 5a6ebd8b9868..6a37f770f8f0 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -41,6 +41,15 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + return 1; +#endif + if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -59,6 +68,15 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, struct pci_host_bridge *bridge = p; pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + return 1; +#endif + return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); } diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 6657d236dc1a..cb0bde35b6a6 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -94,6 +94,7 @@ int vpci_add_handlers(struct pci_dev *pdev) /* We should not get here twice for the same device. */ ASSERT(!pdev->vpci); + ASSERT(pcidevs_locked()); pdev->vpci = xzalloc(struct vpci); if ( !pdev->vpci ) @@ -134,6 +135,8 @@ int vpci_add_virtual_device(struct pci_dev *pdev) pci_sbdf_t sbdf; unsigned long new_dev_number; + ASSERT(pcidevs_locked()); + /* * Each PCI bus supports 32 devices/slots at max or up to 256 when * there are multi-function ones which are not yet supported. @@ -172,10 +175,37 @@ REGISTER_VPCI_INIT(vpci_add_virtual_device, VPCI_PRIORITY_MIDDLE); static void vpci_remove_virtual_device(struct domain *d, const struct pci_dev *pdev) { + ASSERT(pcidevs_locked()); + clear_bit(pdev->vpci->guest_sbdf.dev, &d->vpci_dev_assigned_map); pdev->vpci->guest_sbdf.sbdf = ~0; } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool vpci_translate_virtual_device(struct domain *d, pci_sbdf_t *sbdf) +{ + const struct pci_dev *pdev; + bool found = false; + + pcidevs_lock(); + for_each_pdev( d, pdev ) + { + if ( pdev->vpci->guest_sbdf.sbdf == sbdf->sbdf ) + { + /* Replace virtual SBDF with the physical one. */ + *sbdf = pdev->sbdf; + found = true; + break; + } + } + pcidevs_unlock(); + + return found; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 9cc7071bc0af..d5765301e442 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -274,6 +274,7 @@ static inline void vpci_cancel_pending(const struct pci_dev *pdev) /* Notify vPCI that device is assigned/de-assigned to/from guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev); int vpci_deassign_device(struct domain *d, struct pci_dev *pdev); +bool vpci_translate_virtual_device(struct domain *d, pci_sbdf_t *sbdf); #else static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) {