From patchwork Fri Feb 4 06:34:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F36EFC433EF for ; Fri, 4 Feb 2022 06:35:25 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265005.458274 (Exim 4.92) (envelope-from ) id 1nFsBX-0008Ql-2D; Fri, 04 Feb 2022 06:35:07 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265005.458274; Fri, 04 Feb 2022 06:35:07 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBW-0008PB-Si; Fri, 04 Feb 2022 06:35:06 +0000 Received: by outflank-mailman (input) for mailman id 265005; Fri, 04 Feb 2022 06:35:05 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBV-0008DX-E1 for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:05 +0000 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [2a00:1450:4864:20::12b]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 95396f71-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:03 +0100 (CET) Received: by mail-lf1-x12b.google.com with SMTP id z4so10781282lft.3 for ; Thu, 03 Feb 2022 22:35:03 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:01 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 95396f71-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nEZ0S5ylp2hVxPb59duUWX5SgzLrq0RrV7CsGFnmgls=; b=ljHuqvmKZbx3pFGEj/9BQ+VGUO0+7SGsf8IYTF0oPk5K7zJA+Zr6P+5sPWUUM3wNcm uFX87jQNZi3n/maFzKdEA/l/k2sUOk4VpkAjR909ZkhY72pxTXP2Fyn9dWwaRJkMBUPl eOMzJLCi+HMPk+E6eEuqNzWx1aQ4UEu12T0cOs1IUIiywh3ZkboZbkDy5X62/60XjssH yFzcne58IqIbGWH6pXBQlFnMX7cbxGetb8lKBVjz4H39Rg0MYKcbag/jFj4OnVqEtZdV jf/JsKtTUOIEixmw3rh/gOqAoJREqDDSbHMwWxPDzYh173+NMq0yFXxf3fbLCROk3AnB QbCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nEZ0S5ylp2hVxPb59duUWX5SgzLrq0RrV7CsGFnmgls=; b=zOtFIaLgqCPJHjfoO6CUfXTuQm1xvwAdEPk9uf0qEU1QUnChXnWNYEmm0p+Cgnv6PA AXVwXYlJZEIJbm64tjNafNykRWDmniXtpst7ACOPpPhxZxTjZneCWITRMr7hh0OZuv9t sMDADbzkz5/TSWsYAhBYh8FhKssNP+fUDAXteHWFDYUn5PQWx4w7WhaMiwx8LNvExVyZ HJ7lrbWOO3/6m65PmYzIPHd4B+de8XGuQZ+UFTE8e+qkZx4hZiGxnJZLJwP4E9fBs6F1 eC9d6Khd+peZgkGpDY64JnpKA047tNliFly4agCSchw3/vx84Y2hA1zuhVM0LVQYcsRh pRrg== X-Gm-Message-State: AOAM533EpdA17k7ga4xjQ1m2zWQ/bqdSIClwC1kj47yo5HqN/w7bL2Qd QYb4C5tOFcdbyhjoQEAZ8MNxuWowOxo= X-Google-Smtp-Source: ABdhPJyb52ecDgdNTkD06qSr2ws0sAPXn71t/8qeE3T7ubzYIIgJyTlltqbfnCKWdc26bSW7BjWMnQ== X-Received: by 2002:a05:6512:2354:: with SMTP id p20mr1320036lfu.215.1643956502200; Thu, 03 Feb 2022 22:35:02 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 01/13] xen/pci: arm: add stub for is_memory_hole Date: Fri, 4 Feb 2022 08:34:47 +0200 Message-Id: <20220204063459.680961-2-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add a stub for is_memory_hole which is required for PCI passthrough on Arm. Signed-off-by: Oleksandr Andrushchenko --- Cc: Julien Grall Cc: Stefano Stabellini --- New in v6 --- xen/arch/arm/mm.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index b1eae767c27c..c32e34a182a2 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -1640,6 +1640,12 @@ unsigned long get_upper_mfn_bound(void) return max_page - 1; } +bool is_memory_hole(mfn_t start, mfn_t end) +{ + /* TODO: this needs to be properly implemented. */ + return true; +} + /* * Local variables: * mode: C From patchwork Fri Feb 4 06:34:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3041C4321E for ; Fri, 4 Feb 2022 06:35:27 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265004.458267 (Exim 4.92) (envelope-from ) id 1nFsBW-0008HS-JN; Fri, 04 Feb 2022 06:35:06 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265004.458267; Fri, 04 Feb 2022 06:35:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBW-0008Ga-DK; Fri, 04 Feb 2022 06:35:06 +0000 Received: by outflank-mailman (input) for mailman id 265004; Fri, 04 Feb 2022 06:35:04 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBU-0008DW-TO for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:04 +0000 Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [2a00:1450:4864:20::22f]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 95dfe064-8584-11ec-8f75-fffcc8bd4f1a; Fri, 04 Feb 2022 07:35:04 +0100 (CET) Received: by mail-lj1-x22f.google.com with SMTP id c15so7131145ljf.11 for ; Thu, 03 Feb 2022 22:35:03 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:02 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 95dfe064-8584-11ec-8f75-fffcc8bd4f1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fkt1iiffJfOyIvWsY76Bn/20kVLeJA6ML116mlUONbo=; b=pf1kd98rLRo4oMne6N5TZmeGAzADdmqWwChEtEtmGzVv81uVsjUztGLFhrbZFULVdE 59MTje+KtbOtCXsBULQRsNzE7Z0GKfm8Nw3uoeKeNDxFEQ27YfVvizhGZLVmC+00sDzT GFycA4Y+NWJf8d8B6RgZYRLHCvA5/lghyY0lo9DewojjjiXB12uVVKO8HA+M0HBKO5gO rG+ZOZsp5yjmjQkM3AzPstjUMy4neYoQWjcH+1wdkB/Haj1o90UBHpNfSMawLb1c8u7I v/aUtOlxo0Vho4MIeLtOR1Uz9PjDjFjjDrZjHDMGYclAMJJz1cZI9fBx++zFhtF2eGVt O/mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fkt1iiffJfOyIvWsY76Bn/20kVLeJA6ML116mlUONbo=; b=DRIdD5Oapo/yGKjgmf9wgZWDUYkvxnLCZiIttILa8fmgx3g2hjrpBIBdXZBhW/ITFB rhvwANbGnd7dhx9qEfJBzuKQDsrOIc6M2VqWQZUEeJOh7dxN+WgmwjIaCbuwWQAFDvjb 8irAQRi6uVT5yxcdjtrFH7mVvBJtD9YvUm1T49re6JIqmAEjbAkqZyhlmu/4Uu3mIwSn zRG9CHQ0QPfDrJkf3AZ6WTg0xdfzeDMPJluzaF3tvRIhL+uIfcvtmNqrp3l26P12HV5C T75b3N1hPFQO9dQPL2NMmtwAig033Ulr6udEOjCD4K1rvbxBbEmWJr1Gcx8HdlkxxGNk gG3Q== X-Gm-Message-State: AOAM5303zwEb02k+8Pz8CptI47KHAVGIz4qK+XlNQeU7ccBxWVva6JXl ICfgQpeUtqKyhsUDFW+a6qNu8GgivN4= X-Google-Smtp-Source: ABdhPJxPoAt1JRMWcJlYt+YBJllDqtzQb/8LTegxXLQOdjyAb34lQHAZ/fQnkHFk0AYemOu19Q85wQ== X-Received: by 2002:a2e:a583:: with SMTP id m3mr934695ljp.530.1643956503315; Thu, 03 Feb 2022 22:35:03 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 02/13] rangeset: add RANGESETF_no_print flag Date: Fri, 4 Feb 2022 08:34:48 +0200 Message-Id: <20220204063459.680961-3-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are range sets which should not be printed, so introduce a flag which allows marking those as such. Implement relevant logic to skip such entries while printing. While at it also simplify the definition of the flags by directly defining those without helpers. Suggested-by: Jan Beulich Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Jan Beulich --- Since v5: - comment indentation (Jan) Since v1: - update BUG_ON with new flag - simplify the definition of the flags --- xen/common/rangeset.c | 5 ++++- xen/include/xen/rangeset.h | 5 +++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/xen/common/rangeset.c b/xen/common/rangeset.c index 885b6b15c229..ea27d651723b 100644 --- a/xen/common/rangeset.c +++ b/xen/common/rangeset.c @@ -433,7 +433,7 @@ struct rangeset *rangeset_new( INIT_LIST_HEAD(&r->range_list); r->nr_ranges = -1; - BUG_ON(flags & ~RANGESETF_prettyprint_hex); + BUG_ON(flags & ~(RANGESETF_prettyprint_hex | RANGESETF_no_print)); r->flags = flags; safe_strcpy(r->name, name ?: "(no name)"); @@ -575,6 +575,9 @@ void rangeset_domain_printk( list_for_each_entry ( r, &d->rangesets, rangeset_list ) { + if ( r->flags & RANGESETF_no_print ) + continue; + printk(" "); rangeset_printk(r); printk("\n"); diff --git a/xen/include/xen/rangeset.h b/xen/include/xen/rangeset.h index 135f33f6066f..f7c69394d66a 100644 --- a/xen/include/xen/rangeset.h +++ b/xen/include/xen/rangeset.h @@ -49,8 +49,9 @@ void rangeset_limit( /* Flags for passing to rangeset_new(). */ /* Pretty-print range limits in hexadecimal. */ -#define _RANGESETF_prettyprint_hex 0 -#define RANGESETF_prettyprint_hex (1U << _RANGESETF_prettyprint_hex) +#define RANGESETF_prettyprint_hex (1U << 0) + /* Do not print entries marked with this flag. */ +#define RANGESETF_no_print (1U << 1) bool_t __must_check rangeset_is_empty( const struct rangeset *r); From patchwork Fri Feb 4 06:34:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2ABDC4167D for ; Fri, 4 Feb 2022 06:35:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265006.458294 (Exim 4.92) (envelope-from ) id 1nFsBY-0000VJ-Ea; Fri, 04 Feb 2022 06:35:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265006.458294; Fri, 04 Feb 2022 06:35:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBY-0000Ug-5u; Fri, 04 Feb 2022 06:35:08 +0000 Received: by outflank-mailman (input) for mailman id 265006; Fri, 04 Feb 2022 06:35:06 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBW-0008DW-GL for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:06 +0000 Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [2a00:1450:4864:20::12c]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 96944681-8584-11ec-8f75-fffcc8bd4f1a; Fri, 04 Feb 2022 07:35:05 +0100 (CET) Received: by mail-lf1-x12c.google.com with SMTP id bu18so10750710lfb.5 for ; Thu, 03 Feb 2022 22:35:05 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:04 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 96944681-8584-11ec-8f75-fffcc8bd4f1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yxvQBdV5Q4sDH3B8Pg7aZNA92D9cHrIG2H+0rNtBrr0=; b=VOub8JoCp14IgAzRiaxfC0PpVpvjywdzU8mGd7JwFmaFwxGjA+BjTaDh8w1KVx/A3g 90Eb5m7hpLa2lEdBvxJ14XNBekRwYoxtC5ofat8XoikToYiCq8V1sVYBoi5z/Okh6mEn Nk06PTuxB9YQKcTmHD27MeoSt/qzsD+LPmYpn/3uMCaRPv3UJTqYYHRR9Sbd1q5g/EF9 pCRb8UJFdRl3S/E0lKNT0FqoCYzbAhS13E7tf9fgx7eV3VZEeo/nM2ZM4F95KKzhjxV6 CdElOXTXSDVZc/tztA8uS0V40TcRi2i40CFeqO4mTEmTk1FdOl58riBFXboQYB8l430N VgMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yxvQBdV5Q4sDH3B8Pg7aZNA92D9cHrIG2H+0rNtBrr0=; b=JFB+SR2byGIYjnQ+CGtjt3mCF0UouY4LLiQcKtwA9FfcvFnMDBlubTdH89HSv+i+Bn 6BlFu87ROknkY1hlhIfNyXVD25MJ9MkvY0r61BqPVbiXsraA+iChfLp3UEuUrHGDWLrL l0r1i/rSark1q6oAsmqstuo6x5WzK79bRhVxXjjsJ5JOA+7HrLAfkdd+hS8a6YzWeocB /j+48VUU5+yTZvyH6V0NTWeHOkolaMPYaNeOhsq1+wnzGwLJ0/7nyXNQb61dBjJDg3ab QK9OZydq7Ez+xa9SzBj0CBpZL8uxj3CUQxq53JpqjLj7CmwniEInh6dp5vMlD32kAlc9 E20A== X-Gm-Message-State: AOAM532WJiHnNyCYwTdWx/7KnN+TprbaCgwdsZchW2LolBwZjkz6R3xz P+xB3MgBhrxLeArFjDKGgkbv931VOCs= X-Google-Smtp-Source: ABdhPJzh7UX3hWPz+NmdK0JSfU2SwFsZ5hSTEX8nR+b06eIGCf0NrZK59mb8mm2x/wgotivf/NXbIA== X-Received: by 2002:a05:6512:68a:: with SMTP id t10mr1337219lfe.520.1643956504487; Thu, 03 Feb 2022 22:35:04 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 03/13] vpci: move lock outside of struct vpci Date: Fri, 4 Feb 2022 08:34:49 +0200 Message-Id: <20220204063459.680961-4-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Roger Pau Monné This way the lock can be used (and in a few cases is used right away) to check whether vpci is present, and removal can be performed while holding the lock, in order to make sure there are no accesses to the contents of the vpci struct. Previously removal could race with vpci_read for example, since the lock was dropped prior to freeing pdev->vpci. Signed-off-by: Roger Pau Monné Signed-off-by: Oleksandr Andrushchenko --- Cc: Andrew Cooper Cc: Jan Beulich Cc: Julien Grall Cc: Stefano Stabellini --- New in v5 of this series: this is an updated version of the patch published at https://lore.kernel.org/xen-devel/20180717094830.54806-2-roger.pau@citrix.com/ Changes since v5: - vpci_lock in test_pdev is already initialized to false by default - introduce msix_{get|put} to protect former msix_find's result - add comments to vpci_{add|remove}_registers about pdev->vpci_lock must be held. - do not split code into vpci_remove_device_handlers_locked yet - move INIT_LIST_HEAD outside the locked region (Jan) - stripped out locking optimizations for vpci_{read|write} into a dedicated patch Changes since v2: - fixed pdev->vpci = xzalloc(struct vpci); under spin_lock (Jan) Changes since v1: - Assert that vpci_lock is locked in vpci_remove_device_locked. - Remove double newline. - Shrink critical section in vpci_{read/write}. --- tools/tests/vpci/emul.h | 5 ++- tools/tests/vpci/main.c | 3 +- xen/arch/x86/hvm/vmsi.c | 8 ++--- xen/drivers/passthrough/pci.c | 1 + xen/drivers/vpci/header.c | 21 +++++++---- xen/drivers/vpci/msi.c | 11 ++++-- xen/drivers/vpci/msix.c | 39 ++++++++++++++++----- xen/drivers/vpci/vpci.c | 65 ++++++++++++++++++++++------------- xen/include/xen/pci.h | 1 + xen/include/xen/vpci.h | 3 +- 10 files changed, 106 insertions(+), 51 deletions(-) diff --git a/tools/tests/vpci/emul.h b/tools/tests/vpci/emul.h index 2e1d3057c9d8..d018fb5eef21 100644 --- a/tools/tests/vpci/emul.h +++ b/tools/tests/vpci/emul.h @@ -44,6 +44,7 @@ struct domain { }; struct pci_dev { + bool vpci_lock; struct vpci *vpci; }; @@ -53,10 +54,8 @@ struct vcpu }; extern const struct vcpu *current; -extern const struct pci_dev test_pdev; +extern struct pci_dev test_pdev; -typedef bool spinlock_t; -#define spin_lock_init(l) (*(l) = false) #define spin_lock(l) (*(l) = true) #define spin_unlock(l) (*(l) = false) diff --git a/tools/tests/vpci/main.c b/tools/tests/vpci/main.c index b9a0a6006bb9..3b86ed232eb1 100644 --- a/tools/tests/vpci/main.c +++ b/tools/tests/vpci/main.c @@ -23,7 +23,7 @@ static struct vpci vpci; const static struct domain d; -const struct pci_dev test_pdev = { +struct pci_dev test_pdev = { .vpci = &vpci, }; @@ -158,7 +158,6 @@ main(int argc, char **argv) int rc; INIT_LIST_HEAD(&vpci.handlers); - spin_lock_init(&vpci.lock); VPCI_ADD_REG(vpci_read32, vpci_write32, 0, 4, r0); VPCI_READ_CHECK(0, 4, r0); diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index 13e2a190b439..1f7a37f78264 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -910,14 +910,14 @@ int vpci_msix_arch_print(const struct vpci_msix *msix) { struct pci_dev *pdev = msix->pdev; - spin_unlock(&msix->pdev->vpci->lock); + spin_unlock(&msix->pdev->vpci_lock); process_pending_softirqs(); /* NB: we assume that pdev cannot go away for an alive domain. */ - if ( !pdev->vpci || !spin_trylock(&pdev->vpci->lock) ) + if ( !spin_trylock(&pdev->vpci_lock) ) return -EBUSY; - if ( pdev->vpci->msix != msix ) + if ( !pdev->vpci || pdev->vpci->msix != msix ) { - spin_unlock(&pdev->vpci->lock); + spin_unlock(&pdev->vpci_lock); return -EAGAIN; } } diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index e8b09d77d880..50dec3bb73d0 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -397,6 +397,7 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) *((u8*) &pdev->bus) = bus; *((u8*) &pdev->devfn) = devfn; pdev->domain = NULL; + spin_lock_init(&pdev->vpci_lock); arch_pci_init_pdev(pdev); diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 40ff79c33f8f..bd23c0274d48 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -142,12 +142,13 @@ bool vpci_process_pending(struct vcpu *v) if ( rc == -ERESTART ) return true; - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); + spin_lock(&v->vpci.pdev->vpci_lock); + if ( v->vpci.pdev->vpci ) + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(v->vpci.pdev, + rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, + !rc && v->vpci.rom_only); + spin_unlock(&v->vpci.pdev->vpci_lock); rangeset_destroy(v->vpci.mem); v->vpci.mem = NULL; @@ -285,6 +286,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) continue; } + spin_lock(&tmp->vpci_lock); + if ( !tmp->vpci ) + { + spin_unlock(&tmp->vpci_lock); + continue; + } for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ ) { const struct vpci_bar *bar = &tmp->vpci->header.bars[i]; @@ -303,12 +310,14 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) rc = rangeset_remove_range(mem, start, end); if ( rc ) { + spin_unlock(&tmp->vpci_lock); printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); rangeset_destroy(mem); return rc; } } + spin_unlock(&tmp->vpci_lock); } ASSERT(dev); diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index 5757a7aed20f..e3ce46869dad 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -270,7 +270,7 @@ void vpci_dump_msi(void) rcu_read_lock(&domlist_read_lock); for_each_domain ( d ) { - const struct pci_dev *pdev; + struct pci_dev *pdev; if ( !has_vpci(d) ) continue; @@ -282,8 +282,13 @@ void vpci_dump_msi(void) const struct vpci_msi *msi; const struct vpci_msix *msix; - if ( !pdev->vpci || !spin_trylock(&pdev->vpci->lock) ) + if ( !spin_trylock(&pdev->vpci_lock) ) continue; + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + continue; + } msi = pdev->vpci->msi; if ( msi && msi->enabled ) @@ -323,7 +328,7 @@ void vpci_dump_msi(void) } } - spin_unlock(&pdev->vpci->lock); + spin_unlock(&pdev->vpci_lock); process_pending_softirqs(); } } diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 846f1b8d7038..d1dbfc6e0ffd 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -138,7 +138,7 @@ static void control_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, val); } -static struct vpci_msix *msix_find(const struct domain *d, unsigned long addr) +static struct vpci_msix *msix_get(const struct domain *d, unsigned long addr) { struct vpci_msix *msix; @@ -150,15 +150,29 @@ static struct vpci_msix *msix_find(const struct domain *d, unsigned long addr) for ( i = 0; i < ARRAY_SIZE(msix->tables); i++ ) if ( bars[msix->tables[i] & PCI_MSIX_BIRMASK].enabled && VMSIX_ADDR_IN_RANGE(addr, msix->pdev->vpci, i) ) + { + spin_lock(&msix->pdev->vpci_lock); return msix; + } } return NULL; } +static void msix_put(struct vpci_msix *msix) +{ + if ( !msix ) + return; + + spin_unlock(&msix->pdev->vpci_lock); +} + static int msix_accept(struct vcpu *v, unsigned long addr) { - return !!msix_find(v->domain, addr); + struct vpci_msix *msix = msix_get(v->domain, addr); + + msix_put(msix); + return !!msix; } static bool access_allowed(const struct pci_dev *pdev, unsigned long addr, @@ -186,7 +200,7 @@ static int msix_read(struct vcpu *v, unsigned long addr, unsigned int len, unsigned long *data) { const struct domain *d = v->domain; - struct vpci_msix *msix = msix_find(d, addr); + struct vpci_msix *msix = msix_get(d, addr); const struct vpci_msix_entry *entry; unsigned int offset; @@ -196,7 +210,10 @@ static int msix_read(struct vcpu *v, unsigned long addr, unsigned int len, return X86EMUL_RETRY; if ( !access_allowed(msix->pdev, addr, len) ) + { + msix_put(msix); return X86EMUL_OKAY; + } if ( VMSIX_ADDR_IN_RANGE(addr, msix->pdev->vpci, VPCI_MSIX_PBA) ) { @@ -222,10 +239,10 @@ static int msix_read(struct vcpu *v, unsigned long addr, unsigned int len, break; } + msix_put(msix); return X86EMUL_OKAY; } - spin_lock(&msix->pdev->vpci->lock); entry = get_entry(msix, addr); offset = addr & (PCI_MSIX_ENTRY_SIZE - 1); @@ -254,7 +271,8 @@ static int msix_read(struct vcpu *v, unsigned long addr, unsigned int len, ASSERT_UNREACHABLE(); break; } - spin_unlock(&msix->pdev->vpci->lock); + + msix_put(msix); return X86EMUL_OKAY; } @@ -263,7 +281,7 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, unsigned long data) { const struct domain *d = v->domain; - struct vpci_msix *msix = msix_find(d, addr); + struct vpci_msix *msix = msix_get(d, addr); struct vpci_msix_entry *entry; unsigned int offset; @@ -271,7 +289,10 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, return X86EMUL_RETRY; if ( !access_allowed(msix->pdev, addr, len) ) + { + msix_put(msix); return X86EMUL_OKAY; + } if ( VMSIX_ADDR_IN_RANGE(addr, msix->pdev->vpci, VPCI_MSIX_PBA) ) { @@ -294,10 +315,11 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, } } + msix_put(msix); + return X86EMUL_OKAY; } - spin_lock(&msix->pdev->vpci->lock); entry = get_entry(msix, addr); offset = addr & (PCI_MSIX_ENTRY_SIZE - 1); @@ -370,7 +392,8 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, ASSERT_UNREACHABLE(); break; } - spin_unlock(&msix->pdev->vpci->lock); + + msix_put(msix); return X86EMUL_OKAY; } diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index fb0947179b79..cb2ababa28e3 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -35,12 +35,10 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) -void vpci_remove_device(struct pci_dev *pdev) +static void vpci_remove_device_locked(struct pci_dev *pdev) { - if ( !has_vpci(pdev->domain) ) - return; + ASSERT(spin_is_locked(&pdev->vpci_lock)); - spin_lock(&pdev->vpci->lock); while ( !list_empty(&pdev->vpci->handlers) ) { struct vpci_register *r = list_first_entry(&pdev->vpci->handlers, @@ -50,15 +48,26 @@ void vpci_remove_device(struct pci_dev *pdev) list_del(&r->node); xfree(r); } - spin_unlock(&pdev->vpci->lock); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); pdev->vpci = NULL; } +void vpci_remove_device(struct pci_dev *pdev) +{ + if ( !has_vpci(pdev->domain) ) + return; + + spin_lock(&pdev->vpci_lock); + if ( pdev->vpci ) + vpci_remove_device_locked(pdev); + spin_unlock(&pdev->vpci_lock); +} + int vpci_add_handlers(struct pci_dev *pdev) { + struct vpci *vpci; unsigned int i; int rc = 0; @@ -68,12 +77,14 @@ int vpci_add_handlers(struct pci_dev *pdev) /* We should not get here twice for the same device. */ ASSERT(!pdev->vpci); - pdev->vpci = xzalloc(struct vpci); - if ( !pdev->vpci ) + vpci = xzalloc(struct vpci); + if ( !vpci ) return -ENOMEM; - INIT_LIST_HEAD(&pdev->vpci->handlers); - spin_lock_init(&pdev->vpci->lock); + INIT_LIST_HEAD(&vpci->handlers); + + spin_lock(&pdev->vpci_lock); + pdev->vpci = vpci; for ( i = 0; i < NUM_VPCI_INIT; i++ ) { @@ -83,7 +94,8 @@ int vpci_add_handlers(struct pci_dev *pdev) } if ( rc ) - vpci_remove_device(pdev); + vpci_remove_device_locked(pdev); + spin_unlock(&pdev->vpci_lock); return rc; } @@ -129,6 +141,7 @@ uint32_t vpci_hw_read32(const struct pci_dev *pdev, unsigned int reg, return pci_conf_read32(pdev->sbdf, reg); } +/* Must be called with pdev->vpci_lock held. */ int vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, vpci_write_t *write_handler, unsigned int offset, unsigned int size, void *data) @@ -152,8 +165,6 @@ int vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, r->offset = offset; r->private = data; - spin_lock(&vpci->lock); - /* The list of handlers must be kept sorted at all times. */ list_for_each ( prev, &vpci->handlers ) { @@ -165,25 +176,23 @@ int vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, break; if ( cmp == 0 ) { - spin_unlock(&vpci->lock); xfree(r); return -EEXIST; } } list_add_tail(&r->node, prev); - spin_unlock(&vpci->lock); return 0; } +/* Must be called with pdev->vpci_lock held. */ int vpci_remove_register(struct vpci *vpci, unsigned int offset, unsigned int size) { const struct vpci_register r = { .offset = offset, .size = size }; struct vpci_register *rm; - spin_lock(&vpci->lock); list_for_each_entry ( rm, &vpci->handlers, node ) { int cmp = vpci_register_cmp(&r, rm); @@ -195,14 +204,12 @@ int vpci_remove_register(struct vpci *vpci, unsigned int offset, if ( !cmp && rm->offset == offset && rm->size == size ) { list_del(&rm->node); - spin_unlock(&vpci->lock); xfree(rm); return 0; } if ( cmp <= 0 ) break; } - spin_unlock(&vpci->lock); return -ENOENT; } @@ -311,7 +318,7 @@ static uint32_t merge_result(uint32_t data, uint32_t new, unsigned int size, uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) { const struct domain *d = current->domain; - const struct pci_dev *pdev; + struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; uint32_t data = ~(uint32_t)0; @@ -327,7 +334,12 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) if ( !pdev ) return vpci_read_hw(sbdf, reg, size); - spin_lock(&pdev->vpci->lock); + spin_lock(&pdev->vpci_lock); + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + return vpci_read_hw(sbdf, reg, size); + } /* Read from the hardware or the emulated register handlers. */ list_for_each_entry ( r, &pdev->vpci->handlers, node ) @@ -370,7 +382,7 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) break; ASSERT(data_offset < size); } - spin_unlock(&pdev->vpci->lock); + spin_unlock(&pdev->vpci_lock); if ( data_offset < size ) { @@ -414,7 +426,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, uint32_t data) { const struct domain *d = current->domain; - const struct pci_dev *pdev; + struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; const unsigned long *ro_map = pci_get_ro_map(sbdf.seg); @@ -440,7 +452,14 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, return; } - spin_lock(&pdev->vpci->lock); + spin_lock(&pdev->vpci_lock); + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + vpci_write_hw(sbdf, reg, size, data); + return; + } + /* Write the value to the hardware or emulated registers. */ list_for_each_entry ( r, &pdev->vpci->handlers, node ) @@ -475,7 +494,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, break; ASSERT(data_offset < size); } - spin_unlock(&pdev->vpci->lock); + spin_unlock(&pdev->vpci_lock); if ( data_offset < size ) /* Tailing gap, write the remaining. */ diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index b6d7e454f814..3f60d6c6c6dd 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -134,6 +134,7 @@ struct pci_dev { u64 vf_rlen[6]; /* Data for vPCI. */ + spinlock_t vpci_lock; struct vpci *vpci; }; diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index e8ac1eb39513..f2a7d82ce77b 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -31,7 +31,7 @@ int __must_check vpci_add_handlers(struct pci_dev *dev); /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); -/* Add/remove a register handler. */ +/* Add/remove a register handler. Must be called holding the vpci_lock. */ int __must_check vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, vpci_write_t *write_handler, @@ -60,7 +60,6 @@ bool __must_check vpci_process_pending(struct vcpu *v); struct vpci { /* List of vPCI handlers for a device. */ struct list_head handlers; - spinlock_t lock; #ifdef __XEN__ /* Hide the rest of the vpci struct from the user-space test harness. */ From patchwork Fri Feb 4 06:34:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03F2AC433F5 for ; Fri, 4 Feb 2022 06:35:26 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265007.458299 (Exim 4.92) (envelope-from ) id 1nFsBY-0000Zp-Qn; Fri, 04 Feb 2022 06:35:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265007.458299; Fri, 04 Feb 2022 06:35:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBY-0000Xs-GL; Fri, 04 Feb 2022 06:35:08 +0000 Received: by outflank-mailman (input) for mailman id 265007; Fri, 04 Feb 2022 06:35:07 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBX-0008DX-0q for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:07 +0000 Received: from mail-lf1-x135.google.com (mail-lf1-x135.google.com [2a00:1450:4864:20::135]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 973fa44c-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:06 +0100 (CET) Received: by mail-lf1-x135.google.com with SMTP id z4so10781574lft.3 for ; Thu, 03 Feb 2022 22:35:06 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:05 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 973fa44c-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7lOljSwfVLv7O1Qu1kbWEhEP6CpP7dXjj0XKQulfe6E=; b=Ck6BoSlnQgRISD6huKxphsjKoG7Euj2TC16HbLrf+oVTvsu+WKBI1c3kt8icPCERIB tkkWe/KesrqTP/q7HDoal4nGgy9m/6Ew9lCh7TrJWuYslrgjYmgtc6Ea5jRCMjJNj4LC kg7zdt2/kaKGbAzTDzeW0NoyYPbUHwutKjT6erIRJQAcUqAXsOSafT3+eODU1ugAQHtz Zgn5ZYMrhi45XL/tNMKmf75EiXt2FD+8gRN2p/FUE/n+37KE3I7BSLgLtjIB4Cz6vru+ uXnAwU2XEzrjw0ZZYFZ7eJC2wBKSU2sap05DLurc4bQbHXDNegwVdVDrrSHmvXwPuWv7 4RDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7lOljSwfVLv7O1Qu1kbWEhEP6CpP7dXjj0XKQulfe6E=; b=HBEiXgxoYa/kRj88N8yBGQgrz1YZXomxYTsTsW6YrvaVSI+DeYCHx6Um8vQuQp8BaB AN5IixKLuTiuC++5veHVRQQCPOxiW+cnl+26EXl+RHBSznp9gZuLs+HNLtG7PWoPlMSh QL72+Aq90rnz68+Sj2FR8oB/gRIbnV3g5qNCkTIZ/JfqPc5yidUqeZ/Z6JYQkTu37SiG 93rKPSDtgcpUdZbvS8fHRjVuYYEwXpW9h8ZlI76fTJNs0zTJWCtgQQPcC2AgVHvzhReU 9HuoIYdOmXdP5izkbAEkL3mDKPeZadBoh4RyP5Kwp1yex3zGpOl8mm89n7J6Ui+y1c3i WQeg== X-Gm-Message-State: AOAM532NeogFBNrBMnfGXjBRheB1JCTN9TYaYGl1vEmpIKCUFE3GXMbH YQfzxkilMjbejVimjC/GsMVWnoVXSM8= X-Google-Smtp-Source: ABdhPJyB6MQ7eonV8Yqfxf8f3gGfIt7kmJS1oIR+juotmd4GDOq+j/gxIoib7IA3nxTTvs39ppLLBg== X-Received: by 2002:ac2:5186:: with SMTP id u6mr1392098lfi.519.1643956505619; Thu, 03 Feb 2022 22:35:05 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 04/13] vpci: restrict unhandled read/write operations for guests Date: Fri, 4 Feb 2022 08:34:50 +0200 Message-Id: <20220204063459.680961-5-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko A guest can read and write those registers which are not emulated and have no respective vPCI handlers, so it can access the HW directly. In order to prevent a guest from reads and writes from/to the unhandled registers make sure only hardware domain can access HW directly and restrict guests from doing so. Suggested-by: Roger Pau Monné Signed-off-by: Oleksandr Andrushchenko --- New in v6 --- xen/drivers/vpci/vpci.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index cb2ababa28e3..f8a93e61c08f 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -215,11 +215,15 @@ int vpci_remove_register(struct vpci *vpci, unsigned int offset, } /* Wrappers for performing reads/writes to the underlying hardware. */ -static uint32_t vpci_read_hw(pci_sbdf_t sbdf, unsigned int reg, +static uint32_t vpci_read_hw(bool is_hwdom, pci_sbdf_t sbdf, unsigned int reg, unsigned int size) { uint32_t data; + /* Guest domains are not allowed to read real hardware. */ + if ( !is_hwdom ) + return ~(uint32_t)0; + switch ( size ) { case 4: @@ -260,9 +264,13 @@ static uint32_t vpci_read_hw(pci_sbdf_t sbdf, unsigned int reg, return data; } -static void vpci_write_hw(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, - uint32_t data) +static void vpci_write_hw(bool is_hwdom, pci_sbdf_t sbdf, unsigned int reg, + unsigned int size, uint32_t data) { + /* Guest domains are not allowed to write real hardware. */ + if ( !is_hwdom ) + return; + switch ( size ) { case 4: @@ -322,6 +330,7 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) const struct vpci_register *r; unsigned int data_offset = 0; uint32_t data = ~(uint32_t)0; + bool is_hwdom = is_hardware_domain(d); if ( !size ) { @@ -332,13 +341,13 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) /* Find the PCI dev matching the address. */ pdev = pci_get_pdev_by_domain(d, sbdf.seg, sbdf.bus, sbdf.devfn); if ( !pdev ) - return vpci_read_hw(sbdf, reg, size); + return vpci_read_hw(is_hwdom, sbdf, reg, size); spin_lock(&pdev->vpci_lock); if ( !pdev->vpci ) { spin_unlock(&pdev->vpci_lock); - return vpci_read_hw(sbdf, reg, size); + return vpci_read_hw(is_hwdom, sbdf, reg, size); } /* Read from the hardware or the emulated register handlers. */ @@ -361,7 +370,7 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) { /* Heading gap, read partial content from hardware. */ read_size = r->offset - emu.offset; - val = vpci_read_hw(sbdf, emu.offset, read_size); + val = vpci_read_hw(is_hwdom, sbdf, emu.offset, read_size); data = merge_result(data, val, read_size, data_offset); data_offset += read_size; } @@ -387,7 +396,7 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) if ( data_offset < size ) { /* Tailing gap, read the remaining. */ - uint32_t tmp_data = vpci_read_hw(sbdf, reg + data_offset, + uint32_t tmp_data = vpci_read_hw(is_hwdom, sbdf, reg + data_offset, size - data_offset); data = merge_result(data, tmp_data, size - data_offset, data_offset); @@ -430,6 +439,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, const struct vpci_register *r; unsigned int data_offset = 0; const unsigned long *ro_map = pci_get_ro_map(sbdf.seg); + bool is_hwdom = is_hardware_domain(d); if ( !size ) { @@ -448,7 +458,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, pdev = pci_get_pdev_by_domain(d, sbdf.seg, sbdf.bus, sbdf.devfn); if ( !pdev ) { - vpci_write_hw(sbdf, reg, size, data); + vpci_write_hw(is_hwdom, sbdf, reg, size, data); return; } @@ -456,7 +466,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, if ( !pdev->vpci ) { spin_unlock(&pdev->vpci_lock); - vpci_write_hw(sbdf, reg, size, data); + vpci_write_hw(is_hwdom, sbdf, reg, size, data); return; } @@ -479,7 +489,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, if ( emu.offset < r->offset ) { /* Heading gap, write partial content to hardware. */ - vpci_write_hw(sbdf, emu.offset, r->offset - emu.offset, + vpci_write_hw(is_hwdom, sbdf, emu.offset, r->offset - emu.offset, data >> (data_offset * 8)); data_offset += r->offset - emu.offset; } @@ -498,7 +508,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, if ( data_offset < size ) /* Tailing gap, write the remaining. */ - vpci_write_hw(sbdf, reg + data_offset, size - data_offset, + vpci_write_hw(is_hwdom, sbdf, reg + data_offset, size - data_offset, data >> (data_offset * 8)); } From patchwork Fri Feb 4 06:34:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3408EC4167B for ; Fri, 4 Feb 2022 06:35:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265008.458315 (Exim 4.92) (envelope-from ) id 1nFsBa-00012P-8C; Fri, 04 Feb 2022 06:35:10 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265008.458315; Fri, 04 Feb 2022 06:35:10 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBa-00011Y-2T; Fri, 04 Feb 2022 06:35:10 +0000 Received: by outflank-mailman (input) for mailman id 265008; Fri, 04 Feb 2022 06:35:08 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBY-0008DX-5I for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:08 +0000 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [2a00:1450:4864:20::229]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 97f177f4-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:07 +0100 (CET) Received: by mail-lj1-x229.google.com with SMTP id e17so7167528ljk.5 for ; Thu, 03 Feb 2022 22:35:07 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:06 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 97f177f4-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7XW5WI2LKa0+lHLha6LOvz3mRlOFJHaJBN6i9Y4p6WM=; b=dhahBLefv3vqpJIzC/B6ePyfjpDpPIn86QOziEMIKUWSkc+w0iRCBKAJ6dWQdHnocH QKOJqMlHxXYEJ/mxNf/0sVdGb0tW7CAtp4i/oNw6u8Rr28oCjpPNVgP2WCz+nWhJ7Te2 NmdZfvJ/QtVeUxx23tOWOcEfvFweXLQw30iJt2sTikUH2ss2SUxGXONdjtD4BZXJ3M4t 8j9yN+t5sFF12wbD0Py19VVpXMQwX0RRhg1kPw1G920+tqgtQbA6ZwwwLvDzdKYNVuUX T9nVM7QqK4HsM2wb+uvctYF4gge17KSZ1x0oWs9vA1VcrKsTHR3njYi1Z6r9ZhL+bAxe MXSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7XW5WI2LKa0+lHLha6LOvz3mRlOFJHaJBN6i9Y4p6WM=; b=x2Lrz3Tw9xnAOdSbSY+DdcvxEiF6oicZXECNegNKX67ZDMUeVME41VlwXtPdc/fK7M nhKfYsJ8hl8g1mduv3TvI5VFF1Gtp0i2rieAhhpglyxG+Yz7wGldB2RWRv/Rr5NqtS4f oGd4QQyLqY5iSBM4BF0ZGWTcoeWWDaK8NRTLBLUxeW3MSMM/nK/00yF8BtpaekJ2xxWf bQcxgsym+LU3bsUEp6HUrrD0RENM1hFWQ8AWwefWF5bmQkpRdo73BpBzlBFZXKPDvk3O 0PEGMXRIiGCbDcmjH5VtxCxz5w+IH1xrsc6f9gfM5rwsqapZDVAs6QQxiWeLD/yiXf3W yxYA== X-Gm-Message-State: AOAM533X/WKQqvXQOiVY89c/ogMHJajirMrQ8m/rCXb6JuV/39l/Ze96 rS51qJ354SHfpT90LLT2Ps+m9LuwN2I= X-Google-Smtp-Source: ABdhPJx32wzZ5nqhjR3pnLXjGJB/ispCD26olf2AkIY8XmREw8kF2eQaqvbwXb6rvKqnjeQ7HQ+hKQ== X-Received: by 2002:a2e:a289:: with SMTP id k9mr907186lja.464.1643956506750; Thu, 03 Feb 2022 22:35:06 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 05/13] vpci: add hooks for PCI device assign/de-assign Date: Fri, 4 Feb 2022 08:34:51 +0200 Message-Id: <20220204063459.680961-6-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - do not split code into run_vpci_init - do not check for is_system_domain in vpci_{de}assign_device - do not use vpci_remove_device_handlers_locked and re-allocate pdev->vpci completely - make vpci_deassign_device void Since v4: - de-assign vPCI from the previous domain on device assignment - do not remove handlers in vpci_assign_device as those must not exist at that point Since v3: - remove toolstack roll-back description from the commit message as error are to be handled with proper cleanup in Xen itself - remove __must_check - remove redundant rc check while assigning devices - fix redundant CONFIG_HAS_VPCI check for CONFIG_HAS_VPCI_GUEST_SUPPORT - use REGISTER_VPCI_INIT machinery to run required steps on device init/assign: add run_vpci_init helper Since v2: - define CONFIG_HAS_VPCI_GUEST_SUPPORT so dead code is not compiled for x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/Kconfig | 4 ++++ xen/drivers/passthrough/pci.c | 6 ++++++ xen/drivers/vpci/vpci.c | 27 +++++++++++++++++++++++++++ xen/include/xen/vpci.h | 15 +++++++++++++++ 4 files changed, 52 insertions(+) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47a6..780490cf8e39 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_VPCI + endmenu diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 50dec3bb73d0..88836aab6baf 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -943,6 +943,8 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + vpci_deassign_device(d, pdev); + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1488,6 +1490,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) ASSERT(pdev && (pdev->domain == hardware_domain || pdev->domain == dom_io)); + vpci_deassign_device(pdev->domain, pdev); + rc = pdev_msix_assign(d, pdev); if ( rc ) goto done; @@ -1507,6 +1511,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) pci_to_dev(pdev), flag); } + rc = vpci_assign_device(d, pdev); + done: if ( rc ) printk(XENLOG_G_WARNING "%pd: assign (%pp) failed (%d)\n", diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index f8a93e61c08f..4e774875fa04 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -99,6 +99,33 @@ int vpci_add_handlers(struct pci_dev *pdev) return rc; } + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct domain *d, struct pci_dev *pdev) +{ + int rc; + + if ( !has_vpci(d) ) + return 0; + + rc = vpci_add_handlers(pdev); + if ( rc ) + vpci_deassign_device(d, pdev); + + return rc; +} + +/* Notify vPCI that device is de-assigned from guest. */ +void vpci_deassign_device(struct domain *d, struct pci_dev *pdev) +{ + if ( !has_vpci(d) ) + return; + + vpci_remove_device(pdev); +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index f2a7d82ce77b..246307e6f5d5 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -251,6 +251,21 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) } #endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int vpci_assign_device(struct domain *d, struct pci_dev *pdev); +void vpci_deassign_device(struct domain *d, struct pci_dev *pdev); +#else +static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) +{ + return 0; +}; + +static inline void vpci_deassign_device(struct domain *d, struct pci_dev *pdev) +{ +}; +#endif + #endif /* From patchwork Fri Feb 4 06:34:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2D21C43219 for ; Fri, 4 Feb 2022 06:35:27 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265009.458325 (Exim 4.92) (envelope-from ) id 1nFsBb-0001Mw-KH; Fri, 04 Feb 2022 06:35:11 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265009.458325; Fri, 04 Feb 2022 06:35:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBb-0001Lt-F5; Fri, 04 Feb 2022 06:35:11 +0000 Received: by outflank-mailman (input) for mailman id 265009; Fri, 04 Feb 2022 06:35:09 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBZ-0008DX-Hg for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:09 +0000 Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [2a00:1450:4864:20::22c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 98aeb6f8-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:08 +0100 (CET) Received: by mail-lj1-x22c.google.com with SMTP id e17so7167593ljk.5 for ; Thu, 03 Feb 2022 22:35:08 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:07 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 98aeb6f8-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4b53rA6xzj/VK8mPmETSCKHkM43BtRFV7FZoD/x6CYM=; b=LGFssdbglsHSBRlLC10zAwfwl+lxGwyGLcSEdRXHsHfCtbH4EZFUs1eo+aoLDmBg4S eB6pFpKJGKQyDTv/SUipdDjBH8iXt/3A/eV7ucleaPmF2llLUvbbMFoCgFShlOdgrmE5 IivVoOTRjsuUm/VdB7KPtmsXA9ZYwcx9V1RbdqPeDn0jUaaRojQoDRGJ6z4ZWA9TaktG faQlTO2kJ0JfREXc0+N3NSw8kLhL6bnWJC8ppgKvyBEIHgfvSZmsnkZOh72SlUdfD6qm Z26XqIeZu6GCEF9d4lT6hQIpHAwq0pcEgoN+k/wlaH2N9n2lviiCuaq+NvWUEJnQ4+KW CkIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4b53rA6xzj/VK8mPmETSCKHkM43BtRFV7FZoD/x6CYM=; b=ludS6Bq5lxUDJyLJKJ8L+CWi66r/6ORTsCqx+1w/amxtZDbLTFpSZAfeff8/ruX/rn /9m89XT7xmb59nzqBJwUuB7ELOQyhB2L0Jp4DRcPUdTzt2cgFoa8wR/MJmmlcZUAcQfR 22C+aahPb+c8Q5/9AXtNYbKBIyaoX1foBB18uMQPlHJQuNU5LbQCCOAhDCe4ij9bhTSN Iyf+BSWuGAmfN3CTLL00I24nJinXMdU4dreUZj+JI1Rtri0gn7wGmboUOurUS67SIH6h mjGAgzjE+J00TyicWXEZaLhrqIXxEDkzWAzc0VvED7yfd+C+8u+cwa0fAuWYG04AgJVO s1zg== X-Gm-Message-State: AOAM530Dyhj/fsnMHHgF4Dto7rPTkZlt94RhiU26H68c0e6j4Zxb7L0o LVWXDDBj29dbr05SjhoEUZaJRAfhzfE= X-Google-Smtp-Source: ABdhPJyT9UqVfIhAtTM96uy+/s3bkntSE0jQKHwafHnI3xVxjYIKf9NL0rxmdS5y0dE9VlwJS+zJEg== X-Received: by 2002:a2e:95cb:: with SMTP id y11mr937679ljh.129.1643956507945; Thu, 03 Feb 2022 22:35:07 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 06/13] vpci/header: implement guest BAR register handlers Date: Fri, 4 Feb 2022 08:34:52 +0200 Message-Id: <20220204063459.680961-7-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. All empty, IO and ROM BARs for guests are emulated by returning 0 on reads and ignoring writes: this BARs are special with this respect as their lower bits have special meaning, so returning default ~0 on read may confuse guest OS. Memory decoding is initially disabled when used by guests in order to prevent the BAR being placed on top of a RAM region. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - make sure that the guest set address has the same page offset as the physical address on the host - remove guest_rom_{read|write} as those just implement the default behaviour of the registers not being handled - adjusted comment for struct vpci.addr field - add guest handlers for BARs which are not handled and will otherwise return ~0 on read and ignore writes. The BARs are special with this respect as their lower bits have special meaning, so returning ~0 doesn't seem to be right Since v4: - updated commit message - s/guest_addr/guest_reg Since v3: - squashed two patches: dynamic add/remove handlers and guest BAR handler implementation - fix guest BAR read of the high part of a 64bit BAR (Roger) - add error handling to vpci_assign_device - s/dom%pd/%pd - blank line before return Since v2: - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code has been eliminated from being built on x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment --- xen/drivers/vpci/header.c | 131 +++++++++++++++++++++++++++++++++----- xen/include/xen/vpci.h | 3 + 2 files changed, 118 insertions(+), 16 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index bd23c0274d48..2620a95ff35b 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -406,6 +406,81 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write32(pdev->sbdf, reg, val); } +static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ + struct vpci_bar *bar = data; + bool hi = false; + uint64_t guest_reg = bar->guest_reg; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + guest_reg &= ~(0xffffffffull << (hi ? 32 : 0)); + guest_reg |= (uint64_t)val << (hi ? 32 : 0); + + guest_reg &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; + + /* + * Make sure that the guest set address has the same page offset + * as the physical address on the host or otherwise things won't work as + * expected. + */ + if ( (guest_reg & (~PAGE_MASK & PCI_BASE_ADDRESS_MEM_MASK)) != + (bar->addr & ~PAGE_MASK) ) + { + gprintk(XENLOG_WARNING, + "%pp: ignored BAR %zu write with wrong page offset\n", + &pdev->sbdf, bar - pdev->vpci->header.bars + hi); + return; + } + + bar->guest_reg = guest_reg; +} + +static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + const struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + + return bar->guest_reg >> (hi ? 32 : 0); +} + +static uint32_t guest_bar_ignore_read(const struct pci_dev *pdev, + unsigned int reg, void *data) +{ + return 0; +} + +static int bar_ignore_access(const struct pci_dev *pdev, unsigned int reg, + struct vpci_bar *bar) +{ + if ( is_hardware_domain(pdev->domain) ) + return 0; + + return vpci_add_register(pdev->vpci, guest_bar_ignore_read, NULL, + reg, 4, bar); +} + static void rom_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -462,6 +537,7 @@ static int init_bars(struct pci_dev *pdev) struct vpci_header *header = &pdev->vpci->header; struct vpci_bar *bars = header->bars; int rc; + bool is_hwdom = is_hardware_domain(pdev->domain); switch ( pci_conf_read8(pdev->sbdf, PCI_HEADER_TYPE) & 0x7f ) { @@ -501,8 +577,10 @@ static int init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -516,6 +594,11 @@ static int init_bars(struct pci_dev *pdev) if ( (val & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO ) { bars[i].type = VPCI_BAR_IO; + + rc = bar_ignore_access(pdev, reg, &bars[i]); + if ( rc ) + return rc; + continue; } if ( (val & PCI_BASE_ADDRESS_MEM_TYPE_MASK) == @@ -535,6 +618,11 @@ static int init_bars(struct pci_dev *pdev) if ( size == 0 ) { bars[i].type = VPCI_BAR_EMPTY; + + rc = bar_ignore_access(pdev, reg, &bars[i]); + if ( rc ) + return rc; + continue; } @@ -542,8 +630,10 @@ static int init_bars(struct pci_dev *pdev) bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -551,22 +641,31 @@ static int init_bars(struct pci_dev *pdev) } } - /* Check expansion ROM. */ - rc = pci_size_mem_bar(pdev->sbdf, rom_reg, &addr, &size, PCI_BAR_ROM); - if ( rc > 0 && size ) + /* Check expansion ROM: we do not handle ROM for guests. */ + if ( is_hwdom ) { - struct vpci_bar *rom = &header->bars[num_bars]; + rc = pci_size_mem_bar(pdev->sbdf, rom_reg, &addr, &size, PCI_BAR_ROM); + if ( rc > 0 && size ) + { + struct vpci_bar *rom = &header->bars[num_bars]; - rom->type = VPCI_BAR_ROM; - rom->size = size; - rom->addr = addr; - header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & - PCI_ROM_ADDRESS_ENABLE; + rom->type = VPCI_BAR_ROM; + rom->size = size; + rom->addr = addr; + header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & + PCI_ROM_ADDRESS_ENABLE; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, + rom_reg, 4, rom); + if ( rc ) + rom->type = VPCI_BAR_EMPTY; + } + } + else + { + rc = bar_ignore_access(pdev, rom_reg, &header->bars[num_bars]); if ( rc ) - rom->type = VPCI_BAR_EMPTY; + return rc; } return (cmd & PCI_COMMAND_MEMORY) ? modify_bars(pdev, cmd, false) : 0; diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 246307e6f5d5..270d22b85653 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -66,7 +66,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical (host) address. */ uint64_t addr; + /* Guest view of the BAR: address and lower bits. */ + uint64_t guest_reg; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Fri Feb 4 06:34:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62E54C4332F for ; Fri, 4 Feb 2022 06:35:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265010.458331 (Exim 4.92) (envelope-from ) id 1nFsBc-0001QK-42; Fri, 04 Feb 2022 06:35:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265010.458331; Fri, 04 Feb 2022 06:35:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBb-0001Ou-RL; Fri, 04 Feb 2022 06:35:11 +0000 Received: by outflank-mailman (input) for mailman id 265010; Fri, 04 Feb 2022 06:35:11 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBb-0008DX-2o for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:11 +0000 Received: from mail-lj1-x232.google.com (mail-lj1-x232.google.com [2a00:1450:4864:20::232]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9970619f-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:09 +0100 (CET) Received: by mail-lj1-x232.google.com with SMTP id z7so7175594ljj.4 for ; Thu, 03 Feb 2022 22:35:09 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:08 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9970619f-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7HlIeOQ4u8ilsk2z8tUSqNoEHT6feHwuo6GUwAouwe0=; b=oo/4mkgXCUQW6zXw9NSPzAjcrsksdcI9Ua1MMWdkCTeOF0VmcvI6jgT+91VSZcB/5s ngyIhz7SfBUdVU8AKh4UpTzi2FPCNgvwUO5EoACJphhff7pxujAQfjOoNmMwoJpj0S3N 6ArXKPq94WC6EraXQm5nyJ0de5FUUC7WcUfTZWS6bAE5woxuDwk+9YXtLs2j7osr9wld /7iq7xMwnNWMVgHbDDYY/wogOqysmDurEkT8A4tbDulCqBPoz2AsPwNZ/YVc07R+iRcp N6/cJu/XwfNa/oEIn1TMZrq5AKsVTfChkPngLW257zbFqH7J8RQx9gOkSvSk1d/YtWp7 oADg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7HlIeOQ4u8ilsk2z8tUSqNoEHT6feHwuo6GUwAouwe0=; b=C9nMWgjV0s74BtURBAzvKj62G/pI6rbO9sVyKaFygLI6OY81CYsrNeyJ0tiobfEHX3 SXELeHqrdJSw2sBSgjpL9I5hqWhRyH1XRlaSCmjYVOuLGQPIvCUN7H91bZJYjMVShMTM LZMt6mfrwgqawaJy7G1p9k6pU+3xSxJN5gDD7hGEVvtrdl1OBJfGTUdTyCTvYgKbtxcm aEjE8/dmjpCOZweZFU05d6HIhLXm0nue/ccTjMC4mO911/R3gyKge25eujTTzKN+oVdC vTfv6qM4wcskovpEY1UA3cVD/UMaHSbbB7ScGePoLwzLaAHeFmtTZbypF210G+bTNqgf bfXg== X-Gm-Message-State: AOAM533dUgW17lXih5NkeQS9PIQyt6KQIt//R+kXIFtTkslff8uLwQXu 73XDY/FjZE54UMxZ+P5y9XepolLXZFI= X-Google-Smtp-Source: ABdhPJwLQpGq15e84eC4Z8AwZC/jlpbvigrsAv/ePIiKVlZiNNo/Jgd85WKzCvwf1eJvYavTpfEPIw== X-Received: by 2002:a2e:9918:: with SMTP id v24mr919230lji.230.1643956509105; Thu, 03 Feb 2022 22:35:09 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 07/13] vpci/header: handle p2m range sets per BAR Date: Fri, 4 Feb 2022 08:34:53 +0200 Message-Id: <20220204063459.680961-8-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. As the range sets are now created when a PCI device is added and destroyed when it is removed so make them named and accounted. Note that rangesets were chosen here despite there being only up to 3 separate ranges in each set (typically just 1). But rangeset per BAR was chosen for the ease of implementation and existing code re-usability. This is in preparation of making non-identity mappings in p2m for the MMIOs. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - fix comments - move rangeset allocation to init_bars and only allocate for MAPPABLE BARs - check for overlap with the already setup BAR ranges Since v4: - use named range sets for BARs (Jan) - changes required by the new locking scheme - updated commit message (Jan) Since v3: - re-work vpci_cancel_pending accordingly to the per-BAR handling - s/num_mem_ranges/map_pending and s/uint8_t/bool - ASSERT(bar->mem) in modify_bars - create and destroy the rangesets on add/remove --- xen/drivers/vpci/header.c | 213 ++++++++++++++++++++++++++++---------- xen/drivers/vpci/vpci.c | 17 ++- xen/include/xen/vpci.h | 4 +- 3 files changed, 177 insertions(+), 57 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 2620a95ff35b..0c94504b87d8 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -131,50 +131,85 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + struct pci_dev *pdev = v->vpci.pdev; + + if ( !pdev ) + return false; + + spin_lock(&pdev->vpci_lock); + if ( v->vpci.map_pending ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + int rc; + + if ( rangeset_is_empty(bar->mem) ) + continue; - if ( rc == -ERESTART ) - return true; + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + { + spin_unlock(&pdev->vpci_lock); + return true; + } - spin_lock(&v->vpci.pdev->vpci_lock); - if ( v->vpci.pdev->vpci ) /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci_lock); + modify_decoding(pdev, rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : + v->vpci.cmd, !rc && v->vpci.rom_only); + + if ( rc ) + { + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this + * is safe for Dom0, for DomUs the domain needs to be killed in + * order to avoid leaking stale p2m mappings on failure. + */ + if ( is_hardware_domain(v->domain) ) + vpci_remove_device_locked(pdev); + else + domain_crash(v->domain); + + break; + } + } + + v->vpci.map_pending = false; - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; - if ( rc ) - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. - */ - vpci_remove_device(v->vpci.pdev); } + spin_unlock(&pdev->vpci_lock); return false; } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) - process_pending_softirqs(); - rangeset_destroy(mem); + if ( rangeset_is_empty(bar->mem) ) + continue; + + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + process_pending_softirqs(); + } if ( !rc ) modify_decoding(pdev, cmd, false); @@ -182,7 +217,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only) { struct vcpu *curr = current; @@ -193,7 +228,7 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; + curr->vpci.map_pending = true; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; /* @@ -207,42 +242,60 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; - - if ( !mem ) - return -ENOMEM; + bool map_pending; /* - * Create a rangeset that represents the current device BARs memory region - * and compare it against all the currently active BAR memory regions. If - * an overlap is found, subtract it from the region to be mapped/unmapped. + * Create a rangeset per BAR that represents the current device memory + * region and compare it against all the currently active BAR memory + * regions. If an overlap is found, subtract it from the region to be + * mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + if ( !bar->mem ) + continue; + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) ) continue; - rc = rangeset_add_range(mem, start, end); + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; + } + + /* Check for overlap with the already setup BAR ranges. */ + for ( j = 0; j < i; j++ ) + { + struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove overlapping range [%lx, %lx]: %d\n", + start, end, rc); + goto fail; + } } } @@ -253,14 +306,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + goto fail; + } } } @@ -298,7 +358,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || + if ( !bar->enabled || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against other * BARs in the same device for overlaps, but not against the @@ -307,14 +368,13 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { spin_unlock(&tmp->vpci_lock); printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } spin_unlock(&tmp->vpci_lock); @@ -333,12 +393,28 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + map_pending = false; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + if ( !rangeset_is_empty(header->bars[i].mem) ) + { + map_pending = true; + break; + } + + /* If there's no mapping work write the command register now. */ + if ( !map_pending ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only); return 0; + +fail: + return rc; } static void cmd_write(const struct pci_dev *pdev, unsigned int reg, @@ -529,6 +605,19 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static int bar_add_rangeset(struct pci_dev *pdev, struct vpci_bar *bar, int i) +{ + char str[32]; + + snprintf(str, sizeof(str), "%pp:BAR%d", &pdev->sbdf, i); + + bar->mem = rangeset_new(pdev->domain, str, RANGESETF_no_print); + if ( !bar->mem ) + return -ENOMEM; + + return 0; +} + static int init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -607,6 +696,13 @@ static int init_bars(struct pci_dev *pdev) else bars[i].type = VPCI_BAR_MEM32; + rc = bar_add_rangeset(pdev, &bars[i], i); + if ( rc ) + { + bars[i].type = VPCI_BAR_EMPTY; + return rc; + } + rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size, (i == num_bars - 1) ? PCI_BAR_LAST : 0); if ( rc < 0 ) @@ -659,6 +755,15 @@ static int init_bars(struct pci_dev *pdev) rom_reg, 4, rom); if ( rc ) rom->type = VPCI_BAR_EMPTY; + else + { + rc = bar_add_rangeset(pdev, rom, i); + if ( rc ) + { + rom->type = VPCI_BAR_EMPTY; + return rc; + } + } } } else diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 4e774875fa04..3177f13c1c22 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -35,8 +35,11 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) -static void vpci_remove_device_locked(struct pci_dev *pdev) +void vpci_remove_device_locked(struct pci_dev *pdev) { + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + ASSERT(spin_is_locked(&pdev->vpci_lock)); while ( !list_empty(&pdev->vpci->handlers) ) @@ -48,6 +51,10 @@ static void vpci_remove_device_locked(struct pci_dev *pdev) list_del(&r->node); xfree(r); } + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + rangeset_destroy(header->bars[i].mem); + xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); @@ -94,9 +101,15 @@ int vpci_add_handlers(struct pci_dev *pdev) } if ( rc ) - vpci_remove_device_locked(pdev); + goto fail; + spin_unlock(&pdev->vpci_lock); + return 0; + + fail: + vpci_remove_device_locked(pdev); + spin_unlock(&pdev->vpci_lock); return rc; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 270d22b85653..f1f49db959c7 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -30,6 +30,7 @@ int __must_check vpci_add_handlers(struct pci_dev *dev); /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); +void vpci_remove_device_locked(struct pci_dev *pdev); /* Add/remove a register handler. Must be called holding the vpci_lock. */ int __must_check vpci_add_register(struct vpci *vpci, @@ -71,6 +72,7 @@ struct vpci { /* Guest view of the BAR: address and lower bits. */ uint64_t guest_reg; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -143,9 +145,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + bool map_pending : 1; bool rom_only : 1; }; From patchwork Fri Feb 4 06:34:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2FC1EC433FE for ; Fri, 4 Feb 2022 06:35:26 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265011.458347 (Exim 4.92) (envelope-from ) id 1nFsBe-0001xn-7F; Fri, 04 Feb 2022 06:35:14 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265011.458347; Fri, 04 Feb 2022 06:35:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBd-0001vu-Ru; Fri, 04 Feb 2022 06:35:13 +0000 Received: by outflank-mailman (input) for mailman id 265011; Fri, 04 Feb 2022 06:35:11 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBb-0008DX-OD for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:11 +0000 Received: from mail-lj1-x236.google.com (mail-lj1-x236.google.com [2a00:1450:4864:20::236]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 99ffe2d3-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:10 +0100 (CET) Received: by mail-lj1-x236.google.com with SMTP id q22so7161844ljh.7 for ; Thu, 03 Feb 2022 22:35:10 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:09 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 99ffe2d3-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qtkK/j8pu6kupq3RbAcpqc3jfOmxqW4/nLCIHyxmPhw=; b=LcavmegIjsn1jtf8L7reZvcGqNqvIimlLW1FC6Ke96AWMS6A1lu/GmYLd/9I1A7eOw jPIME4KoFaZrtnuk6in1pPcQT5stbkymritBMZsl10FZgWzoZTCXcMxagp84DfsZihPO RGHQ6aJu9q5yYUUc8Qwq1qITZw9XC9KqwLeO0nqJU9/MdYb4p4qLnWVVbCeoOb9I/hII Vg3reIRVkq4BiSLDhBh/LZekJM9WKWvbbrreZwgvowYHpG5iDUpcB4hlJHFtciVWbF+k cSiUsa8gnlvcsBaOxRu//oTbG897/xj+hDT9fih5k9WgdSxNR6SZxcByWKamdK0yQ+MJ ZKyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qtkK/j8pu6kupq3RbAcpqc3jfOmxqW4/nLCIHyxmPhw=; b=krTo55hg/qcIHx41WTWoZt7hP9NsAKZMFtlrsVKMnuntoAJt/SBKh89r7BXIwGnspl ej1QcDRJWTZ4Ka7ySxPSMMJ7b7KcsTnOWu19Lz3mTgEDrRYV7alJyDFd7f0YDTEg/SdK IyyMdhX0McVPh2rCDvwwMcd6zGwRjjn52jo/DT4vDxz2m23xHwVm1dbw2iGoX4JqUijR oP3qF/FTnHncgFxpkSFm6CLPJhdP+dZCEYB02tvvckYTitkplsL1XLCBfdj6KjozDuVH aATIxqQK71FLpYs0v50KNtj0HRslrF7OtL7pZxeKyIHRABMz18f/yZdHzrokNzESG+Pj F/Rg== X-Gm-Message-State: AOAM5315kovfUAmtwPGswe5OzrrnAydL11URkgEWGXeEBOCS0yH4Q3ED eafc/+FSPEZ4bfCHvN84WyzRN+XyiZ0= X-Google-Smtp-Source: ABdhPJxTXPZEdj2jvQlmavhyN/FVUW2Ujue3KbH3gBqvpnm6iioAFpcnFab7qHHhSkjZUArvQ7YeEw== X-Received: by 2002:a2e:a607:: with SMTP id v7mr936685ljp.248.1643956510257; Thu, 03 Feb 2022 22:35:10 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 08/13] vpci/header: program p2m with guest BAR view Date: Fri, 4 Feb 2022 08:34:54 +0200 Message-Id: <20220204063459.680961-9-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the PCI bus driver in the hardware domain. This way hardware domain sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - remove debug print in map_range callback - remove "identity" from the debug print Since v4: - moved start_{gfn|mfn} calculation into map_range - pass vpci_bar in the map_data instead of start_{gfn|mfn} - s/guest_addr/guest_reg Since v3: - updated comment (Roger) - removed gfn_add(map->start_gfn, rc); which is wrong - use v->domain instead of v->vpci.pdev->domain - removed odd e.g. in comment - s/d%d/%pd in altered code - use gdprintk for map/unmap logs Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 0c94504b87d8..88ca1ad8211d 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,7 @@ struct map_data { struct domain *d; + const struct vpci_bar *bar; bool map; }; @@ -41,8 +42,21 @@ static int map_range(unsigned long s, unsigned long e, void *data, for ( ; ; ) { + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn = _gfn(PFN_DOWN(is_hardware_domain(map->d) + ? map->bar->addr + : map->bar->guest_reg)); + /* Physical start address of the BAR. */ + mfn_t start_mfn = _mfn(PFN_DOWN(map->bar->addr)); unsigned long size = e - s + 1; + /* + * Ranges to be mapped don't always start at the BAR start address, as + * there can be holes or partially consumed ranges. Account for the + * offset of the current address from the BAR start. + */ + start_gfn = gfn_add(start_gfn, s - mfn_x(start_mfn)); + /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +66,8 @@ static int map_range(unsigned long s, unsigned long e, void *data, * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -62,8 +76,8 @@ static int map_range(unsigned long s, unsigned long e, void *data, if ( rc < 0 ) { printk(XENLOG_G_WARNING - "Failed to identity %smap [%lx, %lx] for d%d: %d\n", - map->map ? "" : "un", s, e, map->d->domain_id, rc); + "Failed to %smap [%lx, %lx] for %pd: %d\n", + map->map ? "" : "un", s, e, map->d, rc); break; } ASSERT(rc < size); @@ -154,6 +168,7 @@ bool vpci_process_pending(struct vcpu *v) if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -206,6 +221,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) process_pending_softirqs(); From patchwork Fri Feb 4 06:34:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DFA9BC433EF for ; Fri, 4 Feb 2022 06:35:27 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265012.458353 (Exim 4.92) (envelope-from ) id 1nFsBe-000255-SQ; Fri, 04 Feb 2022 06:35:14 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265012.458353; Fri, 04 Feb 2022 06:35:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBe-00022M-Hy; Fri, 04 Feb 2022 06:35:14 +0000 Received: by outflank-mailman (input) for mailman id 265012; Fri, 04 Feb 2022 06:35:12 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBc-0008DX-K9 for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:12 +0000 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [2a00:1450:4864:20::229]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9aa7edd9-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:11 +0100 (CET) Received: by mail-lj1-x229.google.com with SMTP id o17so7205812ljp.1 for ; Thu, 03 Feb 2022 22:35:11 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:11 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9aa7edd9-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kqe4MtyzAvOGHGKuf3R4wkGH8c0Z7HCCzvLS5umkU1o=; b=aZkmCJKMeQMCzE2uO2NL3NExo6pehR9hggGJIxvr10ezeFkscLBRLAEIPtGHDIlsqF FveGForAVCWOyJlL3/TNTCBHOHo+Q0iyqVUt9stcPGLF6Mxk2yWgzk7Udh7NSFfXHgSv mGKlJEtlUNpEpXKgcI78rxcdD7OZBG30y0EUavYCZkEZvoe9YJ+rCGi+CILkvgXO8xu/ wz9SJIAVk5jW9Ecnomi3T/yn2Mz+ym/5oytSRRlHKvzfNvb7mJPp89O6h5Roik30WQOq uKJVOEQyY9bj7IyjoH31Nbhg+RkJveky4PRhP+/wsOWjwiHZxyRidGJ8s5r75BtavSse pkuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kqe4MtyzAvOGHGKuf3R4wkGH8c0Z7HCCzvLS5umkU1o=; b=AsEv10kFpOKydudwGvE8kQcOvV5LDlnubhJSs6Euvxc1+n1B+i5T2JVeFyu9cC2tYq 1QvjKcQOrAsNCVDl88fo55sOK/EzhN7oC7BdB6iZklEE1ZlvaJaJBPDENKr+4tqIgNEa KI/kTxt/Yv/lsIRphH2zhYfDtY5BvWxaM250zotaOC6sFA3gncKiyor/YLWeTmgNWcKB CJVUrBeEJpQ9imhPSK011AV6eMZ2RV0+5dPRfUKXRRmEYatG1qhOXn0aB7zk1qLfVA8u 8R103oAEH6o6c4/i/ENsxxulxWX4Oh/14J2lGNfN6JJxRSEHZO2YpXiKtsCgTsZstF3f A1Ng== X-Gm-Message-State: AOAM532O27H6J7PQHT8ljmdzqNDO1XcM1mJgOiFCJyWgx9RZbxb1PMxo 2SF8N9vkjE+G4tddnmN//7rxBa8WbtU= X-Google-Smtp-Source: ABdhPJyB7jlJZ7qysvRjbOwP2BLmD+X+2ZSuXBZ47oczLChBA1MjcsRUZxy9FgQ7GWx1wfWjjkim7w== X-Received: by 2002:a2e:bc1c:: with SMTP id b28mr936913ljf.40.1643956511398; Thu, 03 Feb 2022 22:35:11 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 09/13] vpci/header: emulate PCI_COMMAND register for guests Date: Fri, 4 Feb 2022 08:34:55 +0200 Message-Id: <20220204063459.680961-10-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add basic emulation support for guests. At the moment only emulate PCI_COMMAND_INTX_DISABLE bit, the rest is not emulated yet and left as TODO. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - add additional check for MSI-X enabled while altering INTX bit - make sure INTx disabled while guests enable MSI/MSI-X Since v3: - gate more code on CONFIG_HAS_MSI - removed logic for the case when MSI/MSI-X not enabled --- xen/drivers/vpci/header.c | 21 +++++++++++++++++++-- xen/drivers/vpci/msi.c | 4 ++++ xen/drivers/vpci/msix.c | 4 ++++ 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 88ca1ad8211d..33d8c15ae6e8 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -454,6 +454,22 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + /* TODO: Add proper emulation for all bits of the command register. */ + +#ifdef CONFIG_HAS_PCI_MSI + if ( pdev->vpci->msi->enabled || pdev->vpci->msix->enabled ) + { + /* Guest wants to enable INTx. It can't be enabled if MSI/MSI-X enabled. */ + cmd |= PCI_COMMAND_INTX_DISABLE; + } +#endif + + cmd_write(pdev, reg, cmd, data); +} + static void bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -661,8 +677,9 @@ static int init_bars(struct pci_dev *pdev) } /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, + is_hwdom ? cmd_write : guest_cmd_write, + PCI_COMMAND, 2, header); if ( rc ) return rc; diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index e3ce46869dad..90465dcb4831 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -70,6 +70,10 @@ static void control_write(const struct pci_dev *pdev, unsigned int reg, if ( vpci_msi_arch_enable(msi, pdev, vectors) ) return; + + /* Make sure guest doesn't enable INTx while enabling MSI. */ + if ( !is_hardware_domain(pdev->domain) ) + pci_intx(pdev, false); } else vpci_msi_arch_disable(msi, pdev); diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index d1dbfc6e0ffd..4c0e1836b589 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -92,6 +92,10 @@ static void control_write(const struct pci_dev *pdev, unsigned int reg, for ( i = 0; i < msix->max_entries; i++ ) if ( !msix->entries[i].masked && msix->entries[i].updated ) update_entry(&msix->entries[i], pdev, i); + + /* Make sure guest doesn't enable INTx while enabling MSI-X. */ + if ( !is_hardware_domain(pdev->domain) ) + pci_intx(pdev, false); } else if ( !new_enabled && msix->enabled ) { From patchwork Fri Feb 4 06:34:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C54F9C4332F for ; Fri, 4 Feb 2022 06:35:26 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265013.458362 (Exim 4.92) (envelope-from ) id 1nFsBg-0002K8-Bx; Fri, 04 Feb 2022 06:35:16 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265013.458362; Fri, 04 Feb 2022 06:35:16 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBf-0002Gc-DU; Fri, 04 Feb 2022 06:35:15 +0000 Received: by outflank-mailman (input) for mailman id 265013; Fri, 04 Feb 2022 06:35:14 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBd-0008DW-QM for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:13 +0000 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [2a00:1450:4864:20::130]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 9b59e52f-8584-11ec-8f75-fffcc8bd4f1a; Fri, 04 Feb 2022 07:35:13 +0100 (CET) Received: by mail-lf1-x130.google.com with SMTP id z19so10659451lfq.13 for ; Thu, 03 Feb 2022 22:35:13 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:12 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9b59e52f-8584-11ec-8f75-fffcc8bd4f1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FX99xFROYnnqXaxsrJbKwTBFYrf06bSi150/1Wew964=; b=cfEcGqDRT0H+u76W/kO7xJphKRiV6ogCbh4ol7qeJnPisV7JGCgkiVR1sSNGS4CRyM mh8fidCB4ulKXq5h9Smv+lSr4tSwufdbBKxBOxJcPOTv2tF3ohaLs5FRYsAZdqp/QwGU REu1w4eB5e1E3XbosGxE+NjRBsKrwEngfwtIf+UM+sTF2kwFk1mri4BqUPaT8yW/CgwY wLUctjH9xQn93iUyRBgB6cYJTIQC+GW/4RUPonIRmQDjYfRYEFd04iwKj2rr3TRN0hxF P6XUv5qj5Tv+x6thu/brLBbh6YBahufDoBOWqApfgdyuLFuffR4Tc94uaQ8eiDyUEQfX 9OJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FX99xFROYnnqXaxsrJbKwTBFYrf06bSi150/1Wew964=; b=Cjez70YkqzDjVsQK8Msb0C4Am3DQ2qpnAuwFVOTpzBIQSa6FTWpCU9DUsi0ZHcu/XK tbzMPBMFeVhQ9ZyGKjhu4/K6cFyJh9DFojCqYHMQnC00uGXnL5dBumqXXX1vS0XNFgS9 OwsuvmRhTbebroJN5z2CCeJPabLD/TNlMgT4ioqS1S8De9AhDF6uPHhXGTiuFU0FRjQQ XJM9ZCSs0GTOJANL/nCnFo5ONcEOvCcAC6XtHl1SpGuY+jpXoIlbdI791HrQMmOo1uI+ 2W1hHPkhwJxTe8aqHJAIa7tnLSFJWvOiCqxHCFWO2VZhphMXMgSS9LLUFn7fLGOxyacp FDig== X-Gm-Message-State: AOAM531tcuQp3moQqirUboK3mf56If3i32UNdnvRxeFhmMdvKFW0tVyr 0Af1W6h/iD9hcXBhCXJLozp1Q3syffY= X-Google-Smtp-Source: ABdhPJwon9dynpKYpjG0DMccYwOaOzKCe/ToTI783e4Ajn2KtH8i/5XPF0PCpVRt0DG4lKESp5cs0g== X-Received: by 2002:a05:6512:1082:: with SMTP id j2mr1237420lfg.371.1643956512552; Thu, 03 Feb 2022 22:35:12 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 10/13] vpci/header: reset the command register when adding devices Date: Fri, 4 Feb 2022 08:34:56 +0200 Message-Id: <20220204063459.680961-11-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Reset the command register when assigning a PCI device to a guest: according to the PCI spec the PCI_COMMAND register is typically all 0's after reset. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - updated commit message Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 33d8c15ae6e8..407fa2fc4749 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -454,8 +454,7 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } -static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, - uint32_t cmd, void *data) +static uint32_t emulate_cmd_reg(const struct pci_dev *pdev, uint32_t cmd) { /* TODO: Add proper emulation for all bits of the command register. */ @@ -467,7 +466,13 @@ static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, } #endif - cmd_write(pdev, reg, cmd, data); + return cmd; +} + +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + cmd_write(pdev, reg, emulate_cmd_reg(pdev, cmd), data); } static void bar_write(const struct pci_dev *pdev, unsigned int reg, @@ -676,6 +681,10 @@ static int init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } + /* Reset the command register for the guest. */ + if ( !is_hwdom ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, emulate_cmd_reg(pdev, 0)); + /* Setup a handler for the command register. */ rc = vpci_add_register(pdev->vpci, vpci_hw_read16, is_hwdom ? cmd_write : guest_cmd_write, From patchwork Fri Feb 4 06:34:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0AF8C433F5 for ; Fri, 4 Feb 2022 06:35:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265014.458370 (Exim 4.92) (envelope-from ) id 1nFsBh-0002hl-QP; Fri, 04 Feb 2022 06:35:17 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265014.458370; Fri, 04 Feb 2022 06:35:17 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBh-0002dg-El; Fri, 04 Feb 2022 06:35:17 +0000 Received: by outflank-mailman (input) for mailman id 265014; Fri, 04 Feb 2022 06:35:15 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBf-0008DX-6C for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:15 +0000 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [2a00:1450:4864:20::130]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9c0d2cc9-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:14 +0100 (CET) Received: by mail-lf1-x130.google.com with SMTP id o12so10664148lfg.12 for ; Thu, 03 Feb 2022 22:35:14 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:13 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9c0d2cc9-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tGFLOQCwkEQFlI9mFVvLvlv0pMnFdRny8N7SLfH9TJA=; b=YlgiD5lGb3evZ6RR1LePZhPhqzkgUVHIPb2Gy3O72gERgeDAmwDTYaB0XUarTQgsD8 pmdw7nrr7tOpf/VkdnbF851kb5Y8xpkUCSOphdAW69ua0+sdexi8ht29J6xt3629QdYw 8q421az1QnqIpTccWIu2jw9mRqZwIRJJQr9fjZjEvRHqTXgN99wxLu6k7zzlJ+iZexcK j131pVe1Bn0fM1phTaY1dA0B3H4DUjCi07cAHQi7H+H/6xGbYmNtrY/+dguki/2iv26P ps40BdLcc/ktXwgv/LnZ8VTYgZaZEO5Ys52gDX5rQ7GrNIjDR46fhSUVc4ThlvEPF0Vf kRFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tGFLOQCwkEQFlI9mFVvLvlv0pMnFdRny8N7SLfH9TJA=; b=3OSLR+PccHK9fCyIlslFJKh+x68f924fQVGXqIcwpFy68BovlCqqaRP9jKCunkKBna zw7Qok2HaYc3glIJ3iwYzRt+LV8XZwz1waeFrsFvxuD1f+eO0/Kd0EtgDbZmkdT5jkOf RwwWeEnBHB2jRm9qqlsNqB+rBDmYvE1IsODGRTbz+DDFD2/V4iRC+HXnJ6mz+wBXfnWS WzYO822evOxIbpjB2ZlacN44+dYukA0I0b8sWkxxoDhkf8l70WlRyn6j+yApRuMupTIc knl1hqKHhV55AQdpsCP5fHHxNAkLfmA6X/u6pLK1WKvjI5gHoA2PuQ/YN/JEKFndbbRS Ztjw== X-Gm-Message-State: AOAM530G2eU6sZ2vjGp6aIC62W9pCotfYBWy9qe2s03BqwTjMU+MqpBB wT3KG0hBdaK9Btd7cNgv+sHlSD17jgo= X-Google-Smtp-Source: ABdhPJwPzBAmRPwkr5ocrvbHVI7IaKNMChAL/CL0c1126BAXRRe8+v+tzkGTEQ3jLHc1eLnLZCqIZg== X-Received: by 2002:a05:6512:683:: with SMTP id t3mr1289026lfe.61.1643956513698; Thu, 03 Feb 2022 22:35:13 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 11/13] vpci: add initial support for virtual PCI bus topology Date: Fri, 4 Feb 2022 08:34:57 +0200 Message-Id: <20220204063459.680961-12-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Please note, that at the moment only function 0 of a multifunction device can be passed through. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - s/vpci_add_virtual_device/add_virtual_device and make it static - call add_virtual_device from vpci_assign_device and do not use REGISTER_VPCI_INIT machinery - add pcidevs_locked ASSERT - use DECLARE_BITMAP for vpci_dev_assigned_map Since v4: - moved and re-worked guest sbdf initializers - s/set_bit/__set_bit - s/clear_bit/__clear_bit - minor comment fix s/Virtual/Guest/ - added VPCI_MAX_VIRT_DEV constant (PCI_SLOT(~0) + 1) which will be used later for counting the number of MMIO handlers required for a guest (Julien) Since v3: - make use of VPCI_INIT - moved all new code to vpci.c which belongs to it - changed open-coded 31 to PCI_SLOT(~0) - added comments and code to reject multifunction devices with functions other than 0 - updated comment about vpci_dev_next and made it unsigned int - implement roll back in case of error while assigning/deassigning devices - s/dom%pd/%pd Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/drivers/vpci/vpci.c | 74 +++++++++++++++++++++++++++++++++++++++-- xen/include/xen/sched.h | 8 +++++ xen/include/xen/vpci.h | 11 ++++++ 3 files changed, 91 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 3177f13c1c22..7d422d11f83d 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -89,6 +89,9 @@ int vpci_add_handlers(struct pci_dev *pdev) return -ENOMEM; INIT_LIST_HEAD(&vpci->handlers); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + vpci->guest_sbdf.sbdf = ~0; +#endif spin_lock(&pdev->vpci_lock); pdev->vpci = vpci; @@ -114,6 +117,57 @@ int vpci_add_handlers(struct pci_dev *pdev) } #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +static int add_virtual_device(struct pci_dev *pdev) +{ + struct domain *d = pdev->domain; + pci_sbdf_t sbdf = { 0 }; + unsigned long new_dev_number; + + if ( is_hardware_domain(d) ) + return 0; + + ASSERT(pcidevs_locked()); + + /* + * Each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ + if ( pdev->info.is_extfn ) + { + gdprintk(XENLOG_ERR, "%pp: only function 0 passthrough supported\n", + &pdev->sbdf); + return -EOPNOTSUPP; + } + + new_dev_number = find_first_zero_bit(d->vpci_dev_assigned_map, + VPCI_MAX_VIRT_DEV); + if ( new_dev_number >= VPCI_MAX_VIRT_DEV ) + return -ENOSPC; + + __set_bit(new_dev_number, &d->vpci_dev_assigned_map); + + /* + * Both segment and bus number are 0: + * - we emulate a single host bridge for the guest, e.g. segment 0 + * - with bus 0 the virtual devices are seen as embedded + * endpoints behind the root complex + * + * TODO: add support for multi-function devices. + */ + sbdf.devfn = PCI_DEVFN(new_dev_number, 0); + pdev->vpci->guest_sbdf = sbdf; + + return 0; + +} + +static void vpci_remove_virtual_device(struct domain *d, + const struct pci_dev *pdev) +{ + __clear_bit(pdev->vpci->guest_sbdf.dev, &d->vpci_dev_assigned_map); + pdev->vpci->guest_sbdf.sbdf = ~0; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { @@ -124,8 +178,16 @@ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) rc = vpci_add_handlers(pdev); if ( rc ) - vpci_deassign_device(d, pdev); + goto fail; + + rc = add_virtual_device(pdev); + if ( rc ) + goto fail; + return 0; + + fail: + vpci_deassign_device(d, pdev); return rc; } @@ -135,7 +197,15 @@ void vpci_deassign_device(struct domain *d, struct pci_dev *pdev) if ( !has_vpci(d) ) return; - vpci_remove_device(pdev); + spin_lock(&pdev->vpci_lock); + if ( !pdev->vpci ) + goto done; + + vpci_remove_virtual_device(d, pdev); + vpci_remove_device_locked(pdev); + + done: + spin_unlock(&pdev->vpci_lock); } #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 37f78cc4c4c9..3c25e265eaa8 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -444,6 +444,14 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * The bitmap which shows which device numbers are already used by the + * virtual PCI bus topology and is used to assign a unique SBDF to the + * next passed through virtual PCI device. + */ + DECLARE_BITMAP(vpci_dev_assigned_map, VPCI_MAX_VIRT_DEV); +#endif #endif #ifdef CONFIG_HAS_PASSTHROUGH diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index f1f49db959c7..1f04d34a2369 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -21,6 +21,13 @@ typedef int vpci_register_init_t(struct pci_dev *dev); #define VPCI_ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12) +/* + * Maximum number of devices supported by the virtual bus topology: + * each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ +#define VPCI_MAX_VIRT_DEV (PCI_SLOT(~0) + 1) + #define REGISTER_VPCI_INIT(x, p) \ static vpci_register_init_t *const x##_entry \ __used_section(".data.vpci." p) = x @@ -140,6 +147,10 @@ struct vpci { struct vpci_arch_msix_entry arch; } entries[]; } *msix; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* Guest SBDF of the device. */ + pci_sbdf_t guest_sbdf; +#endif #endif }; From patchwork Fri Feb 4 06:34:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734826 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D775C43217 for ; Fri, 4 Feb 2022 06:35:31 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265015.458380 (Exim 4.92) (envelope-from ) id 1nFsBj-0002vd-Gw; Fri, 04 Feb 2022 06:35:19 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265015.458380; Fri, 04 Feb 2022 06:35:19 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBi-0002sJ-Mz; Fri, 04 Feb 2022 06:35:18 +0000 Received: by outflank-mailman (input) for mailman id 265015; Fri, 04 Feb 2022 06:35:16 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBg-0008DX-6Q for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:16 +0000 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [2a00:1450:4864:20::131]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9cb91094-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:15 +0100 (CET) Received: by mail-lf1-x131.google.com with SMTP id i34so10811258lfv.2 for ; Thu, 03 Feb 2022 22:35:15 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:14 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9cb91094-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hIaF8lkau0N10gIhOfyFkvM+7dDNU2pFNiSNz4Bb4DM=; b=BQx5h1U1XzMedlxkvKRWbAoeeDDDuxCa+NYZCC5EXXKvNVb57WykA7geNU2PwwfOZq 9boVsP/FKrdj5QijI5UOmqpiJ/108TlbnER9IZhZmy0RGl5IOE57CosP9uDTvIDuYDbo zHhfShTjr4qt57OIjhKs9SBko9C9BKzPn/3mYPs2jIZxyyICCymb8RGNt8go4CY8Z1Ok zqFmtbBlD3fbVKWSpC72QVUh+9sVJeSCaazwwX1bSLiwuIQHtE6UPH88FeAYFv/JNfLI d2nZcYcEhbTMz7QDE7HkDz/rjBl9F5R4X+98fpvxVKMfjXD2MOxz+NjAY1Zg3+QrrepP mIow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hIaF8lkau0N10gIhOfyFkvM+7dDNU2pFNiSNz4Bb4DM=; b=pzyEKVUMrHCayfrEwdRh9Jj5XXF2LSwKvnNWzy8vUVrrEokCJnW1xFL0sBcEkGuFQY WgONp4bO5QS10r49Vu+YLyfaIlsrh3iLpWJjPXnb90qJNOgIeRDwJkhfRGR++Ca2FTHT CqivcHGv3YCoWelvdVK2PhXaqThz5Di1GGCv+IMFWLIPpxkZFFxnVijyovIDG29o93y6 yhVI2aHj3LKgPj+urdFcui40mUppNBru9Bfbf6Lu7MimivbzKJkwegXARuI4vqaIXazK 27OEJxFme4eZD3UsvAXaJ/8XEuu35Ofk0gV1nI3pWWp/3zVwGRkkfw1sQ6AYZri70h4m h9cg== X-Gm-Message-State: AOAM533fIkCFEKVIDhuO6yVSKqE+lPh27A0VD26PBWehTmA7ofEI1ETf YDJRbJ8oO0EMhzV23I81smev6LTNSfw= X-Google-Smtp-Source: ABdhPJzInhEwoaUav36B1b6oDvhcMTtbaYic6ktmXAldzUwMXmLO5P4HKZ2/mUC2v+QPU1VKXYIkVw== X-Received: by 2002:a05:6512:1697:: with SMTP id bu23mr1338331lfb.392.1643956514817; Thu, 03 Feb 2022 22:35:14 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 12/13] xen/arm: translate virtual PCI bus topology for guests Date: Fri, 4 Feb 2022 08:34:58 +0200 Message-Id: <20220204063459.680961-13-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- Since v5: - add vpci_translate_virtual_device for #ifndef CONFIG_HAS_VPCI_GUEST_SUPPORT case to simplify ifdefery - add ASSERT(!is_hardware_domain(d)); to vpci_translate_virtual_device - reset output register on failed virtual SBDF translation Since v4: - indentation fixes - constify struct domain - updated commit message - updates to the new locking scheme (pdev->vpci_lock) Since v3: - revisit locking - move code to vpci.c Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/vpci.c | 17 +++++++++++++++++ xen/drivers/vpci/vpci.c | 29 +++++++++++++++++++++++++++++ xen/include/xen/vpci.h | 7 +++++++ 3 files changed, 53 insertions(+) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index a9fc5817f94e..84b2b068a0fe 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -41,6 +41,16 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + { + *r = ~0ul; + return 1; + } + if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -59,6 +69,13 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, struct pci_host_bridge *bridge = p; pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + return 1; + return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); } diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 7d422d11f83d..070db7391391 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -168,6 +168,35 @@ static void vpci_remove_virtual_device(struct domain *d, pdev->vpci->guest_sbdf.sbdf = ~0; } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool vpci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf) +{ + struct pci_dev *pdev; + + ASSERT(!is_hardware_domain(d)); + + for_each_pdev( d, pdev ) + { + bool found; + + spin_lock(&pdev->vpci_lock); + found = pdev->vpci && (pdev->vpci->guest_sbdf.sbdf == sbdf->sbdf); + spin_unlock(&pdev->vpci_lock); + + if ( found ) + { + /* Replace guest SBDF with the physical one. */ + *sbdf = pdev->sbdf; + return true; + } + } + + return false; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 1f04d34a2369..f6eb9f2051af 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -271,6 +271,7 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) /* Notify vPCI that device is assigned/de-assigned to/from guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev); void vpci_deassign_device(struct domain *d, struct pci_dev *pdev); +bool vpci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf); #else static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { @@ -280,6 +281,12 @@ static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) static inline void vpci_deassign_device(struct domain *d, struct pci_dev *pdev) { }; + +static inline bool vpci_translate_virtual_device(const struct domain *d, + pci_sbdf_t *sbdf) +{ + return false; +} #endif #endif From patchwork Fri Feb 4 06:34:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12734824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D72CCC433EF for ; Fri, 4 Feb 2022 06:35:30 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.265016.458386 (Exim 4.92) (envelope-from ) id 1nFsBl-0003Ff-62; Fri, 04 Feb 2022 06:35:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 265016.458386; Fri, 04 Feb 2022 06:35:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBk-0003BJ-E5; Fri, 04 Feb 2022 06:35:20 +0000 Received: by outflank-mailman (input) for mailman id 265016; Fri, 04 Feb 2022 06:35:17 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nFsBh-0008DX-6q for xen-devel@lists.xenproject.org; Fri, 04 Feb 2022 06:35:17 +0000 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [2a00:1450:4864:20::229]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 9d5fbd2f-8584-11ec-8eb8-a37418f5ba1a; Fri, 04 Feb 2022 07:35:16 +0100 (CET) Received: by mail-lj1-x229.google.com with SMTP id o17so7206043ljp.1 for ; Thu, 03 Feb 2022 22:35:16 -0800 (PST) Received: from a2klaptop.localdomain ([185.199.97.5]) by smtp.gmail.com with ESMTPSA id f34sm163027lfv.165.2022.02.03.22.35.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Feb 2022 22:35:15 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 9d5fbd2f-8584-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qo4xVV2IOrLU3ayXp+CLSEr+b1lCwCF0zrFQg2DftNA=; b=DW8T5+dKfc/hN9mnCARyk2APUMIqiKK4CcqjIN0Whced814pQ8sYHC/r342zGCp7Gw IBwogyUJD9Ll9XN/ABcSgRxIMazNdqgGYw6lOmF65gWB6mOnkOmc0Xgmhk34CQccPhNo GtIPYj5MTo4WiqPdTXiYUr2UdjamJFF8aitaHwtOtk2vcf/i52oQls8Sya0/R0AaXkAD CG0OaC/BimKmkIEF0RssbmfnB6KDFgkf0q/P4ZQO+yo0tTnJ4zF8wscY/MxfJWeKFx02 rbq+wZ/wpS2fq4B4fhEQYjVdTqlPpD0+jelO1aoh1jAhQbiFmzuHbzY9AGUMIDU4O5D+ ux1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qo4xVV2IOrLU3ayXp+CLSEr+b1lCwCF0zrFQg2DftNA=; b=maYQAknmYI3sU1lx/8bDsOmWYL6R0ezSWud7+oY2B3r4w3KJvkACpfyzb9dh8EGptg M3rd+t1mIV1vMUb0Ud1f332jOk4eugoERLoILfQWj8jKQVZptrVIdvGeXbpuegD2Sswn uIRLYRc6QXerWKAE8LEDxMwUIP83o1w9rLvyoC67uO9O8TF9XQ2Lrs3QljIuYzRXVqW0 MfTJoyy4kLtK2AKHGPTuuuuO/PBTkfzijiRuZdaJQ5JpGKSJTWTbuwDUSkwpVb7dRFf7 Jq3MztAZbU9NSzR7OBS5Erg3oOgg96w97bT+IV4IDaR5G4ZWKuazTGV/6yGkPwbnvZCL XRxw== X-Gm-Message-State: AOAM531hlDQUhcnH/lpWWjcPQxvq3UPhZy7CYUKvs3Fj6lg+jiYQ//bi l1wwBXmDBLkjs3tfyH8qoJ19xW0lioI= X-Google-Smtp-Source: ABdhPJwWj2gtNHZB0Bqn6McKeBHaWxFGAxFvXVwELlj4nHdWpNKInEIKgGnAdCyCL0new9sD8fO1VA== X-Received: by 2002:a2e:82c4:: with SMTP id n4mr893394ljh.163.1643956515947; Thu, 03 Feb 2022 22:35:15 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, artem_mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v6 13/13] xen/arm: account IO handlers for emulated PCI MSI-X Date: Fri, 4 Feb 2022 08:34:59 +0200 Message-Id: <20220204063459.680961-14-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220204063459.680961-1-andr2000@gmail.com> References: <20220204063459.680961-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko At the moment, we always allocate an extra 16 slots for IO handlers (see MAX_IO_HANDLER). So while adding IO trap handlers for the emulated MSI-X registers we need to explicitly tell that we have additional IO handlers, so those are accounted. Signed-off-by: Oleksandr Andrushchenko Acked-by: Julien Grall --- Cc: Julien Grall Cc: Stefano Stabellini --- This actually moved here from the part 2 of the prep work for PCI passthrough on Arm as it seems to be the proper place for it. Since v5: - optimize with IS_ENABLED(CONFIG_HAS_PCI_MSI) since VPCI_MAX_VIRT_DEV is defined unconditionally New in v5 --- xen/arch/arm/vpci.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 84b2b068a0fe..c5902cb9d34d 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -131,6 +131,8 @@ static int vpci_get_num_handlers_cb(struct domain *d, unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) { + unsigned int count; + if ( !has_vpci(d) ) return 0; @@ -151,7 +153,17 @@ unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) * For guests each host bridge requires one region to cover the * configuration space. At the moment, we only expose a single host bridge. */ - return 1; + count = 1; + + /* + * There's a single MSI-X MMIO handler that deals with both PBA + * and MSI-X tables per each PCI device being passed through. + * Maximum number of emulated virtual devices is VPCI_MAX_VIRT_DEV. + */ + if ( IS_ENABLED(CONFIG_HAS_PCI_MSI) ) + count += VPCI_MAX_VIRT_DEV; + + return count; } /*