From patchwork Thu Nov 25 11:02:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3CC08C433EF for ; Thu, 25 Nov 2021 11:03:14 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230874.399126 (Exim 4.92) (envelope-from ) id 1mqCWo-0001eS-LZ; Thu, 25 Nov 2021 11:02:58 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230874.399126; Thu, 25 Nov 2021 11:02:58 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWo-0001d0-FB; Thu, 25 Nov 2021 11:02:58 +0000 Received: by outflank-mailman (input) for mailman id 230874; Thu, 25 Nov 2021 11:02:56 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWm-0001Km-TI for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:02:56 +0000 Received: from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com [2a00:1450:4864:20::12d]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 3e3d595f-4ddf-11ec-9787-a32c541c8605; Thu, 25 Nov 2021 12:02:56 +0100 (CET) Received: by mail-lf1-x12d.google.com with SMTP id b40so15315526lfv.10 for ; Thu, 25 Nov 2021 03:02:55 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.02.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:02:54 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3e3d595f-4ddf-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RCl87Let7v1piBQuF1AwGWgI3o9wVtyo4nCoCPCgQzo=; b=S3Ibnsg8Hp3Woi8/hIXa47CEs0zstSsM52a2powxFZx/nQ3ysic3FE5vVAAI6LSPu4 /bQooWMpp1ftaWMQ7weQIQD58y1c/y8GCvF5SBTE0SflwquVXE1rwiEq3iwAbwmMmxby 2SjioFB9nbWIRc6BPhiahe4udlZB+E2LpacABoDsxLEpZuJKKzRlONFIp9+y8b2uTk6m P9BLS6Sd5tv+OiY+UkAAZYRMCSKeDwsvFJT4TYCVoVrmt8lXS6VQvMcLUAmXzlGs3woD Seffm80pcxa7hKgQpf++cSfKgg3ocUg+HMyPKEmzlHeaz0lGaojRl1kvN5hKFWweYx3G zYvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RCl87Let7v1piBQuF1AwGWgI3o9wVtyo4nCoCPCgQzo=; b=lbwerSWHT+fD6nXHb8auOHrlayHHwMP7iU4VgFogsMjOfaZ5CM2Q3dr+3BU4RWFYFn kEhguVLEYEoWZ1UPvKcVIEUQ3bei9KdmZgMLVoz7eJRkKB2B0EDpoE10E0b8uJq6dz/f FABruD6vRf2zTqU5lWvC0L+OVXw9kKZrlDrO/SRzi4nZlHa5XpBFvUh6esddgG34o47N IJx3NGc24qX6iQZhnHtqxLOHRlNvw4/MpIUQzt6BXzdhOpBUhrMd/+PwkpXvCuOJRvC+ xfPNqynY/ajVzsvp2WOhnKxVGvy66gOGz1RiohW0OngT9ibB/IZpvDqfniHGrV+R+/zl C65w== X-Gm-Message-State: AOAM533oz5WApc/uFIPXs78L8paS18pX21yhsoP6gn/dQQpiSYPzeyoz 4N+jdZa32/w0PC4BvAaTDPpcXn7EMbB+GA== X-Google-Smtp-Source: ABdhPJxRnShat0Qo6e0njAtYNAACw/nqPCn7aYjsxR9gy+3mgzPQ4pkCvYI87GgT3WFpxBgseaGzjA== X-Received: by 2002:a05:6512:3a90:: with SMTP id q16mr22510651lfu.135.1637838175293; Thu, 25 Nov 2021 03:02:55 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 01/14] rangeset: add RANGESETF_no_print flag Date: Thu, 25 Nov 2021 13:02:38 +0200 Message-Id: <20211125110251.2877218-2-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are range sets which should not be printed, so introduce a flag which allows marking those as such. Implement relevant logic to skip such entries while printing. While at it also simplify the definition of the flags by directly defining those without helpers. Suggested-by: Jan Beulich Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Jan Beulich --- Since v1: - update BUG_ON with new flag - simplify the definition of the flags --- xen/common/rangeset.c | 5 ++++- xen/include/xen/rangeset.h | 7 ++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/xen/common/rangeset.c b/xen/common/rangeset.c index 885b6b15c229..ea27d651723b 100644 --- a/xen/common/rangeset.c +++ b/xen/common/rangeset.c @@ -433,7 +433,7 @@ struct rangeset *rangeset_new( INIT_LIST_HEAD(&r->range_list); r->nr_ranges = -1; - BUG_ON(flags & ~RANGESETF_prettyprint_hex); + BUG_ON(flags & ~(RANGESETF_prettyprint_hex | RANGESETF_no_print)); r->flags = flags; safe_strcpy(r->name, name ?: "(no name)"); @@ -575,6 +575,9 @@ void rangeset_domain_printk( list_for_each_entry ( r, &d->rangesets, rangeset_list ) { + if ( r->flags & RANGESETF_no_print ) + continue; + printk(" "); rangeset_printk(r); printk("\n"); diff --git a/xen/include/xen/rangeset.h b/xen/include/xen/rangeset.h index 135f33f6066f..045fcafa8368 100644 --- a/xen/include/xen/rangeset.h +++ b/xen/include/xen/rangeset.h @@ -48,9 +48,10 @@ void rangeset_limit( struct rangeset *r, unsigned int limit); /* Flags for passing to rangeset_new(). */ - /* Pretty-print range limits in hexadecimal. */ -#define _RANGESETF_prettyprint_hex 0 -#define RANGESETF_prettyprint_hex (1U << _RANGESETF_prettyprint_hex) +/* Pretty-print range limits in hexadecimal. */ +#define RANGESETF_prettyprint_hex (1U << 0) +/* Do not print entries marked with this flag. */ +#define RANGESETF_no_print (1U << 1) bool_t __must_check rangeset_is_empty( const struct rangeset *r); From patchwork Thu Nov 25 11:02:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37094C43219 for ; Thu, 25 Nov 2021 11:03:13 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230875.399133 (Exim 4.92) (envelope-from ) id 1mqCWp-0001po-5O; Thu, 25 Nov 2021 11:02:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230875.399133; Thu, 25 Nov 2021 11:02:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWp-0001nR-0t; Thu, 25 Nov 2021 11:02:59 +0000 Received: by outflank-mailman (input) for mailman id 230875; Thu, 25 Nov 2021 11:02:58 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWn-0001K8-Ua for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:02:57 +0000 Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [2a00:1450:4864:20::136]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 3efb707d-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:02:57 +0100 (CET) Received: by mail-lf1-x136.google.com with SMTP id n12so15409277lfe.1 for ; Thu, 25 Nov 2021 03:02:57 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.02.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:02:56 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3efb707d-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yH9Aud+KhB6bv78AcEs/8owAaYp6cfId4snOD7ig8jw=; b=H6hIR2MODuSq0+XyMmOSWqwDEGA5m/qhgk4HpL+cT36B24u7WSDtOZtFBD5Gonc6X9 A2YcV5BwlB5Arrn4G5EifURgaMBJqRZK8IRn5rm1oxoWty2cm/lH5axD2Bi6FdQqJhfI s8qjpY3ANqiUyRszD9eE7H7yP0+Deek5MWp0QAac7sgjB5VtWIUFdHKHBq8ZGt93qqG1 229RojPvgxEpATw+xVDtgFzS7UN5g8j15TDI7tAPT2ETs3oQiRLNXQYqC4XFoP36FqVG DGylLfdG1xpKlBqdIMj5/gGzzv1srJRhrelqRMkgOM5akHquECv4ubNECaLzkUNhTwbD fsyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yH9Aud+KhB6bv78AcEs/8owAaYp6cfId4snOD7ig8jw=; b=5TBQH/0BD50HCTaJQc+hATh1jYbCJMgL9gpVy/C0JcAMwh8RSXxB36dqeX9Uag0PuZ LSBxj9BcnHzH9fjTxCh6TPoqomTKP5TbjP9bW5lo/IhCIFYjFE5sBJ5/g0lMy7zwwNBb qEI+aJ2Fpqh+DPNABcIFK2+hdcPUntW+yNkKKS0ElAuoXYfjQV0YPxu1D5EnJeeLptsi 3Rzc51ms4fNUHJCaNO6bNFJ8WYkAC6uik/ljhLQ423nVrLAsg9yfaV5eBWr2Z5w8X72r K+4YeL1xwxcWKQN10rly6YBZT83ePGUAPekYOsjJIaMRv+DYckraUjAbtemaaU0szYRE FZ7A== X-Gm-Message-State: AOAM532IPiFv6vc+m56L1GiasQapoRNV6bJn+3GEXS9o4MRcH7utmVpY F436DumOkBAh26eKy9U9ugLDgjy4WbzZZQ== X-Google-Smtp-Source: ABdhPJy61wspGeTsK95YYXpRFa8RWw8M9gAX8WvjKHapgNGt7DEmMrCdL8E19Upw8eZ8yKk+WomS5Q== X-Received: by 2002:ac2:5cc7:: with SMTP id f7mr16096375lfq.412.1637838176471; Thu, 25 Nov 2021 03:02:56 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 02/14] vpci: fix function attributes for vpci_process_pending Date: Thu, 25 Nov 2021 13:02:39 +0200 Message-Id: <20211125110251.2877218-3-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko vpci_process_pending is defined with different attributes, e.g. with __must_check if CONFIG_HAS_VPCI enabled and not otherwise. Fix this by defining both of the definitions with __must_check. Fixes: 14583a590783 ("7fbb096bf345 kconfig: don't select VPCI if building a shim-only binary") Signed-off-by: Oleksandr Andrushchenko Reviewed-by: Julien Grall Reviewed-by: Roger Pau Monné --- Cc: Roger Pau Monné New in v4 --- xen/include/xen/vpci.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 9ea66e033f11..3f32de9d7eb3 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -247,7 +247,7 @@ static inline void vpci_write(pci_sbdf_t sbdf, unsigned int reg, ASSERT_UNREACHABLE(); } -static inline bool vpci_process_pending(struct vcpu *v) +static inline bool __must_check vpci_process_pending(struct vcpu *v) { ASSERT_UNREACHABLE(); return false; From patchwork Thu Nov 25 11:02:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FA76C433FE for ; Thu, 25 Nov 2021 11:03:12 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230876.399153 (Exim 4.92) (envelope-from ) id 1mqCWr-0002R6-GO; Thu, 25 Nov 2021 11:03:01 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230876.399153; Thu, 25 Nov 2021 11:03:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWr-0002Qt-Bg; Thu, 25 Nov 2021 11:03:01 +0000 Received: by outflank-mailman (input) for mailman id 230876; Thu, 25 Nov 2021 11:03:00 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWp-0001K8-V5 for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:00 +0000 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [2a00:1450:4864:20::12b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 3fc44399-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:02:58 +0100 (CET) Received: by mail-lf1-x12b.google.com with SMTP id bi37so15338121lfb.5 for ; Thu, 25 Nov 2021 03:02:58 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.02.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:02:57 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3fc44399-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SIXE3W6i3s/PLOl0AkNDVroqqsWXQtCAA5ldmwUu770=; b=D+5JaGFFqqHmDxTsOOorPtGxHkEZFwzBu8u0Ib/1AIjClZhQfGcgdZsbeG+F4KlAu0 qfUdHXdlpt6JhltuKb/SjBVBPmF8vEYJ+joBuI5hJ5hdz+4DdY6ZWSMr0rOD6UDsMKgL ELc3zM3CIXiKloq8zIrjfDrJBzJUS8uTOjHtNP3eUJTh/80ICApR5fR1yUDIRyhQa4ls jmcT7Z15nSOicM6ugh1X1Qwga7Xs5KbM2p8XmfCUKXLMmQ828Sw78qJHUhNYCwIG0UGH Pa6jafKpRcOXpuX8upViqmUOSgeVf4LAWv8f8OL1y1v6SwYo3wNVDw338hvfJCumvgao DL6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SIXE3W6i3s/PLOl0AkNDVroqqsWXQtCAA5ldmwUu770=; b=CpK9TANEogQGpDTWAhuQMjhu3lLloaaQrUe3xrORJAdE/Hv2nwkMDih7126A9gRAl6 AgK70YzpRf1l6Dr2XRo7hzCdo2vFhhqAtCc/AFdfQonnXeP6OW6EwCNCTnvrH/sKoezD DsTogGG7CX+SSd+RAbucWPXpczKsCDCE3osf/XdzX05xamtsYZw4rv6h4ky3dkZSMmbJ ZWqLxgOQxD62rtU4eVSFCSi0m0n3PGGDJGfh+BirX8/Tc/qtvQS6SIugSto/ua0//Htm NlAs9d6/u12X0S7ZAXgAaYFJ9+zUqXt63FhFtCg4R8uyFtgssoETOhrfp6fxRWg/t2OF tWog== X-Gm-Message-State: AOAM531VcCz0ACZ/nzL7FNt61J1M+OFcm1FuDuMHm2ueWSyTd4W+DCcz FtIKuTHWzTT7vvokVlKjFidkCUS1LvNs8dIF X-Google-Smtp-Source: ABdhPJx09MIvNVbWsL5ybmB16oQuTeaKVlA/ijtzUP+FwCqO601HhcDh+t2TR1dIzqqxcX0aPTnfoQ== X-Received: by 2002:a05:6512:39ca:: with SMTP id k10mr23315305lfu.426.1637838177842; Thu, 25 Nov 2021 03:02:57 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko , Ian Jackson Subject: [PATCH v5 03/14] vpci: move lock outside of struct vpci Date: Thu, 25 Nov 2021 13:02:40 +0200 Message-Id: <20211125110251.2877218-4-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Roger Pau Monne This way the lock can be used to check whether vpci is present, and removal can be performed while holding the lock, in order to make sure there are no accesses to the contents of the vpci struct. Previously removal could race with vpci_read for example, since the lock was dropped prior to freeing pdev->vpci. Signed-off-by: Roger Pau Monné Signed-off-by: Oleksandr Andrushchenko --- Cc: Andrew Cooper Cc: Ian Jackson Cc: Jan Beulich Cc: Julien Grall Cc: Stefano Stabellini --- New in v5 of this series: this is an updated version of the patch published at https://lore.kernel.org/xen-devel/20180717094830.54806-2-roger.pau@citrix.com/ Changes since v2: - fixed pdev->vpci = xzalloc(struct vpci); under spin_lock (Jan) Changes since v1: - Assert that vpci_lock is locked in vpci_remove_device_locked. - Remove double newline. - Shrink critical section in vpci_{read/write}. --- tools/tests/vpci/emul.h | 5 ++- tools/tests/vpci/main.c | 4 +-- xen/arch/x86/hvm/vmsi.c | 8 ++--- xen/drivers/passthrough/pci.c | 1 + xen/drivers/vpci/header.c | 21 +++++++---- xen/drivers/vpci/msi.c | 11 ++++-- xen/drivers/vpci/msix.c | 8 ++--- xen/drivers/vpci/vpci.c | 68 +++++++++++++++++++++++------------ xen/include/xen/pci.h | 1 + xen/include/xen/vpci.h | 5 +-- 10 files changed, 85 insertions(+), 47 deletions(-) diff --git a/tools/tests/vpci/emul.h b/tools/tests/vpci/emul.h index 2e1d3057c9d8..d018fb5eef21 100644 --- a/tools/tests/vpci/emul.h +++ b/tools/tests/vpci/emul.h @@ -44,6 +44,7 @@ struct domain { }; struct pci_dev { + bool vpci_lock; struct vpci *vpci; }; @@ -53,10 +54,8 @@ struct vcpu }; extern const struct vcpu *current; -extern const struct pci_dev test_pdev; +extern struct pci_dev test_pdev; -typedef bool spinlock_t; -#define spin_lock_init(l) (*(l) = false) #define spin_lock(l) (*(l) = true) #define spin_unlock(l) (*(l) = false) diff --git a/tools/tests/vpci/main.c b/tools/tests/vpci/main.c index b9a0a6006bb9..26c95b08b6b1 100644 --- a/tools/tests/vpci/main.c +++ b/tools/tests/vpci/main.c @@ -23,7 +23,8 @@ static struct vpci vpci; const static struct domain d; -const struct pci_dev test_pdev = { +struct pci_dev test_pdev = { + .vpci_lock = false, .vpci = &vpci, }; @@ -158,7 +159,6 @@ main(int argc, char **argv) int rc; INIT_LIST_HEAD(&vpci.handlers); - spin_lock_init(&vpci.lock); VPCI_ADD_REG(vpci_read32, vpci_write32, 0, 4, r0); VPCI_READ_CHECK(0, 4, r0); diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index 13e2a190b439..1f7a37f78264 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -910,14 +910,14 @@ int vpci_msix_arch_print(const struct vpci_msix *msix) { struct pci_dev *pdev = msix->pdev; - spin_unlock(&msix->pdev->vpci->lock); + spin_unlock(&msix->pdev->vpci_lock); process_pending_softirqs(); /* NB: we assume that pdev cannot go away for an alive domain. */ - if ( !pdev->vpci || !spin_trylock(&pdev->vpci->lock) ) + if ( !spin_trylock(&pdev->vpci_lock) ) return -EBUSY; - if ( pdev->vpci->msix != msix ) + if ( !pdev->vpci || pdev->vpci->msix != msix ) { - spin_unlock(&pdev->vpci->lock); + spin_unlock(&pdev->vpci_lock); return -EAGAIN; } } diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index a9d31293ac09..286808b25e65 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -328,6 +328,7 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) *((u8*) &pdev->bus) = bus; *((u8*) &pdev->devfn) = devfn; pdev->domain = NULL; + spin_lock_init(&pdev->vpci_lock); arch_pci_init_pdev(pdev); diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 40ff79c33f8f..bd23c0274d48 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -142,12 +142,13 @@ bool vpci_process_pending(struct vcpu *v) if ( rc == -ERESTART ) return true; - spin_lock(&v->vpci.pdev->vpci->lock); - /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, - !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci->lock); + spin_lock(&v->vpci.pdev->vpci_lock); + if ( v->vpci.pdev->vpci ) + /* Disable memory decoding unconditionally on failure. */ + modify_decoding(v->vpci.pdev, + rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, + !rc && v->vpci.rom_only); + spin_unlock(&v->vpci.pdev->vpci_lock); rangeset_destroy(v->vpci.mem); v->vpci.mem = NULL; @@ -285,6 +286,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) continue; } + spin_lock(&tmp->vpci_lock); + if ( !tmp->vpci ) + { + spin_unlock(&tmp->vpci_lock); + continue; + } for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ ) { const struct vpci_bar *bar = &tmp->vpci->header.bars[i]; @@ -303,12 +310,14 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) rc = rangeset_remove_range(mem, start, end); if ( rc ) { + spin_unlock(&tmp->vpci_lock); printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); rangeset_destroy(mem); return rc; } } + spin_unlock(&tmp->vpci_lock); } ASSERT(dev); diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index 5757a7aed20f..e3ce46869dad 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -270,7 +270,7 @@ void vpci_dump_msi(void) rcu_read_lock(&domlist_read_lock); for_each_domain ( d ) { - const struct pci_dev *pdev; + struct pci_dev *pdev; if ( !has_vpci(d) ) continue; @@ -282,8 +282,13 @@ void vpci_dump_msi(void) const struct vpci_msi *msi; const struct vpci_msix *msix; - if ( !pdev->vpci || !spin_trylock(&pdev->vpci->lock) ) + if ( !spin_trylock(&pdev->vpci_lock) ) continue; + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + continue; + } msi = pdev->vpci->msi; if ( msi && msi->enabled ) @@ -323,7 +328,7 @@ void vpci_dump_msi(void) } } - spin_unlock(&pdev->vpci->lock); + spin_unlock(&pdev->vpci_lock); process_pending_softirqs(); } } diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 846f1b8d7038..5310cc3ff520 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -225,7 +225,7 @@ static int msix_read(struct vcpu *v, unsigned long addr, unsigned int len, return X86EMUL_OKAY; } - spin_lock(&msix->pdev->vpci->lock); + spin_lock(&msix->pdev->vpci_lock); entry = get_entry(msix, addr); offset = addr & (PCI_MSIX_ENTRY_SIZE - 1); @@ -254,7 +254,7 @@ static int msix_read(struct vcpu *v, unsigned long addr, unsigned int len, ASSERT_UNREACHABLE(); break; } - spin_unlock(&msix->pdev->vpci->lock); + spin_unlock(&msix->pdev->vpci_lock); return X86EMUL_OKAY; } @@ -297,7 +297,7 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, return X86EMUL_OKAY; } - spin_lock(&msix->pdev->vpci->lock); + spin_lock(&msix->pdev->vpci_lock); entry = get_entry(msix, addr); offset = addr & (PCI_MSIX_ENTRY_SIZE - 1); @@ -370,7 +370,7 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, ASSERT_UNREACHABLE(); break; } - spin_unlock(&msix->pdev->vpci->lock); + spin_unlock(&msix->pdev->vpci_lock); return X86EMUL_OKAY; } diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 657697fe3406..ceaac4516ff8 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -35,12 +35,10 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) -void vpci_remove_device(struct pci_dev *pdev) +static void vpci_remove_device_handlers_locked(struct pci_dev *pdev) { - if ( !has_vpci(pdev->domain) ) - return; + ASSERT(spin_is_locked(&pdev->vpci_lock)); - spin_lock(&pdev->vpci->lock); while ( !list_empty(&pdev->vpci->handlers) ) { struct vpci_register *r = list_first_entry(&pdev->vpci->handlers, @@ -50,15 +48,33 @@ void vpci_remove_device(struct pci_dev *pdev) list_del(&r->node); xfree(r); } - spin_unlock(&pdev->vpci->lock); +} + +void vpci_remove_device_locked(struct pci_dev *pdev) +{ + ASSERT(spin_is_locked(&pdev->vpci_lock)); + + vpci_remove_device_handlers_locked(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); pdev->vpci = NULL; } +void vpci_remove_device(struct pci_dev *pdev) +{ + if ( !has_vpci(pdev->domain) ) + return; + + spin_lock(&pdev->vpci_lock); + if ( pdev->vpci ) + vpci_remove_device_locked(pdev); + spin_unlock(&pdev->vpci_lock); +} + int vpci_add_handlers(struct pci_dev *pdev) { + struct vpci *vpci; unsigned int i; int rc = 0; @@ -68,12 +84,13 @@ int vpci_add_handlers(struct pci_dev *pdev) /* We should not get here twice for the same device. */ ASSERT(!pdev->vpci); - pdev->vpci = xzalloc(struct vpci); - if ( !pdev->vpci ) + vpci = xzalloc(struct vpci); + if ( !vpci ) return -ENOMEM; + spin_lock(&pdev->vpci_lock); + pdev->vpci = vpci; INIT_LIST_HEAD(&pdev->vpci->handlers); - spin_lock_init(&pdev->vpci->lock); for ( i = 0; i < NUM_VPCI_INIT; i++ ) { @@ -83,7 +100,8 @@ int vpci_add_handlers(struct pci_dev *pdev) } if ( rc ) - vpci_remove_device(pdev); + vpci_remove_device_locked(pdev); + spin_unlock(&pdev->vpci_lock); return rc; } @@ -152,8 +170,6 @@ int vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, r->offset = offset; r->private = data; - spin_lock(&vpci->lock); - /* The list of handlers must be kept sorted at all times. */ list_for_each ( prev, &vpci->handlers ) { @@ -165,14 +181,12 @@ int vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, break; if ( cmp == 0 ) { - spin_unlock(&vpci->lock); xfree(r); return -EEXIST; } } list_add_tail(&r->node, prev); - spin_unlock(&vpci->lock); return 0; } @@ -183,7 +197,6 @@ int vpci_remove_register(struct vpci *vpci, unsigned int offset, const struct vpci_register r = { .offset = offset, .size = size }; struct vpci_register *rm; - spin_lock(&vpci->lock); list_for_each_entry ( rm, &vpci->handlers, node ) { int cmp = vpci_register_cmp(&r, rm); @@ -195,14 +208,12 @@ int vpci_remove_register(struct vpci *vpci, unsigned int offset, if ( !cmp && rm->offset == offset && rm->size == size ) { list_del(&rm->node); - spin_unlock(&vpci->lock); xfree(rm); return 0; } if ( cmp <= 0 ) break; } - spin_unlock(&vpci->lock); return -ENOENT; } @@ -311,7 +322,7 @@ static uint32_t merge_result(uint32_t data, uint32_t new, unsigned int size, uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) { const struct domain *d = current->domain; - const struct pci_dev *pdev; + struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; uint32_t data = ~(uint32_t)0; @@ -327,7 +338,12 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) if ( !pdev ) return vpci_read_hw(sbdf, reg, size); - spin_lock(&pdev->vpci->lock); + spin_lock(&pdev->vpci_lock); + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + return vpci_read_hw(sbdf, reg, size); + } /* Read from the hardware or the emulated register handlers. */ list_for_each_entry ( r, &pdev->vpci->handlers, node ) @@ -370,6 +386,7 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) break; ASSERT(data_offset < size); } + spin_unlock(&pdev->vpci_lock); if ( data_offset < size ) { @@ -379,7 +396,6 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) data = merge_result(data, tmp_data, size - data_offset, data_offset); } - spin_unlock(&pdev->vpci->lock); return data & (0xffffffff >> (32 - 8 * size)); } @@ -414,7 +430,7 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, uint32_t data) { const struct domain *d = current->domain; - const struct pci_dev *pdev; + struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; const unsigned long *ro_map = pci_get_ro_map(sbdf.seg); @@ -440,7 +456,14 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, return; } - spin_lock(&pdev->vpci->lock); + spin_lock(&pdev->vpci_lock); + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + vpci_write_hw(sbdf, reg, size, data); + return; + } + /* Write the value to the hardware or emulated registers. */ list_for_each_entry ( r, &pdev->vpci->handlers, node ) @@ -475,13 +498,12 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size, break; ASSERT(data_offset < size); } + spin_unlock(&pdev->vpci_lock); if ( data_offset < size ) /* Tailing gap, write the remaining. */ vpci_write_hw(sbdf, reg + data_offset, size - data_offset, data >> (data_offset * 8)); - - spin_unlock(&pdev->vpci->lock); } /* Helper function to check an access size and alignment on vpci space. */ diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index b6d7e454f814..3f60d6c6c6dd 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -134,6 +134,7 @@ struct pci_dev { u64 vf_rlen[6]; /* Data for vPCI. */ + spinlock_t vpci_lock; struct vpci *vpci; }; diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 3f32de9d7eb3..8b22bdef11d0 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -30,8 +30,9 @@ int __must_check vpci_add_handlers(struct pci_dev *dev); /* Remove all handlers and free vpci related structures. */ void vpci_remove_device(struct pci_dev *pdev); +void vpci_remove_device_locked(struct pci_dev *pdev); -/* Add/remove a register handler. */ +/* Add/remove a register handler. Must be called holding the vpci_lock. */ int __must_check vpci_add_register(struct vpci *vpci, vpci_read_t *read_handler, vpci_write_t *write_handler, @@ -60,7 +61,6 @@ bool __must_check vpci_process_pending(struct vcpu *v); struct vpci { /* List of vPCI handlers for a device. */ struct list_head handlers; - spinlock_t lock; #ifdef __XEN__ /* Hide the rest of the vpci struct from the user-space test harness. */ @@ -231,6 +231,7 @@ static inline int vpci_add_handlers(struct pci_dev *pdev) } static inline void vpci_remove_device(struct pci_dev *pdev) { } +static inline void vpci_remove_device_locked(struct pci_dev *pdev) { } static inline void vpci_dump_msi(void) { } From patchwork Thu Nov 25 11:02:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF649C43217 for ; Thu, 25 Nov 2021 11:03:12 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230877.399159 (Exim 4.92) (envelope-from ) id 1mqCWs-0002XH-0E; Thu, 25 Nov 2021 11:03:02 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230877.399159; Thu, 25 Nov 2021 11:03:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWr-0002Uk-PA; Thu, 25 Nov 2021 11:03:01 +0000 Received: by outflank-mailman (input) for mailman id 230877; Thu, 25 Nov 2021 11:03:00 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWq-0001Km-DK for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:00 +0000 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [2a00:1450:4864:20::130]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 40673b5c-4ddf-11ec-9787-a32c541c8605; Thu, 25 Nov 2021 12:02:59 +0100 (CET) Received: by mail-lf1-x130.google.com with SMTP id n12so15409529lfe.1 for ; Thu, 25 Nov 2021 03:02:59 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.02.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:02:58 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 40673b5c-4ddf-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7kzoaEt6IOOYgPm1yLVSMOcEeyel0wJP73UNzPtLoAI=; b=J7c62W6SQ1rWnkjJWulpfBcyPUfKdE/+Jcyqho4A2lXAbkjJISdmAyJhVhmrqhFfCD sBlU8pcYaH7OZ3oZwgKGmXYkcGqlEb8mwLaIN1O94wzuLcxO6dmd1kzNp73BbKX4hHkg UEYLMLifl8f88XrGdvTNmzNYIBolNacU+y+F49X4YAlT89s9x+WxyIRh1U4Q4DE0i5CQ 8c27FKLoYlVyoznJc2hLa5n9+ww6v6wGNIzr0lZS3sgt4wc8waC+f+afm9eV4pPuyFEw 7ByiCf05YD+K6KrDSi3HPV6IvOkcWMSWch+h98S8tJJ6GGNN9Ctm2561xOC/MRyDA1gF /NIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7kzoaEt6IOOYgPm1yLVSMOcEeyel0wJP73UNzPtLoAI=; b=y0Ra6Muia4tgAZmiofolkv8udmaacozHAPutY4yID9f1nRIA3xddzTOaeYhD2PAfl1 w9ZsLZ0YKOKqPHXLqsQDppOIYyY+YJ14uPkIVfZy/jv5xbaz8UG8C28DTtl5CeG7pM4p olRun4yMqsddZMBxJumjIbPbGbtMoHPwnfZNmJZLHvPaZ8gNgObse6Qp4WJgjMDqzZ69 7aSQ4p2L1vj7AW3gW51FTNOb0kjMo4myo9fA1TIbPak7eE5MgZtGqNU06Wktugby1cPM ypB0UTee9CTFNMtklGWyzw5+DDmBIJQIcXksxtv+w9L7GftrQaE8cVfHJALLzxXAuVc3 Y0CQ== X-Gm-Message-State: AOAM530RnILgj7r71xySDKuVG++hWfROKaEqoAD3DFr+/Dlll3QUkkXH X8sekm0YzW/LCiUKNgQRzG4Y2ceov17eQ3Da X-Google-Smtp-Source: ABdhPJwyW9PROrjGv8m7q1noHVv59lv+CBLisM5gGjx3LBOvlGDfmhbBF7oxhVinCIR6CAKyXD07SQ== X-Received: by 2002:ac2:4828:: with SMTP id 8mr23352757lft.332.1637838178981; Thu, 25 Nov 2021 03:02:58 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 04/14] vpci: cancel pending map/unmap on vpci removal Date: Thu, 25 Nov 2021 13:02:41 +0200 Message-Id: <20211125110251.2877218-5-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a vPCI is removed for a PCI device it is possible that we have scheduled a delayed work for map/unmap operations for that device. For example, the following scenario can illustrate the problem: pci_physdev_op pci_add_device init_bars -> modify_bars -> defer_map -> raise_softirq(SCHEDULE_SOFTIRQ) iommu_add_device <- FAILS vpci_remove_device -> xfree(pdev->vpci) leave_hypervisor_to_guest vpci_process_pending: v->vpci.mem != NULL; v->vpci.pdev->vpci == NULL For the hardware domain we continue execution as the worse that could happen is that MMIO mappings are left in place when the device has been deassigned. For unprivileged domains that get a failure in the middle of a vPCI {un}map operation we need to destroy them, as we don't know in which state the p2m is. This can only happen in vpci_process_pending for DomUs as they won't be allowed to call pci_add_device. Signed-off-by: Oleksandr Andrushchenko --- Cc: Roger Pau Monné --- Since v4: - crash guest domain if map/unmap operation didn't succeed - re-work vpci cancel work to cancel work on all vCPUs - use new locking scheme with pdev->vpci_lock New in v4 Fixes: 86dbcf6e30cb ("vpci: cancel pending map/unmap on vpci removal") --- Signed-off-by: Oleksandr Andrushchenko --- xen/drivers/vpci/header.c | 49 ++++++++++++++++++++++++++++++--------- xen/drivers/vpci/vpci.c | 2 ++ xen/include/xen/pci.h | 5 ++++ xen/include/xen/vpci.h | 6 +++++ 4 files changed, 51 insertions(+), 11 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index bd23c0274d48..ba333fb2f9b0 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -131,7 +131,13 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { - if ( v->vpci.mem ) + struct pci_dev *pdev = v->vpci.pdev; + + if ( !pdev ) + return false; + + spin_lock(&pdev->vpci_lock); + if ( !pdev->vpci_cancel_pending && v->vpci.mem ) { struct map_data data = { .d = v->domain, @@ -140,32 +146,53 @@ bool vpci_process_pending(struct vcpu *v) int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); if ( rc == -ERESTART ) + { + spin_unlock(&pdev->vpci_lock); return true; + } - spin_lock(&v->vpci.pdev->vpci_lock); - if ( v->vpci.pdev->vpci ) + if ( pdev->vpci ) /* Disable memory decoding unconditionally on failure. */ - modify_decoding(v->vpci.pdev, + modify_decoding(pdev, rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, !rc && v->vpci.rom_only); - spin_unlock(&v->vpci.pdev->vpci_lock); - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; if ( rc ) + { /* * FIXME: in case of failure remove the device from the domain. * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain will likely need to be - * killed in order to avoid leaking stale p2m mappings on - * failure. + * safe for Dom0, for DomUs the domain needs to be killed in order + * to avoid leaking stale p2m mappings on failure. */ - vpci_remove_device(v->vpci.pdev); + if ( is_hardware_domain(v->domain) ) + vpci_remove_device_locked(pdev); + else + domain_crash(v->domain); + } } + spin_unlock(&pdev->vpci_lock); return false; } +void vpci_cancel_pending_locked(struct pci_dev *pdev) +{ + struct vcpu *v; + + ASSERT(spin_is_locked(&pdev->vpci_lock)); + + /* Cancel any pending work now on all vCPUs. */ + for_each_vcpu( pdev->domain, v ) + { + if ( v->vpci.mem && (v->vpci.pdev == pdev) ) + { + rangeset_destroy(v->vpci.mem); + v->vpci.mem = NULL; + } + } +} + static int __init apply_map(struct domain *d, const struct pci_dev *pdev, struct rangeset *mem, uint16_t cmd) { diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index ceaac4516ff8..37103e207635 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -54,7 +54,9 @@ void vpci_remove_device_locked(struct pci_dev *pdev) { ASSERT(spin_is_locked(&pdev->vpci_lock)); + pdev->vpci_cancel_pending = true; vpci_remove_device_handlers_locked(pdev); + vpci_cancel_pending_locked(pdev); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 3f60d6c6c6dd..52d302ac5f35 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -135,6 +135,11 @@ struct pci_dev { /* Data for vPCI. */ spinlock_t vpci_lock; + /* + * Set if PCI device is being removed now and we need to cancel any + * pending map/unmap operations. + */ + bool vpci_cancel_pending; struct vpci *vpci; }; diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 8b22bdef11d0..cfff87e5801e 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -57,6 +57,7 @@ uint32_t vpci_hw_read32(const struct pci_dev *pdev, unsigned int reg, * should not run. */ bool __must_check vpci_process_pending(struct vcpu *v); +void vpci_cancel_pending_locked(struct pci_dev *pdev); struct vpci { /* List of vPCI handlers for a device. */ @@ -253,6 +254,11 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) ASSERT_UNREACHABLE(); return false; } + +static inline void vpci_cancel_pending_locked(struct pci_dev *pdev) +{ + ASSERT_UNREACHABLE(); +} #endif #endif From patchwork Thu Nov 25 11:02:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D06CBC4321E for ; Thu, 25 Nov 2021 11:03:13 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230878.399176 (Exim 4.92) (envelope-from ) id 1mqCWu-00036N-Mm; Thu, 25 Nov 2021 11:03:04 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230878.399176; Thu, 25 Nov 2021 11:03:04 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWu-00035p-FM; Thu, 25 Nov 2021 11:03:04 +0000 Received: by outflank-mailman (input) for mailman id 230878; Thu, 25 Nov 2021 11:03:02 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWs-0001K8-8b for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:02 +0000 Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [2a00:1450:4864:20::12b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 41629d9c-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:03:01 +0100 (CET) Received: by mail-lf1-x12b.google.com with SMTP id b40so15316060lfv.10 for ; Thu, 25 Nov 2021 03:03:01 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.02.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:02:59 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 41629d9c-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zlqkD9o6TFSW+Z9K8vbwXV06GGEnSpeGFpgXT4iChkc=; b=YDQqX6VgoUvas8T8U7uF/ZLbFzxhShXfl9FTIgBLp8Q+oRbGtdqWaYd21MvW81oXtN fW7Bd5DWXl28WqLF6KJ+d/ELkGV3pRxGU25kkgGylCZqSSdkoMzdedK3pwRa0RDwOlJG 1GD7o9W9TxuYIbLvINCio03CCNYqrNxoJzZx04Ok5b9G2SLzi78SxzwcBQ5zM3ICsvPk XEccye87xQtrzBAaNhnLDOcZlaWW2euSsvnqYk6DCkooXq54LT/OpPU4otO6j5lO41mT sbamG8EVXMbRhcK5kGLapbetYx9q+49PKfZT3eoWtSjk/APfmI1spEoC1nRW020gdGWm 5gBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zlqkD9o6TFSW+Z9K8vbwXV06GGEnSpeGFpgXT4iChkc=; b=3NuhXg5tYWyHQksLCay99pgS0c0ev4NbFkN+w3da56Z/KsxUETRZbXBNPeThjK4gvr 2C+wYy/blDO8T7UpG8woSHCpYa5emtmFWmJzMsCb+aQcCwQrGS1IoClOYysvJvoSSYoj 4NXVeutR27/yt+3zwvjHBB5obZukD7wfqkiWvQszJ5RZvI44h07UJk7RjUNYlnFAbb3n z2lK5ti/lHnw8AJS2Ddxuym1QgWkH9CViVI+maDf8eRtp1SObLW0NsJlewZQsSMRnThb /V0kAg5c3ETF2rOYinTqcjTRDh+Qo/BxtHxI8azJpWgxXiDqrppISBwOX4xCNN6EUuDJ I6tA== X-Gm-Message-State: AOAM531+cGalkxWUw+NjZJrTqlcAgKXgWccIgghv15gb2/Q6pPMBQIDb TdIrrxDIZG5WCGJvnKuw8lLXT/Kr9ovsNg== X-Google-Smtp-Source: ABdhPJyO225gyuzeTIpsa5rB1IrCqUbVttrUzf067KVOPkjsXGLJoVBo9SPRvxvDMrWxlwUzJj7+2Q== X-Received: by 2002:a19:ac46:: with SMTP id r6mr23521138lfc.42.1637838180416; Thu, 25 Nov 2021 03:03:00 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 05/14] vpci: add hooks for PCI device assign/de-assign Date: Thu, 25 Nov 2021 13:02:42 +0200 Message-Id: <20211125110251.2877218-6-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko When a PCI device gets assigned/de-assigned some work on vPCI side needs to be done for that device. Introduce a pair of hooks so vPCI can handle that. Signed-off-by: Oleksandr Andrushchenko --- Since v4: - de-assign vPCI from the previous domain on device assignment - do not remove handlers in vpci_assign_device as those must not exist at that point Since v3: - remove toolstack roll-back description from the commit message as error are to be handled with proper cleanup in Xen itself - remove __must_check - remove redundant rc check while assigning devices - fix redundant CONFIG_HAS_VPCI check for CONFIG_HAS_VPCI_GUEST_SUPPORT - use REGISTER_VPCI_INIT machinery to run required steps on device init/assign: add run_vpci_init helper Since v2: - define CONFIG_HAS_VPCI_GUEST_SUPPORT so dead code is not compiled for x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - extended the commit message --- xen/drivers/Kconfig | 4 +++ xen/drivers/passthrough/pci.c | 10 ++++++ xen/drivers/vpci/vpci.c | 61 +++++++++++++++++++++++++++++------ xen/include/xen/vpci.h | 16 +++++++++ 4 files changed, 82 insertions(+), 9 deletions(-) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47a6..780490cf8e39 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + depends on HAS_VPCI + endmenu diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 286808b25e65..d9ef91571adf 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -874,6 +874,10 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus, if ( ret ) goto out; + ret = vpci_deassign_device(d, pdev); + if ( ret ) + goto out; + if ( pdev->domain == hardware_domain ) pdev->quarantine = false; @@ -1429,6 +1433,10 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) ASSERT(pdev && (pdev->domain == hardware_domain || pdev->domain == dom_io)); + rc = vpci_deassign_device(pdev->domain, pdev); + if ( rc ) + goto done; + rc = pdev_msix_assign(d, pdev); if ( rc ) goto done; @@ -1446,6 +1454,8 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) rc = hd->platform_ops->assign_device(d, devfn, pci_to_dev(pdev), flag); } + rc = vpci_assign_device(d, pdev); + done: if ( rc ) printk(XENLOG_G_WARNING "%pd: assign (%pp) failed (%d)\n", diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 37103e207635..a9e9e8ec438c 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -74,12 +74,26 @@ void vpci_remove_device(struct pci_dev *pdev) spin_unlock(&pdev->vpci_lock); } -int vpci_add_handlers(struct pci_dev *pdev) +static int run_vpci_init(struct pci_dev *pdev) { - struct vpci *vpci; unsigned int i; int rc = 0; + for ( i = 0; i < NUM_VPCI_INIT; i++ ) + { + rc = __start_vpci_array[i](pdev); + if ( rc ) + break; + } + + return rc; +} + +int vpci_add_handlers(struct pci_dev *pdev) +{ + struct vpci *vpci; + int rc; + if ( !has_vpci(pdev->domain) ) return 0; @@ -94,19 +108,48 @@ int vpci_add_handlers(struct pci_dev *pdev) pdev->vpci = vpci; INIT_LIST_HEAD(&pdev->vpci->handlers); - for ( i = 0; i < NUM_VPCI_INIT; i++ ) - { - rc = __start_vpci_array[i](pdev); - if ( rc ) - break; - } - + rc = run_vpci_init(pdev); if ( rc ) vpci_remove_device_locked(pdev); spin_unlock(&pdev->vpci_lock); return rc; } + +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned to guest. */ +int vpci_assign_device(struct domain *d, struct pci_dev *pdev) +{ + int rc; + + /* It only makes sense to assign for hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + spin_lock(&pdev->vpci_lock); + rc = run_vpci_init(pdev); + spin_unlock(&pdev->vpci_lock); + if ( rc ) + vpci_deassign_device(d, pdev); + + return rc; +} + +/* Notify vPCI that device is de-assigned from guest. */ +int vpci_deassign_device(struct domain *d, struct pci_dev *pdev) +{ + /* It only makes sense to de-assign from hwdom or guest domain. */ + if ( is_system_domain(d) || !has_vpci(d) ) + return 0; + + spin_lock(&pdev->vpci_lock); + vpci_remove_device_handlers_locked(pdev); + spin_unlock(&pdev->vpci_lock); + + return 0; +} +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + #endif /* __XEN__ */ static int vpci_register_cmp(const struct vpci_register *r1, diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index cfff87e5801e..ed127a08a953 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -261,6 +261,22 @@ static inline void vpci_cancel_pending_locked(struct pci_dev *pdev) } #endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +/* Notify vPCI that device is assigned/de-assigned to/from guest. */ +int vpci_assign_device(struct domain *d, struct pci_dev *pdev); +int vpci_deassign_device(struct domain *d, struct pci_dev *pdev); +#else +static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) +{ + return 0; +}; + +static inline int vpci_deassign_device(struct domain *d, struct pci_dev *pdev) +{ + return 0; +}; +#endif + #endif /* From patchwork Thu Nov 25 11:02:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 10D06C4167B for ; Thu, 25 Nov 2021 11:03:15 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230879.399182 (Exim 4.92) (envelope-from ) id 1mqCWv-0003Dp-Eb; Thu, 25 Nov 2021 11:03:05 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230879.399182; Thu, 25 Nov 2021 11:03:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWv-0003AW-3g; Thu, 25 Nov 2021 11:03:05 +0000 Received: by outflank-mailman (input) for mailman id 230879; Thu, 25 Nov 2021 11:03:03 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWt-0001K8-CC for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:03 +0000 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [2a00:1450:4864:20::130]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 4218a690-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:03:02 +0100 (CET) Received: by mail-lf1-x130.google.com with SMTP id bu18so15489161lfb.0 for ; Thu, 25 Nov 2021 03:03:02 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:01 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4218a690-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=O+IzydKHdJygtefHSBhDUYSHyy89TbaBO8s7u4ieLkg=; b=ph97ktYAdzOhWjuIz9sTV0tBn0nTxfzF89jNMYPHNw9sIrPDgKIX/dn9UYZUWn5Qvo WQp/qb9S6xSR3MONRkDU8o0RKYtBC7ZxF6LUTnU6c0ThCCeGxFE1aglOLTQWPFq2mNMv ZM0C9nb1n0hTRYksRTNy7KVdrVqCMWtq3ua+76LKq04OoQlF4Q4Ro62wCPBmEBC4AzMP OATAC0f3dkRk/CGBFFTlS/9l8crAtSMmEKFFd49S0EtZZ0ELznLgxugFIiXqU5Bh5NWl emjxazNH2BrNCDxDrkpkxRIjLHyZeoFnnUxP8vX4ru0P1U4+Idyh5/wBPg/wiMpoiS7I loWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O+IzydKHdJygtefHSBhDUYSHyy89TbaBO8s7u4ieLkg=; b=GJPxTqDGKHs/ihoFM2nP4jOYVEVut6Eck1Nvz3tQVnPBeKi6G2L0jU8GRWO/r525ND AfqCT0uWvTmGixp/NulizKxLBZDQglOj9Pzg6Nm+UAdpDoZnTbe+EgZ95joxO7PA+beY DMW7emomuB6gI4BVmYawt45cE0EJT6E4sI3kOCh3yzhTc8mBhqsWz7IikxLwcng4HfVh uLhp70vFn75kgPfKJavepmY2Zx7pzBAXkL6qRT5UJs4nDWnb4hSOCx5a+ycGy/QZ0R8h epSPWYxpW1uOPRRtp3ttzVJZiC+6R7U7yUmTDOlQsjKXK7zmaZskL6c4ERmTD9K72V+C Wzig== X-Gm-Message-State: AOAM530PZjXkQ2eXRM7PM6zrmNAdWwubi1pa5hjsOoiqzKfOh7iAI2L0 uZUn0ha/EhAvja6uFCD84REOlhusQI4AQA== X-Google-Smtp-Source: ABdhPJy8wI74HkKqpkhUCTmVZjQSZRS/HDZXqAarTk1y6geijO/HfiIwnyEwRYpA2iUuDDNcFUpddQ== X-Received: by 2002:a05:6512:3ca1:: with SMTP id h33mr22339207lfv.640.1637838181712; Thu, 25 Nov 2021 03:03:01 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 06/14] vpci/header: implement guest BAR register handlers Date: Thu, 25 Nov 2021 13:02:43 +0200 Message-Id: <20211125110251.2877218-7-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add relevant vpci register handlers when assigning PCI device to a domain and remove those when de-assigning. This allows having different handlers for different domains, e.g. hwdom and other guests. Emulate guest BAR register values: this allows creating a guest view of the registers and emulates size and properties probe as it is done during PCI device enumeration by the guest. ROM BAR is only handled for the hardware domain and for guest domains there is a stub: at the moment PCI expansion ROM handling is supported for x86 only and it might not be used by other architectures without emulating x86. Other use-cases may include using that expansion ROM before Xen boots, hence no emulation is needed in Xen itself. Or when a guest wants to use the ROM code which seems to be rare. Signed-off-by: Oleksandr Andrushchenko --- Since v4: - updated commit message - s/guest_addr/guest_reg Since v3: - squashed two patches: dynamic add/remove handlers and guest BAR handler implementation - fix guest BAR read of the high part of a 64bit BAR (Roger) - add error handling to vpci_assign_device - s/dom%pd/%pd - blank line before return Since v2: - remove unneeded ifdefs for CONFIG_HAS_VPCI_GUEST_SUPPORT as more code has been eliminated from being built on x86 Since v1: - constify struct pci_dev where possible - do not open code is_system_domain() - simplify some code3. simplify - use gdprintk + error code instead of gprintk - gate vpci_bar_{add|remove}_handlers with CONFIG_HAS_VPCI_GUEST_SUPPORT, so these do not get compiled for x86 - removed unneeded is_system_domain check - re-work guest read/write to be much simpler and do more work on write than read which is expected to be called more frequently - removed one too obvious comment --- xen/drivers/vpci/header.c | 72 +++++++++++++++++++++++++++++++++++---- xen/include/xen/vpci.h | 3 ++ 2 files changed, 69 insertions(+), 6 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index ba333fb2f9b0..8880d34ebf8e 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -433,6 +433,48 @@ static void bar_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write32(pdev->sbdf, reg, val); } +static void guest_bar_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ + struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + else + { + val &= PCI_BASE_ADDRESS_MEM_MASK; + val |= bar->type == VPCI_BAR_MEM32 ? PCI_BASE_ADDRESS_MEM_TYPE_32 + : PCI_BASE_ADDRESS_MEM_TYPE_64; + val |= bar->prefetchable ? PCI_BASE_ADDRESS_MEM_PREFETCH : 0; + } + + bar->guest_reg &= ~(0xffffffffull << (hi ? 32 : 0)); + bar->guest_reg |= (uint64_t)val << (hi ? 32 : 0); + + bar->guest_reg &= ~(bar->size - 1) | ~PCI_BASE_ADDRESS_MEM_MASK; +} + +static uint32_t guest_bar_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + const struct vpci_bar *bar = data; + bool hi = false; + + if ( bar->type == VPCI_BAR_MEM64_HI ) + { + ASSERT(reg > PCI_BASE_ADDRESS_0); + bar--; + hi = true; + } + + return bar->guest_reg >> (hi ? 32 : 0); +} + static void rom_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -481,6 +523,17 @@ static void rom_write(const struct pci_dev *pdev, unsigned int reg, rom->addr = val & PCI_ROM_ADDRESS_MASK; } +static void guest_rom_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t val, void *data) +{ +} + +static uint32_t guest_rom_read(const struct pci_dev *pdev, unsigned int reg, + void *data) +{ + return 0xffffffff; +} + static int init_bars(struct pci_dev *pdev) { uint16_t cmd; @@ -489,6 +542,7 @@ static int init_bars(struct pci_dev *pdev) struct vpci_header *header = &pdev->vpci->header; struct vpci_bar *bars = header->bars; int rc; + bool is_hwdom = is_hardware_domain(pdev->domain); switch ( pci_conf_read8(pdev->sbdf, PCI_HEADER_TYPE) & 0x7f ) { @@ -528,8 +582,10 @@ static int init_bars(struct pci_dev *pdev) if ( i && bars[i - 1].type == VPCI_BAR_MEM64_LO ) { bars[i].type = VPCI_BAR_MEM64_HI; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, - 4, &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -569,8 +625,10 @@ static int init_bars(struct pci_dev *pdev) bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, bar_write, reg, 4, - &bars[i]); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_bar_read, + is_hwdom ? bar_write : guest_bar_write, + reg, 4, &bars[i]); if ( rc ) { pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); @@ -590,8 +648,10 @@ static int init_bars(struct pci_dev *pdev) header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & PCI_ROM_ADDRESS_ENABLE; - rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rom_write, rom_reg, - 4, rom); + rc = vpci_add_register(pdev->vpci, + is_hwdom ? vpci_hw_read32 : guest_rom_read, + is_hwdom ? rom_write : guest_rom_write, + rom_reg, 4, rom); if ( rc ) rom->type = VPCI_BAR_EMPTY; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index ed127a08a953..0a73b14a92dc 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -68,7 +68,10 @@ struct vpci { struct vpci_header { /* Information about the PCI BARs of this device. */ struct vpci_bar { + /* Physical view of the BAR. */ uint64_t addr; + /* Guest view of the BAR: address and lower bits. */ + uint64_t guest_reg; uint64_t size; enum { VPCI_BAR_EMPTY, From patchwork Thu Nov 25 11:02:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D3ABC433FE for ; Thu, 25 Nov 2021 11:03:19 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230881.399198 (Exim 4.92) (envelope-from ) id 1mqCWx-0003nv-U5; Thu, 25 Nov 2021 11:03:07 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230881.399198; Thu, 25 Nov 2021 11:03:07 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWx-0003mn-L4; Thu, 25 Nov 2021 11:03:07 +0000 Received: by outflank-mailman (input) for mailman id 230881; Thu, 25 Nov 2021 11:03:05 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWv-0001K8-9h for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:05 +0000 Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [2a00:1450:4864:20::12f]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 42e42885-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:03:03 +0100 (CET) Received: by mail-lf1-x12f.google.com with SMTP id bi37so15338748lfb.5 for ; Thu, 25 Nov 2021 03:03:03 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:02 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 42e42885-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0goulj3ILS644tMO0YQq9bXnR7rW90qkHRTjF7tQQeI=; b=UrJyj4FUG1x0ALg9aBFoWluXc+IWI6+0EtF3UzJs890N/iRq9y5FZKVzwwC4gBXECI nGmd69BrCY6TWf2NqUKbjDqorFirlsTLuvE9P8VZj0H+oRudkXtSlnzZBZnyoNV29js/ W5pkmWeN8OGJ8ua/UKfe8R97sHbWv3zEWQbDmUV0MfKNMdmqShf1ei/rLkIV0Ewa321h LuM4G7ti8ZT9Ke76YcoJVQ2ZYy9rl14oFUzH/KlQTPcsUrTPNZQKpacmDa3fZopAxdIA AoVcWwFyzcsy3lgPPh2vC/kiCBj2yoDjsRvhoqbBdUSq0/e7PNWFYTI0tsmUCL4D4uQI 681A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0goulj3ILS644tMO0YQq9bXnR7rW90qkHRTjF7tQQeI=; b=XfFLWWTwXkce4/2v9+teLPSdwrT5BSVOewOn71QQi++uJQiq/JaI3Ylmrk2WVTKVp4 yMQRI2UNKGSyZCKsyfXifySvHRXRvt5SVf30QnM7oWiWvMAk/0cxwDgEHeq1FdKuZJ/D MJUnx0k8CpDUZNA4Ee32rplyKqdIcusjJ2mgVxH3UafHXbuaaUT4b+wB42udHJlPXXlt TUCFWNes93Ug1ytofyFoKNLOiPX5/1HvN6N6T0iz2/jTmqAUYEeJzRvYtf7VNdQP5XOj 1PAaCQvZSHmBg9SWyXRylG9+9Fv23T692RVCl+nWAxMACSRTMtX5fQmWDjhnMwtgT0OI BDKA== X-Gm-Message-State: AOAM532DEHW0ea5NQjLhBEEEE8oWRGkDcIBCMA6SrjmS32Bjmj00LIc1 THhap8OyRj4wK2RXWmjJZYEPCAyoKi9mTg== X-Google-Smtp-Source: ABdhPJwTcrZIWmWIhIhIwdty/jDoU586b73+ZP2UFSd0S42UDaitxbH9WPRBAHPwrG/NuXfedaESIg== X-Received: by 2002:a05:6512:3b9c:: with SMTP id g28mr23026819lfv.651.1637838183031; Thu, 25 Nov 2021 03:03:03 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 07/14] vpci/header: handle p2m range sets per BAR Date: Thu, 25 Nov 2021 13:02:44 +0200 Message-Id: <20211125110251.2877218-8-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Instead of handling a single range set, that contains all the memory regions of all the BARs and ROM, have them per BAR. As the range sets are now created when a PCI device is added and destroyed when it is removed so make them named and accounted. Note that rangesets were chosen here despite there being only up to 3 separate ranges in each set (typically just 1). But rangeset per BAR was chosen for the ease of implementation and existing code re-usability. This is in preparation of making non-identity mappings in p2m for the MMIOs/ROM. Signed-off-by: Oleksandr Andrushchenko --- Since v4: - use named range sets for BARs (Jan) - changes required by the new locking scheme - updated commit message (Jan) Since v3: - re-work vpci_cancel_pending accordingly to the per-BAR handling - s/num_mem_ranges/map_pending and s/uint8_t/bool - ASSERT(bar->mem) in modify_bars - create and destroy the rangesets on add/remove --- xen/drivers/vpci/header.c | 190 +++++++++++++++++++++++++++----------- xen/drivers/vpci/vpci.c | 30 +++++- xen/include/xen/vpci.h | 3 +- 3 files changed, 166 insertions(+), 57 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 8880d34ebf8e..cc49aa68886f 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -137,45 +137,86 @@ bool vpci_process_pending(struct vcpu *v) return false; spin_lock(&pdev->vpci_lock); - if ( !pdev->vpci_cancel_pending && v->vpci.mem ) + if ( !pdev->vpci ) + { + spin_unlock(&pdev->vpci_lock); + return false; + } + + if ( !pdev->vpci_cancel_pending && v->vpci.map_pending ) { struct map_data data = { .d = v->domain, .map = v->vpci.cmd & PCI_COMMAND_MEMORY, }; - int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data); + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; - if ( rc == -ERESTART ) + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - spin_unlock(&pdev->vpci_lock); - return true; - } + struct vpci_bar *bar = &header->bars[i]; + int rc; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_consume_ranges(bar->mem, map_range, &data); + + if ( rc == -ERESTART ) + { + spin_unlock(&pdev->vpci_lock); + return true; + } - if ( pdev->vpci ) /* Disable memory decoding unconditionally on failure. */ - modify_decoding(pdev, - rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, + modify_decoding(pdev, rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd, !rc && v->vpci.rom_only); - if ( rc ) - { - /* - * FIXME: in case of failure remove the device from the domain. - * Note that there might still be leftover mappings. While this is - * safe for Dom0, for DomUs the domain needs to be killed in order - * to avoid leaking stale p2m mappings on failure. - */ - if ( is_hardware_domain(v->domain) ) - vpci_remove_device_locked(pdev); - else - domain_crash(v->domain); + if ( rc ) + { + /* + * FIXME: in case of failure remove the device from the domain. + * Note that there might still be leftover mappings. While this is + * safe for Dom0, for DomUs the domain needs to be killed in order + * to avoid leaking stale p2m mappings on failure. + */ + if ( is_hardware_domain(v->domain) ) + vpci_remove_device_locked(pdev); + else + domain_crash(v->domain); + + break; + } } + + v->vpci.map_pending = false; } spin_unlock(&pdev->vpci_lock); return false; } +static void vpci_bar_remove_ranges(const struct pci_dev *pdev) +{ + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + int rc; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, 0, ~0ULL); + if ( !rc ) + printk(XENLOG_ERR + "%pd %pp failed to remove range set for BAR: %d\n", + pdev->domain, &pdev->sbdf, rc); + } +} + void vpci_cancel_pending_locked(struct pci_dev *pdev) { struct vcpu *v; @@ -185,23 +226,33 @@ void vpci_cancel_pending_locked(struct pci_dev *pdev) /* Cancel any pending work now on all vCPUs. */ for_each_vcpu( pdev->domain, v ) { - if ( v->vpci.mem && (v->vpci.pdev == pdev) ) + if ( v->vpci.map_pending && (v->vpci.pdev == pdev) ) { - rangeset_destroy(v->vpci.mem); - v->vpci.mem = NULL; + vpci_bar_remove_ranges(pdev); + v->vpci.map_pending = false; } } } static int __init apply_map(struct domain *d, const struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd) + uint16_t cmd) { struct map_data data = { .d = d, .map = true }; - int rc; + struct vpci_header *header = &pdev->vpci->header; + int rc = 0; + unsigned int i; + + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; - while ( (rc = rangeset_consume_ranges(mem, map_range, &data)) == -ERESTART ) - process_pending_softirqs(); - rangeset_destroy(mem); + if ( rangeset_is_empty(bar->mem) ) + continue; + + while ( (rc = rangeset_consume_ranges(bar->mem, map_range, + &data)) == -ERESTART ) + process_pending_softirqs(); + } if ( !rc ) modify_decoding(pdev, cmd, false); @@ -209,7 +260,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, } static void defer_map(struct domain *d, struct pci_dev *pdev, - struct rangeset *mem, uint16_t cmd, bool rom_only) + uint16_t cmd, bool rom_only) { struct vcpu *curr = current; @@ -220,7 +271,7 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, * started for the same device if the domain is not well-behaved. */ curr->vpci.pdev = pdev; - curr->vpci.mem = mem; + curr->vpci.map_pending = true; curr->vpci.cmd = cmd; curr->vpci.rom_only = rom_only; /* @@ -234,42 +285,40 @@ static void defer_map(struct domain *d, struct pci_dev *pdev, static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) { struct vpci_header *header = &pdev->vpci->header; - struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; const struct vpci_msix *msix = pdev->vpci->msix; - unsigned int i; + unsigned int i, j; int rc; - - if ( !mem ) - return -ENOMEM; + bool map_pending; /* - * Create a rangeset that represents the current device BARs memory region + * Create a rangeset per BAR that represents the current device memory region * and compare it against all the currently active BAR memory regions. If * an overlap is found, subtract it from the region to be mapped/unmapped. * - * First fill the rangeset with all the BARs of this device or with the ROM + * First fill the rangesets with all the BARs of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { - const struct vpci_bar *bar = &header->bars[i]; + struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + ASSERT(bar->mem); + if ( !MAPPABLE_BAR(bar) || (rom_only ? bar->type != VPCI_BAR_ROM : (bar->type == VPCI_BAR_ROM && !header->rom_enabled)) ) continue; - rc = rangeset_add_range(mem, start, end); + rc = rangeset_add_range(bar->mem, start, end); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } @@ -280,14 +329,21 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) { - printk(XENLOG_G_WARNING - "Failed to remove MSIX table [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &header->bars[j]; + + if ( rangeset_is_empty(bar->mem) ) + continue; + + rc = rangeset_remove_range(bar->mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING + "Failed to remove MSIX table [%lx, %lx]: %d\n", + start, end, rc); + goto fail; + } } } @@ -325,7 +381,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || + if ( !bar->enabled || + !rangeset_overlaps_range(bar->mem, start, end) || /* * If only the ROM enable bit is toggled check against other * BARs in the same device for overlaps, but not against the @@ -334,14 +391,13 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) continue; - rc = rangeset_remove_range(mem, start, end); + rc = rangeset_remove_range(bar->mem, start, end); if ( rc ) { spin_unlock(&tmp->vpci_lock); printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", start, end, rc); - rangeset_destroy(mem); - return rc; + goto fail; } } spin_unlock(&tmp->vpci_lock); @@ -360,12 +416,36 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * will always be to establish mappings and process all the BARs. */ ASSERT((cmd & PCI_COMMAND_MEMORY) && !rom_only); - return apply_map(pdev->domain, pdev, mem, cmd); + return apply_map(pdev->domain, pdev, cmd); } - defer_map(dev->domain, dev, mem, cmd, rom_only); + /* Find out how many memory ranges has left after MSI and overlaps. */ + map_pending = false; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + if ( !rangeset_is_empty(header->bars[i].mem) ) + { + map_pending = true; + break; + } + + /* + * There are cases when PCI device, root port for example, has neither + * memory space nor IO. In this case PCI command register write is + * missed resulting in the underlying PCI device not functional, so: + * - if there are no regions write the command register now + * - if there are regions then defer work and write later on + */ + if ( !map_pending ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd); + else + defer_map(dev->domain, dev, cmd, rom_only); return 0; + +fail: + /* Destroy all the ranges we may have added. */ + vpci_bar_remove_ranges(pdev); + return rc; } static void cmd_write(const struct pci_dev *pdev, unsigned int reg, diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index a9e9e8ec438c..98b12a61be6f 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -52,11 +52,16 @@ static void vpci_remove_device_handlers_locked(struct pci_dev *pdev) void vpci_remove_device_locked(struct pci_dev *pdev) { + struct vpci_header *header = &pdev->vpci->header; + unsigned int i; + ASSERT(spin_is_locked(&pdev->vpci_lock)); pdev->vpci_cancel_pending = true; vpci_remove_device_handlers_locked(pdev); vpci_cancel_pending_locked(pdev); + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + rangeset_destroy(header->bars[i].mem); xfree(pdev->vpci->msix); xfree(pdev->vpci->msi); xfree(pdev->vpci); @@ -92,6 +97,8 @@ static int run_vpci_init(struct pci_dev *pdev) int vpci_add_handlers(struct pci_dev *pdev) { struct vpci *vpci; + struct vpci_header *header; + unsigned int i; int rc; if ( !has_vpci(pdev->domain) ) @@ -108,11 +115,32 @@ int vpci_add_handlers(struct pci_dev *pdev) pdev->vpci = vpci; INIT_LIST_HEAD(&pdev->vpci->handlers); + header = &pdev->vpci->header; + for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) + { + struct vpci_bar *bar = &header->bars[i]; + char str[32]; + + snprintf(str, sizeof(str), "%pp:BAR%d", &pdev->sbdf, i); + bar->mem = rangeset_new(pdev->domain, str, RANGESETF_no_print); + if ( !bar->mem ) + { + rc = -ENOMEM; + goto fail; + } + } + rc = run_vpci_init(pdev); if ( rc ) - vpci_remove_device_locked(pdev); + goto fail; + spin_unlock(&pdev->vpci_lock); + return 0; + + fail: + vpci_remove_device_locked(pdev); + spin_unlock(&pdev->vpci_lock); return rc; } diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 0a73b14a92dc..18319fc329f9 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -73,6 +73,7 @@ struct vpci { /* Guest view of the BAR: address and lower bits. */ uint64_t guest_reg; uint64_t size; + struct rangeset *mem; enum { VPCI_BAR_EMPTY, VPCI_BAR_IO, @@ -147,9 +148,9 @@ struct vpci { struct vpci_vcpu { /* Per-vcpu structure to store state while {un}mapping of PCI BARs. */ - struct rangeset *mem; struct pci_dev *pdev; uint16_t cmd; + bool map_pending : 1; bool rom_only : 1; }; From patchwork Thu Nov 25 11:02:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0CF0C433EF for ; Thu, 25 Nov 2021 11:03:17 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230882.399203 (Exim 4.92) (envelope-from ) id 1mqCWy-0003ui-SZ; Thu, 25 Nov 2021 11:03:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230882.399203; Thu, 25 Nov 2021 11:03:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWy-0003t2-F3; Thu, 25 Nov 2021 11:03:08 +0000 Received: by outflank-mailman (input) for mailman id 230882; Thu, 25 Nov 2021 11:03:06 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWw-0001K8-9Z for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:06 +0000 Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [2a00:1450:4864:20::12e]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 43a627ac-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:03:05 +0100 (CET) Received: by mail-lf1-x12e.google.com with SMTP id l22so15333068lfg.7 for ; Thu, 25 Nov 2021 03:03:05 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:03 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 43a627ac-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=k/RGv6ybx7YyFsmgyX3yYGViHePRS8H6axFbMKtL+NM=; b=NH67NlrIH5Ybk480ToPEOvLqg5URZvwh/QGlPI8eMNf57TsY3odxpYlTb88dfC5rc1 vtZGDVob6WzDb0yGJTMiQrB86sIkeC12pSc6OK9Gd9xzPhiVpvMFbbIqsJ7VyMQ8gYN3 RtMpDxzQuLpy5AqSFpYcTG9dGF2a7Xh0kKnsG9e0CgtskzJ57voB431EW4ImqgJ+p3xF 5ZxQNO9gEJV/hQpLwRYdfKDtsTcuyqxf4KnRxosTYQ+LG99subpv2MiB2w61JB+DCh7Z Tt4g8QTzy5AXgiYsTFuqRned8Q9sFtdiHhe4J0Wz9buf3/8htcNr27kif7q0o2uZoCNy RR/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=k/RGv6ybx7YyFsmgyX3yYGViHePRS8H6axFbMKtL+NM=; b=ickXIXci+3txfz1fCi7MIcVoCF51Ov1CQVUIPpn5KwwhIkOUJtsfsB9yZRgDS8isoQ Z09bNPZiwC93Wg+5hx/yT9rDNZP+3pzr20EnaPdLGyHUtjO3ZWmdVKCzHtn+Q5J5vElt RUpbCtiY7NUJvV6mY41w6xu53WIGOH/pjWDZD674V3rBy107XyZSLaT2PKPoCTCFVr4U Zr/xuIKfQZugNiCNrybGt/R6XP3xa42n88eWZmGSyl8lm7BNgF7/nO2jhxFfhbo2FMRs 6IOKdRVKVr5vzoMQdS/UVz0V3hfWRJCwMIb30jnD6vn65cqZ/rNlLRC2QIRTEJ2lh67q 9N4A== X-Gm-Message-State: AOAM530z60LGIJEmRWhROXr26HB4n2kWrMxCzx89+JsCcQqRy0kA1GAr XylYuZ2IF8IH8HbV0+fMVLMTbCLDZ8d9dA== X-Google-Smtp-Source: ABdhPJx/6zGeavqUGNS7ApB3Y8UGu0V+f7KGhetYpG+N9ZJazQ9xzusOsoLlz7Ch+XQYRoT1COfbFQ== X-Received: by 2002:a05:6512:3212:: with SMTP id d18mr22988968lfe.285.1637838184244; Thu, 25 Nov 2021 03:03:04 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 08/14] vpci/header: program p2m with guest BAR view Date: Thu, 25 Nov 2021 13:02:45 +0200 Message-Id: <20211125110251.2877218-9-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value as set up by the PCI bus driver in the hardware domain. This way hardware domain sees physical BAR values and guest sees emulated ones. Signed-off-by: Oleksandr Andrushchenko --- Since v4: - moved start_{gfn|mfn} calculation into map_range - pass vpci_bar in the map_data instead of start_{gfn|mfn} - s/guest_addr/guest_reg Since v3: - updated comment (Roger) - removed gfn_add(map->start_gfn, rc); which is wrong - use v->domain instead of v->vpci.pdev->domain - removed odd e.g. in comment - s/d%d/%pd in altered code - use gdprintk for map/unmap logs Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- --- xen/drivers/vpci/header.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index cc49aa68886f..b0499d32c5d8 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -30,6 +30,7 @@ struct map_data { struct domain *d; + const struct vpci_bar *bar; bool map; }; @@ -41,8 +42,25 @@ static int map_range(unsigned long s, unsigned long e, void *data, for ( ; ; ) { + /* Start address of the BAR as seen by the guest. */ + gfn_t start_gfn = _gfn(PFN_DOWN(is_hardware_domain(map->d) + ? map->bar->addr + : map->bar->guest_reg)); + /* Physical start address of the BAR. */ + mfn_t start_mfn = _mfn(PFN_DOWN(map->bar->addr)); unsigned long size = e - s + 1; + /* + * Ranges to be mapped don't always start at the BAR start address, as + * there can be holes or partially consumed ranges. Account for the + * offset of the current address from the BAR start. + */ + start_gfn = gfn_add(start_gfn, s - mfn_x(start_mfn)); + + gdprintk(XENLOG_G_DEBUG, + "%smap [%lx, %lx] -> %#"PRI_gfn" for %pd\n", + map->map ? "" : "un", s, e, gfn_x(start_gfn), + map->d); /* * ARM TODOs: * - On ARM whether the memory is prefetchable or not should be passed @@ -52,8 +70,10 @@ static int map_range(unsigned long s, unsigned long e, void *data, * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, start_gfn, + size, _mfn(s)) + : unmap_mmio_regions(map->d, start_gfn, + size, _mfn(s)); if ( rc == 0 ) { *c += size; @@ -62,8 +82,8 @@ static int map_range(unsigned long s, unsigned long e, void *data, if ( rc < 0 ) { printk(XENLOG_G_WARNING - "Failed to identity %smap [%lx, %lx] for d%d: %d\n", - map->map ? "" : "un", s, e, map->d->domain_id, rc); + "Failed to identity %smap [%lx, %lx] for %pd: %d\n", + map->map ? "" : "un", s, e, map->d, rc); break; } ASSERT(rc < size); @@ -160,6 +180,7 @@ bool vpci_process_pending(struct vcpu *v) if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; rc = rangeset_consume_ranges(bar->mem, map_range, &data); if ( rc == -ERESTART ) @@ -249,6 +270,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, if ( rangeset_is_empty(bar->mem) ) continue; + data.bar = bar; while ( (rc = rangeset_consume_ranges(bar->mem, map_range, &data)) == -ERESTART ) process_pending_softirqs(); From patchwork Thu Nov 25 11:02:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1EA1C433F5 for ; Thu, 25 Nov 2021 11:03:18 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230883.399208 (Exim 4.92) (envelope-from ) id 1mqCWz-00046y-RR; Thu, 25 Nov 2021 11:03:09 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230883.399208; Thu, 25 Nov 2021 11:03:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWz-000440-H8; Thu, 25 Nov 2021 11:03:09 +0000 Received: by outflank-mailman (input) for mailman id 230883; Thu, 25 Nov 2021 11:03:07 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWx-0001K8-9f for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:07 +0000 Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [2a00:1450:4864:20::132]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 447957b9-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:03:06 +0100 (CET) Received: by mail-lf1-x132.google.com with SMTP id t26so15312528lfk.9 for ; Thu, 25 Nov 2021 03:03:06 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:05 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 447957b9-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Nyrz1DyhOvYhqBK8nvrcaWou5z7OLEnK+mhXV6s0HlU=; b=V870hY+Si8qenQfHmDNp7t1IDTMRn/UwCIGehQvYQgixBMY8M4Fn9NrqqUta2QNPUu 04vENHAecm1kgPQnPRIaIjCThiEhj/mz2ERJh3tdh5oqauIihr1RacIP+ZJBvkdOyS2o S3dlDXnvjOzZdzlincmq4yYNkLxpS4ZoOtvoRSyepL0mMHIOtSEFOgso6PXsDjeYWrfF KgikvrmgQGQpw9DYZHd/M4L43ZL+sMN9OxuC88t/utyEBmNjy3xk3XQrdYmrHiQ6bDNk ZuTBLTUsAdzW0LHJbcz2X0MR1mbzQeuR+wXtdW5NmFSfojX9v2m4whQIXkomngrqcwT1 3EFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Nyrz1DyhOvYhqBK8nvrcaWou5z7OLEnK+mhXV6s0HlU=; b=LjGj2kFjW4Wnx8AonA4vXwfNtrzhC6gVI0wheBte7HC9TFEPkpCYJb/AoyTLhIgb2D LfUpSGkULk2yShLs69zxwC2PU40mTLsNJ3+UFMmY/MM4wDrZior/Dpgno92/B56qE7bT mtB03v+vimgK7j/+bKVw+dD4WZ+LSUszP+eErcn6gSnm+SExHQSjhw3EZ8EY3VTp3gqT VfLxe0n7RySjwfnPK5jBi50o4Px59opX+JjwIiWIrD9tL1oZfea+QenLKp/pZSWCL8La Hv2QALNx3tbz0f/Y3g2yY+40Bytbmkejs1UL1GTnMZmR1FnDnI3cntBiRtptkyqCDIdT gRHA== X-Gm-Message-State: AOAM530fsijH0QITmffTL3H0oMz4fHYwp0M4wLqFDlW7Lftw9etKF3hq rEVF43N30phAy+gCqOOhjKBm4uQtRLVo6A== X-Google-Smtp-Source: ABdhPJzGjzT6zakGE3+HWjoenO7aqNkiIWjxzh5Xrz+J75cQSaEJ9+zwzgD/iqnblW5fj+SeYObF8A== X-Received: by 2002:a05:6512:a92:: with SMTP id m18mr1738441lfu.306.1637838185738; Thu, 25 Nov 2021 03:03:05 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 09/14] vpci/header: emulate PCI_COMMAND register for guests Date: Thu, 25 Nov 2021 13:02:46 +0200 Message-Id: <20211125110251.2877218-10-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Add basic emulation support for guests. At the moment only emulate PCI_COMMAND_INTX_DISABLE bit, the rest is not emulated yet and left as TODO. Signed-off-by: Oleksandr Andrushchenko --- Since v3: - gate more code on CONFIG_HAS_MSI - removed logic for the case when MSI/MSI-X not enabled --- xen/drivers/vpci/header.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index b0499d32c5d8..2e44055946b0 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -491,6 +491,22 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + /* TODO: Add proper emulation for all bits of the command register. */ + +#ifdef CONFIG_HAS_PCI_MSI + if ( pdev->vpci->msi->enabled ) + { + /* Guest wants to enable INTx. It can't be enabled if MSI/MSI-X enabled. */ + cmd |= PCI_COMMAND_INTX_DISABLE; + } +#endif + + cmd_write(pdev, reg, cmd, data); +} + static void bar_write(const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -663,8 +679,9 @@ static int init_bars(struct pci_dev *pdev) } /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + rc = vpci_add_register(pdev->vpci, vpci_hw_read16, + is_hwdom ? cmd_write : guest_cmd_write, + PCI_COMMAND, 2, header); if ( rc ) return rc; From patchwork Thu Nov 25 11:02:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40F65C433EF for ; Thu, 25 Nov 2021 11:03:21 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230885.399220 (Exim 4.92) (envelope-from ) id 1mqCX2-0004bL-1m; Thu, 25 Nov 2021 11:03:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230885.399220; Thu, 25 Nov 2021 11:03:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX1-0004Z0-Ny; Thu, 25 Nov 2021 11:03:11 +0000 Received: by outflank-mailman (input) for mailman id 230885; Thu, 25 Nov 2021 11:03:09 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCWy-0001K8-TG for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:09 +0000 Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [2a00:1450:4864:20::22e]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 4587c4b2-4ddf-11ec-a9d2-d9f7a1cc8784; Thu, 25 Nov 2021 12:03:08 +0100 (CET) Received: by mail-lj1-x22e.google.com with SMTP id z8so11729537ljz.9 for ; Thu, 25 Nov 2021 03:03:08 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:06 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4587c4b2-4ddf-11ec-a9d2-d9f7a1cc8784 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v/1hCqpbICaVhImJYbNH1Pzo/XjLvxlkPWIsYcGJ7iE=; b=dsQ7fIdBbFEQQs2Kb7aJIZHQf8yS7jdmZpbsZmPLPp+D7gCSFnBnthiYhk1usVP3iJ E+V2JEJKVrOee7Q3nN7Yw/eX/axIZ+2aiUWwRx1VTZVi31YlDTtSMTNVqa+n9DdRtG/I swcHYM1ITfFItZHRJToOjmUTej2VquSRTl5ERXDfaum/IKjKfEXkrEnZWgjDaH8L4VDf BoQ3eUxD2XyEFuasUzNVu6cVgKl5TeNElm1vSaRCJxR/8oUt96kKvjy0HURaDw7v7lym RIVLa2CnK5NnCk7WJzHYT52/tdH+fy9TZy1r5MPMrcCay4qaA76sW3vlG21bIKq2iOx/ 5ZhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v/1hCqpbICaVhImJYbNH1Pzo/XjLvxlkPWIsYcGJ7iE=; b=RjRI40w77MzBmTWENzUrTOxAFQgOfr9lIDS6xf1nEi1sM0CGyrYlpIUp+Fk1RYQ3Ml vYOYebtqf+7oZvsJa0ew1IM3kGcJJ7jX0doz2TwT9QFRMu2joVRqc4BEsyaLCdYZmfml IBqbrln1mhb6HUqmc4z0SwKeVKThMxb1jJzm381w9+sdxNAx2c2p+bG0oyjNR+HaxWuu BCCEgqRB/qiUvYH2QaRsKtXw38nYOt/OKfPoEkO7ZkWeEERfBL/PUBv1udsM9Oy32+YR 2REFN6nYptPTaYQSbGzXIyHEZKM2MaAx+uNXUj3CzpZ4m/olEp5UWv38SV4521K9KWwj qY8w== X-Gm-Message-State: AOAM531J47EHe7PVPnoGkAFndrW+OsEfAZJVy/J/slwmXfXkU9cgFQ2K 0CxftU4Ry0lVZ52C+oRK93KkXtFj8R6ekA== X-Google-Smtp-Source: ABdhPJzS3EW6c2oIc2r405IHk30Ssbig4XaZwvPXuufl47eVPHl5A6F8Pvxh9m5QoqHaFnustCOBYg== X-Received: by 2002:a05:651c:1035:: with SMTP id w21mr22964089ljm.278.1637838186915; Thu, 25 Nov 2021 03:03:06 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 10/14] vpci/header: reset the command register when adding devices Date: Thu, 25 Nov 2021 13:02:47 +0200 Message-Id: <20211125110251.2877218-11-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Reset the command register when passing through a PCI device: it is possible that when passing through a PCI device its memory decoding bits in the command register are already set. Thus, a guest OS may not write to the command register to update memory decoding, so guest mappings (guest's view of the BARs) are left not updated. Signed-off-by: Oleksandr Andrushchenko --- Since v1: - do not write 0 to the command register, but respect host settings. --- xen/drivers/vpci/header.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 2e44055946b0..41dda3c43d56 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -491,8 +491,7 @@ static void cmd_write(const struct pci_dev *pdev, unsigned int reg, pci_conf_write16(pdev->sbdf, reg, cmd); } -static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, - uint32_t cmd, void *data) +static uint32_t emulate_cmd_reg(const struct pci_dev *pdev, uint32_t cmd) { /* TODO: Add proper emulation for all bits of the command register. */ @@ -504,7 +503,13 @@ static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, } #endif - cmd_write(pdev, reg, cmd, data); + return cmd; +} + +static void guest_cmd_write(const struct pci_dev *pdev, unsigned int reg, + uint32_t cmd, void *data) +{ + cmd_write(pdev, reg, emulate_cmd_reg(pdev, cmd), data); } static void bar_write(const struct pci_dev *pdev, unsigned int reg, @@ -678,6 +683,10 @@ static int init_bars(struct pci_dev *pdev) return -EOPNOTSUPP; } + /* Reset the command register for the guest. */ + if ( !is_hwdom ) + pci_conf_write16(pdev->sbdf, PCI_COMMAND, emulate_cmd_reg(pdev, 0)); + /* Setup a handler for the command register. */ rc = vpci_add_register(pdev->vpci, vpci_hw_read16, is_hwdom ? cmd_write : guest_cmd_write, From patchwork Thu Nov 25 11:02:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 71C7AC433F5 for ; Thu, 25 Nov 2021 11:03:22 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230886.399230 (Exim 4.92) (envelope-from ) id 1mqCX3-0004ne-IV; Thu, 25 Nov 2021 11:03:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230886.399230; Thu, 25 Nov 2021 11:03:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX2-0004kM-MJ; Thu, 25 Nov 2021 11:03:12 +0000 Received: by outflank-mailman (input) for mailman id 230886; Thu, 25 Nov 2021 11:03:10 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX0-0001Km-0F for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:10 +0000 Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [2a00:1450:4864:20::22d]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 4618b0c5-4ddf-11ec-9787-a32c541c8605; Thu, 25 Nov 2021 12:03:09 +0100 (CET) Received: by mail-lj1-x22d.google.com with SMTP id 207so11701807ljf.10 for ; Thu, 25 Nov 2021 03:03:09 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:07 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4618b0c5-4ddf-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v3OzM+kUo0HWtvX10bY+8EGI7EL/11lwc+lbMCf7qNk=; b=jUEMnIzqEnMb2tTxWY2ERzXP+gz/bY9eHslY4VclicZg2ER7+KwvDffPj9x2ZLn7sw ffHw1O40qtnwyNXxYJmMRn4tIPHZomVIPBa55zou05gEwjanHcE3Gh+C9cCEQ0htPb6R PNre5uj/tKGcREoM+UiPIBf0MvRJIYNqly5uJlhBcN6jORMDBbI8cTGxuBcXCsOSUQR9 8hPIWOS7r5EuaK7MlJNL8PzuHvFBrSNMn28dzcmJUjj3nO9NJ08hvNDG4MDuyap8Mz9E xbWnpqnOOnONLnR79ZiQipALs1THJokDQ4StQg6ngoIaCuja8Rp7RK44GO3GGDvj7P3J l+GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v3OzM+kUo0HWtvX10bY+8EGI7EL/11lwc+lbMCf7qNk=; b=qKOQqwAWx+FYaS3fdhfAMxY0zIwS9zniOi2EocZ1Z2bazz4Zvzz7iNM8egVuMuubnJ OEbL6uCQAmO88KzJ0Vq0qgQZVgGgKWIsMhSBh1EgzcwjJN9nIoIMy1RO3kZHTX6kG4zo ipXEXSgukVmsxeOut6jF5QTysnpf+2EOw0B0HWuF/JFOX7KtNkNiM1ccQxMhNWYqnHO9 mZbQT+9TNnZ1C3Su9VJYF4spJFA9Dn13onJmn1Cu2bmZ4OgdzMGE9OfU2Io1FOVbACc+ wTAI8WZ4QO7tw4hx9nmnH8XrSM0yu6KI+BO6nYjFK4PzPuPaDtX1xEqWV3cXydptK1Sh WS2g== X-Gm-Message-State: AOAM532X7pKIoDWMzSDpSXwrjqgseFqRHTkoUyQqNARHELXoxwv1qGoC 0LYy+fRMCi54wcg87RWo1gCjyGhTfpaQVA== X-Google-Smtp-Source: ABdhPJzIu53i+hoVd9Ngr9ybt0u7H25mEZU+kuIkiFrLep98sVOIIkcpRt6BfJXfbtsVLlRksnUejg== X-Received: by 2002:a2e:bc1b:: with SMTP id b27mr24175465ljf.91.1637838188551; Thu, 25 Nov 2021 03:03:08 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 11/14] vpci: add initial support for virtual PCI bus topology Date: Thu, 25 Nov 2021 13:02:48 +0200 Message-Id: <20211125110251.2877218-12-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Please note, that at the moment only function 0 of a multifunction device can be passed through. Signed-off-by: Oleksandr Andrushchenko --- Since v4: - moved and re-worked guest sbdf initializers - s/set_bit/__set_bit - s/clear_bit/__clear_bit - minor comment fix s/Virtual/Guest/ - added VPCI_MAX_VIRT_DEV constant (PCI_SLOT(~0) + 1) which will be used later for counting the number of MMIO handlers required for a guest (Julien) Since v3: - make use of VPCI_INIT - moved all new code to vpci.c which belongs to it - changed open-coded 31 to PCI_SLOT(~0) - added comments and code to reject multifunction devices with functions other than 0 - updated comment about vpci_dev_next and made it unsigned int - implement roll back in case of error while assigning/deassigning devices - s/dom%pd/%pd Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/drivers/vpci/vpci.c | 51 +++++++++++++++++++++++++++++++++++++++++ xen/include/xen/sched.h | 8 +++++++ xen/include/xen/vpci.h | 11 +++++++++ 3 files changed, 70 insertions(+) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 98b12a61be6f..c2fb4d4db233 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -114,6 +114,9 @@ int vpci_add_handlers(struct pci_dev *pdev) spin_lock(&pdev->vpci_lock); pdev->vpci = vpci; INIT_LIST_HEAD(&pdev->vpci->handlers); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + pdev->vpci->guest_sbdf.sbdf = ~0; +#endif header = &pdev->vpci->header; for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) @@ -145,6 +148,53 @@ int vpci_add_handlers(struct pci_dev *pdev) } #ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +int vpci_add_virtual_device(struct pci_dev *pdev) +{ + struct domain *d = pdev->domain; + pci_sbdf_t sbdf = { 0 }; + unsigned long new_dev_number; + + /* + * Each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ + if ( pdev->info.is_extfn ) + { + gdprintk(XENLOG_ERR, "%pp: only function 0 passthrough supported\n", + &pdev->sbdf); + return -EOPNOTSUPP; + } + + new_dev_number = find_first_zero_bit(&d->vpci_dev_assigned_map, + VPCI_MAX_VIRT_DEV); + if ( new_dev_number >= VPCI_MAX_VIRT_DEV ) + return -ENOSPC; + + __set_bit(new_dev_number, &d->vpci_dev_assigned_map); + + /* + * Both segment and bus number are 0: + * - we emulate a single host bridge for the guest, e.g. segment 0 + * - with bus 0 the virtual devices are seen as embedded + * endpoints behind the root complex + * + * TODO: add support for multi-function devices. + */ + sbdf.devfn = PCI_DEVFN(new_dev_number, 0); + pdev->vpci->guest_sbdf = sbdf; + + return 0; + +} +REGISTER_VPCI_INIT(vpci_add_virtual_device, VPCI_PRIORITY_MIDDLE); + +static void vpci_remove_virtual_device(struct domain *d, + const struct pci_dev *pdev) +{ + __clear_bit(pdev->vpci->guest_sbdf.dev, &d->vpci_dev_assigned_map); + pdev->vpci->guest_sbdf.sbdf = ~0; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { @@ -171,6 +221,7 @@ int vpci_deassign_device(struct domain *d, struct pci_dev *pdev) return 0; spin_lock(&pdev->vpci_lock); + vpci_remove_virtual_device(d, pdev); vpci_remove_device_handlers_locked(pdev); spin_unlock(&pdev->vpci_lock); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 28146ee404e6..10bff103317c 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -444,6 +444,14 @@ struct domain #ifdef CONFIG_HAS_PCI struct list_head pdev_list; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * The bitmap which shows which device numbers are already used by the + * virtual PCI bus topology and is used to assign a unique SBDF to the + * next passed through virtual PCI device. + */ + unsigned long vpci_dev_assigned_map; +#endif #endif #ifdef CONFIG_HAS_PASSTHROUGH diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 18319fc329f9..e5258bd7ce90 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -21,6 +21,13 @@ typedef int vpci_register_init_t(struct pci_dev *dev); #define VPCI_ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12) +/* + * Maximum number of devices supported by the virtual bus topology: + * each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ +#define VPCI_MAX_VIRT_DEV (PCI_SLOT(~0) + 1) + #define REGISTER_VPCI_INIT(x, p) \ static vpci_register_init_t *const x##_entry \ __used_section(".data.vpci." p) = x @@ -143,6 +150,10 @@ struct vpci { struct vpci_arch_msix_entry arch; } entries[]; } *msix; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* Guest SBDF of the device. */ + pci_sbdf_t guest_sbdf; +#endif #endif }; From patchwork Thu Nov 25 11:02:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCBCEC433F5 for ; Thu, 25 Nov 2021 11:03:24 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230889.399238 (Exim 4.92) (envelope-from ) id 1mqCX6-0005N3-19; Thu, 25 Nov 2021 11:03:16 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230889.399238; Thu, 25 Nov 2021 11:03:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX5-0005GA-7q; Thu, 25 Nov 2021 11:03:15 +0000 Received: by outflank-mailman (input) for mailman id 230889; Thu, 25 Nov 2021 11:03:12 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX2-0001Km-0Y for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:12 +0000 Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [2a00:1450:4864:20::22b]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 46f9c8c8-4ddf-11ec-9787-a32c541c8605; Thu, 25 Nov 2021 12:03:10 +0100 (CET) Received: by mail-lj1-x22b.google.com with SMTP id z8so11729740ljz.9 for ; Thu, 25 Nov 2021 03:03:10 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:09 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 46f9c8c8-4ddf-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kZqEhK0lSTPPH7s4XeLss9am7aDHkDnrG+VGANSSqhs=; b=kxE0nc7+ZIhSzeKFBn/ycXd/0/Fr6R/JY8BouykvGtlyhqqwj+RmypY5J4aiJjr4WR ZHsjSLNe55PEe1TKhubs2FGmmQIYWGNyjWSkVj+6bL2VYs3AGJL+rwpq1yonyTJxKz41 w0a4EUQD678fE3UElGs9IxfXC/pTd6fHT2Njol3wd/JxXOSyhCZCTe8r5o20qPqX+Jr3 5lQmtmJl6Ygh9qkKTx+jd33mYwf9kHwLhPrpX3RsnMHWUqp6OE2bmjYuxCVHEMPPXoxt ThykhCQjC201gZLo/mCvED+la3mOewg66QIINFh1r/WDQXGir9CXf8hZz8zrTH0eBcuO iVmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kZqEhK0lSTPPH7s4XeLss9am7aDHkDnrG+VGANSSqhs=; b=lE0oStvMsK91m/GbK9/YfPsTh5runSVCnOy07pESdRM7N8ElftLKpx2koYYxVOhZwL Cuq0OIguYZXuy/j8hsbzBF2+oKCjEcTjm4fETvWx8qQzFnu/vN7dk8kkoM9YEb2FpKot RHP0EXvs+9TOopA8sZEgJGnCXwpiT7MvRSdiLvUt/jla1KdGfZozof57Eng0a3vBOwsv bLWwI7ZryrFlY+dr/k4cmF0kFtlEH2vWUzZd0Tv2rGuEvzdblM6Y5QADTYKcrCV6H4sA FeeeWa2Cgo09b2pXi8rEkUs2T3eNKaeaAM7lNzF+PNmHA1WOaI5X1ZKP5aqN245Yh3uP 3iqA== X-Gm-Message-State: AOAM532fNg3ejqvpX6fHurQLvaJsEm0OihqJrGCmACevnMeWQi9a5am/ ExyoF/XGfmNwEq7RX2WnEnGQAgSi7XluOA== X-Google-Smtp-Source: ABdhPJzBvgipfEIhTxTfN3KLHKclXOeyvsJMOM3BP4PRkB6a8NA6Vn42fUxKGhF7gYGXSo2CxNAiYA== X-Received: by 2002:a2e:9f15:: with SMTP id u21mr23980508ljk.132.1637838189955; Thu, 25 Nov 2021 03:03:09 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 12/14] xen/arm: translate virtual PCI bus topology for guests Date: Thu, 25 Nov 2021 13:02:49 +0200 Message-Id: <20211125110251.2877218-13-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko --- Since v4: - indentation fixes - constify struct domain - updated commit message - updates to the new locking scheme (pdev->vpci_lock) Since v3: - revisit locking - move code to vpci.c Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/vpci.c | 18 ++++++++++++++++++ xen/drivers/vpci/vpci.c | 27 +++++++++++++++++++++++++++ xen/include/xen/vpci.h | 1 + 3 files changed, 46 insertions(+) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 8e801f275879..3d134f42d07e 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -41,6 +41,15 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + return 1; +#endif + if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -59,6 +68,15 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, struct pci_host_bridge *bridge = p; pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + if ( !bridge && !vpci_translate_virtual_device(v->domain, &sbdf) ) + return 1; +#endif + return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); } diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index c2fb4d4db233..bdc8c63f73fa 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -195,6 +195,33 @@ static void vpci_remove_virtual_device(struct domain *d, pdev->vpci->guest_sbdf.sbdf = ~0; } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool vpci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf) +{ + struct pci_dev *pdev; + + for_each_pdev( d, pdev ) + { + bool found; + + spin_lock(&pdev->vpci_lock); + found = pdev->vpci && (pdev->vpci->guest_sbdf.sbdf == sbdf->sbdf); + spin_unlock(&pdev->vpci_lock); + + if ( found ) + { + /* Replace guest SBDF with the physical one. */ + *sbdf = pdev->sbdf; + return true; + } + } + + return false; +} + /* Notify vPCI that device is assigned to guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index e5258bd7ce90..21d76929391f 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -280,6 +280,7 @@ static inline void vpci_cancel_pending_locked(struct pci_dev *pdev) /* Notify vPCI that device is assigned/de-assigned to/from guest. */ int vpci_assign_device(struct domain *d, struct pci_dev *pdev); int vpci_deassign_device(struct domain *d, struct pci_dev *pdev); +bool vpci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf); #else static inline int vpci_assign_device(struct domain *d, struct pci_dev *pdev) { From patchwork Thu Nov 25 11:02:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8FFFFC433EF for ; Thu, 25 Nov 2021 11:03:26 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230890.399246 (Exim 4.92) (envelope-from ) id 1mqCX7-0005il-KS; Thu, 25 Nov 2021 11:03:17 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230890.399246; Thu, 25 Nov 2021 11:03:17 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX6-0005az-RF; Thu, 25 Nov 2021 11:03:16 +0000 Received: by outflank-mailman (input) for mailman id 230890; Thu, 25 Nov 2021 11:03:13 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX3-0001Km-0V for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:13 +0000 Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [2a00:1450:4864:20::12c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 47e1a854-4ddf-11ec-9787-a32c541c8605; Thu, 25 Nov 2021 12:03:12 +0100 (CET) Received: by mail-lf1-x12c.google.com with SMTP id r26so15322225lfn.8 for ; Thu, 25 Nov 2021 03:03:12 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:10 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 47e1a854-4ddf-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=L3Qpmr6FqZYgDpc5qkrdE2inG3hirALh0cTqSNvXeb0=; b=A86ZGJDWrW5wsHgmN6KiObnttHnIdUH0wAEP75NY/1CAZXj9JACsj72sxaSPAyqpeL 4eRRX7X8olSqt0PoP3cJ7++u6PbI7vkusgzoRpyKPNXRy8wgMIqJ53DsMhRB8lyRHaHk Bdfpepnrd+Smph0/u8UyLU94866oVm9OFMA5Ll7+S3hq5rAPeyq/C1fKpVAdXYDspytn Rg1N0jc96f7WDkWXJxjJ+wTMrSjsMyeOIYLF4dJg3RAMn1nhECzTTH4g7M99qbEmlrcZ KzLZM+hp1JnQTImPHBTCoGo+tzKuPDYoFaWiBkYw/40v5mg4+vve+g0wqGlr/Qkx8O+r F13g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L3Qpmr6FqZYgDpc5qkrdE2inG3hirALh0cTqSNvXeb0=; b=Ck8pL32pqvZxngmmZmMrdmoOB2z3WSgP6mlgQR1V09JVZqrWf1fltZZKG1o1ofSYLe cJ2JG+gvFlsD3SVMyKfs2qcP1ZLcHvlfGqVAg+SqZsaOwsYEVtt7bcCtZxM3DGiU5C5j ikpUKjVWAn5RuuIihjHaba6gvLSij5OUBVuuofEHGKEYL0t11dFqAKtNQV9jF7esi+Yg wRvm6gQ2nf8SzEuol6L14ofF9NdRjMiGKUGFunJKymgnHMtvaHGNN7dOXSd4FqzAJpnc 2CX+DSrR7lnEQhqZwkY5PK8s8BpvS1aBi7QeBTmWe/UrupFxbLCKVHSZPRDHc7vX1V4b lacA== X-Gm-Message-State: AOAM531+0J7TJDabg1sZDxrvxF+hvlWoc3yU8XyLS76N02B3vnpUi2Ie keAucg0FM+UK8D6Do9JAqKpCZwn9cCK9zw== X-Google-Smtp-Source: ABdhPJyrj13RzkBKxdB8sdeoYBM8yJ42d6o945D4TIYQ05dDgukaapyLHMVXDFefL29quZTh6Cw7jg== X-Received: by 2002:ac2:4c4d:: with SMTP id o13mr22059581lfk.196.1637838191512; Thu, 25 Nov 2021 03:03:11 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 13/14] xen/arm: account IO handlers for emulated PCI MSI-X Date: Thu, 25 Nov 2021 13:02:50 +0200 Message-Id: <20211125110251.2877218-14-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko At the moment, we always allocate an extra 16 slots for IO handlers (see MAX_IO_HANDLER). So while adding IO trap handlers for the emulated MSI-X registers we need to explicitly tell that we have additional IO handlers, so those are accounted. Signed-off-by: Oleksandr Andrushchenko --- Cc: Julien Grall Cc: Stefano Stabellini --- This actually moved here from the part 2 of the prep work for PCI passthrough on Arm as it seems to be the proper place for it. New in v5 --- xen/arch/arm/vpci.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 3d134f42d07e..902f8491e030 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -134,6 +134,8 @@ static int vpci_get_num_handlers_cb(struct domain *d, unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) { + unsigned int count; + if ( !has_vpci(d) ) return 0; @@ -145,7 +147,18 @@ unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) } /* For a single emulated host bridge's configuration space. */ - return 1; + count = 1; + +#ifdef CONFIG_HAS_PCI_MSI + /* + * There's a single MSI-X MMIO handler that deals with both PBA + * and MSI-X tables per each PCI device being passed through. + * Maximum number of emulated virtual devices is VPCI_MAX_VIRT_DEV. + */ + count += VPCI_MAX_VIRT_DEV; +#endif + + return count; } /* From patchwork Thu Nov 25 11:02:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Oleksandr Andrushchenko X-Patchwork-Id: 12638907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA67CC433F5 for ; Thu, 25 Nov 2021 11:03:28 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.230892.399252 (Exim 4.92) (envelope-from ) id 1mqCX8-0005yk-SV; Thu, 25 Nov 2021 11:03:18 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 230892.399252; Thu, 25 Nov 2021 11:03:18 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX7-0005uS-W4; Thu, 25 Nov 2021 11:03:18 +0000 Received: by outflank-mailman (input) for mailman id 230892; Thu, 25 Nov 2021 11:03:15 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mqCX5-0001Km-12 for xen-devel@lists.xenproject.org; Thu, 25 Nov 2021 11:03:15 +0000 Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [2a00:1450:4864:20::12a]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 488f0cd3-4ddf-11ec-9787-a32c541c8605; Thu, 25 Nov 2021 12:03:13 +0100 (CET) Received: by mail-lf1-x12a.google.com with SMTP id l22so15334070lfg.7 for ; Thu, 25 Nov 2021 03:03:13 -0800 (PST) Received: from a2klaptop.epam.com (host-176-36-245-220.b024.la.net.ua. [176.36.245.220]) by smtp.gmail.com with ESMTPSA id bt10sm235165lfb.193.2021.11.25.03.03.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Nov 2021 03:03:12 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 488f0cd3-4ddf-11ec-9787-a32c541c8605 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4HaiT/SE5PB8xSapqdBhw3R3QW3El4s7e5lKw8ZTREI=; b=bfnw8BzGMv/ai6lLmKkEDQIRsRtot4KIqLNG5xgb4M7L96xuEszQVAbb6jR453kFvb nNGmlVvyf3pR6Xav47zlgkrwQg2ZyVmdYoeGNl0PNkYBiGldZn3CRAevXvMBKJyhVCZo T90GjY0T2MiSabXU+jnCwN640V8uz5DjPme1+Op4TZvuglH9++DON0craD3VqeY5w9a8 B+OGZ0BuuFIF21AfjeB/pxAbrr0LmXvU888Wa3Cwyd85kswt2+qWRSynb1snzGTI16L6 KFVfDqKYHKgVtcwBf1nsKP15lCEmQOoexcbOVBbxR4+KlkS9kfqvFS0U02qHes7Cq9se mQoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4HaiT/SE5PB8xSapqdBhw3R3QW3El4s7e5lKw8ZTREI=; b=EF2C7smyVYXsxhsrrYtNtxJZf3+a47SF9KdFa/Gueam25XpqHlLayhHNABqIWcnEsh e+qyt954YSmXkSnLWEfdBejEV3kjE0pnUIk2WCkQtUnGym2i/eI3YO0IIpfcVgAPG3JZ f7alvZUzY6VsISCwlvjfyHs/1QE+aFLB8DDMz+sdUY/iC4KX1Anl/yp+X6baneyxOE/v v3qX77vobhzb1e6dy6BK86d4xs4tv72lYqefMjQd96kmTPTqDj/5t+hr9Aih18nXiBeV 8JOfyiOfwQrSHjF0z6JL4kp1+0CEM9A2iWag8CTLGENUIm0nIcG8f9zE5L0Px2dXwCv/ GnDg== X-Gm-Message-State: AOAM533VE9mfeXYxbJ6aWzbHaI+MItIS4E+/h2+9rQJge8mG2+HVPt3F ZkviSXw6lWqLAXlZml0r04AibfFlYYbQjA== X-Google-Smtp-Source: ABdhPJxHJg03JdEEks4CztqRyth39Hkg5mIrysSqR/7r+OomeU2Fdzj0/2zlZtidUcwKwYWgE2T33Q== X-Received: by 2002:ac2:58c3:: with SMTP id u3mr22100069lfo.103.1637838192718; Thu, 25 Nov 2021 03:03:12 -0800 (PST) From: Oleksandr Andrushchenko To: xen-devel@lists.xenproject.org Cc: julien@xen.org, sstabellini@kernel.org, oleksandr_tyshchenko@epam.com, volodymyr_babchuk@epam.com, Artem_Mygaiev@epam.com, roger.pau@citrix.com, jbeulich@suse.com, andrew.cooper3@citrix.com, george.dunlap@citrix.com, paul@xen.org, bertrand.marquis@arm.com, rahul.singh@arm.com, Oleksandr Andrushchenko Subject: [PATCH v5 14/14] vpci: add TODO for the registers not explicitly handled Date: Thu, 25 Nov 2021 13:02:51 +0200 Message-Id: <20211125110251.2877218-15-andr2000@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211125110251.2877218-1-andr2000@gmail.com> References: <20211125110251.2877218-1-andr2000@gmail.com> MIME-Version: 1.0 From: Oleksandr Andrushchenko For unprivileged guests vpci_{read|write} need to be re-worked to not passthrough accesses to the registers not explicitly handled by the corresponding vPCI handlers: without fixing that passthrough to guests is completely unsafe as Xen allows them full access to the registers. Xen needs to be sure that every register a guest accesses is not going to cause the system to malfunction, so Xen needs to keep a list of the registers it is safe for a guest to access. For example, we should only expose the PCI capabilities that we know are safe for a guest to use, i.e.: MSI and MSI-X initially. The rest of the capabilities should be blocked from guest access, unless we audit them and declare safe for a guest to access. As a reference we might want to look at the approach currently used by QEMU in order to do PCI passthrough. A very limited set of PCI capabilities known to be safe for untrusted access are exposed to the guest and registers need to be explicitly handled or else access is rejected. Xen needs a fairly similar model in vPCI or else none of this will be safe for unprivileged access. Add the corresponding TODO comment to highlight there is a problem that needs to be fixed. Suggested-by: Roger Pau Monné Suggested-by: Jan Beulich Signed-off-by: Oleksandr Andrushchenko --- New in v5 --- xen/drivers/vpci/vpci.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index bdc8c63f73fa..4fb77d08825a 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -493,6 +493,29 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsigned int reg, unsigned int size) if ( !pdev->vpci ) { spin_unlock(&pdev->vpci_lock); + /* + * TODO: for unprivileged guests vpci_{read|write} need to be re-worked + * to not passthrough accesses to the registers not explicitly handled + * by the corresponding vPCI handlers: without fixing that passthrough + * to guests is completely unsafe as Xen allows them full access to + * the registers. + * + * Xen needs to be sure that every register a guest accesses is not + * going to cause the system to malfunction, so Xen needs to keep a + * list of the registers it is safe for a guest to access. + * + * For example, we should only expose the PCI capabilities that we know + * are safe for a guest to use, i.e.: MSI and MSI-X initially. + * The rest of the capabilities should be blocked from guest access, + * unless we audit them and declare safe for a guest to access. + * + * As a reference we might want to look at the approach currently used + * by QEMU in order to do PCI passthrough. A very limited set of PCI + * capabilities known to be safe for untrusted access are exposed to the + * guest and registers need to be explicitly handled or else access is + * rejected. Xen needs a fairly similar model in vPCI or else none of + * this will be safe for unprivileged access. + */ return vpci_read_hw(sbdf, reg, size); }