From patchwork Mon Jul 15 05:19:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A44DFC2BD09 for ; Mon, 15 Jul 2024 05:20:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE86-0008G0-9l; Mon, 15 Jul 2024 01:20:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE7z-0008DH-MC for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:00 -0400 Received: from mail-yb1-xb36.google.com ([2607:f8b0:4864:20::b36]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE7w-0007Qd-Lg for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:19:59 -0400 Received: by mail-yb1-xb36.google.com with SMTP id 3f1490d57ef6-e026a2238d8so3673879276.0 for ; Sun, 14 Jul 2024 22:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020783; x=1721625583; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=kPbp030IzWtL72MPt2loFrgmtqrQcgrAaOwGHqsGR/E=; b=VEJteOypy3zMmF83N2dKEplvaOb5Zx3sfUtkjE5VZbrR4fvhJ20vcv93t2k53isbt7 h7bwsehTqegWoW7d3nBm1tqWtZLftZYxC+D82Yi8SRYxunvo7Z2hXD8HJmcY5cERizEi 3jeTGqEYHk3jtEl3trZYgi025hNiZ3cePmR0zG09gfe2T49KoAiCaosH2RxWAv/eA5t2 GbSagMV3+VdKSN1mj+ew57cQIiuMdL2MbZCdxR3riqVGH6v5IA2+MBv0Wv7pBmaZ1iIj pnZnmZl+VoLAKB8TOrV0s15VwTo2if9flwE8mxN7x3Rsr+x29eMEs4h6uPKLMpGFR8lA tpzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020783; x=1721625583; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kPbp030IzWtL72MPt2loFrgmtqrQcgrAaOwGHqsGR/E=; b=uykd8rmwuN5svvg+u5TVc3noTZGB/zf1oqYFSj9owEEM6M7+prN+ApdegKlRLx8s2H kIi/Li1eaLZNTa7k5y4uZ/oJuUGNJ9Wp7jBag/xwarG2bbvSpsIMIiapqFhzOh+fPhCK Ntpsc69EWjRyCzzrPNG7Qi10aFqUmw6/+32Q6l2mQW1jQh32C1bjVSANwF8bMgSDO4ZI DjnjSSxbEtgZSOb0qoBIKPrFYBMEbOZI70qHSzLsp/P8Tltj7Dfv9fWcFrC5y91Me0jo Q8t4mgS4wrv+AYe3X8pDu2DpNXWisl0bNbFZT3qFlgSBnwCSkQJTFLe01k7c2T7r1Wnl uqMA== X-Gm-Message-State: AOJu0YzwAPb8nH9F9tFQjK9j7Ye4UePLEXZLrtkknhgb7YitpSd9Sa9h GEUBL/cEo0477TrWHD+b+ww20TmqiAfuPn3v+Pb+gXh7KmS/goVbtx+LU4bf5I8= X-Google-Smtp-Source: AGHT+IGHXfcBxvdInsneKiUR+e5sjfrNXmNmtZydJVSeJK4EZmxi2MMNUc4i6EeXIUCD9znliGOMyA== X-Received: by 2002:a5b:601:0:b0:dfa:4d12:7ee0 with SMTP id 3f1490d57ef6-e041b12b51emr20929361276.49.1721020783219; Sun, 14 Jul 2024 22:19:43 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-78e34d2c4edsm2656912a12.49.2024.07.14.22.19.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:19:42 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:07 +0900 Subject: [PATCH v5 1/8] hw/pci: Do not add ROM BAR for SR-IOV VF MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-1-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::b36; envelope-from=akihiko.odaki@daynix.com; helo=mail-yb1-xb36.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A SR-IOV VF cannot have a ROM BAR. Co-developed-by: Yui Washizu Signed-off-by: Akihiko Odaki --- hw/pci/pci.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/hw/pci/pci.c b/hw/pci/pci.c index ca8fb5383765..497a057b79f9 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -2412,6 +2412,14 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom, return; } + if (pci_is_vf(pdev)) { + if (pdev->rom_bar == ON_OFF_AUTO_ON) { + error_setg(errp, "ROM BAR cannot be enabled for SR-IOV VF"); + } + + return; + } + if (load_file || pdev->romsize == UINT32_MAX) { path = qemu_find_file(QEMU_FILE_TYPE_BIOS, pdev->romfile); if (path == NULL) { From patchwork Mon Jul 15 05:19:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89085C3DA45 for ; Mon, 15 Jul 2024 05:20:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE8P-0000Pn-DC; Mon, 15 Jul 2024 01:20:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE87-0008Ll-N5 for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:08 -0400 Received: from mail-oi1-x22d.google.com ([2607:f8b0:4864:20::22d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE7y-0007RS-8Y for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:07 -0400 Received: by mail-oi1-x22d.google.com with SMTP id 5614622812f47-3d9e13ef9aaso2515037b6e.1 for ; Sun, 14 Jul 2024 22:19:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020788; x=1721625588; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=YW/hY5XCH9ysRbsD3Qw09WbyPo+KgPIedwlSdOeEQ/I=; b=VzZmLZmr7q2xwEIUbv44GKi5Vl6mvNijx/Ig0yyaCubVFmuaMXQZcbuZfSM5WCchTJ Kx9DOo8z9RODFOcTNuBfdeJ09an6OetkDNqwkL+wJUFEeHrEpdBuTPE3YTQi3oHyMj4S HFM6KB+giuBLPSbEMK+V2U/DNFhvzHeG4sLovs4I8O6/LmrbHCbVQaJfOGSuAheQRiqq KGqsj+su2FXawj4IBCrC5HPTGu+zlK/zcbndHDuPdxg0wEgmQq63TyfnN4rCqP30x2IQ ZboAFhPL0U9d1ASAs+g92qoOyLqOrpxBgHzPL4TDRnxNL4YYtIBkWEQiHUYWvxW0GJ8Q 1esw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020788; x=1721625588; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YW/hY5XCH9ysRbsD3Qw09WbyPo+KgPIedwlSdOeEQ/I=; b=klkMom2N37dAEGNZ9uRvclaqJHqpomgyp8qP4yUNzB4JrNRB+Z5yr4VAKAh1sAp6ca RThFI1CfIR/ZNAQXzon3En18xIyTWfsiLp4QgH8ohm0aEagQy1IDDhYUYQGuIx2DT7H+ 0pV7pSVDa5BEtq2WE/zfnyGFP7k/nie0zFZf7knXyPB4rclrufiDY9/IZekKv0cl24YS Qep/IReVTKn76ws5cteE2/yoMVenTsr+hbJdlSAmYsTyowKZkgUMRMWWxuyVfDnj++o9 RPMup+hp/jO1MTpR56k7ZpS4tqYWUyM5CqG326TscthDGwLT0FY8Pq58rtyYx4n23LwS O4ZQ== X-Gm-Message-State: AOJu0YxZqUF7LTc1PIRKtWjI1p04uCCeE15d6gMRrKndcTNKiseg1qm0 Ekatpa2fxZA94xUgAoF+jCtDFkWcQymFzQnebTU9UGmwjxSuID/TVpoigIX8MLw= X-Google-Smtp-Source: AGHT+IFN2JQLdponyCJBZ1jucvrPQlDscMYYstuf9XCTzad79EXooRWCe+QA57JVUTSx+igDfVwE5g== X-Received: by 2002:a05:6870:82a4:b0:255:1819:b444 with SMTP id 586e51a60fabf-25eae75ec76mr15200830fac.7.1721020788160; Sun, 14 Jul 2024 22:19:48 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-70b7ecd301csm3437136b3a.220.2024.07.14.22.19.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:19:47 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:08 +0900 Subject: [PATCH v5 2/8] hw/pci: Fix SR-IOV VF number calculation MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-2-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::22d; envelope-from=akihiko.odaki@daynix.com; helo=mail-oi1-x22d.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org pci_config_get_bar_addr() had a division by vf_stride. vf_stride needs to be non-zero when there are multiple VFs, but the specification does not prohibit to make it zero when there is only one VF. Do not perform the division for the first VF to avoid division by zero. Signed-off-by: Akihiko Odaki --- hw/pci/pci.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/pci/pci.c b/hw/pci/pci.c index 497a057b79f9..ae7137d70579 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -1490,7 +1490,11 @@ static pcibus_t pci_config_get_bar_addr(PCIDevice *d, int reg, pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_OFFSET); uint16_t vf_stride = pci_get_word(pf->config + sriov_cap + PCI_SRIOV_VF_STRIDE); - uint32_t vf_num = (d->devfn - (pf->devfn + vf_offset)) / vf_stride; + uint32_t vf_num = d->devfn - (pf->devfn + vf_offset); + + if (vf_num) { + vf_num /= vf_stride; + } if (type & PCI_BASE_ADDRESS_MEM_TYPE_64) { new_addr = pci_get_quad(pf->config + bar); From patchwork Mon Jul 15 05:19:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8ABA9C3DA45 for ; Mon, 15 Jul 2024 05:21:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE8V-0001K7-NL; Mon, 15 Jul 2024 01:20:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE8Q-0000gJ-DL for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:26 -0400 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE7y-0007Rw-8R for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:10 -0400 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-70b07bdbfbcso3599032b3a.0 for ; Sun, 14 Jul 2024 22:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020793; x=1721625593; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=ZZGRu8v6oxuF32W/xrA/69BjX55/YFUQg7PXZsmlE7o=; b=Ak4EJU3VSqk1eCVWAUb0+PbMC+ctopGi9UG+dTE0YWMpwdVHFLnlfC2w1mSPB/muIc zcYrSV2TAyLC0dpFHTvAnrHbSsZbsziJfENZJ0OmMxFxmM4ecp+z16BN8gJCroJKr2bA 02LTDoVZjqJ1J2tJ2e2baQ2ZCpR5YxNf26QRf5FoCrlgoKT1tNVqPXmaw3EBuH8PppGy tZdi3T8/FIkmyBolzREeptdG3LCZrVHN0s2xAlodNcE/n5mSSs4Z24H5BY/Rfs9MUPT8 ysi8N4WOatshT/uZU6PeTDv44rL0HS0agx/tsMzjJQq0L7NiqGEXFro4WvYYYGmtiW0y fz4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020793; x=1721625593; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZZGRu8v6oxuF32W/xrA/69BjX55/YFUQg7PXZsmlE7o=; b=lGe5tmhlbV8KtefONo+v460mdQ52ZDaHPF/5UUICkKRkEXALcLGFVanOj47nhG7vCe l22mbbbWTV3CrWS99C+6yfHBW1LEE67VRiTyAkBM8iIixIQKaKHEKbGULWSaouhQD3z1 ZaK5ryWdIwiE7U1Vnbw5s8Sq9h24V+rcN127hKY472eD6S7ZOFn4QH89PWr/Dp1ki69Y BDg+tAfm1rD5aw/c/nRBPNQGjxpE2ydHZsZX8+eozqT+SmCheHJpKr8glSrkGR9FVlHk Lo5EJ+5gsReT0TApIxlwgaiu4NjHUD0Ii9SjzPrt/DMZ/a3DKwqeU9ika8NISvyENwNo xraA== X-Gm-Message-State: AOJu0YwlnszXhyqSTeN7arfWWEVDBoXzJW1Cx0mpqRG7Wp1XHCRRHsRY r7CFZilVMweJwtKrwTTK+DLC1QQ/ySgJ2s1ZL0T7NG32uBE6iAfsa0dfgX395r8= X-Google-Smtp-Source: AGHT+IG4YjG0ntIffx9f8EzYgy5fvK5gEa4orev9+ojliP35f+0nuEj+8HiWLZO6H79kpdfZStR7vA== X-Received: by 2002:a05:6a20:cfa9:b0:1c2:8d1b:2776 with SMTP id adf61e73a8af0-1c3bed16a76mr12920506637.20.1721020793302; Sun, 14 Jul 2024 22:19:53 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-70b7ecd362dsm3394705b3a.215.2024.07.14.22.19.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:19:53 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:09 +0900 Subject: [PATCH v5 3/8] pcie_sriov: Ensure PF and VF are mutually exclusive MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-3-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::42f; envelope-from=akihiko.odaki@daynix.com; helo=mail-pf1-x42f.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A device cannot be a SR-IOV PF and a VF at the same time. Signed-off-by: Akihiko Odaki --- hw/pci/pcie_sriov.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c index 56523ab4e833..ec8fc0757b92 100644 --- a/hw/pci/pcie_sriov.c +++ b/hw/pci/pcie_sriov.c @@ -42,6 +42,11 @@ bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, uint8_t *cfg = dev->config + offset; uint8_t *wmask; + if (pci_is_vf(dev)) { + error_setg(errp, "a device cannot be a SR-IOV PF and a VF at the same time"); + return false; + } + if (total_vfs) { uint16_t ari_cap = pcie_find_capability(dev, PCI_EXT_CAP_ID_ARI); uint16_t first_vf_devfn = dev->devfn + vf_offset; From patchwork Mon Jul 15 05:19:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6E77C2BD09 for ; Mon, 15 Jul 2024 05:21:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE88-0008Oe-Gi; Mon, 15 Jul 2024 01:20:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE82-0008FB-UK for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:06 -0400 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE80-0007Sg-SW for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:02 -0400 Received: by mail-pg1-x52f.google.com with SMTP id 41be03b00d2f7-7669d62b5bfso2226314a12.1 for ; Sun, 14 Jul 2024 22:20:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020799; x=1721625599; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=eIrCsb1ntTQZ4u2QiCIc/D2bj4QGskaeiB3kw7Zorrg=; b=ncqbhm6eHRDzmRZflRXi7eqrkjbrD+Dl88qNh/oUrlWzjVVyyF3N1u1oiClj/K7SF8 q7yAm15R43pxUSfMukKWSvivYgBjs7trccqT25Pjex0OImxO7dZCdPOyHpjTh/SnC2pP 9YNmo0LI+71W0keNAZS7y3QMI8scdNxE6tYIXQp5HsPa6cl881mN1EkaMomv43cmY8PD mf4mVBkozz1G4K4lz5b86lecZXzb+/xSBduzB5EBhCanuTeNJP0HNpBCp4zVWxJkY94q ma53AT9IN/l74t/d8/SwA70IfBAFN1T/fI2COlbuTWj4nrx3GfAcpo2T3nhGZnIJ4KRj Lu/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020799; x=1721625599; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eIrCsb1ntTQZ4u2QiCIc/D2bj4QGskaeiB3kw7Zorrg=; b=HILa0o0v67vE0jxubSOQAwu6TFWUnoLvJLrxGh1HdQbTEzd3Cv3cwxXr28gnBDDE3n 4y6oTNReTBrlOaJKuYk+zdXQJZV9xs3FfSipuOzngZitkRzJdwSeUBaqsgQUd6ZLqrFk PoRZ+xQJVwG6cj/JpGqop4l3dYwAW/HMNNcpvXDGSlbF6Jb5fHv1pQTjXUi4H2DLSaSD YZU45zVfOCuerW5zFOBySeQEhzpFVzMcCQvAKB6fB9oNr8GkIrKXamu7Rahb/IXbvKVf 8UL+AAKa0ywN25sYlWeY4SOlfY8LQgK782YSlhxjh9LYb6KYDDCuNlarV+fc5XAAMPPu B4Hg== X-Gm-Message-State: AOJu0YxePa2s+C3w9c2M/XuNDC7dq7C6ajPrBjPhe6peSXMAraVVDfQN WH5vtnOpofAMbDyV7bUNEo5wB4L2pS8GxnCZ4M+d68TJ5j0QjtRzIZ96SsIw8dc= X-Google-Smtp-Source: AGHT+IFib7LEFM18p1VEiwK50F5jIM8zxzKC7ZaQb9YCDW0n1iSyWhpeQ8eQ4kfcat+6QasdswQ2zw== X-Received: by 2002:a05:6a20:7b2a:b0:1c2:8f07:579e with SMTP id adf61e73a8af0-1c2984d7835mr14019965637.52.1721020798928; Sun, 14 Jul 2024 22:19:58 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2cacd3e5a87sm5430253a91.14.2024.07.14.22.19.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:19:58 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:10 +0900 Subject: [PATCH v5 4/8] pcie_sriov: Check PCI Express for SR-IOV PF MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-4-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::52f; envelope-from=akihiko.odaki@daynix.com; helo=mail-pg1-x52f.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org SR-IOV requires PCI Express. Signed-off-by: Akihiko Odaki --- hw/pci/pcie_sriov.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c index ec8fc0757b92..3af0cc7d560a 100644 --- a/hw/pci/pcie_sriov.c +++ b/hw/pci/pcie_sriov.c @@ -42,6 +42,11 @@ bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, uint8_t *cfg = dev->config + offset; uint8_t *wmask; + if (!pci_is_express(dev)) { + error_setg(errp, "PCI Express is required for SR-IOV PF"); + return false; + } + if (pci_is_vf(dev)) { error_setg(errp, "a device cannot be a SR-IOV PF and a VF at the same time"); return false; From patchwork Mon Jul 15 05:19:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5351C2BD09 for ; Mon, 15 Jul 2024 05:21:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE8U-00018N-Fz; Mon, 15 Jul 2024 01:20:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE8Q-0000gD-Cj for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:26 -0400 Received: from mail-ot1-x32b.google.com ([2607:f8b0:4864:20::32b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE86-0007dr-0q for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:08 -0400 Received: by mail-ot1-x32b.google.com with SMTP id 46e09a7af769-708adad61f8so2055381a34.1 for ; Sun, 14 Jul 2024 22:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020803; x=1721625603; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=Qybwk3OW5qaGVUA/oEO6VnAFeQ3iixWcj0s4ESw4IO8=; b=VJ8AhaqaCER/iTXs91nrT//y8XDg//uddeiHb97m9DMm5CPU+NZZ+7ZC+dAaQAcMmo 2AMMDL4XGnSHdCFs7QzTQHXaKW6Q0nv8StVRAgaTq+K38W7MHByzzSHRKfIZI/SWjflZ fYQeaFdSd888Bn4Gggc2H8gXKtHsVZMie7cHkpAZnCkoyQU0xwzRl8oZwYUGJTJ7mMNZ PZyMt6/nqefdmpW8JFeXtNVuna1cR5UR3+g6zZIQQ58IbW2W5f3aLu5uZvlQnL5OSWOV 8Ct+ft1CSSqjWxfImLLKOmuVVPSqabsyxZj6VWLjrXdZEHkxPO5mNReOOV1iveliQW3d J+nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020803; x=1721625603; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qybwk3OW5qaGVUA/oEO6VnAFeQ3iixWcj0s4ESw4IO8=; b=bm6CclSpnJW9cgbdI7GH6jM+I8nKy69VMlUZc//sl15XtmaCMpFBi7c+fwrzdjSWtI 5A7qUqjKWkdKmxwDsNQyocqT7lE8qWfJWBqTOhIRB/lZSg5UsA7dCdxDaeWcHKSutCw9 TP6WQeQMbMUY0UzrhpoqxdEdiVDMl6f1/F3dfSll4Qx5ZuQnKNfWjb3mTV82JBEkAdPz 4cB/FkBGa83b6rDicXXQWNKWRWzqNU8vP3Jbg9Hiv/aMAqMnc5g9eQ279jL2v8Tipg1V CtAul9aMgVyI+BfWP0CQuzx2+5dJpFVLfRpvksT2cwZSOd8i9IkPV8HXBDFGMUBMabOP CWpg== X-Gm-Message-State: AOJu0Yyk02osKbLHELHsm5jv9JoqOLAheE3LXEmzNsaGNktxLvvqMx1X 1HSPEt1blCusOphlVsUrhyTAexJ0WwtKjLPNNuRo3gjsCNYGEeAp3bp0wxrCQz4= X-Google-Smtp-Source: AGHT+IEMo5MRbjeRqJlgtgic1dhobI312RHDg3mD4SOTmvZ0ueQBTwFXErrKBkOI3e61samFoU2kbw== X-Received: by 2002:a05:6830:18f1:b0:703:6703:8909 with SMTP id 46e09a7af769-70375a055ccmr21149158a34.11.1721020803572; Sun, 14 Jul 2024 22:20:03 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-70b7eb9e285sm3422482b3a.8.2024.07.14.22.20.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:20:03 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:11 +0900 Subject: [PATCH v5 5/8] pcie_sriov: Allow user to create SR-IOV device MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-5-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::32b; envelope-from=akihiko.odaki@daynix.com; helo=mail-ot1-x32b.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A user can create a SR-IOV device by specifying the PF with the sriov-pf property of the VFs. The VFs must be added before the PF. A user-creatable VF must have PCIDeviceClass::sriov_vf_user_creatable set. Such a VF cannot refer to the PF because it is created before the PF. A PF that user-creatable VFs can be attached calls pcie_sriov_pf_init_from_user_created_vfs() during realization and pcie_sriov_pf_exit() when exiting. Signed-off-by: Akihiko Odaki --- include/hw/pci/pci_device.h | 6 +- include/hw/pci/pcie_sriov.h | 18 +++ hw/pci/pci.c | 62 ++++++---- hw/pci/pcie_sriov.c | 288 +++++++++++++++++++++++++++++++++++--------- 4 files changed, 291 insertions(+), 83 deletions(-) diff --git a/include/hw/pci/pci_device.h b/include/hw/pci/pci_device.h index 49b341ce2e27..14d391333aff 100644 --- a/include/hw/pci/pci_device.h +++ b/include/hw/pci/pci_device.h @@ -37,6 +37,8 @@ struct PCIDeviceClass { uint16_t subsystem_id; /* only for header type = 0 */ const char *romfile; /* rom bar */ + + bool sriov_vf_user_creatable; }; enum PCIReqIDType { @@ -160,6 +162,8 @@ struct PCIDevice { /* ID of standby device in net_failover pair */ char *failover_pair_id; uint32_t acpi_index; + + char *sriov_pf; }; static inline int pci_intx(PCIDevice *pci_dev) @@ -192,7 +196,7 @@ static inline int pci_is_express_downstream_port(const PCIDevice *d) static inline int pci_is_vf(const PCIDevice *d) { - return d->exp.sriov_vf.pf != NULL; + return d->sriov_pf || d->exp.sriov_vf.pf != NULL; } static inline uint32_t pci_config_size(const PCIDevice *d) diff --git a/include/hw/pci/pcie_sriov.h b/include/hw/pci/pcie_sriov.h index c5d2d318d330..f75b8f22ee92 100644 --- a/include/hw/pci/pcie_sriov.h +++ b/include/hw/pci/pcie_sriov.h @@ -18,6 +18,7 @@ typedef struct PCIESriovPF { uint8_t vf_bar_type[PCI_NUM_REGIONS]; /* Store type for each VF bar */ PCIDevice **vf; /* Pointer to an array of num_vfs VF devices */ + bool vf_user_created; /* If VFs are created by user */ } PCIESriovPF; typedef struct PCIESriovVF { @@ -40,6 +41,23 @@ void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num, void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num, MemoryRegion *memory); +/** + * pcie_sriov_pf_init_from_user_created_vfs() - Initialize PF with user-created + * VFs. + * @dev: A PCIe device being realized. + * @offset: The offset of the SR-IOV capability. + * @errp: pointer to Error*, to store an error if it happens. + * + * Return: The size of added capability. 0 if the user did not create VFs. + * -1 if failed. + */ +int16_t pcie_sriov_pf_init_from_user_created_vfs(PCIDevice *dev, + uint16_t offset, + Error **errp); + +bool pcie_sriov_register_device(PCIDevice *dev, Error **errp); +void pcie_sriov_unregister_device(PCIDevice *dev); + /* * Default (minimal) page size support values * as required by the SR/IOV standard: diff --git a/hw/pci/pci.c b/hw/pci/pci.c index ae7137d70579..0368e448992a 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -138,6 +138,7 @@ static Property pci_props[] = { QEMU_PCIE_ERR_UNC_MASK_BITNR, true), DEFINE_PROP_BIT("x-pcie-ari-nextfn-1", PCIDevice, cap_present, QEMU_PCIE_ARI_NEXTFN_1_BITNR, false), + DEFINE_PROP_STRING("sriov-pf", PCIDevice, sriov_pf), DEFINE_PROP_END_OF_LIST() }; @@ -1012,13 +1013,8 @@ static void pci_init_multifunction(PCIBus *bus, PCIDevice *dev, Error **errp) dev->config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION; } - /* - * With SR/IOV and ARI, a device at function 0 need not be a multifunction - * device, as it may just be a VF that ended up with function 0 in - * the legacy PCI interpretation. Avoid failing in such cases: - */ - if (pci_is_vf(dev) && - dev->exp.sriov_vf.pf->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { + /* SR/IOV is not handled here. */ + if (pci_is_vf(dev)) { return; } @@ -1051,7 +1047,8 @@ static void pci_init_multifunction(PCIBus *bus, PCIDevice *dev, Error **errp) } /* function 0 indicates single function, so function > 0 must be NULL */ for (func = 1; func < PCI_FUNC_MAX; ++func) { - if (bus->devices[PCI_DEVFN(slot, func)]) { + PCIDevice *device = bus->devices[PCI_DEVFN(slot, func)]; + if (device && !pci_is_vf(device)) { error_setg(errp, "PCI: %x.0 indicates single function, " "but %x.%x is already populated.", slot, slot, func); @@ -1336,6 +1333,7 @@ static void pci_qdev_unrealize(DeviceState *dev) pci_unregister_io_regions(pci_dev); pci_del_option_rom(pci_dev); + pcie_sriov_unregister_device(pci_dev); if (pc->exit) { pc->exit(pci_dev); @@ -1367,7 +1365,6 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num, pcibus_t size = memory_region_size(memory); uint8_t hdr_type; - assert(!pci_is_vf(pci_dev)); /* VFs must use pcie_sriov_vf_register_bar */ assert(region_num >= 0); assert(region_num < PCI_NUM_REGIONS); assert(is_power_of_2(size)); @@ -1378,7 +1375,6 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num, assert(hdr_type != PCI_HEADER_TYPE_BRIDGE || region_num < 2); r = &pci_dev->io_regions[region_num]; - r->addr = PCI_BAR_UNMAPPED; r->size = size; r->type = type; r->memory = memory; @@ -1386,22 +1382,35 @@ void pci_register_bar(PCIDevice *pci_dev, int region_num, ? pci_get_bus(pci_dev)->address_space_io : pci_get_bus(pci_dev)->address_space_mem; - wmask = ~(size - 1); - if (region_num == PCI_ROM_SLOT) { - /* ROM enable bit is writable */ - wmask |= PCI_ROM_ADDRESS_ENABLE; - } - - addr = pci_bar(pci_dev, region_num); - pci_set_long(pci_dev->config + addr, type); + if (pci_is_vf(pci_dev)) { + PCIDevice *pf = pci_dev->exp.sriov_vf.pf; + assert(!pf || type == pf->exp.sriov_pf.vf_bar_type[region_num]); - if (!(r->type & PCI_BASE_ADDRESS_SPACE_IO) && - r->type & PCI_BASE_ADDRESS_MEM_TYPE_64) { - pci_set_quad(pci_dev->wmask + addr, wmask); - pci_set_quad(pci_dev->cmask + addr, ~0ULL); + r->addr = pci_bar_address(pci_dev, region_num, r->type, r->size); + if (r->addr != PCI_BAR_UNMAPPED) { + memory_region_add_subregion_overlap(r->address_space, + r->addr, r->memory, 1); + } } else { - pci_set_long(pci_dev->wmask + addr, wmask & 0xffffffff); - pci_set_long(pci_dev->cmask + addr, 0xffffffff); + r->addr = PCI_BAR_UNMAPPED; + + wmask = ~(size - 1); + if (region_num == PCI_ROM_SLOT) { + /* ROM enable bit is writable */ + wmask |= PCI_ROM_ADDRESS_ENABLE; + } + + addr = pci_bar(pci_dev, region_num); + pci_set_long(pci_dev->config + addr, type); + + if (!(r->type & PCI_BASE_ADDRESS_SPACE_IO) && + r->type & PCI_BASE_ADDRESS_MEM_TYPE_64) { + pci_set_quad(pci_dev->wmask + addr, wmask); + pci_set_quad(pci_dev->cmask + addr, ~0ULL); + } else { + pci_set_long(pci_dev->wmask + addr, wmask & 0xffffffff); + pci_set_long(pci_dev->cmask + addr, 0xffffffff); + } } } @@ -2162,6 +2171,11 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp) } } + if (!pcie_sriov_register_device(pci_dev, errp)) { + pci_qdev_unrealize(DEVICE(pci_dev)); + return; + } + /* * A PCIe Downstream Port that do not have ARI Forwarding enabled must * associate only Device 0 with the device attached to the bus diff --git a/hw/pci/pcie_sriov.c b/hw/pci/pcie_sriov.c index 3af0cc7d560a..0c875e61fe96 100644 --- a/hw/pci/pcie_sriov.c +++ b/hw/pci/pcie_sriov.c @@ -20,6 +20,8 @@ #include "qapi/error.h" #include "trace.h" +static GHashTable *pfs; + static void unparent_vfs(PCIDevice *dev, uint16_t total_vfs) { for (uint16_t i = 0; i < total_vfs; i++) { @@ -31,14 +33,49 @@ static void unparent_vfs(PCIDevice *dev, uint16_t total_vfs) dev->exp.sriov_pf.vf = NULL; } -bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, - const char *vfname, uint16_t vf_dev_id, - uint16_t init_vfs, uint16_t total_vfs, - uint16_t vf_offset, uint16_t vf_stride, - Error **errp) +static void clear_ctrl_vfe(PCIDevice *dev) +{ + uint8_t *ctrl = dev->config + dev->exp.sriov_cap + PCI_SRIOV_CTRL; + pci_set_word(ctrl, pci_get_word(ctrl) & ~PCI_SRIOV_CTRL_VFE); +} + +static void register_vfs(PCIDevice *dev) +{ + uint16_t num_vfs; + uint16_t i; + uint16_t sriov_cap = dev->exp.sriov_cap; + + assert(sriov_cap > 0); + num_vfs = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF); + if (num_vfs > pci_get_word(dev->config + sriov_cap + PCI_SRIOV_TOTAL_VF)) { + clear_ctrl_vfe(dev); + return; + } + + trace_sriov_register_vfs(dev->name, PCI_SLOT(dev->devfn), + PCI_FUNC(dev->devfn), num_vfs); + for (i = 0; i < num_vfs; i++) { + pci_set_enabled(dev->exp.sriov_pf.vf[i], true); + } +} + +static void unregister_vfs(PCIDevice *dev) +{ + uint16_t i; + uint8_t *cfg = dev->config + dev->exp.sriov_cap; + + trace_sriov_unregister_vfs(dev->name, PCI_SLOT(dev->devfn), + PCI_FUNC(dev->devfn)); + for (i = 0; i < pci_get_word(cfg + PCI_SRIOV_TOTAL_VF); i++) { + pci_set_enabled(dev->exp.sriov_pf.vf[i], false); + } +} + +static bool pcie_sriov_pf_init_common(PCIDevice *dev, uint16_t offset, + uint16_t vf_dev_id, uint16_t init_vfs, + uint16_t total_vfs, uint16_t vf_offset, + uint16_t vf_stride, Error **errp) { - BusState *bus = qdev_get_parent_bus(&dev->qdev); - int32_t devfn = dev->devfn + vf_offset; uint8_t *cfg = dev->config + offset; uint8_t *wmask; @@ -100,6 +137,28 @@ bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, qdev_prop_set_bit(&dev->qdev, "multifunction", true); + return true; +} + +bool pcie_sriov_pf_init(PCIDevice *dev, uint16_t offset, + const char *vfname, uint16_t vf_dev_id, + uint16_t init_vfs, uint16_t total_vfs, + uint16_t vf_offset, uint16_t vf_stride, + Error **errp) +{ + BusState *bus = qdev_get_parent_bus(&dev->qdev); + int32_t devfn = dev->devfn + vf_offset; + + if (pfs && g_hash_table_contains(pfs, dev->qdev.id)) { + error_setg(errp, "attaching user-created SR-IOV VF unsupported"); + return false; + } + + if (!pcie_sriov_pf_init_common(dev, offset, vf_dev_id, init_vfs, + total_vfs, vf_offset, vf_stride, errp)) { + return false; + } + dev->exp.sriov_pf.vf = g_new(PCIDevice *, total_vfs); for (uint16_t i = 0; i < total_vfs; i++) { @@ -129,7 +188,22 @@ void pcie_sriov_pf_exit(PCIDevice *dev) { uint8_t *cfg = dev->config + dev->exp.sriov_cap; - unparent_vfs(dev, pci_get_word(cfg + PCI_SRIOV_TOTAL_VF)); + if (dev->exp.sriov_pf.vf_user_created) { + uint16_t ven_id = pci_get_word(dev->config + PCI_VENDOR_ID); + uint16_t total_vfs = pci_get_word(dev->config + PCI_SRIOV_TOTAL_VF); + uint16_t vf_dev_id = pci_get_word(dev->config + PCI_SRIOV_VF_DID); + + unregister_vfs(dev); + + for (uint16_t i = 0; i < total_vfs; i++) { + dev->exp.sriov_pf.vf[i]->exp.sriov_vf.pf = NULL; + + pci_config_set_vendor_id(dev->exp.sriov_pf.vf[i]->config, ven_id); + pci_config_set_device_id(dev->exp.sriov_pf.vf[i]->config, vf_dev_id); + } + } else { + unparent_vfs(dev, pci_get_word(cfg + PCI_SRIOV_TOTAL_VF)); + } } void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num, @@ -162,74 +236,172 @@ void pcie_sriov_pf_init_vf_bar(PCIDevice *dev, int region_num, void pcie_sriov_vf_register_bar(PCIDevice *dev, int region_num, MemoryRegion *memory) { - PCIIORegion *r; - PCIBus *bus = pci_get_bus(dev); uint8_t type; - pcibus_t size = memory_region_size(memory); - assert(pci_is_vf(dev)); /* PFs must use pci_register_bar */ - assert(region_num >= 0); - assert(region_num < PCI_NUM_REGIONS); + assert(dev->exp.sriov_vf.pf); type = dev->exp.sriov_vf.pf->exp.sriov_pf.vf_bar_type[region_num]; - if (!is_power_of_2(size)) { - error_report("%s: PCI region size must be a power" - " of two - type=0x%x, size=0x%"FMT_PCIBUS, - __func__, type, size); - exit(1); - } - - r = &dev->io_regions[region_num]; - r->memory = memory; - r->address_space = - type & PCI_BASE_ADDRESS_SPACE_IO - ? bus->address_space_io - : bus->address_space_mem; - r->size = size; - r->type = type; - - r->addr = pci_bar_address(dev, region_num, r->type, r->size); - if (r->addr != PCI_BAR_UNMAPPED) { - memory_region_add_subregion_overlap(r->address_space, - r->addr, r->memory, 1); - } + return pci_register_bar(dev, region_num, type, memory); } -static void clear_ctrl_vfe(PCIDevice *dev) +static gint compare_vf_devfns(gconstpointer a, gconstpointer b) { - uint8_t *ctrl = dev->config + dev->exp.sriov_cap + PCI_SRIOV_CTRL; - pci_set_word(ctrl, pci_get_word(ctrl) & ~PCI_SRIOV_CTRL_VFE); + return (*(PCIDevice **)a)->devfn - (*(PCIDevice **)b)->devfn; } -static void register_vfs(PCIDevice *dev) +int16_t pcie_sriov_pf_init_from_user_created_vfs(PCIDevice *dev, + uint16_t offset, + Error **errp) { - uint16_t num_vfs; + GPtrArray *pf; + PCIDevice **vfs; + BusState *bus = qdev_get_parent_bus(DEVICE(dev)); + uint16_t ven_id = pci_get_word(dev->config + PCI_VENDOR_ID); + uint16_t vf_dev_id; + uint16_t vf_offset; + uint16_t vf_stride; uint16_t i; - uint16_t sriov_cap = dev->exp.sriov_cap; - assert(sriov_cap > 0); - num_vfs = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF); - if (num_vfs > pci_get_word(dev->config + sriov_cap + PCI_SRIOV_TOTAL_VF)) { - clear_ctrl_vfe(dev); - return; + if (!pfs || !dev->qdev.id) { + return 0; } - trace_sriov_register_vfs(dev->name, PCI_SLOT(dev->devfn), - PCI_FUNC(dev->devfn), num_vfs); - for (i = 0; i < num_vfs; i++) { - pci_set_enabled(dev->exp.sriov_pf.vf[i], true); + pf = g_hash_table_lookup(pfs, dev->qdev.id); + if (!pf) { + return 0; + } + + if (pf->len > UINT16_MAX) { + error_setg(errp, "too many VFs"); + return -1; + } + + g_ptr_array_sort(pf, compare_vf_devfns); + vfs = (void *)pf->pdata; + + if (vfs[0]->devfn <= dev->devfn) { + error_setg(errp, "a VF function number is less than the PF function number"); + return -1; + } + + vf_dev_id = pci_get_word(vfs[0]->config + PCI_DEVICE_ID); + vf_offset = vfs[0]->devfn - dev->devfn; + vf_stride = pf->len < 2 ? 0 : vfs[1]->devfn - vfs[0]->devfn; + + for (i = 0; i < pf->len; i++) { + if (bus != qdev_get_parent_bus(&vfs[i]->qdev)) { + error_setg(errp, "SR-IOV VF parent bus mismatches with PF"); + return -1; + } + + if (ven_id != pci_get_word(vfs[i]->config + PCI_VENDOR_ID)) { + error_setg(errp, "SR-IOV VF vendor ID mismatches with PF"); + return -1; + } + + if (vf_dev_id != pci_get_word(vfs[i]->config + PCI_DEVICE_ID)) { + error_setg(errp, "inconsistent SR-IOV VF device IDs"); + return -1; + } + + for (size_t j = 0; j < PCI_NUM_REGIONS; j++) { + if (vfs[i]->io_regions[j].size != vfs[0]->io_regions[j].size || + vfs[i]->io_regions[j].type != vfs[0]->io_regions[j].type) { + error_setg(errp, "inconsistent SR-IOV BARs"); + return -1; + } + } + + if (vfs[i]->devfn - vfs[0]->devfn != vf_stride * i) { + error_setg(errp, "inconsistent SR-IOV stride"); + return -1; + } + } + + if (!pcie_sriov_pf_init_common(dev, offset, vf_dev_id, pf->len, + pf->len, vf_offset, vf_stride, errp)) { + return -1; + } + + for (i = 0; i < pf->len; i++) { + vfs[i]->exp.sriov_vf.pf = dev; + vfs[i]->exp.sriov_vf.vf_number = i; + + /* set vid/did according to sr/iov spec - they are not used */ + pci_config_set_vendor_id(vfs[i]->config, 0xffff); + pci_config_set_device_id(vfs[i]->config, 0xffff); } + + dev->exp.sriov_pf.vf = vfs; + dev->exp.sriov_pf.vf_user_created = true; + + for (i = 0; i < PCI_NUM_REGIONS; i++) { + PCIIORegion *region = &vfs[0]->io_regions[i]; + + if (region->size) { + pcie_sriov_pf_init_vf_bar(dev, i, region->type, region->size); + } + } + + return PCI_EXT_CAP_SRIOV_SIZEOF; } -static void unregister_vfs(PCIDevice *dev) +bool pcie_sriov_register_device(PCIDevice *dev, Error **errp) { - uint16_t i; - uint8_t *cfg = dev->config + dev->exp.sriov_cap; + if (!dev->exp.sriov_pf.vf && dev->qdev.id && + pfs && g_hash_table_contains(pfs, dev->qdev.id)) { + error_setg(errp, "attaching user-created SR-IOV VF unsupported"); + return false; + } - trace_sriov_unregister_vfs(dev->name, PCI_SLOT(dev->devfn), - PCI_FUNC(dev->devfn)); - for (i = 0; i < pci_get_word(cfg + PCI_SRIOV_TOTAL_VF); i++) { - pci_set_enabled(dev->exp.sriov_pf.vf[i], false); + if (dev->sriov_pf) { + PCIDevice *pci_pf; + GPtrArray *pf; + + if (!PCI_DEVICE_GET_CLASS(dev)->sriov_vf_user_creatable) { + error_setg(errp, "user cannot create SR-IOV VF with this device type"); + return false; + } + + if (!pci_is_express(dev)) { + error_setg(errp, "PCI Express is required for SR-IOV VF"); + return false; + } + + if (!pci_qdev_find_device(dev->sriov_pf, &pci_pf)) { + error_setg(errp, "PCI device specified as SR-IOV PF already exists"); + return false; + } + + if (!pfs) { + pfs = g_hash_table_new_full(g_str_hash, g_str_equal, g_free, NULL); + } + + pf = g_hash_table_lookup(pfs, dev->sriov_pf); + if (!pf) { + pf = g_ptr_array_new(); + g_hash_table_insert(pfs, g_strdup(dev->sriov_pf), pf); + } + + g_ptr_array_add(pf, dev); + } + + return true; +} + +void pcie_sriov_unregister_device(PCIDevice *dev) +{ + if (dev->sriov_pf && pfs) { + GPtrArray *pf = g_hash_table_lookup(pfs, dev->sriov_pf); + + if (pf) { + g_ptr_array_remove_fast(pf, dev); + + if (!pf->len) { + g_hash_table_remove(pfs, dev->sriov_pf); + g_ptr_array_free(pf, FALSE); + } + } } } @@ -316,7 +488,7 @@ void pcie_sriov_pf_add_sup_pgsize(PCIDevice *dev, uint16_t opt_sup_pgsize) uint16_t pcie_sriov_vf_number(PCIDevice *dev) { - assert(pci_is_vf(dev)); + assert(dev->exp.sriov_vf.pf); return dev->exp.sriov_vf.vf_number; } From patchwork Mon Jul 15 05:19:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18429C2BD09 for ; Mon, 15 Jul 2024 05:21:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE8W-0001Uz-Sp; Mon, 15 Jul 2024 01:20:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE8S-0000re-0a for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:29 -0400 Received: from mail-oi1-x236.google.com ([2607:f8b0:4864:20::236]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE8H-0007hA-RS for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:27 -0400 Received: by mail-oi1-x236.google.com with SMTP id 5614622812f47-3dab3b85623so963455b6e.2 for ; Sun, 14 Jul 2024 22:20:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020808; x=1721625608; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=vdR/tAAVHxSpgFFmSFp64O9usJu3eSZIZ8gMew9CW0c=; b=yMKRNvO/412j2m4tgLw0ysuznne3+xzXMr6qAQeW//i2Shl9S3HP8sU1YUhVDillFq KjG50WagSWfUlR52ag7z1vtrVbeOIqA7iHfjaMy+Om95npM4HO76Si/0bVl1no8MGTl9 mgtxYTW3q/cMfuFMR+6xn7E0/h33495JpxuSLgtwaCspmZCaC1kKwpQgv6TPeNeXKI1/ M9cuXNJvvLoIY4zNo86lmDpUkyWKL1XUqf3x/J3QCGtvUFUbLA93emUEPvZ/mCQ1ZM5F 5Glv+Y4SPCf9CBH/nKBxEnR+f/kL4RiTYtOphQitS7LTyzOXCYrTt4QX2klOqxghSWkS YdPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020808; x=1721625608; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vdR/tAAVHxSpgFFmSFp64O9usJu3eSZIZ8gMew9CW0c=; b=YQEKzr7no+ulfNZscL5TyuKLvNCQpBbwdkJW7V9SYu3gIi06irFixEVGNJWw9Ba5sU YtsRdMDbBMYOiSKcSmQ/9sosHBZiGtOAkNdb6/cwRvskEBm+ecYtGyonMuJixMBV03/E xXMNzW8njZ85s6UKyL0j2tkpDBwLPyUxjMS5qqLMzQSZDwMViE6QkUYPZ0bxy8OkY/+m 4gg2EvRYRo9bC8rtjdNSLL6MAjdxLN0J6Nf/xzf/OUwPuCs5lCvIHX3l0XdFB3N6UtZb t2x/tSjKUBLpRvfM3+fgDTPkM+0pZADHW6+XMiBCID9X+MreReRG91hoSVVXFJeQOXFE SFXw== X-Gm-Message-State: AOJu0Ywx8mx9LbUK9sb1+A9y5SWXN3qQKEQarRtCHAq7X3cTfYxX6tKm qfWZtTzsjoxxj6koVIyLHpNTTRBBfNWpHCxNxIMpfEux0SyVE/R8HptZXgtZ8yw= X-Google-Smtp-Source: AGHT+IGgO0GaQBGK1+McyNpKJeJkC+M8Gp1+szDrNR0E95izawSAiQ8dyXyho98fpVeSPKk3U4Qg3g== X-Received: by 2002:a05:6808:23d6:b0:3da:b22d:a5ab with SMTP id 5614622812f47-3dab22da976mr8228651b6e.49.1721020808213; Sun, 14 Jul 2024 22:20:08 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-70b7ebb6adesm3407865b3a.73.2024.07.14.22.20.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:20:07 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:12 +0900 Subject: [PATCH v5 6/8] virtio-pci: Implement SR-IOV PF MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-6-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::236; envelope-from=akihiko.odaki@daynix.com; helo=mail-oi1-x236.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Allow user to attach SR-IOV VF to a virtio-pci PF. Signed-off-by: Akihiko Odaki --- include/hw/virtio/virtio-pci.h | 1 + hw/virtio/virtio-pci.c | 20 +++++++++++++++----- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h index 9e67ba38c748..34539f2f6722 100644 --- a/include/hw/virtio/virtio-pci.h +++ b/include/hw/virtio/virtio-pci.h @@ -152,6 +152,7 @@ struct VirtIOPCIProxy { uint32_t modern_io_bar_idx; uint32_t modern_mem_bar_idx; int config_cap; + uint16_t last_pcie_cap_offset; uint32_t flags; bool disable_modern; bool ignore_backend_features; diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 9534730bba19..0c8fcc5627d5 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -1955,6 +1955,7 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp) uint8_t *config; uint32_t size; VirtIODevice *vdev = virtio_bus_get_device(bus); + int16_t res; /* * Virtio capabilities present without @@ -2100,6 +2101,14 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp) pci_register_bar(&proxy->pci_dev, proxy->legacy_io_bar_idx, PCI_BASE_ADDRESS_SPACE_IO, &proxy->bar); } + + res = pcie_sriov_pf_init_from_user_created_vfs(&proxy->pci_dev, + proxy->last_pcie_cap_offset, + errp); + if (res > 0) { + proxy->last_pcie_cap_offset += res; + virtio_add_feature(&vdev->host_features, VIRTIO_F_SR_IOV); + } } static void virtio_pci_device_unplugged(DeviceState *d) @@ -2187,7 +2196,7 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp) if (pcie_port && pci_is_express(pci_dev)) { int pos; - uint16_t last_pcie_cap_offset = PCI_CONFIG_SPACE_SIZE; + proxy->last_pcie_cap_offset = PCI_CONFIG_SPACE_SIZE; pos = pcie_endpoint_cap_init(pci_dev, 0); assert(pos > 0); @@ -2207,9 +2216,9 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp) pci_set_word(pci_dev->config + pos + PCI_PM_PMC, 0x3); if (proxy->flags & VIRTIO_PCI_FLAG_AER) { - pcie_aer_init(pci_dev, PCI_ERR_VER, last_pcie_cap_offset, + pcie_aer_init(pci_dev, PCI_ERR_VER, proxy->last_pcie_cap_offset, PCI_ERR_SIZEOF, NULL); - last_pcie_cap_offset += PCI_ERR_SIZEOF; + proxy->last_pcie_cap_offset += PCI_ERR_SIZEOF; } if (proxy->flags & VIRTIO_PCI_FLAG_INIT_DEVERR) { @@ -2234,9 +2243,9 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp) } if (proxy->flags & VIRTIO_PCI_FLAG_ATS) { - pcie_ats_init(pci_dev, last_pcie_cap_offset, + pcie_ats_init(pci_dev, proxy->last_pcie_cap_offset, proxy->flags & VIRTIO_PCI_FLAG_ATS_PAGE_ALIGNED); - last_pcie_cap_offset += PCI_EXT_CAP_ATS_SIZEOF; + proxy->last_pcie_cap_offset += PCI_EXT_CAP_ATS_SIZEOF; } if (proxy->flags & VIRTIO_PCI_FLAG_INIT_FLR) { @@ -2263,6 +2272,7 @@ static void virtio_pci_exit(PCIDevice *pci_dev) bool pcie_port = pci_bus_is_express(pci_get_bus(pci_dev)) && !pci_bus_is_root(pci_get_bus(pci_dev)); + pcie_sriov_pf_exit(&proxy->pci_dev); msix_uninit_exclusive_bar(pci_dev); if (proxy->flags & VIRTIO_PCI_FLAG_AER && pcie_port && pci_is_express(pci_dev)) { From patchwork Mon Jul 15 05:19:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF582C2BD09 for ; Mon, 15 Jul 2024 05:21:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE8V-0001J8-K0; Mon, 15 Jul 2024 01:20:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE8S-0000uF-Dg for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:29 -0400 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE8P-0007i0-P0 for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:28 -0400 Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-70b702be5e4so1721705b3a.0 for ; Sun, 14 Jul 2024 22:20:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020814; x=1721625614; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=NlrY553tddcDoCWOe56+tRoCgkm3F8OvMmopjLbpLLI=; b=fdAYiZ93VHpy3qejRXnu/jUQKxnNEdsjXtnEyA2q+36v1/4XFmPq0g3RSrCA5r9FJO zCR3LODh+VJBJo7QfhiD+FZ5whNeBKgSVSc2EowRQIB/3/8cGDB1wDOKwOccZJ6ecCMI xhDepq9aoubBzM2uXkocXGGAUGguyhh13UL9kguyC835wqVrn3oqn2Ho8jyNJ5MI6nph PqZDOsdOVvm6X1tREjVmwrEc7pbQ8jC1HRAMVers2YypPuwVlU0AC0NKvdD72PAuHlUj 3JzbOSzC+MZev3wWVIv2V9xY1883llYIkuuQKuqcXVmlIVUVZTUewdLNx6LDBdcCui7b id0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020814; x=1721625614; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NlrY553tddcDoCWOe56+tRoCgkm3F8OvMmopjLbpLLI=; b=Ar/VqPjgbnK+AYazu0h0+lHui7QzP5C7cUKNN678/dKgUU4/kovw3RCQCW2G2a45w8 AVGbNjM0G4Efh+McjHRjnoYHGZ0+sU3//5r7bq3aW9Lxv98PO/MDxSk8aV0qpwBNZFw2 yVWtsHDptV3q1ixen7BY3e4dDJFh7mK2aFU7NImohqOkvCl8uCYm/0VJR7U0N1MDPvZi 1ncCm/PMe9YuaXU25bEgLjsRsvLkDUUIkQSvSFEhfOoUt8VGwDo/QQOekAGPPyQZ9Ulu bkGBHtoXRV0afYnrpQsjNQIb9sEma3l2r8YO6zskkwTIT+Z/N/+0yxoIjIGY8X9Q/Wyr L+DA== X-Gm-Message-State: AOJu0Yx9R/WKRFYbhkBrGMa01hOtKee98HIFSBLE7H1O5OipsB5Hpi12 cOJtkDQ952TCLNfdGW4EeymErGKleeOOQe08rRxLOgw9rEeDr9+jNGT9fQUkV9A= X-Google-Smtp-Source: AGHT+IErokbw4zB+9xbYFnB4kBHSAh+U7fYTlr6MWbvurqXNSIAG0wa5vkdyMS5ChXICEfbirrXRvg== X-Received: by 2002:a05:6a21:32a1:b0:1c0:e997:7081 with SMTP id adf61e73a8af0-1c29822cff8mr23862825637.29.1721020813751; Sun, 14 Jul 2024 22:20:13 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2cacd419e5asm5437630a91.28.2024.07.14.22.20.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:20:13 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:13 +0900 Subject: [PATCH v5 7/8] virtio-net: Implement SR-IOV VF MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-7-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::42f; envelope-from=akihiko.odaki@daynix.com; helo=mail-pf1-x42f.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A virtio-net device can be added as a SR-IOV VF to another virtio-pci device that will be the PF. Signed-off-by: Akihiko Odaki --- hw/virtio/virtio-net-pci.c | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/virtio/virtio-net-pci.c b/hw/virtio/virtio-net-pci.c index e03543a70a75..dba4987d6e04 100644 --- a/hw/virtio/virtio-net-pci.c +++ b/hw/virtio/virtio-net-pci.c @@ -75,6 +75,7 @@ static void virtio_net_pci_class_init(ObjectClass *klass, void *data) k->device_id = PCI_DEVICE_ID_VIRTIO_NET; k->revision = VIRTIO_PCI_ABI_VERSION; k->class_id = PCI_CLASS_NETWORK_ETHERNET; + k->sriov_vf_user_creatable = true; set_bit(DEVICE_CATEGORY_NETWORK, dc->categories); device_class_set_props(dc, virtio_net_properties); vpciklass->realize = virtio_net_pci_realize; From patchwork Mon Jul 15 05:19:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akihiko Odaki X-Patchwork-Id: 13732900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A87DDC3DA4B for ; Mon, 15 Jul 2024 05:21:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sTE8W-0001Qu-D0; Mon, 15 Jul 2024 01:20:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sTE8U-00013e-2j for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:30 -0400 Received: from mail-oi1-x233.google.com ([2607:f8b0:4864:20::233]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sTE8R-0007jK-LL for qemu-devel@nongnu.org; Mon, 15 Jul 2024 01:20:29 -0400 Received: by mail-oi1-x233.google.com with SMTP id 5614622812f47-3d9e13ef8edso2539963b6e.2 for ; Sun, 14 Jul 2024 22:20:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daynix-com.20230601.gappssmtp.com; s=20230601; t=1721020820; x=1721625620; darn=nongnu.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=RSyxXdQYMZg83yy/c3D8saUoTuohOw2eIYAXKlv1nkY=; b=jD4XVfdiyW3FJs8L80oeqr3UMclQrDS2Oht8y5XuMApwmMwjNbdMW2oDojFwcsLUor AxaQpxnUIzNkBWwG++5/EjHskoQiZal0AAjsNOkcCfWbiyKxNR/3dpZY+so4xhFhjWwU +WCD0Gjtsu/dvDsrywAbZt2/OwtUJOOXTxBD9Ky+TPUHx2wWH4BUz7Saipk3GvrLqw56 EYj9syBdooSzuCcPtgv56Bu/tZqmPGXqvt2C4hEdX/2Ad3n1hJST4ouQLG8SjmSXik0W 2UdjlyQCbb4PclduBN1i/n24fd0FW4nSGRgJD59IGqTY/Cb9qtI9JV6mkzgrLy2+1I8T Yfrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721020820; x=1721625620; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RSyxXdQYMZg83yy/c3D8saUoTuohOw2eIYAXKlv1nkY=; b=b+5+Ma5J3777xH7zileDk4hFfnVkpiI/mussC7NxxNviLEk3W31rN/mvup4CECshdO j1+eCzdLGVH8vnXGDCd9eVQLYxJfWBV1Jd3TR4cj5rXHOHEtdcbRd4jr5CQM9BHlr9Hh pfp13IYtG+ZaP2aTc/eh8JxGC5gOGtWpGMEyEll6Hzvgx8/8xCJZG05eUgM3DwrZArQ3 Q5v4Pz+8KJlwRRJXnm9idjG3W/dgQevo7KC2QJX/STlJTw8ibgB2T28gtPJ0amGGsJ1R q8dMJwrLrt5OBY7aVOYI/AAxhlND3sIjFrMUwyBMJMcX5B3iJk28sIiNRpmKLH4X5D5J PGOQ== X-Gm-Message-State: AOJu0YxXriksbkfitl9PB/eaIJ7qYBcvE/9enSrWp8hnJJjQ7+in43dS aXiKSYPD3TByUkeMwab6WW7zrFbL0fh/wRY9nG92EVUR8oZmRgHsLXGMsl1tKG800DUMqvQ6/pV IMYM= X-Google-Smtp-Source: AGHT+IGOeeOYo1mFPDGIODgqEMs+PesSFDuNNohkqj2KLkcAtBKb1XGMWVzdmJ5KT19m9Cq9/IsSRA== X-Received: by 2002:a05:6808:179e:b0:3d9:2986:5a3b with SMTP id 5614622812f47-3d93c085bb7mr22043699b6e.37.1721020818843; Sun, 14 Jul 2024 22:20:18 -0700 (PDT) Received: from localhost ([157.82.204.135]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-78e32b6be78sm2105831a12.15.2024.07.14.22.20.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 14 Jul 2024 22:20:18 -0700 (PDT) From: Akihiko Odaki Date: Mon, 15 Jul 2024 14:19:14 +0900 Subject: [PATCH v5 8/8] docs: Document composable SR-IOV device MIME-Version: 1.0 Message-Id: <20240715-sriov-v5-8-3f5539093ffc@daynix.com> References: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> In-Reply-To: <20240715-sriov-v5-0-3f5539093ffc@daynix.com> To: "Michael S. Tsirkin" , Marcel Apfelbaum , Alex Williamson , =?utf-8?q?C=C3=A9dric_Le_Goa?= =?utf-8?q?ter?= , Paolo Bonzini , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eduardo Habkost , Jason Wang , Sriram Yagnaraman , Keith Busch , Klaus Jensen Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Yui Washizu , Akihiko Odaki X-Mailer: b4 0.14-dev-fd6e3 Received-SPF: none client-ip=2607:f8b0:4864:20::233; envelope-from=akihiko.odaki@daynix.com; helo=mail-oi1-x233.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Signed-off-by: Akihiko Odaki --- MAINTAINERS | 1 + docs/system/index.rst | 1 + docs/system/sriov.rst | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 6725913c8b3a..ca0222e5528b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2008,6 +2008,7 @@ F: hw/pci-bridge/* F: qapi/pci.json F: docs/pci* F: docs/specs/*pci* +F: docs/system/sriov.rst PCIE DOE M: Huai-Cheng Kuo diff --git a/docs/system/index.rst b/docs/system/index.rst index c21065e51932..718e9d3c56bb 100644 --- a/docs/system/index.rst +++ b/docs/system/index.rst @@ -39,3 +39,4 @@ or Hypervisor.Framework. multi-process confidential-guest-support vm-templating + sriov diff --git a/docs/system/sriov.rst b/docs/system/sriov.rst new file mode 100644 index 000000000000..a851a66a4b8b --- /dev/null +++ b/docs/system/sriov.rst @@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +Compsable SR-IOV device +======================= + +SR-IOV (Single Root I/O Virtualization) is an optional extended capability of a +PCI Express device. It allows a single physical function (PF) to appear as +multiple virtual functions (VFs) for the main purpose of eliminating software +overhead in I/O from virtual machines. + +There are devices with predefined SR-IOV configurations, but it is also possible +to compose an SR-IOV device yourself. Composing an SR-IOV device is currently +only supported by virtio-net-pci. + +Users can configure an SR-IOV-capable virtio-net device by adding +virtio-net-pci functions to a bus. Below is a command line example: + +.. code-block:: shell + + -netdev user,id=n -netdev user,id=o + -netdev user,id=p -netdev user,id=q + -device pcie-root-port,id=b + -device virtio-net-pci,bus=b,addr=0x0.0x3,netdev=q,sriov-pf=f + -device virtio-net-pci,bus=b,addr=0x0.0x2,netdev=p,sriov-pf=f + -device virtio-net-pci,bus=b,addr=0x0.0x1,netdev=o,sriov-pf=f + -device virtio-net-pci,bus=b,addr=0x0.0x0,netdev=n,id=f + +The VFs specify the paired PF with ``sriov-pf`` property. The PF must be +added after all VFs. It is the user's responsibility to ensure that VFs have +function numbers larger than one of the PF, and that the function numbers +have a consistent stride. + +You may also need to perform additional steps to activate the SR-IOV feature on +your guest. For Linux, refer to [1]_. + +.. [1] https://docs.kernel.org/PCI/pci-iov-howto.html