From patchwork Fri Feb 25 03:09:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhang, Tianfei" X-Patchwork-Id: 12759577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC57BC433EF for ; Fri, 25 Feb 2022 03:15:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236940AbiBYDNw (ORCPT ); Thu, 24 Feb 2022 22:13:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230117AbiBYDNq (ORCPT ); Thu, 24 Feb 2022 22:13:46 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E1BD64BE7; Thu, 24 Feb 2022 19:13:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1645758796; x=1677294796; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=peqfZ+rFk7ZwJ07YEOVnePiWyPNcAj7SeTjZgu1wjWo=; b=mKTR7uZ+ydvOMo1yALM61PTrA/p2IB7YdtLmPADCk6wkgzcHYlxrndUE YyaMT/+BXHkjEMFZQcQ/6E0DzT3GWdF0CQCHKcvGaKYn1W5A3u02xycvO jSIfLJmUaJrWTkF6H9t5jq6DAFxjQ+9GGyuFnN1F/pqxVE5QYyitB6Kw8 813Sfc9aXpJ0XXScON239lcz1r2KlHKyYYJhPt4YggrUiXCZjE9Gdp6G8 qItU74Ap2bVLlIYFwaZhwmCuHY/wIziCFBN8r7eh/N1AYVTFLHDnsRFMb Yb8LSCCidmPAFT1Zk+G5uYes+jTqw10ZMPmAoMglh1hw7PX+6uzakRndv w==; X-IronPort-AV: E=McAfee;i="6200,9189,10268"; a="252602981" X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="252602981" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2022 19:13:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="684512442" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by fmsmga001.fm.intel.com with ESMTP; 24 Feb 2022 19:13:13 -0800 From: Tianfei zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: corbet@lwn.net, Matthew Gerlach , Tianfei Zhang Subject: [PATCH v2 1/5] fpga: dfl: Allow for ports without specific bar space. Date: Thu, 24 Feb 2022 22:09:58 -0500 Message-Id: <20220225031002.261264-2-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220225031002.261264-1-tianfei.zhang@intel.com> References: <20220225031002.261264-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fpga@vger.kernel.org From: Matthew Gerlach In IOFS, there is a Port device for each PR slot, like Port control, Port user clock control and Port errors, those feature devices are linked with DFL. The DFL of Port device was located in PCIe Bar 0 MMIO space by default, but it also can put into any PCIe Bar space. If the BarID (3bits field) in PORTn_OFFSET register set to invalid means that DFL of Port device is located in the Bar 0 by default, in this case, it don't need add the Bar 0 into dfl list twice. --- v2: use FME_HDR_NO_PORT_BAR instead of PCI_STD_NUM_BARS. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei Zhang --- drivers/fpga/dfl-pci.c | 6 ++++++ drivers/fpga/dfl.h | 1 + 2 files changed, 7 insertions(+) diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c index 4d68719e608f..33545c999c06 100644 --- a/drivers/fpga/dfl-pci.c +++ b/drivers/fpga/dfl-pci.c @@ -258,6 +258,12 @@ static int find_dfls_by_default(struct pci_dev *pcidev, */ bar = FIELD_GET(FME_PORT_OFST_BAR_ID, v); offset = FIELD_GET(FME_PORT_OFST_DFH_OFST, v); + if (bar >= FME_HDR_NO_PORT_BAR) { + dev_dbg(&pcidev->dev, "skipping port without specific BAR space %d\n", + bar); + continue; + } + start = pci_resource_start(pcidev, bar) + offset; len = pci_resource_len(pcidev, bar) - offset; diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h index 53572c7aced0..1fd493e82dd8 100644 --- a/drivers/fpga/dfl.h +++ b/drivers/fpga/dfl.h @@ -91,6 +91,7 @@ #define FME_HDR_PORT_OFST(n) (0x38 + ((n) * 0x8)) #define FME_HDR_BITSTREAM_ID 0x60 #define FME_HDR_BITSTREAM_MD 0x68 +#define FME_HDR_NO_PORT_BAR 7 /* FME Fab Capability Register Bitfield */ #define FME_CAP_FABRIC_VERID GENMASK_ULL(7, 0) /* Fabric version ID */ From patchwork Fri Feb 25 03:09:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhang, Tianfei" X-Patchwork-Id: 12759579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A698EC433FE for ; Fri, 25 Feb 2022 03:15:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236949AbiBYDNz (ORCPT ); Thu, 24 Feb 2022 22:13:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236942AbiBYDNu (ORCPT ); Thu, 24 Feb 2022 22:13:50 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAB047DA91; Thu, 24 Feb 2022 19:13:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1645758798; x=1677294798; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KdPQ6SeVdfaExi695kgdsiaNjw0SiA/pAv18ILKo/QM=; b=DMgGq/JzrK3Nq7ONjagHmKCGNUQCMwZoVzK3sNZ5FG88gMQqAvIEqfFX FgOE76eVlghnmQefikgXPkcfYhfD9R8kTQSZpTcQ4hQK7QzNKguszoLQ5 fjwBmR6fk9uA/xGKvCT1Yxj93rD1i7pHi8vX9fyYoYNI+IWvr2CDrJqZ2 ZP0vFzYCxdjfeE0CDVUHRyVXmTMsS0w0eUNLsAWFmjsO5QNco7PknQ60a W4srsR00e5nyYKYlsZJPTvYpSlEjuBggVfofnJLQ/dPHpCSAwiUj19dsY nJTvZGq6SLkhEJKLpepDFLDDHC5A/MYdIn/q4jGfjmkm8OVroyHcMNm5I w==; X-IronPort-AV: E=McAfee;i="6200,9189,10268"; a="252602990" X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="252602990" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2022 19:13:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="684512463" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by fmsmga001.fm.intel.com with ESMTP; 24 Feb 2022 19:13:16 -0800 From: Tianfei zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: corbet@lwn.net, Tianfei zhang Subject: [PATCH v2 2/5] fpga: dfl: add features in dfl_fpga_cdev Date: Thu, 24 Feb 2022 22:09:59 -0500 Message-Id: <20220225031002.261264-3-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220225031002.261264-1-tianfei.zhang@intel.com> References: <20220225031002.261264-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fpga@vger.kernel.org Introducing features in dfl_fpga_cdev during DFL enumeration. On IOFS, we will add more extensions or features in DFL in future, so adding a new member "features"in dfl_fpga_cdev. For example, in the legacy model, the AFU was connected to Port device, but in "multiple VFs per PR slot" model, the AFU or PR slot without connected to Port device directly, so in this model, we only can access the resource of AFU or PR slot via VFs. In this patch, we introducing a new flags DFL_FEAT_PORT_CONNECTED_AFU to distinguish them. Signed-off-by: Tianfei zhang --- drivers/fpga/dfl.c | 6 +++++- drivers/fpga/dfl.h | 5 +++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index 6bff39ff21a0..9b7a01a4af04 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -1130,6 +1130,7 @@ static void build_info_complete(struct build_feature_devs_info *binfo) static int parse_feature_fiu(struct build_feature_devs_info *binfo, resource_size_t ofst) { + struct dfl_fpga_cdev *cdev = binfo->cdev; int ret = 0; u32 offset; u16 id; @@ -1166,8 +1167,11 @@ static int parse_feature_fiu(struct build_feature_devs_info *binfo, v = readq(binfo->ioaddr + NEXT_AFU); offset = FIELD_GET(NEXT_AFU_NEXT_DFH_OFST, v); - if (offset) + if (offset) { + if (dfh_id_to_type(id) == PORT_ID) + cdev->features |= DFL_FEAT_PORT_CONNECTED_AFU; return parse_feature_afu(binfo, offset); + } dev_dbg(binfo->dev, "No AFUs detected on FIU %d\n", id); diff --git a/drivers/fpga/dfl.h b/drivers/fpga/dfl.h index 1fd493e82dd8..6171bcdcb3c5 100644 --- a/drivers/fpga/dfl.h +++ b/drivers/fpga/dfl.h @@ -461,6 +461,9 @@ int dfl_fpga_enum_info_add_irq(struct dfl_fpga_enum_info *info, unsigned int nr_irqs, int *irq_table); void dfl_fpga_enum_info_free(struct dfl_fpga_enum_info *info); +/* in legacy model, the AFU was connected to Port device */ +#define DFL_FEAT_PORT_CONNECTED_AFU BIT_ULL(0) + /** * struct dfl_fpga_cdev - container device of DFL based FPGA * @@ -470,6 +473,7 @@ void dfl_fpga_enum_info_free(struct dfl_fpga_enum_info *info); * @lock: mutex lock to protect the port device list. * @port_dev_list: list of all port feature devices under this container device. * @released_port_num: released port number under this container device. + * @features: features discovered during DFL enumeration. */ struct dfl_fpga_cdev { struct device *parent; @@ -478,6 +482,7 @@ struct dfl_fpga_cdev { struct mutex lock; struct list_head port_dev_list; int released_port_num; + u64 features; }; struct dfl_fpga_cdev * From patchwork Fri Feb 25 03:10:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhang, Tianfei" X-Patchwork-Id: 12759578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96E18C433F5 for ; Fri, 25 Feb 2022 03:15:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236957AbiBYDN5 (ORCPT ); Thu, 24 Feb 2022 22:13:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236945AbiBYDNy (ORCPT ); Thu, 24 Feb 2022 22:13:54 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C34FA74618; Thu, 24 Feb 2022 19:13:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1645758801; x=1677294801; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UwdCvXzSdTBDzvJOc2d5fc844vP3KlSqJURgIRaEWqs=; b=FMmsHONSgVQ9dSUzhiiW8rpdlNKz8FaqEzHDJvKwDsh4ulXSuTGTpV4t fUxcx83oNSG/icbTawHHsQJpWxU1s5t8UWMAqaR7YWsi7HB4oGe/x3sY+ SI6GsJYqPdYEGCXu94eJlGSAkx5OR8wFYcY2zpqFoVqp+tNExgCql3GUh /FGoQMiUXnVxpxNcRY/ZYhxmlchb597TLUpp3HKrbhFurArYzQwUdag27 PgUZ3fFKW/uOsXaU1dwo+xaC1UKPqTBwAXMLptDmSscsNgmSNadSaTipk PXZ8zn64nHvSFsKijjy7ztOLyFZS1AJOCbOU4dd97JlIsqZpWJkpss9d3 g==; X-IronPort-AV: E=McAfee;i="6200,9189,10268"; a="252602995" X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="252602995" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2022 19:13:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="684512482" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by fmsmga001.fm.intel.com with ESMTP; 24 Feb 2022 19:13:18 -0800 From: Tianfei zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: corbet@lwn.net, Tianfei zhang , Matthew Gerlach Subject: [PATCH v2 3/5] fpga: dfl: fix VF creation in IOFS Date: Thu, 24 Feb 2022 22:10:00 -0500 Message-Id: <20220225031002.261264-4-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220225031002.261264-1-tianfei.zhang@intel.com> References: <20220225031002.261264-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fpga@vger.kernel.org In IOFS legacy model, there is only 1 Port device related to 1 VF, the flag DFL_FEAT_PORT_CONNECTED_AFU will take notes for this model. In legacy model, it need to check the released port number match VF device number or not. But in "Multiple VFs per PR slot" model, the Port device would not connected to AFU/PR slot, so we don't need to release the Port device before creating the VFs. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei zhang --- drivers/fpga/dfl.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index 9b7a01a4af04..afa9311c30d5 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -1708,11 +1708,13 @@ int dfl_fpga_cdev_config_ports_vf(struct dfl_fpga_cdev *cdev, int num_vfs) mutex_lock(&cdev->lock); /* - * can't turn multiple ports into 1 VF device, only 1 port for 1 VF - * device, so if released port number doesn't match VF device number, - * then reject the request with -EINVAL error code. + * In the IOFS legacy model, it can't turn multiple ports into 1 VF + * device, because only 1 port conneced to 1 VF device, so if released + * port number doesn't match VF device number, then reject the request + * with -EINVAL error code. */ - if (cdev->released_port_num != num_vfs) { + if ((cdev->features & DFL_FEAT_PORT_CONNECTED_AFU) && + cdev->released_port_num != num_vfs) { ret = -EINVAL; goto done; } From patchwork Fri Feb 25 03:10:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhang, Tianfei" X-Patchwork-Id: 12759581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0633C433EF for ; Fri, 25 Feb 2022 03:16:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233737AbiBYDPI (ORCPT ); Thu, 24 Feb 2022 22:15:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236954AbiBYDN4 (ORCPT ); Thu, 24 Feb 2022 22:13:56 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFB3E7DA91; Thu, 24 Feb 2022 19:13:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1645758804; x=1677294804; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+6kBKmZZxOFTdF4b8FToTP7Rqc52r8lXs4pLygIIL88=; b=d82ae4h7WDvgclgehNofPHRWwAnzkHwGg0eYlzjiT/7gN0lmVsXxui/G oxzArSMVesZlBYl40HDy5jSXZk+bCeXHvnISfDSq4BQkSIsckPQixP0me rqHPrgyPytOhPemcDQbf09eTfT/N480R+rnvaEG4Ln+xmHv+elKnqmUVc ERkHPS250OXQe6HKRfJiCw8OzA4r+h7M+50oYnTowuIDQDn8BIqcZeZSa 8uIB+sDAEF0zxPGNR+/rEbWULmCP89gCYC+5PMa0jzi3JVax2QJgHspGb Hv+hbVbT7lm0G9/RiycYT5J5bUY+Aon1wXfftCmtC4BkSE8Y//kMeDN+l g==; X-IronPort-AV: E=McAfee;i="6200,9189,10268"; a="252603001" X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="252603001" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2022 19:13:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="684512494" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by fmsmga001.fm.intel.com with ESMTP; 24 Feb 2022 19:13:21 -0800 From: Tianfei zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: corbet@lwn.net, Matthew Gerlach , Tianfei Zhang Subject: [PATCH v2 4/5] fpga: dfl: Handle dfl's starting with AFU Date: Thu, 24 Feb 2022 22:10:01 -0500 Message-Id: <20220225031002.261264-5-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220225031002.261264-1-tianfei.zhang@intel.com> References: <20220225031002.261264-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fpga@vger.kernel.org From: Matthew Gerlach Allow for a Device Feature List (DFL) to start with a Device Feature Header (DFH) of type Accelerator Function Unit (AFU) by doing nothing. This allows for PCIe VFs to be created. Signed-off-by: Matthew Gerlach Signed-off-by: Tianfei Zhang --- drivers/fpga/dfl-pci.c | 7 ++++++- drivers/fpga/dfl.c | 22 +++++++++++++--------- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/drivers/fpga/dfl-pci.c b/drivers/fpga/dfl-pci.c index 33545c999c06..e7d58e7b1bbd 100644 --- a/drivers/fpga/dfl-pci.c +++ b/drivers/fpga/dfl-pci.c @@ -275,7 +275,12 @@ static int find_dfls_by_default(struct pci_dev *pcidev, dfl_fpga_enum_info_add_dfl(info, start, len); } else { - ret = -ENODEV; + v = readq(base + DFH); + if (FIELD_GET(DFH_TYPE, v) != DFH_TYPE_AFU) { + dev_info(&pcidev->dev, "Unknown feature type 0x%llx id 0x%llx\n", + FIELD_GET(DFH_TYPE, v), FIELD_GET(DFH_ID, v)); + ret = -ENODEV; + } } /* release I/O mappings for next step enumeration */ diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index afa9311c30d5..8a71ff6c3e2f 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -900,9 +900,11 @@ static void build_info_free(struct build_feature_devs_info *binfo) dfl_id_free(feature_dev_id_type(binfo->feature_dev), binfo->feature_dev->id); - list_for_each_entry_safe(finfo, p, &binfo->sub_features, node) { - list_del(&finfo->node); - kfree(finfo); + if (!list_empty(&binfo->sub_features)) { + list_for_each_entry_safe(finfo, p, &binfo->sub_features, node) { + list_del(&finfo->node); + kfree(finfo); + } } } @@ -1445,12 +1447,14 @@ dfl_fpga_feature_devs_enumerate(struct dfl_fpga_enum_info *info) * start enumeration for all feature devices based on Device Feature * Lists. */ - list_for_each_entry(dfl, &info->dfls, node) { - ret = parse_feature_list(binfo, dfl->start, dfl->len); - if (ret) { - remove_feature_devs(cdev); - build_info_free(binfo); - goto unregister_region_exit; + if (!list_empty(&info->dfls)) { + list_for_each_entry(dfl, &info->dfls, node) { + ret = parse_feature_list(binfo, dfl->start, dfl->len); + if (ret) { + remove_feature_devs(cdev); + build_info_free(binfo); + goto unregister_region_exit; + } } } From patchwork Fri Feb 25 03:10:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhang, Tianfei" X-Patchwork-Id: 12759580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE89EC433F5 for ; Fri, 25 Feb 2022 03:15:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236974AbiBYDOS (ORCPT ); Thu, 24 Feb 2022 22:14:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236965AbiBYDN7 (ORCPT ); Thu, 24 Feb 2022 22:13:59 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 510127DA98; Thu, 24 Feb 2022 19:13:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1645758807; x=1677294807; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rH1pJ8ne/FKJWQirIoIp2z6yrhy7FoypqadKznLIPjE=; b=e6zD/M8itgBK6qi1uesdav5jgzgmaVetyzpgPx0Zx32fsan74oOMRFc4 /EMw+D0u5bjOHtufoBjkXm+PMotx1emL/daN7Y/HJ7hPpP5Z0Rd4dwEck foYCcOdgFVSvRqTWqlWfcYRSfl7+5yNxE6bWaiKwVnhBd8LAALHDsmYmp hrOFR7K1g/4pV4wIOxNNsF5A+uBUswzH3RxcrcOK7kgJCa9b2MwdEI56Q /uM0H4vpmYh3khClb7zVJkAEnoI/fz4U2xf2O20oGEQXnpVRNpXs2BFXV ZqYjch9SYhuWrDC05Rq/WEbYPs0iotWT50XJ66vNPO1VxBk7r6NE6Hmw6 A==; X-IronPort-AV: E=McAfee;i="6200,9189,10268"; a="252603002" X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="252603002" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2022 19:13:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,135,1643702400"; d="scan'208";a="684512515" Received: from unknown (HELO localhost.localdomain.sh.intel.com) ([10.238.175.107]) by fmsmga001.fm.intel.com with ESMTP; 24 Feb 2022 19:13:24 -0800 From: Tianfei zhang To: hao.wu@intel.com, trix@redhat.com, mdf@kernel.org, yilun.xu@intel.com, linux-fpga@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: corbet@lwn.net, Tianfei zhang Subject: [PATCH v2 5/5] Documentation: fpga: dfl: add description of IOFS Date: Thu, 24 Feb 2022 22:10:02 -0500 Message-Id: <20220225031002.261264-6-tianfei.zhang@intel.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20220225031002.261264-1-tianfei.zhang@intel.com> References: <20220225031002.261264-1-tianfei.zhang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fpga@vger.kernel.org This patch adds description about IOFS support for DFL. --- v2: * Fixs some typos. * Adds more detail description about the models of AFU access which supported in IOFS. Signed-off-by: Tianfei zhang --- Documentation/fpga/dfl.rst | 113 +++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/Documentation/fpga/dfl.rst b/Documentation/fpga/dfl.rst index ef9eec71f6f3..4c3ac6f452bc 100644 --- a/Documentation/fpga/dfl.rst +++ b/Documentation/fpga/dfl.rst @@ -556,6 +556,119 @@ new DFL feature via UIO direct access, its feature id should be added to the driver's id_table. +Intel Open FPGA stack +===================== + +Intel Open FPGA stack aka IOFS, Intel's version of a common core set of +RTL to allow customers to easily interface to logic and IP on the FPGA. +IOFS leverages the DFL for the implementation of the FPGA RTL design. + +IOFS designs allow for the arrangement of software interfaces across multiple +PCIe endpoints. Some of these interfaces may be PFs defined in the static region +that connect to interfaces in an IP that is loaded via Partial Reconfiguration (PR). +And some of these interfaces may be VFs defined in the PR region that can be +reconfigured by the end-user. Furthermore, these PFs/VFs may also be arranged +using a DFL such that features may be discovered and accessed in user space +(with the aid of a generic kernel driver like vfio-pci). The diagram below depicts +an example design with two PFs and two VFs. In this example, it will export the +management functions via PF0, PF1 will bind with virtio-net driver presenting itself +as a network interface to the OS. The other functions, VF0 and VF1, leverage VFIO +to export the MMIO space to an application or assign to a VM. +:: + + +-----------------+ +--------------+ +-------------+ +------------+ + | FPGA Management | | VirtIO | | User App | | Virtual | + | App | | App | | | | Machine | + +--------+--------+ +------+-------+ +------+------+ +-----+------+ + | | | | + +--------+--------+ +------+-------+ +------+------+ | + | DFL Driver | |VirtIO driver | | VFIO | | + +--------+--------+--+------+-------+ +------+------+ | + | | | | + | | | | + +--------+--------+ +------+-------+ +------+------+ +----+------+ + | PF0 | | PF1 | | PF0_VF0 | | PF0_VF1 | + +-----------------+ +--------------+ +-------------+ +-----------+ + +As accelerators are specialized hardware, they are typically limited in the +number installed in a given system. Many use cases require them to be shared +across multiple software contexts or threads of software execution, either +through partitioning of individual dedicated resources, or virtualization of +shared resources. On IOFS, it provides several models to share the AFU +resources via PR mechanism and hardware-based virtualization schemes. + +1. Legacy model. + In legacy FPGA card platforms (like Intel PAC N3000 or N5000 Card),there is + a notion that the boundary between the AFU and the shell is also the unit of + PR for those FPGA platforms. In this model, it can only able to handle a + single context, because it only has one PR engine, and one PR region which + has an associated Port device. +2. Multiple VFs per PR slot. + In this model, available AFU resources may allow instantiation of many of VFs + which has a dedicated PCIe function with their own dedicated MMIO space, or + partition a region of MMIO space on a single PCIe function. + In this model, the Port device would not connected to AFU/PR slot, so we don't + need to release the Port device before creating the VFs. For DFL's view, the AFU + will not connect to Port device, so the Next_AFU pointer in FIU feature header + of port device points to NULL in this model. On the other hand, each VF can start + with an AFU feature header without connected to a FIU Port feature header. +3. Micro-Personas in AFU. + IOFS introducing a new concept to extend the FPGA usage, Micro-Personas in + AFU. It finds some downsides of the legacy model to be unacceptable, because + this may be desirable by a customer who intends to switch out one accelerator + for another accelerator without having to reconfigure the entire FPGA. + Micro-Personas allow the developer to designate their own AFU-defined PR + regions. In this model the unit of PR is not the entire AFU, instead + the unit of PR can be any size block or blocks inside the AFU. + In this model, it has PR capability includes one PR engine and multiple PR regions, + and each PR region has an associated port gasket. A PR region may also be + referred to as a PR slot. Port gasket is similar with port device in legacy + model which include the port control, port user clock control and port errors. + +IOFS provides the diversity for access the AFU resource to RTL developer. +An IP designer may choose to add more than one PF for interfacing with IP +on the FPGA and choose different model to access the AFU resource. + +There is one reference architecture design using the "Multiple VFs per PR slot" +model for IOFS as illustrated below. In this reference design, it exports the +FPGA management functions via PF0. PF1 will bind with virtio-net driver +presenting itself as a network interface to the OS. PF2 will bound to the +vfio-pci driver allowing the user space software to discover and interface +with the specific workload like diagnostic test. To access the AFU resource, +it uses SR-IOV to partition workload interfaces across various VFs. +:: + + +----------------------+ + | PF/VF mux/demux | + +--+--+-----+------+-+-+ + | | | | | + +------------------------+ | | | | + PF0 | +---------+ +-+ | | + +---+---+ | +---+----+ | | + | DFH | | | DFH | | | + +-------+ +-----+----+ +--------+ | | + | FME | | VirtIO | | Test | | | + +---+---+ +----------+ +--------+ | | + | PF1 PF2 | | + | | | + | +----------+ | + | | ++ + | | | + | | PF0_VF0 | PF0_VF1 + | +-----------------+-----------+------------+ + | | +-----+-----------+--------+ | + | | | | | | | + | | +------+ | +--+ -+ +--+---+ | | + | | | Port | | | DFH | | DFH | | | + +-----------+ +------+ | +-----+ +------+ | | + | | | DEV | | DEV | | | + | | +-----+ +------+ | | + | | PR Slot | | + | +--------------------------+ | + | Port Gasket | + +------------------------------------------+ + + Open discussion =============== FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration