From patchwork Fri May 19 17:06:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3111C7EE2D for ; Fri, 19 May 2023 17:10:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230361AbjESRKB (ORCPT ); Fri, 19 May 2023 13:10:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231747AbjESRJh (ORCPT ); Fri, 19 May 2023 13:09:37 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 457BAE5A for ; Fri, 19 May 2023 10:09:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516173; x=1716052173; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XH+4wicTto2YbIj4/yaAwqnjMtxZfvX+jAwmC6BNbrQ=; b=ivj+6ZmJvu/hwAxEYHvs6glrKIKdhzGYJYdSP+iIji/NpfWuZxKPhUHj T32WdtWa7kuHUrU+CKTv/ow14P1LyrYFWyrerPYVOdcpCOZSD6NVPf2sO qVg7MQ8qkW92qna3vGi/zeMa2ErHsW59/9QC1i7/hKOxoMcNysBWn2s0U vX+drrxQ/j2aUGou5aeGzaQr3RzV+UdkMwzZQFFUnCa+jxlIIrb3KY2gp /aHmtFswqL9WXnZcACSDL5sRfVxhDv1KC8+4nXE4a/wouKw0Gev6HCPU5 ZfwdJphwK3dK/XL+5vJUkJHVPR8kJ8JNL4n9FK86HSUgpmNml+qJlMGER w==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="355655263" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="355655263" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="792427503" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="792427503" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:41 -0700 Subject: [PATCH v6 01/11] cxl: Add callback to parse the DSMAS subtables from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:06:40 -0700 Message-ID: <168451600072.3470703.11316174773607021145.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Provide a callback function to the CDAT parser in order to parse the Device Scoped Memory Affinity Structure (DSMAS). Each DSMAS structure contains the DPA range and its associated attributes in each entry. See the CDAT specification for details. The device handle and the DPA range is saved and to be associated with the DSLBIS locality data when the DSLBIS entries are parsed. The list is a local list. When the total path performance data is calculated and storred this list can be discarded. Coherent Device Attribute Table 1.03 2.1 Device Scoped memory Affinity Structure (DSMAS) Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v6: - Adjust dsmas offset w/o common header. - Move dsmas list destroy to core/cdat. (Jonathan) v5: - Update commit log to indicate what list is used for. (Jonathan, Dan) - Use acpi_table_parse_cdat() - Isolate cdat code behind CONFIG_ACPI v3: - Add spec section number. (Alison) - Remove cast from void *. (Alison) - Refactor cxl_port_probe() block. (Alison) - Move CDAT parse to cxl_endpoint_port_probe() v2: - Add DSMAS table size check. (Lukas) - Use local DSMAS header for LE handling. - Remove dsmas lock. (Jonathan) - Fix handle size (Jonathan) - Add LE to host conversion for DSMAS address and length. - Make dsmas_list local --- drivers/cxl/Kconfig | 1 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/cdat.c | 58 +++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 23 ++++++++++++++++++ drivers/cxl/port.c | 12 +++++++++ 5 files changed, 95 insertions(+) create mode 100644 drivers/cxl/core/cdat.c diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index ff4e78117b31..5420130ddeba 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -3,6 +3,7 @@ menuconfig CXL_BUS tristate "CXL (Compute Express Link) Devices Support" depends on PCI select PCI_DOE + select ACPI_TABLES_LIB help CXL is a bus that is electrically compatible with PCI Express, but layers three protocols on that signalling (CXL.io, CXL.cache, and diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile index ca4ae31d8f57..01fb5f6982b3 100644 --- a/drivers/cxl/core/Makefile +++ b/drivers/cxl/core/Makefile @@ -12,5 +12,6 @@ cxl_core-y += memdev.o cxl_core-y += mbox.o cxl_core-y += pci.o cxl_core-y += hdm.o +cxl_core-$(CONFIG_ACPI_TABLES_LIB) += cdat.o cxl_core-$(CONFIG_TRACING) += trace.o cxl_core-$(CONFIG_CXL_REGION) += region.o diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c new file mode 100644 index 000000000000..f70a3a146901 --- /dev/null +++ b/drivers/cxl/core/cdat.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2023 Intel Corporation. All rights reserved. */ +#include +#include "cxlpci.h" +#include "cxl.h" + +static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg, + const unsigned long end) +{ + struct acpi_cdat_header *hdr = &header->cdat; + struct acpi_cdat_dsmas *dsmas; + int size = sizeof(*hdr) + sizeof(*dsmas); + struct list_head *dsmas_list = arg; + struct dsmas_entry *dent; + u16 len; + + len = le16_to_cpu((__force __le16)hdr->length); + if (len != size || (unsigned long)hdr + len > end) { + pr_warn("Malformed DSMAS table length: (%u:%u)\n", size, len); + return -EINVAL; + } + + /* Skip common header */ + dsmas = (struct acpi_cdat_dsmas *)(hdr + 1); + + dent = kzalloc(sizeof(*dent), GFP_KERNEL); + if (!dent) + return -ENOMEM; + + dent->handle = dsmas->dsmad_handle; + dent->dpa_range.start = le64_to_cpu((__force __le64)dsmas->dpa_base_address); + dent->dpa_range.end = le64_to_cpu((__force __le64)dsmas->dpa_base_address) + + le64_to_cpu((__force __le64)dsmas->dpa_length) - 1; + list_add_tail(&dent->list, dsmas_list); + + return 0; +} + +int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) +{ + return cdat_table_parse(ACPI_CDAT_TYPE_DSMAS, cdat_dsmas_handler, + list, port->cdat.table); +} +EXPORT_SYMBOL_NS_GPL(cxl_cdat_endpoint_process, CXL); + +void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list) +{ + struct dsmas_entry *dentry, *n; + + list_for_each_entry_safe(dentry, n, dsmas_list, list) { + list_del(&dentry->list); + kfree(dentry); + } +} +EXPORT_SYMBOL_NS_GPL(cxl_cdat_dsmas_list_destroy, CXL); + +MODULE_IMPORT_NS(CXL); + diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 4577d808ac6d..f119f030e2e8 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -791,6 +792,28 @@ static inline struct cxl_dax_region *to_cxl_dax_region(struct device *dev) } #endif +/* CDAT related bits */ +struct dsmas_entry { + struct list_head list; + struct range dpa_range; + u8 handle; +}; + +#ifdef CONFIG_ACPI_TABLES_LIB +int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list); +void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list); +#else +static inline int cxl_cdat_endpoint_process(struct cxl_port *port, + struct list_head *list) +{ + return -EOPNOTSUPP; +} + +static inline void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list) +{ +} +#endif + /* * Unit test builds overrides this to __weak, find the 'strong' version * of these symbols in tools/testing/cxl/. diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 432b42caf3f9..6b73e7b8660e 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -133,6 +133,18 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) device_for_each_child(&port->dev, root, discover_region); put_device(&root->dev); + if (port->cdat.table) { + LIST_HEAD(dsmas_list); + + rc = cxl_cdat_endpoint_process(port, &dsmas_list); + if (rc < 0) + dev_dbg(&port->dev, "Failed to parse CDAT: %d\n", rc); + + /* Performance data processing */ + + cxl_cdat_dsmas_list_destroy(&dsmas_list); + } + return 0; } From patchwork Fri May 19 17:06:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC3F1C7EE29 for ; Fri, 19 May 2023 17:10:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229747AbjESRJ3 (ORCPT ); Fri, 19 May 2023 13:09:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230480AbjESRJC (ORCPT ); Fri, 19 May 2023 13:09:02 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0FA01705 for ; Fri, 19 May 2023 10:08:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516118; x=1716052118; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jjGKx2gla6Qd9r3Lj4J6pwZJuuB1yYj0apRSBqvEKKc=; b=Iw31fw/WeXFE/Yr9hB7a4p6jmj7+RyOnqxXtqNlGQh2XcDq8Cj6cmDAg rF+iMDTmiD6o5ueZ8rmKNyU5o2xIEnFErfTsQ2ZRIWPMZNLhgeb335tWC 3eKvv6splVIIFK6qajnEl1D6A7VHlsk30mUwLib9IPfmjggZE7WFfRzNP QHPrI5oYovK1/ute92HMASZwwEVTgk8dy0S9mK4eFhsc+y/7OGgpP68oP TkQJrzHWT6RPo6FXGYri98Pi0aVU9IPXAdC19S59GnXxW0/oEHEG+DDuu bRLn8oo57GBcgZosavUF8CGlp3QStuY7etCYHWGm3CZ15bz9acPZ2eFB0 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="438778231" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="438778231" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="876910767" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="876910767" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:46 -0700 Subject: [PATCH v6 02/11] cxl: Add callback to parse the DSLBIS subtable from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:06:46 -0700 Message-ID: <168451600640.3470703.3461483051303767270.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Provide a callback to parse the Device Scoped Latency and Bandwidth Information Structure (DSLBIS) in the CDAT structures. The DSLBIS contains the bandwidth and latency information that's tied to a DSMAS handle. The driver will retrieve the read and write latency and bandwidth associated with the DSMAS which is tied to a DPA range. Coherent Device Attribute Table 1.03 2.1 Device Scoped Latency and Bandwidth Information Structure (DSLBIS) Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v6: - Clarify cxl_cdat_endpoint_process() error paths for readability. (Jonathan) - Adjust dslbis ptr from header. v5: - Remove macro for common headers. (Jonathan) - Use acpi_table_parse_cdat(). - Remove unlikely(). (Dan) v3: - Added spec section in commit header. (Alison) - Remove void * recast. (Alison) - Rework comment. (Alison) - Move CDAT parse to cxl_endpoint_port_probe() - Convert to use 'struct node_hmem_attrs' v2: - Add size check to DSLIBIS table. (Lukas) - Remove unnecessary entry type check. (Jonathan) - Move data_type check to after match. (Jonathan) - Skip unknown data type. (Jonathan) - Add overflow check for unit multiply. (Jonathan) - Use dev_warn() when entries parsing fail. (Jonathan) --- drivers/cxl/core/cdat.c | 102 ++++++++++++++++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 2 + 2 files changed, 102 insertions(+), 2 deletions(-) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index f70a3a146901..464e69a9c8e4 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2023 Intel Corporation. All rights reserved. */ #include +#include #include "cxlpci.h" #include "cxl.h" @@ -36,10 +37,107 @@ static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg, return 0; } +static void cxl_access_coordinate_set(struct access_coordinate *coord, + int access, unsigned int val) +{ + switch (access) { + case ACPI_HMAT_ACCESS_LATENCY: + coord->read_latency = val; + coord->write_latency = val; + break; + case ACPI_HMAT_READ_LATENCY: + coord->read_latency = val; + break; + case ACPI_HMAT_WRITE_LATENCY: + coord->write_latency = val; + break; + case ACPI_HMAT_ACCESS_BANDWIDTH: + coord->read_bandwidth = val; + coord->write_bandwidth = val; + break; + case ACPI_HMAT_READ_BANDWIDTH: + coord->read_bandwidth = val; + break; + case ACPI_HMAT_WRITE_BANDWIDTH: + coord->write_bandwidth = val; + break; + } +} + +static int cdat_dslbis_handler(union acpi_subtable_headers *header, void *arg, + const unsigned long end) +{ + struct acpi_cdat_header *hdr = &header->cdat; + struct acpi_cdat_dslbis *dslbis; + int size = sizeof(*hdr) + sizeof(*dslbis); + struct list_head *dsmas_list = arg; + struct dsmas_entry *dent; + u16 len; + + len = le16_to_cpu((__force __le16)hdr->length); + if (len != size || (unsigned long)hdr + len > end) { + pr_warn("Malformed DSLBIS table length: (%u:%u)\n", size, len); + return -EINVAL; + } + + /* Skip common header */ + dslbis = (struct acpi_cdat_dslbis *)(hdr + 1); + + /* Skip unrecognized data type */ + if (dslbis->data_type > ACPI_HMAT_WRITE_BANDWIDTH) + return 0; + + list_for_each_entry(dent, dsmas_list, list) { + __le64 le_base; + __le16 le_val; + u64 val; + int rc; + + if (dslbis->handle != dent->handle) + continue; + + /* Not a memory type, skip */ + if ((dslbis->flags & ACPI_HMAT_MEMORY_HIERARCHY) != + ACPI_HMAT_MEMORY) + return 0; + + le_base = (__force __le64)dslbis->entry_base_unit; + le_val = (__force __le16)dslbis->entry[0]; + rc = check_mul_overflow(le64_to_cpu(le_base), + le16_to_cpu(le_val), &val); + if (rc) + pr_warn("DSLBIS value overflowed.\n"); + + cxl_access_coordinate_set(&dent->coord, dslbis->data_type, val); + break; + } + + return 0; +} + +static int cdat_table_parse_output(int rc) +{ + if (rc < 0) + return rc; + if (rc == 0) + return -ENOENT; + + return 0; +} + int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) { - return cdat_table_parse(ACPI_CDAT_TYPE_DSMAS, cdat_dsmas_handler, - list, port->cdat.table); + int rc; + + rc = cdat_table_parse(ACPI_CDAT_TYPE_DSMAS, cdat_dsmas_handler, + list, port->cdat.table); + rc = cdat_table_parse_output(rc); + if (rc) + return rc; + + rc = cdat_table_parse(ACPI_CDAT_TYPE_DSLBIS, cdat_dslbis_handler, + list, port->cdat.table); + return cdat_table_parse_output(rc); } EXPORT_SYMBOL_NS_GPL(cxl_cdat_endpoint_process, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index f119f030e2e8..87b9e80d834c 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -797,6 +798,7 @@ struct dsmas_entry { struct list_head list; struct range dpa_range; u8 handle; + struct access_coordinate coord; }; #ifdef CONFIG_ACPI_TABLES_LIB From patchwork Fri May 19 17:06:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62F1BC7EE2E for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230162AbjESRJb (ORCPT ); Fri, 19 May 2023 13:09:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231898AbjESRJO (ORCPT ); Fri, 19 May 2023 13:09:14 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24F0410C3 for ; Fri, 19 May 2023 10:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516137; x=1716052137; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Rp2tUyYEe1sr89CvFY7nY5xayliz4gA00CfzX8Wu+NQ=; b=TS84vASLZYx2U4Tj/I/TVIkdVhWtFn7MgNxZSfXgWXVZrq8z0vIEj9eP n1dfdEoycqBpu6SLs8mhi3xXkpTXn6Eib1rDyTCEhu+2EjFheeTVfzgd4 H3SvtqYoSW9wG2OomNNN42IF7pHtizq5i9i4tbq+V+7lNV1J+lP15c+LZ 9bpV4gQ1Di1tBictWcefVMl5AyJci5kjipg6rcmaqGpyRB/3iHs/6EYOG MiIbg0+tOG5/CTod+9bncfW5bz6DOnoJDc4irP0lRvRbi4tKzwq25e/oA pDeswqp8IAsyH/PHRWbuqS8C5HvW3e6VFSKOXFbOdon+uoYP9fJiOLEvH g==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="438778242" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="438778242" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="876910831" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="876910831" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:52 -0700 Subject: [PATCH v6 03/11] cxl: Add callback to parse the SSLBIS subtable from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:06:52 -0700 Message-ID: <168451601243.3470703.9340908897464530349.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Provide a callback to parse the Switched Scoped Latency and Bandwidth Information Structure (SSLBIS) in the CDAT structures. The SSLBIS contains the bandwidth and latency information that's tied to the CXL switch that the data table has been read from. The extracted values are stored to the cxl_dport correlated by the port_id depending on the SSLBIS entry. Coherent Device Attribute Table 1.03 2.1 Switched Scoped Latency and Bandwidth Information Structure (DSLBIS) Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v6: - Adjust sslbis ptr w/o common header. v5: - Store data to cxl_dport directly instead. (Dan) - Use acpi_table_parse_cdat(). v3: - Add spec section in commit header (Alison) - Move CDAT parse to cxl_switch_port_probe() - Use 'struct node_hmem_attrs' --- drivers/cxl/core/cdat.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 8 ++++ drivers/cxl/port.c | 6 +++ 3 files changed, 107 insertions(+) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index 464e69a9c8e4..73ac96e754a0 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -152,5 +152,98 @@ void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list) } EXPORT_SYMBOL_NS_GPL(cxl_cdat_dsmas_list_destroy, CXL); +static int cdat_sslbis_handler(union acpi_subtable_headers *header, void *arg, + const unsigned long end) +{ + struct acpi_cdat_sslbis *sslbis; + int size = sizeof(header->cdat) + sizeof(*sslbis); + struct cxl_port *port = arg; + struct device *dev = &port->dev; + struct acpi_cdat_sslbe *entry; + int remain, entries, i; + u16 len; + + len = le16_to_cpu((__force __le16)header->cdat.length); + remain = len - size; + if (!remain || remain % sizeof(*entry) || + (unsigned long)header + len > end) { + dev_warn(dev, "Malformed SSLBIS table length: (%u)\n", len); + return -EINVAL; + } + + /* Skip common header */ + sslbis = (struct acpi_cdat_sslbis *)((unsigned long)header + + sizeof(header->cdat)); + + /* Unrecognized data type, we can skip */ + if (sslbis->data_type > ACPI_HMAT_WRITE_BANDWIDTH) + return 0; + + entries = remain / sizeof(*entry); + entry = (struct acpi_cdat_sslbe *)((unsigned long)header + sizeof(*sslbis)); + + for (i = 0; i < entries; i++) { + u16 x = le16_to_cpu((__force __le16)entry->portx_id); + u16 y = le16_to_cpu((__force __le16)entry->porty_id); + __le64 le_base; + __le16 le_val; + struct cxl_dport *dport; + unsigned long index; + u16 dsp_id; + u64 val; + + switch (x) { + case ACPI_CDAT_SSLBIS_US_PORT: + dsp_id = y; + break; + case ACPI_CDAT_SSLBIS_ANY_PORT: + switch (y) { + case ACPI_CDAT_SSLBIS_US_PORT: + dsp_id = x; + break; + case ACPI_CDAT_SSLBIS_ANY_PORT: + dsp_id = ACPI_CDAT_SSLBIS_ANY_PORT; + break; + default: + dsp_id = y; + break; + } + break; + default: + dsp_id = x; + break; + } + + le_base = (__force __le64)sslbis->entry_base_unit; + le_val = (__force __le16)entry->latency_or_bandwidth; + + if (check_mul_overflow(le64_to_cpu(le_base), + le16_to_cpu(le_val), &val)) + dev_warn(dev, "SSLBIS value overflowed!\n"); + + xa_for_each(&port->dports, index, dport) { + if (dsp_id == ACPI_CDAT_SSLBIS_ANY_PORT || + dsp_id == dport->port_id) + cxl_access_coordinate_set(&dport->coord, + sslbis->data_type, + val); + } + + entry++; + } + + return 0; +} + +int cxl_cdat_switch_process(struct cxl_port *port) +{ + int rc; + + rc = cdat_table_parse(ACPI_CDAT_TYPE_SSLBIS, cdat_sslbis_handler, + port, port->cdat.table); + return cdat_table_parse_output(rc); +} +EXPORT_SYMBOL_NS_GPL(cxl_cdat_switch_process, CXL); + MODULE_IMPORT_NS(CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 87b9e80d834c..d18306b55a2d 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -600,6 +600,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) * @rcrb: base address for the Root Complex Register Block * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port + * @coord: access coordinates (performance) for switch from CDAT */ struct cxl_dport { struct device *dport; @@ -608,6 +609,7 @@ struct cxl_dport { resource_size_t rcrb; bool rch; struct cxl_port *port; + struct access_coordinate coord; }; /** @@ -804,6 +806,7 @@ struct dsmas_entry { #ifdef CONFIG_ACPI_TABLES_LIB int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list); void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list); +int cxl_cdat_switch_process(struct cxl_port *port); #else static inline int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) @@ -814,6 +817,11 @@ static inline int cxl_cdat_endpoint_process(struct cxl_port *port, static inline void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list) { } + +static inline int cxl_cdat_switch_process(struct cxl_port *port) +{ + return -EOPNOTSUPP; +} #endif /* diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 6b73e7b8660e..0f9c25645e1d 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -78,6 +78,12 @@ static int cxl_switch_port_probe(struct cxl_port *port) return PTR_ERR(cxlhdm); } + if (port->cdat.table) { + rc = cxl_cdat_switch_process(port); + if (rc < 0) + dev_warn(&port->dev, "Failed to parse SSLBIS: %d\n", rc); + } + if (rc == 1) { dev_dbg(&port->dev, "Fallback to passthrough decoder\n"); return devm_cxl_add_passthrough_decoder(port); From patchwork Fri May 19 17:06:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85A15C7EE30 for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230201AbjESRJc (ORCPT ); Fri, 19 May 2023 13:09:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231966AbjESRJU (ORCPT ); Fri, 19 May 2023 13:09:20 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEF9310FC for ; Fri, 19 May 2023 10:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516142; x=1716052142; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AOLNKCkZDQfLCajSjjJoQ9mLKKEd/as7Y99SJRVxRSI=; b=mno1+fFSwQeaRejVQok8ctFl+vzr3/8bcbYjmjeTCoY15sC7d3vcWv26 kAEzyzYYBhVMwHtsoBvLQ/BXOl9LCxBSvMalovnZpqOgLEdtnieQjSG+Z y4BdN1tWYnuDmS0t19LTemVfUyMc1DJoyszVsIW3ADbVGuzhwpuacw+SC rJSnnUOtdAYpN85mI3lG7zVYllL8MyiprPQBPurQ985g9IgUmVbnyNbrW YK+Mwyirc6xmU/T42ck27YKR3AVvlwWdcSNeWBhkUySYNB3Ff1Z7dbJyk WW9AFvI7RYBHxOuWzB/RNliIdV/qihpH9qcyVef3LQpXQB7EUJnijAdO/ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="438778274" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="438778274" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="876910884" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="876910884" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:06:58 -0700 Subject: [PATCH v6 04/11] cxl: Add support for _DSM Function for retrieving QTG ID From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:06:58 -0700 Message-ID: <168451601833.3470703.277986088556971387.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL spec v3.0 9.17.3 CXL Root Device Specific Methods (_DSM) Add support to retrieve QTG ID via ACPI _DSM call. The _DSM call requires an input of an ACPI package with 4 dwords (read latency, write latency, read bandwidth, write bandwidth). The call returns a package with 1 WORD that provides the max supported QTG ID and a package that may contain 0 or more WORDs as the recommended QTG IDs in the recommended order. Create a cxl_root container for the root cxl_port and provide a callback ->get_qos_class() in order to retrieve the QoS class. For the ACPI case, the _DSM helper is used to retrieve the QTG ID and returned. A devm_cxl_add_root() function is added for root port setup and registration of the cxl_root callback operation(s). Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang caluclate caculates --- v5: - Make the helper a callback for the CXL root. (Dan) - Drop the addition of core/acpi.c. (Dan) - Add endiness handling. (Jonathan) - Refactor error exits. (Jonathan) - Update evaluate function description. (Jonathan) - Make uuid static. (Dan) v2: - Reorder var declaration and use C99 style. (Jonathan) - Allow >2 ACPI objects in package for future expansion. (Jonathan) - Check QTG IDs against MAX QTG ID provided by output package. (Jonathan) --- drivers/cxl/acpi.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++- drivers/cxl/core/port.c | 41 ++++++++++++- drivers/cxl/cxl.h | 36 ++++++++++++ 3 files changed, 215 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index e063df2bf876..8247df06d683 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -17,6 +17,10 @@ struct cxl_cxims_data { u64 xormaps[]; }; +static const guid_t acpi_cxl_qtg_id_guid = + GUID_INIT(0xF365F9A6, 0xA7DE, 0x4071, + 0xA6, 0x6A, 0xB4, 0x0C, 0x0B, 0x4F, 0x8E, 0x52); + /* * Find a targets entry (n) in the host bridge interleave list. * CXL Specification 3.0 Table 9-22 @@ -194,6 +198,140 @@ struct cxl_cfmws_context { int id; }; +/** + * cxl_acpi_evaluate_qtg_dsm - Retrieve QTG ids via ACPI _DSM + * @handle: ACPI handle + * @input: bandwidth and latency data + * + * Return: qos_class output or ERRPTR of -errno + * + * Issue QTG _DSM with accompanied bandwidth and latency data in order to get + * the QTG IDs that are suitable for the performance point in order of most + * suitable to least suitable. Return first QTG ID. + */ +static struct qos_class * +cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, struct qtg_dsm_input *input) +{ + union acpi_object *out_obj, *out_buf, *pkg; + union acpi_object in_buf = { + .buffer = { + .type = ACPI_TYPE_BUFFER, + .pointer = (u8 *)input, + .length = cpu_to_le32(sizeof(*input)), + }, + }; + union acpi_object in_obj = { + .package = { + .type = ACPI_TYPE_PACKAGE, + .count = cpu_to_le32(1), + .elements = &in_buf + }, + }; + struct qos_class *output; + u16 max_qtg; + __le16 *ent; + int entries; + int i, len; + int rc = 0; + + out_obj = acpi_evaluate_dsm(handle, &acpi_cxl_qtg_id_guid, 1, 1, &in_obj); + if (!out_obj) + return ERR_PTR(-ENXIO); + + if (out_obj->type != ACPI_TYPE_PACKAGE) { + rc = -ENXIO; + goto out; + } + + /* Check Max QTG ID */ + pkg = &out_obj->package.elements[0]; + if (pkg->type != ACPI_TYPE_BUFFER) { + rc = -ENXIO; + goto out; + } + + if (le32_to_cpu(pkg->buffer.length) != sizeof(u16)) { + rc = -ENXIO; + goto out; + } + + max_qtg = le16_to_cpu(*(__le16 *)pkg->buffer.pointer); + + /* Retrieve QTG IDs package */ + pkg = &out_obj->package.elements[1]; + if (pkg->type != ACPI_TYPE_PACKAGE) { + rc = -ENXIO; + goto out; + } + + out_buf = &pkg->package.elements[0]; + if (out_buf->type != ACPI_TYPE_BUFFER) { + rc = -ENXIO; + goto out; + } + + len = le32_to_cpu(out_buf->buffer.length); + + /* It's legal to have 0 QTG entries */ + if (len == 0) { + rc = -EEXIST; + goto out; + } + + /* Malformed package, not multiple of WORD size */ + if (len % sizeof(__le16)) { + rc = -ENXIO; + goto out; + } + + entries = len / sizeof(__le16); + output = kmalloc(sizeof(*output) + entries * sizeof(int), GFP_KERNEL); + if (!output) { + rc = -ENOMEM; + goto out; + } + + ent = (__le16 *)out_buf->buffer.pointer; + for (i = 0; i < entries; i++) { + int qtg_id = le16_to_cpu(ent[i]); + + if (qtg_id > max_qtg) + pr_warn("QTG ID %u greater than MAX %u\n", + qtg_id, max_qtg); + + output->entries[i] = qtg_id; + } + output->nr = entries; + +out: + ACPI_FREE(out_obj); + if (rc) + output = ERR_PTR(rc); + return output; +} + +static struct qos_class * +cxl_acpi_get_qos_class(struct cxl_port *root_port, struct qtg_dsm_input *input) +{ + acpi_handle handle; + struct device *dev; + + dev = root_port->uport; + + if (!dev_is_platform(dev)) + return ERR_PTR(-ENODEV); + + handle = ACPI_HANDLE(dev); + if (!handle) + return ERR_PTR(-ENODEV); + + return cxl_acpi_evaluate_qtg_dsm(handle, input); +} + +static const struct cxl_root_ops acpi_root_ops = { + .get_qos_class = cxl_acpi_get_qos_class, +}; + static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, const unsigned long end) { @@ -631,6 +769,7 @@ static int cxl_acpi_probe(struct platform_device *pdev) { int rc; struct resource *cxl_res; + struct cxl_root *cxl_root; struct cxl_port *root_port; struct device *host = &pdev->dev; struct acpi_device *adev = ACPI_COMPANION(host); @@ -650,9 +789,10 @@ static int cxl_acpi_probe(struct platform_device *pdev) cxl_res->end = -1; cxl_res->flags = IORESOURCE_MEM; - root_port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL); - if (IS_ERR(root_port)) - return PTR_ERR(root_port); + cxl_root = devm_cxl_add_root(host, &acpi_root_ops); + if (IS_ERR(cxl_root)) + return PTR_ERR(cxl_root); + root_port = &cxl_root->port; rc = bus_for_each_dev(adev->dev.bus, NULL, root_port, add_host_bridge_dport); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 374c9b93daf2..fbef16dee3ed 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -525,7 +525,10 @@ static void cxl_port_release(struct device *dev) xa_destroy(&port->dports); xa_destroy(&port->regions); ida_free(&cxl_port_ida, port->id); - kfree(port); + if (is_cxl_root(port)) + kfree(to_cxl_root(port)); + else + kfree(port); } static const struct attribute_group *cxl_port_attribute_groups[] = { @@ -628,13 +631,22 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, resource_size_t component_reg_phys, struct cxl_dport *parent_dport) { + struct cxl_root *cxl_root = NULL; struct cxl_port *port; struct device *dev; int rc; - port = kzalloc(sizeof(*port), GFP_KERNEL); - if (!port) - return ERR_PTR(-ENOMEM); + /* No parent_dport, root cxl_port */ + if (!parent_dport) { + cxl_root = kzalloc(sizeof(*cxl_root), GFP_KERNEL); + if (!cxl_root) + return ERR_PTR(-ENOMEM); + port = &cxl_root->port; + } else { + port = kzalloc(sizeof(*port), GFP_KERNEL); + if (!port) + return ERR_PTR(-ENOMEM); + } rc = ida_alloc(&cxl_port_ida, GFP_KERNEL); if (rc < 0) @@ -692,7 +704,10 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, return port; err: - kfree(port); + if (cxl_root) + kfree(cxl_root); + else + kfree(port); return ERR_PTR(rc); } @@ -778,6 +793,22 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, } EXPORT_SYMBOL_NS_GPL(devm_cxl_add_port, CXL); +struct cxl_root *devm_cxl_add_root(struct device *host, + const struct cxl_root_ops *ops) +{ + struct cxl_root *cxl_root; + struct cxl_port *port; + + port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL); + if (IS_ERR(port)) + return (struct cxl_root *)port; + + cxl_root = to_cxl_root(port); + cxl_root->ops = ops; + return cxl_root; +} +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_root, CXL); + struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port) { /* There is no pci_bus associated with a CXL platform-root port */ diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index d18306b55a2d..34e6e19875cd 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -586,6 +586,40 @@ struct cxl_port { bool cdat_available; }; +struct qtg_dsm_input { + __le32 rd_lat; + __le32 wr_lat; + __le32 rd_bw; + __le32 wr_bw; +}; + +struct qos_class { + int nr; + int entries[]; +}; + +struct cxl_root_ops { + struct qos_class *(*get_qos_class)(struct cxl_port *root_port, + struct qtg_dsm_input *input); +}; + +/** + * struct cxl_root - logical collection of root cxl_port items + * + * @port: cxl_port member + * @ops: cxl root operations + */ +struct cxl_root { + struct cxl_port port; + const struct cxl_root_ops *ops; +}; + +static inline struct cxl_root * +to_cxl_root(const struct cxl_port *port) +{ + return container_of(port, struct cxl_root, port); +} + static inline struct cxl_dport * cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) { @@ -665,6 +699,8 @@ struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port); struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys, struct cxl_dport *parent_dport); +struct cxl_root *devm_cxl_add_root(struct device *host, + const struct cxl_root_ops *ops); struct cxl_port *find_cxl_root(struct cxl_port *port); int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd); void cxl_bus_rescan(void); From patchwork Fri May 19 17:07:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 730A6C7EE31 for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230250AbjESRJd (ORCPT ); Fri, 19 May 2023 13:09:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231993AbjESRJW (ORCPT ); Fri, 19 May 2023 13:09:22 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AA36A0 for ; Fri, 19 May 2023 10:09:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516145; x=1716052145; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3keKEe/mlFnJmSwEjG50x4v/sl0A/bu1oiI/SayW5lY=; b=f1ui/3MRdoQOicbFnUkpEsl5bS3IPcJlB8InieYucmnyKeR7tT2SRMEC KFeG+Cvy9ExmGy4wupmZjTpij4NqJSRlv1uAxUelA6dESIStXz0kEUE8t F3UHq3lcOcE11hNhMKpb4y6Jp7s6nfrYveEVCfxL/++6wM7y3NgZnUVup LVl7er3A8s8AeAaoXxCy/47MxkH2q0Bp+2ql+3ogqsPK6OwnIkaz80gPX XZoO0VH7ZGlS1fx3uj4Nzk86hpjEaOA36vkpfsfp21YY+D0iPicnbSub0 ejVzWG6135GVBGzMfU3Ea9VV0gQHGsqRrexcE2XFL7T1NElPbhdCADFuA Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="438778303" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="438778303" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="876910950" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="876910950" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:04 -0700 Subject: [PATCH v6 05/11] cxl: Calculate and store PCI link latency for the downstream ports From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:04 -0700 Message-ID: <168451602432.3470703.1573168128194966298.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org The latency is calculated by dividing the flit size over the bandwidth. Add support to retrieve the flit size for the CXL switch device and calculate the latency of the PCIe link. Cache the latency number with cxl_dport. Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v5: - Store the latency numbers under dports. (Dan) - Use defines instead of magic numbers. (Jonathan) v2: - Fix commit log issues. (Jonathan) - Fix var declaration issues. (Jonathan) --- drivers/cxl/core/pci.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/port.c | 6 ++++ drivers/cxl/cxl.h | 4 +++ drivers/cxl/cxlpci.h | 15 ++++++++++ drivers/cxl/pci.c | 13 -------- 5 files changed, 98 insertions(+), 13 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 206c4a31bd09..cfb55f85a168 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include #include #include #include @@ -783,3 +784,75 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, return PCI_ERS_RESULT_NEED_RESET; } EXPORT_SYMBOL_NS_GPL(cxl_error_detected, CXL); + +extern const unsigned char pcie_link_speed[]; + +static enum pci_bus_speed get_link_speed(struct pci_dev *pdev) +{ + u16 linkstat; + int err; + + err = pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &linkstat); + if (err) + return -EINVAL; + + return pcie_link_speed[linkstat & PCI_EXP_LNKSTA_CLS]; +} + +static int pci_bus_speed_to_mbps(enum pci_bus_speed speed) +{ + switch (speed) { + case PCIE_SPEED_2_5GT: + return 2500; + case PCIE_SPEED_5_0GT: + return 5000; + case PCIE_SPEED_8_0GT: + return 8000; + case PCIE_SPEED_16_0GT: + return 16000; + case PCIE_SPEED_32_0GT: + return 32000; + case PCIE_SPEED_64_0GT: + return 64000; + default: + break; + } + + return -EINVAL; +} + +static int cxl_flit_size(struct pci_dev *pdev) +{ + if (cxl_pci_flit_256(pdev)) + return 256; + + return 68; +} + +/** + * cxl_pci_get_latency - calculate the link latency for the PCIe link + * @pdev - PCI device + * + * return: calculated latency or 0 for no latency + * + * CXL Memory Device SW Guide v1.0 2.11.4 Link latency calculation + * Link latency = LinkPropagationLatency + FlitLatency + RetimerLatency + * LinkProgationLatency is negligible, so 0 will be used + * RetimerLatency is assumed to be negligible and 0 will be used + * FlitLatency = FlitSize / LinkBandwidth + * FlitSize is defined by spec. CXL rev3.0 4.2.1. + * 68B flit is used up to 32GT/s. >32GT/s, 256B flit size is used. + * The FlitLatency is converted to picoseconds. + */ +long cxl_pci_get_latency(struct pci_dev *pdev) +{ + long bw; + + bw = pci_bus_speed_to_mbps(get_link_speed(pdev)); + if (bw < 0) + return 0; + bw /= BITS_PER_BYTE; + + return cxl_flit_size(pdev) * MEGA / bw; +} +EXPORT_SYMBOL_NS_GPL(cxl_pci_get_latency, CXL); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index fbef16dee3ed..cd7c22e7e387 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -750,6 +750,9 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host, if (rc) return ERR_PTR(rc); + if (parent_dport && dev_is_pci(uport)) + port->pci_latency = cxl_pci_get_latency(to_pci_dev(uport)); + return port; err: @@ -1007,6 +1010,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, if (rc) return ERR_PTR(rc); + if (dev_is_pci(dport_dev)) + dport->link_latency = cxl_pci_get_latency(to_pci_dev(dport_dev)); + return dport; } diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 34e6e19875cd..3d0c8c63f6d6 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -562,6 +562,7 @@ struct cxl_dax_region { * @depth: How deep this port is relative to the root. depth 0 is the root. * @cdat: Cached CDAT data * @cdat_available: Should a CDAT attribute be available in sysfs + * @pci_latency: Upstream latency in picoseconds */ struct cxl_port { struct device dev; @@ -584,6 +585,7 @@ struct cxl_port { size_t length; } cdat; bool cdat_available; + long pci_latency; }; struct qtg_dsm_input { @@ -635,6 +637,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port * @coord: access coordinates (performance) for switch from CDAT + * @link_latency: calculated PCIe downstream latency */ struct cxl_dport { struct device *dport; @@ -644,6 +647,7 @@ struct cxl_dport { bool rch; struct cxl_port *port; struct access_coordinate coord; + long link_latency; }; /** diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 7c02e55b8042..1772cd226108 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -84,6 +84,19 @@ struct cdat_entry_header { __le16 length; } __packed; +/* + * CXL v3.0 6.2.3 Table 6-4 + * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits + * mode, otherwise it's 68B flits mode. + */ +static inline bool cxl_pci_flit_256(struct pci_dev *pdev) +{ + u16 lnksta2; + + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); + return lnksta2 & PCI_EXP_LNKSTA2_FLIT; +} + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -92,4 +105,6 @@ void read_cdat_data(struct cxl_port *port); void cxl_cor_error_detected(struct pci_dev *pdev); pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, pci_channel_state_t state); +long cxl_pci_get_latency(struct pci_dev *pdev); + #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 0872f2233ed0..5f1b2f7239bc 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -365,19 +365,6 @@ static bool is_cxl_restricted(struct pci_dev *pdev) return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END; } -/* - * CXL v3.0 6.2.3 Table 6-4 - * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits - * mode, otherwise it's 68B flits mode. - */ -static bool cxl_pci_flit_256(struct pci_dev *pdev) -{ - u16 lnksta2; - - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); - return lnksta2 & PCI_EXP_LNKSTA2_FLIT; -} - static int cxl_pci_ras_unmask(struct pci_dev *pdev) { struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus); From patchwork Fri May 19 17:07:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C430FC7EE39 for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230423AbjESRJe (ORCPT ); Fri, 19 May 2023 13:09:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232052AbjESRJ1 (ORCPT ); Fri, 19 May 2023 13:09:27 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80F821A1 for ; Fri, 19 May 2023 10:09:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516154; x=1716052154; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HQE7u7MtWpYtQfOgdXuDYgh4z8uJ6pdIS40kuv+tT/I=; b=aH32AkGnvo8cqYMjYc06fIO+sXZ2z27xEJmclTGWFBXvfaipG2pI4FPE tHQ/a4AAQmvetCB8b54OIn1sghLAJIWHY6Ph0hEHtNV5YZc+j8qGrG3sR 8ALu+YNttvWAoK41skHIeuxV6R7aMzfX5aT4kmGoDcue9+dtwOeFfNYiF BX29hILq0uZYcFE6WjlQJ14rTtLgTawHE4VaEo2BUP441JVtau0DWoG2g /cXXsnAgRp/F0HPBO0h96nUabgXeYdRbfNxTDgu9G9hDjuovc0EB/9cT9 ruPGnx/TMpIPB+3K8tkVocgOL8mrDMu6sVoBrMzTmjSSYpYh/ybOukEKg Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="438778326" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="438778326" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="876911011" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="876911011" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:10 -0700 Subject: [PATCH v6 06/11] cxl: Store the access coordinates for the generic ports From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:10 -0700 Message-ID: <168451603029.3470703.13447576080193633262.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Each CXL host bridge is represented by an ACPI0016 device. A generic port device handle that is an ACPI device is represented by a string of ACPI0016 device HID and UID. Create a device handle from the ACPI device and retrieve the access coordinates from the stored memory targets. The access coordinates are stored under the cxl_dport that is associated with the CXL host bridge. The access coordinates struct is dynamically allocated under cxl_dport in order for code later on to detect whether the data exists or not. Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v6: - Change memcpy to strncpy. ACPI HID and UID are strings. --- drivers/cxl/acpi.c | 22 ++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 24 insertions(+) diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 8247df06d683..0c22cc9dbcba 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -557,8 +557,26 @@ static int cxl_get_chbcr(union acpi_subtable_headers *header, void *arg, return 0; } +static int get_genport_coordinates(struct device *dev, struct cxl_dport *dport) +{ + struct acpi_device *hb = to_cxl_host_bridge(NULL, dev); + u8 handle[ACPI_SRAT_DEVICE_HANDLE_SIZE] = { 0 }; + int rc; + + /* ACPI spec 6.5 table 5.65 */ + strncpy(handle, acpi_device_hid(hb), 8); + strncpy(&handle[8], acpi_device_uid(hb), 4); + + rc = acpi_get_genport_coordinates(handle, &dport->hb_access); + if (rc) + return rc; + + return 0; +} + static int add_host_bridge_dport(struct device *match, void *arg) { + int ret; acpi_status rc; struct device *bridge; unsigned long long uid; @@ -614,6 +632,10 @@ static int add_host_bridge_dport(struct device *match, void *arg) if (IS_ERR(dport)) return PTR_ERR(dport); + ret = get_genport_coordinates(match, dport); + if (ret) + dev_dbg(match, "Failed to get generic port perf coordinates.\n"); + return 0; } diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 3d0c8c63f6d6..2369e91add63 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -636,6 +636,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) * @rcrb: base address for the Root Complex Register Block * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port + * @genport_coord: access coordinates (performance) from ACPI generic port * @coord: access coordinates (performance) for switch from CDAT * @link_latency: calculated PCIe downstream latency */ @@ -646,6 +647,7 @@ struct cxl_dport { resource_size_t rcrb; bool rch; struct cxl_port *port; + struct access_coordinate hb_access; struct access_coordinate coord; long link_latency; }; From patchwork Fri May 19 17:07:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4590C77B75 for ; Fri, 19 May 2023 17:10:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229887AbjESRKA (ORCPT ); Fri, 19 May 2023 13:10:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232062AbjESRJ2 (ORCPT ); Fri, 19 May 2023 13:09:28 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C305B1BC for ; Fri, 19 May 2023 10:09:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516160; x=1716052160; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aE5miGtJG8jKLnDMya5ffMgbV588qPbaGaXeSjWleXs=; b=Er2v82lsXqhm7x9VuPrgi0KyymRnLAEOr+Vsn4MPqoJT7Sy0fHkLhvx2 FmpgWG8cKhZ56bIHF1QE5qyExey6S+w1ScLkbq6VophaFWW5SBiamFzGt qbKBGYSfOu7FGAc59XKfqaZzAd6ym3fAmI1tOdjtcZ5VVB2eocpjPeS7n CujhZHFF+f/KMan4fI2s7PK9tG4YasRqAjnqK9dYdNxt7RFc99rz81CMS jwc0ga+Pu4Moi8R5RayqqbW/i8kjQxhzQ2AIUU+iISLwBhgzc46xiiMuG cRgh1EsbYHVfumhUY/2wGdd3Ut25sglhLE5RGkzA3151hp1ZyULmYtgK9 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="438778354" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="438778354" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="876911075" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="876911075" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:16 -0700 Subject: [PATCH v6 07/11] cxl: Add helper function that calculate performance data for downstream ports From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:16 -0700 Message-ID: <168451603625.3470703.5688640606970207460.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org The CDAT information from the switch, Switch Scoped Latency and Bandwidth Information Strucutre (SSLBIS), is parsed and stored under a cxl_dport based on the correlated downstream port id from the SSLBIS entry. Walk the entire CXL port paths and collect all the performance data. Also pick up the link latency number that's stored under the dports. The entire path PCIe bandwidth can be retrieved using the pcie_bandwidth_available() call. Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v6: - Change min_t() to min(). (Jonathan) --- drivers/cxl/core/port.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 3 ++ 2 files changed, 69 insertions(+) diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index cd7c22e7e387..c051c48f835f 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -1949,6 +1950,71 @@ bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd) } EXPORT_SYMBOL_NS_GPL(schedule_cxl_memdev_detach, CXL); +static void combine_coordinates(struct access_coordinate *c1, + struct access_coordinate *c2) +{ + if (c2->write_bandwidth) + c1->write_bandwidth = min(c1->write_bandwidth, + c2->write_bandwidth); + c1->write_latency += c2->write_latency; + + if (c2->read_bandwidth) + c1->read_bandwidth = min(c1->read_bandwidth, + c2->read_bandwidth); + c1->read_latency += c2->read_latency; +} + +/** + * cxl_endpoint_get_perf_coordinates - Retrieve performance numbers stored in dports + * of CXL path + * @port: endpoint cxl_port + * @coord: output performance data + * + * Return: errno on failure, 0 on success. + */ +int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, + struct access_coordinate *coord) +{ + struct access_coordinate c = { + .read_bandwidth = UINT_MAX, + .write_bandwidth = UINT_MAX, + }; + struct cxl_port *iter = port; + struct cxl_dport *dport; + struct pci_dev *pdev; + unsigned int bw; + + if (!is_cxl_endpoint(port)) + return -EINVAL; + + dport = iter->parent_dport; + while (iter && !is_cxl_root(iter)) { + combine_coordinates(&c, &dport->coord); + c.write_latency += dport->link_latency; + c.read_latency += dport->link_latency; + + combine_coordinates(&c, &dport->hb_access); + + iter = to_cxl_port(iter->dev.parent); + dport = iter->parent_dport; + } + + /* Get the calculated PCI paths bandwidth */ + pdev = to_pci_dev(port->uport->parent); + bw = pcie_bandwidth_available(pdev, NULL, NULL, NULL); + if (bw == 0) + return -ENXIO; + bw /= BITS_PER_BYTE; + + c.write_bandwidth = min(c.write_bandwidth, bw); + c.read_bandwidth = min(c.read_bandwidth, bw); + + *coord = c; + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_endpoint_get_perf_coordinates, CXL); + /* for user tooling to ensure port disable work has completed */ static ssize_t flush_store(const struct bus_type *bus, const char *buf, size_t count) { diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 2369e91add63..f5c152eb55c0 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -866,6 +866,9 @@ static inline int cxl_cdat_switch_process(struct cxl_port *port) } #endif +int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, + struct access_coordinate *coord); + /* * Unit test builds overrides this to __weak, find the 'strong' version * of these symbols in tools/testing/cxl/. From patchwork Fri May 19 17:07:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C387CC77B7A for ; Fri, 19 May 2023 17:10:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230060AbjESRJa (ORCPT ); Fri, 19 May 2023 13:09:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231165AbjESRJD (ORCPT ); Fri, 19 May 2023 13:09:03 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8620A18F for ; Fri, 19 May 2023 10:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516120; x=1716052120; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lnwjzWJBwpZEUaH7obwI9lyJo1Yvgd3Hs8oNYc9ZqcM=; b=EZDO+Vm+Mmx+XhuUp/i8bFB5sQxQH70QDohd9ZdEt6WrabaYOgUh9s4j /UwvQ1UjoBpBeDt/p7JrWtak1vRlV8Abipb1u7oQbXLZeF24MmR0djTc1 l7vhcYwL/S7c4MYYyqaoXVWArFV1JUYffDk8VYFr5wyEXhbHUGe1pj3Rq uV8yeD47s7kz7vYochFQPX+AMZxmjywci1oEqN2K72G1RT+3RY33VxyzO t+JoF2O//ECXQfpA/42uq6WWFoKeSodBicFmxeNx35tuoquzBg4oeYXvh /EFG1soa63uwzYWmTzCefUtNmxBDz/Ozu6o2KA+fdUpHg0x3vZ9TbWQ4B w==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="337018596" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="337018596" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="846951202" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="846951202" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:23 -0700 Subject: [PATCH v6 08/11] cxl: Compute the entire CXL path latency and bandwidth data From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:22 -0700 Message-ID: <168451604218.3470703.3152270825986749312.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL Memory Device SW Guide [1] rev1.0 2.11.2 provides instruction on how to calculate latency and bandwidth for CXL memory device. Calculate minimum bandwidth and total latency for the path from the CXL device to the root port. The QTG id is retrieved by providing the performance data as input and calling the root port callback ->get_qos_class(). The retrieved id is stored with the cxl_port of the CXL device. For example for a device that is directly attached to a host bus: Total Latency = Device Latency (from CDAT) + Dev to Host Bus (HB) Link Latency + Generic Port Latency Min Bandwidth = Min bandwidth for link bandwidth between HB and CXL device, device CDAT bandwidth, and Generic Port Bandwidth For a device that has a switch in between host bus and CXL device: Total Latency = Device (CDAT) Latency + Dev to Switch Link Latency + Switch (CDAT) Latency + Switch to HB Link Latency + Generic Port Latency Min Bandwidth = Min bandwidth for link bandwidth between CXL device to CXL switch, CXL device CDAT bandwidth, CXL switch CDAT bandwidth, CXL switch to HB bandwidth, and Generic Port Bandwidth. [1]: https://cdrdv2-public.intel.com/643805/643805_CXL%20Memory%20Device%20SW%20Guide_Rev1p0.pdf Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v5: - Use new API call cxl_endpoint_get_perf_coordinates(). - Use root_port->get_qos_class() (Dan) - Add endieness handling to DSM input. --- drivers/cxl/core/cdat.c | 1 + drivers/cxl/cxl.h | 1 + drivers/cxl/port.c | 57 +++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 56 insertions(+), 3 deletions(-) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index 73ac96e754a0..3ddb2629515f 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -147,6 +147,7 @@ void cxl_cdat_dsmas_list_destroy(struct list_head *dsmas_list) list_for_each_entry_safe(dentry, n, dsmas_list, list) { list_del(&dentry->list); + kfree(dentry->qos_class); kfree(dentry); } } diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index f5c152eb55c0..495b6271d9a5 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -843,6 +843,7 @@ struct dsmas_entry { struct range dpa_range; u8 handle; struct access_coordinate coord; + struct qos_class *qos_class; }; #ifdef CONFIG_ACPI_TABLES_LIB diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 0f9c25645e1d..03af92217192 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -57,6 +57,53 @@ static int discover_region(struct device *dev, void *root) return 0; } +static int cxl_port_perf_data_calculate(struct cxl_port *port, + struct list_head *dsmas_list) +{ + struct qos_class *qos_class; + struct access_coordinate c; + struct qtg_dsm_input input; + struct cxl_port *root_port; + struct cxl_root *cxl_root; + struct dsmas_entry *dent; + int rc; + + rc = cxl_endpoint_get_perf_coordinates(port, &c); + if (rc) { + dev_dbg(&port->dev, "Failed to retrieve perf coordinates.\n"); + return rc; + } + + root_port = find_cxl_root(port); + cxl_root = to_cxl_root(root_port); + if (!cxl_root->ops || !cxl_root->ops->get_qos_class) + return -EOPNOTSUPP; + + list_for_each_entry(dent, dsmas_list, list) { + dent->coord.read_latency = dent->coord.read_latency + + c.read_latency; + dent->coord.write_latency = dent->coord.write_latency + + c.write_latency; + dent->coord.read_bandwidth = min_t(int, c.read_bandwidth, + dent->coord.read_bandwidth); + dent->coord.write_bandwidth = min_t(int, c.write_bandwidth, + dent->coord.write_bandwidth); + + input.rd_lat = cpu_to_le32(dent->coord.read_latency); + input.wr_lat = cpu_to_le32(dent->coord.write_latency); + input.rd_bw = cpu_to_le32(dent->coord.read_bandwidth); + input.wr_bw = cpu_to_le32(dent->coord.write_bandwidth); + + qos_class = cxl_root->ops->get_qos_class(root_port, &input); + if (IS_ERR(qos_class)) + continue; + + dent->qos_class = qos_class; + } + + return 0; +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -143,10 +190,14 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) LIST_HEAD(dsmas_list); rc = cxl_cdat_endpoint_process(port, &dsmas_list); - if (rc < 0) + if (rc < 0) { dev_dbg(&port->dev, "Failed to parse CDAT: %d\n", rc); - - /* Performance data processing */ + } else { + rc = cxl_port_perf_data_calculate(port, &dsmas_list); + if (rc) + dev_dbg(&port->dev, + "Failed to do perf coord calculations.\n"); + } cxl_cdat_dsmas_list_destroy(&dsmas_list); } From patchwork Fri May 19 17:07:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A539C7EE2D for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230025AbjESRJa (ORCPT ); Fri, 19 May 2023 13:09:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231550AbjESRJG (ORCPT ); Fri, 19 May 2023 13:09:06 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85FAF10DD for ; Fri, 19 May 2023 10:08:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516125; x=1716052125; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mc/8fniOlq92Ar//AdNTCWxqpnwvSxBw30oA/VqtSGo=; b=Eg3EEi8Q/sxjuZJNQEtaNaUBLSQwG8noylKrytUFRm1OHWLWMQrOVNe1 3mvXFS0kEAlKNQPDa5jy35ndM4LazEy2TrjfzZNv3YTaeTih4Po370e8i GEFT/SwZWqKqgV+dw82K2gp+9jMHdUHLfES28Bqacwu46vNGL7WkPkeKZ CqBPHs3ENRpP4yoZl4UrfYq0YWqQMiZxezZo0YDw5GzLWlbrfKo2rXinN GQQEpPMjja80V7h5UOFo5mX5kK5HaElFM0EAjYyz0swbKGgJBaHhqqUyQ dfj0fwMiApPcM8mQG3AmUKUMBwIXds+W6jY7USljt8OTZPE9UcaxL/wfs w==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="337018603" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="337018603" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="846951221" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="846951221" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:29 -0700 Subject: [PATCH v6 09/11] cxl: Store QTG IDs and related info to the CXL memory device context From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:28 -0700 Message-ID: <168451604884.3470703.10173844932484539394.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Once the QTG ID _DSM is executed successfully, the QTG ID is retrieved from the return package. Create a list of entries in the cxl_memdev context and store the QTG ID and the associated DPA range. This information can be exposed to user space via sysfs in order to help region setup for hot-plugged CXL memory devices. Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v6: - Store entire QTG ID list v4: - Remove unused qos_list from cxl_md v3: - Move back to QTG ID per partition --- drivers/cxl/core/mbox.c | 1 + drivers/cxl/cxlmem.h | 23 +++++++++++++++++++++++ drivers/cxl/port.c | 38 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index 2c8dc7e2b84d..35941a306ea8 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1260,6 +1260,7 @@ struct cxl_dev_state *cxl_dev_state_create(struct device *dev) mutex_init(&cxlds->mbox_mutex); mutex_init(&cxlds->event.log_lock); cxlds->dev = dev; + INIT_LIST_HEAD(&cxlds->perf_list); return cxlds; } diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index a2845a7a69d8..708d60c5ffe1 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -5,6 +5,7 @@ #include #include #include +#include #include "cxl.h" /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ @@ -254,6 +255,21 @@ struct cxl_poison_state { struct mutex lock; /* Protect reads of poison list */ }; +/** + * struct perf_prop - performance property entry + * @list - list entry + * @dpa_range - range for DPA address + * @coord - QoS performance data (i.e. latency, bandwidth) + * @qos_class - QoS Class cookies + */ +struct perf_prop_entry { + struct list_head list; + struct range dpa_range; + struct access_coordinate coord; + /* Do not add members below this, contains flex array */ + struct qos_class qos_class; +}; + /** * struct cxl_dev_state - The driver device state * @@ -292,6 +308,9 @@ struct cxl_poison_state { * @event: event log driver state * @poison: poison driver state info * @mbox_send: @dev specific transport for transmitting mailbox commands + * @ram_qos_class: QoS class cookies for volatile region + * @pmem_qos_class: QoS class cookies for persistent region + * @perf_list: performance data entries list * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for * details on capacity parameters. @@ -325,6 +344,10 @@ struct cxl_dev_state { u64 next_volatile_bytes; u64 next_persistent_bytes; + struct qos_class *ram_qos_class; + struct qos_class *pmem_qos_class; + struct list_head perf_list; + resource_size_t component_reg_phys; u64 serial; diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 03af92217192..e5d7ad5b1e16 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -104,6 +104,42 @@ static int cxl_port_perf_data_calculate(struct cxl_port *port, return 0; } +static void cxl_memdev_set_qtg(struct cxl_dev_state *cxlds, struct list_head *dsmas_list) +{ + struct range pmem_range = { + .start = cxlds->pmem_res.start, + .end = cxlds->pmem_res.end, + }; + struct range ram_range = { + .start = cxlds->ram_res.start, + .end = cxlds->ram_res.end, + }; + struct perf_prop_entry *perf; + struct dsmas_entry *dent; + + list_for_each_entry(dent, dsmas_list, list) { + perf = devm_kzalloc(cxlds->dev, + sizeof(*perf) + dent->qos_class->nr * sizeof(int), + GFP_KERNEL); + if (!perf) + return; + + perf->dpa_range = dent->dpa_range; + perf->coord = dent->coord; + perf->qos_class = *dent->qos_class; + list_add_tail(&perf->list, &cxlds->perf_list); + + if (resource_size(&cxlds->ram_res) && + range_contains(&ram_range, &dent->dpa_range) && + !cxlds->ram_qos_class) + cxlds->ram_qos_class = &perf->qos_class; + else if (resource_size(&cxlds->pmem_res) && + range_contains(&pmem_range, &dent->dpa_range) && + !cxlds->pmem_qos_class) + cxlds->pmem_qos_class = &perf->qos_class; + } +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -197,6 +233,8 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) if (rc) dev_dbg(&port->dev, "Failed to do perf coord calculations.\n"); + else + cxl_memdev_set_qtg(cxlds, &dsmas_list); } cxl_cdat_dsmas_list_destroy(&dsmas_list); From patchwork Fri May 19 17:07:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B40E2C7EE32 for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230268AbjESRJd (ORCPT ); Fri, 19 May 2023 13:09:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231991AbjESRJV (ORCPT ); Fri, 19 May 2023 13:09:21 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9876610FA for ; Fri, 19 May 2023 10:09:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516143; x=1716052143; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C+IRYEmeky+jiKMTFQYjJ2nrfEME3gDPoL+KaHbx/BI=; b=iHuPo6VKJxkce5nUOwx9Ebi6oqItnRplgA76KfAS/4Dacy/4xOJ+jyNQ 9F16INjxFdNDqvRH0vI1v+dcuk4ZH8MlSeEUBAZuHVBAbh2WAKnxbEhzh 49nLSJPHaO6VADgsyeqCRd1T3glsnb1HQL67omwnYLqCJ0ZuxPzk5iDX9 RGGp4cga67ecrB4fchcLD7Y+lTP+CL0g6dy+8+H+693C97oMNLH9GwgKh 63vASSsFsZh/NguD1kpVmhOXiujCkHHHT/mpS4DLmiVBtTjEtzOLG6nFB ZdNVQOnqAijjmZkqxE/2sIJOP8TmfI/SoLWKePtTrICIWIaCohcdewT1d Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="337018625" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="337018625" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="846951245" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="846951245" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:34 -0700 Subject: [PATCH v6 10/11] cxl: Export sysfs attributes for memory device QoS class From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Dan Williams , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:34 -0700 Message-ID: <168451605449.3470703.14744276101463857790.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Export qos_class sysfs attributes for the CXL memory device. The QoS clas should show up as /sys/bus/cxl/devices/memX/ram/qos_class for the volatile partition and /sys/bus/cxl/devices/memX/pmem/qos_class for the persistent partition. The QTG ID is retrieved via _DSM after supplying the calculated bandwidth and latency for the entire CXL path from device to the CPU. This ID is used to match up to the root decoder QoS class to determine which CFMWS the memory range of a hotplugged CXL mem device should be assigned under. While there may be multiple DSMAS exported by the device CDAT, the driver will only expose the first QTG ID per partition in sysfs for now. In the future when multiple QTG IDs are necessary, they can be exposed. [1] [1]: https://lore.kernel.org/linux-cxl/167571650007.587790.10040913293130712882.stgit@djiang5-mobl3.local/T/#md2a47b1ead3e1ba08f50eab29a4af1aed1d215ab Suggested-by: Dan Williams Signed-off-by: Dave Jiang --- v6: - Provide full ordered QTG IDs from _DSM. (Jonathan) v5: - Change qtg_id to qos_class v4: - Change kernel version for documentation to v6.5 v3: - Expand description of qtg_id. (Alison) --- Documentation/ABI/testing/sysfs-bus-cxl | 34 +++++++++++++++++++ drivers/cxl/core/memdev.c | 55 +++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index ccfc7ecc61f5..7a4cacd382f9 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -28,6 +28,23 @@ Description: Payload in the CXL-2.0 specification. +What: /sys/bus/cxl/devices/memX/ram/qos_class +Date: May, 2023 +KernelVersion: v6.5 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) For CXL host platforms that support "QoS Telemmetry" + this attribute conveys a comma delimited list of platform + specific cookies that identifies a QoS performance class + for the volatile partition of the CXL mem device. These + class-ids can be compared against a similar "qos_class" + published for a root decoder. While it is not required + that the endpoints map their local memory-class to a + matching platform class, mismatches are not recommended + and there are platform specific side-effects that may + result. + + What: /sys/bus/cxl/devices/memX/pmem/size Date: December, 2020 KernelVersion: v5.12 @@ -38,6 +55,23 @@ Description: Payload in the CXL-2.0 specification. +What: /sys/bus/cxl/devices/memX/pmem/qos_class +Date: May, 2023 +KernelVersion: v6.5 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) For CXL host platforms that support "QoS Telemmetry" + this attribute conveys a comma delimited list of platform + specific cookies that identifies a QoS performance class + for the persistent partition of the CXL mem device. These + class-ids can be compared against a similar "qos_class" + published for a root decoder. While it is not required + that the endpoints map their local memory-class to a + matching platform class, mismatches are not recommended + and there are platform specific side-effects that may + result. + + What: /sys/bus/cxl/devices/memX/serial Date: January, 2022 KernelVersion: v5.18 diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 057a43267290..9366257e3183 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -77,6 +77,32 @@ static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL); +static ssize_t ram_qos_class_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + struct qos_class *qos_class = cxlds->ram_qos_class; + int count, i; + + if (!qos_class) + return 0; + + for (i = 0, count = 0; i < qos_class->nr; i++) { + count += sysfs_emit_at(buf, count, "%d", + qos_class->entries[i]); + if (i + 1 == qos_class->nr) + count += sysfs_emit_at(buf, count, "\n"); + else + count += sysfs_emit_at(buf, count, ", "); + } + + return count; +} + +static struct device_attribute dev_attr_ram_qos_class = + __ATTR(qos_class, 0444, ram_qos_class_show, NULL); + static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -90,6 +116,33 @@ static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL); +static ssize_t pmem_qos_class_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + struct qos_class *qos_class = cxlds->pmem_qos_class; + int count, i; + + if (!qos_class) + return 0; + + for (i = 0, count = 0; i < qos_class->nr; i++) { + count += sysfs_emit_at(buf, count, "%d", + qos_class->entries[i]); + + if (i + 1 == qos_class->nr) + count += sysfs_emit_at(buf, count, "\n"); + else + count += sysfs_emit_at(buf, count, ", "); + } + + return count; +} + +static struct device_attribute dev_attr_pmem_qos_class = + __ATTR(qos_class, 0444, pmem_qos_class_show, NULL); + static ssize_t serial_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -344,11 +397,13 @@ static struct attribute *cxl_memdev_attributes[] = { static struct attribute *cxl_memdev_pmem_attributes[] = { &dev_attr_pmem_size.attr, + &dev_attr_pmem_qos_class.attr, NULL, }; static struct attribute *cxl_memdev_ram_attributes[] = { &dev_attr_ram_size.attr, + &dev_attr_ram_qos_class.attr, NULL, }; From patchwork Fri May 19 17:07:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13248647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94916C7EE33 for ; Fri, 19 May 2023 17:10:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230318AbjESRJe (ORCPT ); Fri, 19 May 2023 13:09:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232023AbjESRJZ (ORCPT ); Fri, 19 May 2023 13:09:25 -0400 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6339E49 for ; Fri, 19 May 2023 10:09:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684516146; x=1716052146; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TpcyBClL2xkW6mFLLKXtc8VKmjh6vSpL8VUbBIor44Y=; b=V43vnsahUFLtiMFMTETUj7mEr2RFKSaLdH2Vp9jZTb4rpvUhZYXwKOPE IV5ENKjHjTyRUJfqiE37m4vVTMjWCi5cv0PIFLBWuDRLt0FNyV37rkvYG 2vZW1d3dZ4eip+U+7klWdOyW9wioAUOSxYYk/8iujLqx64OsgeoeXohCM skT5YmqKgYWAHyuUcvClxT9fLFquN4ZbmaIEPDkhCc5grA5h8pntrqU8q 6H4MWNW+ZBGShEw5pyLTedpo07aFDfraNTESQnl2EK8Qffz/tt4PBMgEj HaAtMjtU6Yvb4Vujb1n1GonTmckBnxTL32D6IZrvnundk2Z05ywMa3T9P g==; X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="337018639" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="337018639" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10715"; a="846951250" X-IronPort-AV: E=Sophos;i="6.00,177,1681196400"; d="scan'208";a="846951250" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.212.29.189]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2023 10:07:40 -0700 Subject: [PATCH v6 11/11] cxl/mem: Add debugfs output for QTG related data From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Dan Williams , Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Fri, 19 May 2023 10:07:40 -0700 Message-ID: <168451606013.3470703.17517034050660255743.stgit@djiang5-mobl3> In-Reply-To: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> References: <168451588868.3470703.3527256859632103687.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Add debugfs output to /sys/kernel/debug/cxl/memX/qtgmap The debugfs attribute will dump out all the DSMAS ranges and the associated QTG ID exported by the CXL device CDAT. Suggested-by: Dan Williams Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v4: - Use cxlds->qos_list instead of the stray cxlmd->qos_list --- Documentation/ABI/testing/debugfs-cxl | 13 +++++++++++++ MAINTAINERS | 1 + drivers/cxl/mem.c | 26 ++++++++++++++++++++++++++ 3 files changed, 40 insertions(+) diff --git a/Documentation/ABI/testing/debugfs-cxl b/Documentation/ABI/testing/debugfs-cxl index fe61d372e3fa..e2b5b583c031 100644 --- a/Documentation/ABI/testing/debugfs-cxl +++ b/Documentation/ABI/testing/debugfs-cxl @@ -33,3 +33,16 @@ Description: device cannot clear poison from the address, -ENXIO is returned. The clear_poison attribute is only visible for devices supporting the capability. + + +What: /sys/kernel/debug/cxl/memX/qtg_map +Date: Mar, 2023 +KernelVersion: v6.5 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) Entries of all Device Physical Address (DPA) ranges + provided by the device Coherent Device Attributes Table (CDAT) + Device Scoped Memory Affinity Structure (DSMAS) entries with + the matching QoS Throttling Group (QTG) id calculated from the + latency and bandwidth of the CXL path from the memory device + to the CPU. diff --git a/MAINTAINERS b/MAINTAINERS index e0ad886d3163..0437b07df026 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5188,6 +5188,7 @@ M: Ben Widawsky M: Dan Williams L: linux-cxl@vger.kernel.org S: Maintained +F: Documentation/ABI/testing/debugfs-cxl F: drivers/cxl/ F: include/uapi/linux/cxl_mem.h diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 519edd0eb196..99bb68ae21ac 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -45,6 +45,31 @@ static int cxl_mem_dpa_show(struct seq_file *file, void *data) return 0; } +static int cxl_mem_qtg_show(struct seq_file *file, void *data) +{ + struct device *dev = file->private; + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + struct perf_prop_entry *perf; + + list_for_each_entry(perf, &cxlds->perf_list, list) { + struct qos_class *qos_class = &perf->qos_class; + int i; + + seq_printf(file, "%08llx-%08llx : QoS Class: ", + perf->dpa_range.start, perf->dpa_range.end); + for (i = 0; i < qos_class->nr; i++) { + seq_printf(file, "%d", qos_class->entries[i]); + if (i + 1 == qos_class->nr) + seq_puts(file, "\n"); + else + seq_puts(file, ", "); + } + } + + return 0; +} + static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, struct cxl_dport *parent_dport) { @@ -140,6 +165,7 @@ static int cxl_mem_probe(struct device *dev) dentry = cxl_debugfs_create_dir(dev_name(dev)); debugfs_create_devm_seqfile(dev, "dpamem", dentry, cxl_mem_dpa_show); + debugfs_create_devm_seqfile(dev, "qtgmap", dentry, cxl_mem_qtg_show); if (test_bit(CXL_POISON_ENABLED_INJECT, cxlds->poison.enabled_cmds)) debugfs_create_file("inject_poison", 0200, dentry, cxlmd,