From patchwork Mon May 8 20:46:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ACB5C77B7F for ; Mon, 8 May 2023 20:47:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232525AbjEHUrC (ORCPT ); Mon, 8 May 2023 16:47:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230125AbjEHUrB (ORCPT ); Mon, 8 May 2023 16:47:01 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 874AC4ECF for ; Mon, 8 May 2023 13:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578820; x=1715114820; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BcqKT+UhKNUQksbe6xIXUe8nlh5CDuzSwNs608X3Sxs=; b=JzRO1KaEBi/DLK2qzRFVOplNhTDIhrtGmfhwVQAgCQCbDFJRppSXmxqx z99dmusp3oi9HKkFcZwPOS8YeQrBjwCH570eGTcNBPFLqP9pssNg2k7yu n/5vEbri4U/imPAgfX3S1HIE6XNw8wUgsIh2NQL+l6dszvXlzvQOqXp7Z VXKYvrjhrmJ9EA1zGbcozgBGunH7M4myay5ZYZonBjFIe35B//R7QpINq 800GL2kOFKUFvhs9w0ZfNrNZ6Y3cKh/uVq+EDFaLnhrXIqZDJcUMYzSv9 SfxL+c8cyg8dksI/RfcDrKouBkdeUsajC+ETdmzOGoxUth8T/njnC9YeB A==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="330113990" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="330113990" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="763531160" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="763531160" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:00 -0700 Subject: [PATCH v5 01/14] cxl: Add callback to parse the DSMAS subtables from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:46:59 -0700 Message-ID: <168357881963.2756219.1853631182045246376.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Provide a callback function to the CDAT parser in order to parse the Device Scoped Memory Affinity Structure (DSMAS). Each DSMAS structure contains the DPA range and its associated attributes in each entry. See the CDAT specification for details. The device handle and the DPA range is saved and to be associated with the DSLBIS locality data when the DSLBIS entries are parsed. The list is a local list. When the total path performance data is calculated and storred this list can be discarded. Coherent Device Attribute Table 1.03 2.1 Device Scoped memory Affinity Structure (DSMAS) Signed-off-by: Dave Jiang --- v5: - Update commit log to indicate what list is used for. (Jonathan, Dan) - Use acpi_table_parse_cdat() - Isolate cdat code behind CONFIG_ACPI v3: - Add spec section number. (Alison) - Remove cast from void *. (Alison) - Refactor cxl_port_probe() block. (Alison) - Move CDAT parse to cxl_endpoint_port_probe() v2: - Add DSMAS table size check. (Lukas) - Use local DSMAS header for LE handling. - Remove dsmas lock. (Jonathan) - Fix handle size (Jonathan) - Add LE to host conversion for DSMAS address and length. - Make dsmas_list local --- drivers/cxl/core/Makefile | 1 + drivers/cxl/core/cdat.c | 40 ++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 18 ++++++++++++++++++ drivers/cxl/port.c | 22 ++++++++++++++++++++++ 4 files changed, 81 insertions(+) create mode 100644 drivers/cxl/core/cdat.c diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile index ca4ae31d8f57..98ddfd110f9b 100644 --- a/drivers/cxl/core/Makefile +++ b/drivers/cxl/core/Makefile @@ -12,5 +12,6 @@ cxl_core-y += memdev.o cxl_core-y += mbox.o cxl_core-y += pci.o cxl_core-y += hdm.o +cxl_core-$(CONFIG_ACPI) += cdat.o cxl_core-$(CONFIG_TRACING) += trace.o cxl_core-$(CONFIG_CXL_REGION) += region.o diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c new file mode 100644 index 000000000000..61979f0789aa --- /dev/null +++ b/drivers/cxl/core/cdat.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2023 Intel Corporation. All rights reserved. */ +#include +#include "cxlpci.h" +#include "cxl.h" + +static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg, + const unsigned long end) +{ + struct acpi_cdat_dsmas *dsmas = (struct acpi_cdat_dsmas *)header; + struct list_head *dsmas_list = arg; + struct dsmas_entry *dent; + u16 len; + + len = le16_to_cpu((__force __le16)dsmas->header.length); + if (len != sizeof(*dsmas) || (unsigned long)header + len > end) { + pr_warn("Malformed DSMAS table length: (%lu:%u)\n", + (unsigned long)sizeof(*dsmas), len); + return -EINVAL; + } + + dent = kzalloc(sizeof(*dent), GFP_KERNEL); + if (!dent) + return -ENOMEM; + + dent->handle = dsmas->dsmad_handle; + dent->dpa_range.start = le64_to_cpu((__force __le64)dsmas->dpa_base_address); + dent->dpa_range.end = le64_to_cpu((__force __le64)dsmas->dpa_base_address) + + le64_to_cpu((__force __le64)dsmas->dpa_length) - 1; + list_add_tail(&dent->list, dsmas_list); + + return 0; +} + +int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) +{ + return acpi_table_parse_cdat(ACPI_CDAT_TYPE_DSMAS, cdat_dsmas_handler, + list, port->cdat.table); +} +EXPORT_SYMBOL_NS_GPL(cxl_cdat_endpoint_process, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 4577d808ac6d..dda7238b47f5 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -791,6 +792,23 @@ static inline struct cxl_dax_region *to_cxl_dax_region(struct device *dev) } #endif +/* CDAT related bits */ +struct dsmas_entry { + struct list_head list; + struct range dpa_range; + u8 handle; +}; + +#ifdef CONFIG_ACPI +int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list); +#else +static inline int cxl_cdat_endpoint_process(struct cxl_port *port, + struct list_head *list) +{ + return -EOPNOTSUPP; +} +#endif + /* * Unit test builds overrides this to __weak, find the 'strong' version * of these symbols in tools/testing/cxl/. diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index a49f5eb149f1..da023feaa6e2 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -57,6 +57,16 @@ static int discover_region(struct device *dev, void *root) return 0; } +static void dsmas_list_destroy(struct list_head *dsmas_list) +{ + struct dsmas_entry *dentry, *n; + + list_for_each_entry_safe(dentry, n, dsmas_list, list) { + list_del(&dentry->list); + kfree(dentry); + } +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -131,6 +141,18 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) device_for_each_child(&port->dev, root, discover_region); put_device(&root->dev); + if (port->cdat.table) { + LIST_HEAD(dsmas_list); + + rc = cxl_cdat_endpoint_process(port, &dsmas_list); + if (rc < 0) + dev_dbg(&port->dev, "Failed to parse CDAT: %d\n", rc); + + /* Performance data processing */ + + dsmas_list_destroy(&dsmas_list); + } + return 0; } From patchwork Mon May 8 20:47:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D29AC7EE22 for ; Mon, 8 May 2023 20:47:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230125AbjEHUrb (ORCPT ); Mon, 8 May 2023 16:47:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229492AbjEHUra (ORCPT ); Mon, 8 May 2023 16:47:30 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCF05558B for ; Mon, 8 May 2023 13:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578849; x=1715114849; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KKnLJ1gTg6Ha5w8u7thHJMT+8jkS9xQ1gEWVCrc/His=; b=SIloT9QY8/MIRknR1RhTrMwOiIZFNmX5QEZRsac/vAcXU+GaahqPbpEx gs+krVd2glYgwsgQ17W7UwDADy5Y286HRoYgC1h4gdnbqgcHuc/h6UT4o RwljDRBJo0UrkIFQN/5begjnKKWHb5XLs2L1cRVMbFmWOZy5YrDeng8zp EnyKd1+0WSUegAyV4+EXS4WpwqCnG5IwNTnjs5ZBK4Xw+jCPqsnT+gjFd wudJBZHBNu5hhqaACfQNNBEtSO7lvj2Ylos/3piOsM6pU8+5s9gluQoMO YmfbCnEAuQFlXlMg+DSVjYedt5MUkoEBoKPqnUfnsMnG2HzZdiHam2Fzc g==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="338976599" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="338976599" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="731431659" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="731431659" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:05 -0700 Subject: [PATCH v5 02/14] cxl: Add callback to parse the DSLBIS subtable from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:05 -0700 Message-ID: <168357882541.2756219.12547742982939088156.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Provide a callback to parse the Device Scoped Latency and Bandwidth Information Structure (DSLBIS) in the CDAT structures. The DSLBIS contains the bandwidth and latency information that's tied to a DSMAS handle. The driver will retrieve the read and write latency and bandwidth associated with the DSMAS which is tied to a DPA range. Coherent Device Attribute Table 1.03 2.1 Device Scoped Latency and Bandwidth Information Structure (DSLBIS) Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v5: - Remove macro for common headers. (Jonathan) - Use acpi_table_parse_cdat(). - Remove unlikely(). (Dan) v3: - Added spec section in commit header. (Alison) - Remove void * recast. (Alison) - Rework comment. (Alison) - Move CDAT parse to cxl_endpoint_port_probe() - Convert to use 'struct node_hmem_attrs' v2: - Add size check to DSLIBIS table. (Lukas) - Remove unnecessary entry type check. (Jonathan) - Move data_type check to after match. (Jonathan) - Skip unknown data type. (Jonathan) - Add overflow check for unit multiply. (Jonathan) - Use dev_warn() when entries parsing fail. (Jonathan) --- drivers/cxl/core/cdat.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++- drivers/cxl/cxl.h | 2 + 2 files changed, 90 insertions(+), 2 deletions(-) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index 61979f0789aa..6e14d04c0453 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2023 Intel Corporation. All rights reserved. */ #include +#include #include "cxlpci.h" #include "cxl.h" @@ -32,9 +33,94 @@ static int cdat_dsmas_handler(union acpi_subtable_headers *header, void *arg, return 0; } +static void cxl_access_coordinate_set(struct access_coordinate *coord, + int access, unsigned int val) +{ + switch (access) { + case ACPI_HMAT_ACCESS_LATENCY: + coord->read_latency = val; + coord->write_latency = val; + break; + case ACPI_HMAT_READ_LATENCY: + coord->read_latency = val; + break; + case ACPI_HMAT_WRITE_LATENCY: + coord->write_latency = val; + break; + case ACPI_HMAT_ACCESS_BANDWIDTH: + coord->read_bandwidth = val; + coord->write_bandwidth = val; + break; + case ACPI_HMAT_READ_BANDWIDTH: + coord->read_bandwidth = val; + break; + case ACPI_HMAT_WRITE_BANDWIDTH: + coord->write_bandwidth = val; + break; + } +} + +static int cdat_dslbis_handler(union acpi_subtable_headers *header, void *arg, + const unsigned long end) +{ + struct acpi_cdat_dslbis *dslbis = (struct acpi_cdat_dslbis *)header; + struct list_head *dsmas_list = arg; + struct dsmas_entry *dent; + u16 len; + + len = le16_to_cpu((__force __le16)dslbis->header.length); + if (len != sizeof(*dslbis) || (unsigned long)header + len > end) { + pr_warn("Malformed DSLBIS table length: (%lu:%u)\n", + (unsigned long)sizeof(*dslbis), len); + return -EINVAL; + } + + /* Skip unrecognized data type */ + if (dslbis->data_type > ACPI_HMAT_WRITE_BANDWIDTH) + return 0; + + list_for_each_entry(dent, dsmas_list, list) { + u64 val; + int rc; + + if (dslbis->handle != dent->handle) + continue; + + /* Not a memory type, skip */ + if ((dslbis->flags & ACPI_HMAT_MEMORY_HIERARCHY) != + ACPI_HMAT_MEMORY) + return 0; + + rc = check_mul_overflow(le64_to_cpu((__force __le64)dslbis->entry_base_unit), + le16_to_cpu((__force __le16)dslbis->entry[0]), &val); + if (rc) + pr_warn("DSLBIS value overflowed.\n"); + + cxl_access_coordinate_set(&dent->coord, dslbis->data_type, val); + break; + } + + return 0; +} + int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) { - return acpi_table_parse_cdat(ACPI_CDAT_TYPE_DSMAS, cdat_dsmas_handler, - list, port->cdat.table); + int rc; + + rc = acpi_table_parse_cdat(ACPI_CDAT_TYPE_DSMAS, cdat_dsmas_handler, + list, port->cdat.table); + if (rc <= 0) { + if (rc == 0) + rc = -ENOENT; + return rc; + } + + rc = acpi_table_parse_cdat(ACPI_CDAT_TYPE_DSLBIS, + cdat_dslbis_handler, + list, port->cdat.table); + if (rc == 0) + rc = -ENOENT; + + return rc; } EXPORT_SYMBOL_NS_GPL(cxl_cdat_endpoint_process, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index dda7238b47f5..ca3d0d74f2e5 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -797,6 +798,7 @@ struct dsmas_entry { struct list_head list; struct range dpa_range; u8 handle; + struct access_coordinate coord; }; #ifdef CONFIG_ACPI From patchwork Mon May 8 20:47:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D0ABC77B7F for ; Mon, 8 May 2023 20:47:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230137AbjEHUrd (ORCPT ); Mon, 8 May 2023 16:47:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229492AbjEHUrc (ORCPT ); Mon, 8 May 2023 16:47:32 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E64255587 for ; Mon, 8 May 2023 13:47:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578850; x=1715114850; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nNHJZbx6SzN4jOJjU2bW6mMF7KMFWeDCyYy+tiNgg+Q=; b=evQ0z1pWpoA7hejhUjkB/dBbx7Wm5QKdX7v0xp3cYJhl06zztvvqV7DA vkE3v8mxCK3t/M9Bh6AgqqbgJ2Yvntsy3w6zSC5OzOYE/vGr+3silU/Qw DiBgDO1wmy+TNawI/DaqBJD60/aAIyv9OUDU6N8bCelvXLwSuaOQ9C5f9 xmLZLaBWOOrZhQ363lyVsdWUePFccT5++QZ/VqQxhzDZi0ijpTao0KaPl UJEcU9MpzyipwlZRoFrYgxmq/mUF31grgk0HK2PW98uZpmgiWccrif4fK 1DNAXRuPGU8lK246dyB19MtkV9kBQJ3r1ayxUTyDI9EtfK0IWTdT193tC A==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="338976604" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="338976604" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="731431665" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="731431665" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:12 -0700 Subject: [PATCH v5 03/14] cxl: Add callback to parse the SSLBIS subtable from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:11 -0700 Message-ID: <168357883158.2756219.14426990857899261700.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Provide a callback to parse the Switched Scoped Latency and Bandwidth Information Structure (SSLBIS) in the CDAT structures. The SSLBIS contains the bandwidth and latency information that's tied to the CXL switch that the data table has been read from. The extracted values are stored to the cxl_dport correlated by the port_id depending on the SSLBIS entry. Coherent Device Attribute Table 1.03 2.1 Switched Scoped Latency and Bandwidth Information Structure (DSLBIS) Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v5: - Store data to cxl_dport directly instead. (Dan) - Use acpi_table_parse_cdat(). v3: - Add spec section in commit header (Alison) - Move CDAT parse to cxl_switch_port_probe() - Use 'struct node_hmem_attrs' --- drivers/cxl/core/cdat.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 8 ++++ drivers/cxl/port.c | 12 ++++++ include/acpi/actbl1.h | 3 ++ 4 files changed, 110 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index 6e14d04c0453..37b135c900d5 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -124,3 +124,91 @@ int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) return rc; } EXPORT_SYMBOL_NS_GPL(cxl_cdat_endpoint_process, CXL); + +static int cdat_sslbis_handler(union acpi_subtable_headers *header, void *arg, + const unsigned long end) +{ + struct acpi_cdat_sslbis *sslbis = (struct acpi_cdat_sslbis *)header; + struct acpi_cdat_sslbe *entry; + struct cxl_port *port = arg; + struct device *dev = &port->dev; + int remain, entries, i; + u16 len; + + len = le16_to_cpu((__force __le16)sslbis->header.length); + remain = len - sizeof(*sslbis); + if (!remain || remain % sizeof(*entry) || + (unsigned long)header + len > end) { + dev_warn(dev, "Malformed SSLBIS table length: (%u)\n", len); + return -EINVAL; + } + + /* Unrecognized data type, we can skip */ + if (sslbis->data_type > ACPI_HMAT_WRITE_BANDWIDTH) + return 0; + + entries = remain / sizeof(*entry); + entry = (struct acpi_cdat_sslbe *)((unsigned long)header + sizeof(*sslbis)); + + for (i = 0; i < entries; i++) { + u16 x = le16_to_cpu(entry->portx_id); + u16 y = le16_to_cpu(entry->porty_id); + struct cxl_dport *dport; + unsigned long index; + u16 dsp_id; + u64 val; + + switch (x) { + case ACPI_CDAT_SSLBIS_US_PORT: + dsp_id = y; + break; + case ACPI_CDAT_SSLBIS_ANY_PORT: + switch (y) { + case ACPI_CDAT_SSLBIS_US_PORT: + dsp_id = x; + break; + case ACPI_CDAT_SSLBIS_ANY_PORT: + dsp_id = ACPI_CDAT_SSLBIS_ANY_PORT; + break; + default: + dsp_id = y; + break; + } + break; + default: + dsp_id = x; + break; + } + + if (check_mul_overflow(le64_to_cpu(sslbis->entry_base_unit), + le16_to_cpu(entry->latency_or_bandwidth), + &val)) + dev_warn(dev, "SSLBIS value overflowed!\n"); + + xa_for_each(&port->dports, index, dport) { + if (dsp_id == ACPI_CDAT_SSLBIS_ANY_PORT || + dsp_id == dport->port_id) + cxl_access_coordinate_set(&dport->coord, + sslbis->data_type, + val); + } + + entry++; + } + + return 0; +} + +int cxl_cdat_switch_process(struct cxl_port *port) +{ + int rc; + + rc = acpi_table_parse_cdat(ACPI_CDAT_TYPE_SSLBIS, + cdat_sslbis_handler, + port, port->cdat.table); + if (rc == 0) + rc = -ENOENT; + + return rc; +} +EXPORT_SYMBOL_NS_GPL(cxl_cdat_switch_process, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index ca3d0d74f2e5..3e8020e0a132 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -600,6 +600,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) * @rcrb: base address for the Root Complex Register Block * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port + * @coord: access coordinates (performance) for switch from CDAT */ struct cxl_dport { struct device *dport; @@ -608,6 +609,7 @@ struct cxl_dport { resource_size_t rcrb; bool rch; struct cxl_port *port; + struct access_coordinate coord; }; /** @@ -803,12 +805,18 @@ struct dsmas_entry { #ifdef CONFIG_ACPI int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list); +int cxl_cdat_switch_process(struct cxl_port *port); #else static inline int cxl_cdat_endpoint_process(struct cxl_port *port, struct list_head *list) { return -EOPNOTSUPP; } + +static inline int cxl_cdat_switch_process(struct cxl_port *port) +{ + return -EOPNOTSUPP; +} #endif /* diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index da023feaa6e2..c5a24b75bf03 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -86,7 +86,17 @@ static int cxl_switch_port_probe(struct cxl_port *port) if (IS_ERR(cxlhdm)) return PTR_ERR(cxlhdm); - return devm_cxl_enumerate_decoders(cxlhdm, NULL); + rc = devm_cxl_enumerate_decoders(cxlhdm, NULL); + if (rc < 0) + return rc; + + if (port->cdat.table) { + rc = cxl_cdat_switch_process(port); + if (rc < 0) + dev_warn(&port->dev, "Failed to parse SSLBIS: %d\n", rc); + } + + return 0; } static int cxl_endpoint_port_probe(struct cxl_port *port) diff --git a/include/acpi/actbl1.h b/include/acpi/actbl1.h index 8ea7e5d64bc1..82def138a7e4 100644 --- a/include/acpi/actbl1.h +++ b/include/acpi/actbl1.h @@ -419,6 +419,9 @@ struct acpi_cdat_sslbis { u64 entry_base_unit; }; +#define ACPI_CDAT_SSLBIS_US_PORT 0x0100 +#define ACPI_CDAT_SSLBIS_ANY_PORT 0xffff + /* Sub-subtable for above, sslbe_entries field */ struct acpi_cdat_sslbe { From patchwork Mon May 8 20:47:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5C39C7EE22 for ; Mon, 8 May 2023 20:47:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232313AbjEHUrh (ORCPT ); Mon, 8 May 2023 16:47:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229492AbjEHUrh (ORCPT ); Mon, 8 May 2023 16:47:37 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 926145595 for ; Mon, 8 May 2023 13:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578855; x=1715114855; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=x/GVrcQ1czroa7QRamE1h4cfrfbMMGSwGHjJO0O1OlY=; b=P/ll0iGuFTABgFYpchcU3ejm5Fg3gtnGxMLAqL5Hn0FnT5i5J31uAQZF rbxpmHahpMeqpQo9AlBhCh9o3lcgVycZgPMrebjafGtkzm3td7bJkAj8s nz43JG6PNgY+Ay5zP7uGqwwYwEIXricRvlnrr+rT6KFGuCxxEda14/nt1 G9diBvAqv476XJhq5/zZhzJB+0IZ53iGlsd4/dXFvT4/xoChq5pNLdaTg ECGuwi/FQdvMFUOe3Q+Fb8BKP3oU8/khFBNad9vbHaQykssQ62MM8bbCF U+KmwzQMU2etM6kt/GO5xAxpxb5Rjlg0br+fek8UjPUj7wecgRWc/4ovr w==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="338976673" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="338976673" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="731431675" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="731431675" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:18 -0700 Subject: [PATCH v5 04/14] cxl: Add support for _DSM Function for retrieving QTG ID From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:17 -0700 Message-ID: <168357883767.2756219.11563272201242678864.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL spec v3.0 9.17.3 CXL Root Device Specific Methods (_DSM) Add support to retrieve QTG ID via ACPI _DSM call. The _DSM call requires an input of an ACPI package with 4 dwords (read latency, write latency, read bandwidth, write bandwidth). The call returns a package with 1 WORD that provides the max supported QTG ID and a package that may contain 0 or more WORDs as the recommended QTG IDs in the recommended order. Create a cxl_root container for the root cxl_port and provide a callback ->get_qos_class() in order to retrieve the QoS class. For the ACPI case, the _DSM helper is used to retrieve the QTG ID and returned. A devm_cxl_add_root() function is added for root port setup and registration of the cxl_root callback operation(s). Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v5: - Make the helper a callback for the CXL root. (Dan) - Drop the addition of core/acpi.c. (Dan) - Add endiness handling. (Jonathan) - Refactor error exits. (Jonathan) - Update evaluate function description. (Jonathan) - Make uuid static. (Dan) v2: - Reorder var declaration and use C99 style. (Jonathan) - Allow >2 ACPI objects in package for future expansion. (Jonathan) - Check QTG IDs against MAX QTG ID provided by output package. (Jonathan) --- drivers/cxl/acpi.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++- drivers/cxl/core/port.c | 41 +++++++++++++-- drivers/cxl/cxl.h | 33 ++++++++++++ 3 files changed, 192 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index e063df2bf876..f9b35e8fe810 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -17,6 +17,10 @@ struct cxl_cxims_data { u64 xormaps[]; }; +static const guid_t acpi_cxl_qtg_id_guid = + GUID_INIT(0xF365F9A6, 0xA7DE, 0x4071, + 0xA6, 0x6A, 0xB4, 0x0C, 0x0B, 0x4F, 0x8E, 0x52); + /* * Find a targets entry (n) in the host bridge interleave list. * CXL Specification 3.0 Table 9-22 @@ -194,6 +198,120 @@ struct cxl_cfmws_context { int id; }; +/** + * cxl_acpi_evaluate_qtg_dsm - Retrieve QTG ids via ACPI _DSM + * @handle: ACPI handle + * @input: bandwidth and latency data + * + * Issue QTG _DSM with accompanied bandwidth and latency data in order to get + * the QTG IDs that are suitable for the performance point in order of most + * suitable to least suitable. Return first QTG ID. + */ +static int cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, + struct qtg_dsm_input *input) +{ + union acpi_object *out_obj, *out_buf, *pkg; + union acpi_object in_buf = { + .buffer = { + .type = ACPI_TYPE_BUFFER, + .pointer = (u8 *)input, + .length = cpu_to_le32(sizeof(*input)), + }, + }; + union acpi_object in_obj = { + .package = { + .type = ACPI_TYPE_PACKAGE, + .count = cpu_to_le32(1), + .elements = &in_buf + }, + }; + u16 max_qtg; + int rc = 0; + int len; + + out_obj = acpi_evaluate_dsm(handle, &acpi_cxl_qtg_id_guid, 1, 1, &in_obj); + if (!out_obj) + return -ENXIO; + + if (out_obj->type != ACPI_TYPE_PACKAGE) { + rc = -ENXIO; + goto out; + } + + /* Check Max QTG ID */ + pkg = &out_obj->package.elements[0]; + if (pkg->type != ACPI_TYPE_BUFFER) { + rc = -ENXIO; + goto out; + } + + if (le32_to_cpu(pkg->buffer.length) != sizeof(u16)) { + rc = -ENXIO; + goto out; + } + + max_qtg = le16_to_cpu(*(__le16 *)pkg->buffer.pointer); + + /* Retrieve QTG IDs package */ + pkg = &out_obj->package.elements[1]; + if (pkg->type != ACPI_TYPE_PACKAGE) { + rc = -ENXIO; + goto out; + } + + out_buf = &pkg->package.elements[0]; + if (out_buf->type != ACPI_TYPE_BUFFER) { + rc = -ENXIO; + goto out; + } + + len = le32_to_cpu(out_buf->buffer.length); + + /* It's legal to have 0 QTG entries */ + if (len == 0) { + rc = -EEXIST; + goto out; + } + + /* Malformed package, not multiple of WORD size */ + if (len % sizeof(u16)) { + rc = -ENXIO; + goto out; + } + + rc = le16_to_cpu(*(__le16 *)out_buf->buffer.pointer); + if (rc > max_qtg) { + rc = -ERANGE; + pr_warn("QTG ID %u greater than MAX %u\n", rc, max_qtg); + } + +out: + ACPI_FREE(out_obj); + return rc; +} + +static int cxl_acpi_get_qos_class(struct cxl_port *root_port, + struct qtg_dsm_input *input) +{ + acpi_handle handle; + struct device *dev; + + dev = root_port->uport; + + if (!dev_is_platform(dev)) + return -ENODEV; + + handle = ACPI_HANDLE(dev); + if (!handle) + return -ENODEV; + + return cxl_acpi_evaluate_qtg_dsm(handle, input); +} + +static const struct cxl_root_ops acpi_root_ops = { + .get_qos_class = cxl_acpi_get_qos_class, +}; + static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, const unsigned long end) { @@ -631,6 +749,7 @@ static int cxl_acpi_probe(struct platform_device *pdev) { int rc; struct resource *cxl_res; + struct cxl_root *cxl_root; struct cxl_port *root_port; struct device *host = &pdev->dev; struct acpi_device *adev = ACPI_COMPANION(host); @@ -650,9 +769,10 @@ static int cxl_acpi_probe(struct platform_device *pdev) cxl_res->end = -1; cxl_res->flags = IORESOURCE_MEM; - root_port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL); - if (IS_ERR(root_port)) - return PTR_ERR(root_port); + cxl_root = devm_cxl_add_root(host, &acpi_root_ops); + if (IS_ERR(cxl_root)) + return PTR_ERR(cxl_root); + root_port = &cxl_root->port; rc = bus_for_each_dev(adev->dev.bus, NULL, root_port, add_host_bridge_dport); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index a0130aeb8d42..60693c3f85f6 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -526,7 +526,10 @@ static void cxl_port_release(struct device *dev) xa_destroy(&port->dports); xa_destroy(&port->regions); ida_free(&cxl_port_ida, port->id); - kfree(port); + if (is_cxl_root(port)) + kfree(to_cxl_root(port)); + else + kfree(port); } static const struct attribute_group *cxl_port_attribute_groups[] = { @@ -629,13 +632,22 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, resource_size_t component_reg_phys, struct cxl_dport *parent_dport) { + struct cxl_root *cxl_root = NULL; struct cxl_port *port; struct device *dev; int rc; - port = kzalloc(sizeof(*port), GFP_KERNEL); - if (!port) - return ERR_PTR(-ENOMEM); + /* No parent_dport, root cxl_port */ + if (!parent_dport) { + cxl_root = kzalloc(sizeof(*cxl_root), GFP_KERNEL); + if (!cxl_root) + return ERR_PTR(-ENOMEM); + port = &cxl_root->port; + } else { + port = kzalloc(sizeof(*port), GFP_KERNEL); + if (!port) + return ERR_PTR(-ENOMEM); + } rc = ida_alloc(&cxl_port_ida, GFP_KERNEL); if (rc < 0) @@ -693,7 +705,10 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, return port; err: - kfree(port); + if (cxl_root) + kfree(cxl_root); + else + kfree(port); return ERR_PTR(rc); } @@ -779,6 +794,22 @@ struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, } EXPORT_SYMBOL_NS_GPL(devm_cxl_add_port, CXL); +struct cxl_root *devm_cxl_add_root(struct device *host, + const struct cxl_root_ops *ops) +{ + struct cxl_root *cxl_root; + struct cxl_port *port; + + port = devm_cxl_add_port(host, host, CXL_RESOURCE_NONE, NULL); + if (IS_ERR(port)) + return (struct cxl_root *)port; + + cxl_root = to_cxl_root(port); + cxl_root->ops = ops; + return cxl_root; +} +EXPORT_SYMBOL_NS_GPL(devm_cxl_add_root, CXL); + struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port) { /* There is no pci_bus associated with a CXL platform-root port */ diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 3e8020e0a132..16fc14d43aa4 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -586,6 +586,30 @@ struct cxl_port { bool cdat_available; }; +struct qtg_dsm_input; + +struct cxl_root_ops { + int (*get_qos_class)(struct cxl_port *root_port, + struct qtg_dsm_input *input); +}; + +/** + * struct cxl_root - logical collection of root cxl_port items + * + * @port: cxl_port member + * @ops: cxl root operations + */ +struct cxl_root { + struct cxl_port port; + const struct cxl_root_ops *ops; +}; + +static inline struct cxl_root * +to_cxl_root(const struct cxl_port *port) +{ + return container_of(port, struct cxl_root, port); +} + static inline struct cxl_dport * cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) { @@ -665,6 +689,8 @@ struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port); struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys, struct cxl_dport *parent_dport); +struct cxl_root *devm_cxl_add_root(struct device *host, + const struct cxl_root_ops *ops); struct cxl_port *find_cxl_root(struct cxl_port *port); int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd); void cxl_bus_rescan(void); @@ -819,6 +845,13 @@ static inline int cxl_cdat_switch_process(struct cxl_port *port) } #endif +struct qtg_dsm_input { + __le32 rd_lat; + __le32 wr_lat; + __le32 rd_bw; + __le32 wr_bw; +}; + /* * Unit test builds overrides this to __weak, find the 'strong' version * of these symbols in tools/testing/cxl/. From patchwork Mon May 8 20:47:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4E66C77B7F for ; Mon, 8 May 2023 20:47:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233106AbjEHUro (ORCPT ); Mon, 8 May 2023 16:47:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232812AbjEHUrn (ORCPT ); Mon, 8 May 2023 16:47:43 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 236515595 for ; Mon, 8 May 2023 13:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578862; x=1715114862; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rQeFJ6p2RIiXIBSpW5Kp0u605VCmW8d6CRcNKyE8hxQ=; b=WlSgI1BSw3AiwPwdM8vObNhQCRdhPsKwCKPD07t/5xxZWJwhazAx/XUj XtCYfy8ScAZAhQ+HwJBwpW3jmKEZh02DcOy0nZVNXNeKlrIdAcRJwiTPh A10V6oKC4n4Bftb7wKU/r/WHjRmNj/KnkQmeV4QGTZFzMMDmxtFr80sfc hbpnXBHVWSWjnxQM2PbNTyGfamAksOx1oi5cGKQLCwU6qQclHc0O++V5g 0u7HpuHXM8FxVGARksHXbWSYRULUX8bRIjn0ZvbED63nxr2r8Wl2CxpTO TwBvtmsnadki4j94lr1ZA4ODyELju7fSC2vzmcwvx5vHKzshKrcu9Yvnb w==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="338976780" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="338976780" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="731431683" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="731431683" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:24 -0700 Subject: [PATCH v5 05/14] cxl: Calculate and store PCI link latency for the downstream ports From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Jonathan Cameron , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:23 -0700 Message-ID: <168357884387.2756219.5365448705913389325.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org The latency is calculated by dividing the flit size over the bandwidth. Add support to retrieve the flit size for the CXL switch device and calculate the latency of the PCIe link. Cache the latency number with cxl_dport. Reviewed-by: Jonathan Cameron Signed-off-by: Dave Jiang --- v5: - Store the latency numbers under dports. (Dan) - Use defines instead of magic numbers. (Jonathan) v2: - Fix commit log issues. (Jonathan) - Fix var declaration issues. (Jonathan) --- drivers/cxl/core/pci.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/port.c | 6 ++++ drivers/cxl/cxl.h | 4 +++ drivers/cxl/cxlpci.h | 15 ++++++++++ drivers/cxl/pci.c | 13 -------- 5 files changed, 98 insertions(+), 13 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 58051154ab1a..6c99a964eb54 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include #include #include #include @@ -718,3 +719,75 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, return PCI_ERS_RESULT_NEED_RESET; } EXPORT_SYMBOL_NS_GPL(cxl_error_detected, CXL); + +extern const unsigned char pcie_link_speed[]; + +static enum pci_bus_speed get_link_speed(struct pci_dev *pdev) +{ + u16 linkstat; + int err; + + err = pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &linkstat); + if (err) + return -EINVAL; + + return pcie_link_speed[linkstat & PCI_EXP_LNKSTA_CLS]; +} + +static int pci_bus_speed_to_mbps(enum pci_bus_speed speed) +{ + switch (speed) { + case PCIE_SPEED_2_5GT: + return 2500; + case PCIE_SPEED_5_0GT: + return 5000; + case PCIE_SPEED_8_0GT: + return 8000; + case PCIE_SPEED_16_0GT: + return 16000; + case PCIE_SPEED_32_0GT: + return 32000; + case PCIE_SPEED_64_0GT: + return 64000; + default: + break; + } + + return -EINVAL; +} + +static int cxl_flit_size(struct pci_dev *pdev) +{ + if (cxl_pci_flit_256(pdev)) + return 256; + + return 68; +} + +/** + * cxl_pci_get_latency - calculate the link latency for the PCIe link + * @pdev - PCI device + * + * return: calculated latency or 0 for no latency + * + * CXL Memory Device SW Guide v1.0 2.11.4 Link latency calculation + * Link latency = LinkPropagationLatency + FlitLatency + RetimerLatency + * LinkProgationLatency is negligible, so 0 will be used + * RetimerLatency is assumed to be negligible and 0 will be used + * FlitLatency = FlitSize / LinkBandwidth + * FlitSize is defined by spec. CXL rev3.0 4.2.1. + * 68B flit is used up to 32GT/s. >32GT/s, 256B flit size is used. + * The FlitLatency is converted to picoseconds. + */ +long cxl_pci_get_latency(struct pci_dev *pdev) +{ + long bw; + + bw = pci_bus_speed_to_mbps(get_link_speed(pdev)); + if (bw < 0) + return 0; + bw /= BITS_PER_BYTE; + + return cxl_flit_size(pdev) * MEGA / bw; +} +EXPORT_SYMBOL_NS_GPL(cxl_pci_get_latency, CXL); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 60693c3f85f6..e4f75847b851 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -751,6 +751,9 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host, if (rc) return ERR_PTR(rc); + if (parent_dport && dev_is_pci(uport)) + port->pci_latency = cxl_pci_get_latency(to_pci_dev(uport)); + return port; err: @@ -1008,6 +1011,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev, if (rc) return ERR_PTR(rc); + if (dev_is_pci(dport_dev)) + dport->link_latency = cxl_pci_get_latency(to_pci_dev(dport_dev)); + return dport; } diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 16fc14d43aa4..c1e2c3703a63 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -562,6 +562,7 @@ struct cxl_dax_region { * @depth: How deep this port is relative to the root. depth 0 is the root. * @cdat: Cached CDAT data * @cdat_available: Should a CDAT attribute be available in sysfs + * @pci_latency: Upstream latency in picoseconds */ struct cxl_port { struct device dev; @@ -584,6 +585,7 @@ struct cxl_port { size_t length; } cdat; bool cdat_available; + long pci_latency; }; struct qtg_dsm_input; @@ -625,6 +627,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port * @coord: access coordinates (performance) for switch from CDAT + * @link_latency: calculated PCIe downstream latency */ struct cxl_dport { struct device *dport; @@ -634,6 +637,7 @@ struct cxl_dport { bool rch; struct cxl_port *port; struct access_coordinate coord; + long link_latency; }; /** diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 0465ef963cd6..84c9f73c7d92 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -82,6 +82,19 @@ struct cdat_entry_header { __le16 length; } __packed; +/* + * CXL v3.0 6.2.3 Table 6-4 + * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits + * mode, otherwise it's 68B flits mode. + */ +static inline bool cxl_pci_flit_256(struct pci_dev *pdev) +{ + u16 lnksta2; + + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); + return lnksta2 & PCI_EXP_LNKSTA2_FLIT; +} + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -90,4 +103,6 @@ void read_cdat_data(struct cxl_port *port); void cxl_cor_error_detected(struct pci_dev *pdev); pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, pci_channel_state_t state); +long cxl_pci_get_latency(struct pci_dev *pdev); + #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index ea38bd49b0cf..ed39d133b70d 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -365,19 +365,6 @@ static bool is_cxl_restricted(struct pci_dev *pdev) return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END; } -/* - * CXL v3.0 6.2.3 Table 6-4 - * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits - * mode, otherwise it's 68B flits mode. - */ -static bool cxl_pci_flit_256(struct pci_dev *pdev) -{ - u16 lnksta2; - - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); - return lnksta2 & PCI_EXP_LNKSTA2_FLIT; -} - static int cxl_pci_ras_unmask(struct pci_dev *pdev) { struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus); From patchwork Mon May 8 20:47:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29D54C7EE26 for ; Mon, 8 May 2023 20:47:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233120AbjEHUrs (ORCPT ); Mon, 8 May 2023 16:47:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbjEHUrr (ORCPT ); Mon, 8 May 2023 16:47:47 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C155859D3 for ; Mon, 8 May 2023 13:47:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578865; x=1715114865; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MxDcpaop7BEJXbAJJ/mIUTokUJIg6NVkc2lo++oUdkM=; b=ONLCQ3pwzEkTRC9ci+aXKMas8gsbVlHLqhJxoQKUf8STgNRPFV2mtaQ1 yK/E89/gENG+5CqnqkrA2nsdzHqf2B3XWfoNS9kOHM/Iwh+dvueH+iVbT KAKGaV12ZmQXBhj7mr1IhNUlV9o1K8OUhhPDfNbMxtEB9j7soW/RhaFGH uLJ4Y2xE1S/eAETPRwjLsgOtzi8mJR56EoF4QmKx7XnYJNQG6xvfKOSdO PEJhL9IxhFKjddWWXF5Peqcd+XdGfFQoMU7KrsnfuMWpqp+fFAUDBh9GU XA6xq9ZokrRp6H4gLnSfvFj6sisjL4sG4Ao7+B/m2XV6KY7SLeQ5gBfyH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="338976835" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="338976835" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="731431702" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="731431702" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:30 -0700 Subject: [PATCH v5 06/14] cxl: Store the access coordinates for the generic ports From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:29 -0700 Message-ID: <168357884996.2756219.11067819935569345137.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Each CXL host bridge is represented by an ACPI0016 device. A generic port device handle that is an ACPI device is represented by a string of ACPI0016 device HID and UID. Create a device handle from the ACPI device and retrieve the access coordinates from the stored memory targets. The access coordinates are stored under the cxl_dport that is associated with the CXL host bridge. Signed-off-by: Dave Jiang --- drivers/cxl/acpi.c | 28 ++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 30 insertions(+) diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index f9b35e8fe810..675a4f423f4b 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -537,8 +537,26 @@ static int cxl_get_chbcr(union acpi_subtable_headers *header, void *arg, return 0; } +static int get_genport_coordinates(struct device *dev, struct cxl_dport *dport) +{ + struct acpi_device *hb = to_cxl_host_bridge(NULL, dev); + u8 handle[ACPI_SRAT_DEVICE_HANDLE_SIZE] = { 0 }; + int rc; + + /* ACPI spec 6.5 tABLE 5.65 */ + memcpy(handle, acpi_device_hid(hb), 8); + memcpy(&handle[8], acpi_device_uid(hb), 4); + + rc = acpi_get_genport_coordinates(handle, dport->genport_coord); + if (rc) + return rc; + + return 0; +} + static int add_host_bridge_dport(struct device *match, void *arg) { + int ret; acpi_status rc; struct device *bridge; unsigned long long uid; @@ -594,6 +612,16 @@ static int add_host_bridge_dport(struct device *match, void *arg) if (IS_ERR(dport)) return PTR_ERR(dport); + dport->genport_coord = devm_kzalloc(dport->dport, + sizeof(*dport->genport_coord), + GFP_KERNEL); + if (!dport->genport_coord) + return -ENOMEM; + + ret = get_genport_coordinates(match, dport); + if (ret) + dev_dbg(match, "Failed to get generic port perf coordinates.\n"); + return 0; } diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index c1e2c3703a63..033b822a20f2 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -626,6 +626,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev) * @rcrb: base address for the Root Complex Register Block * @rch: Indicate whether this dport was enumerated in RCH or VH mode * @port: reference to cxl_port that contains this downstream port + * @genport_coord: access coordinates (performance) from ACPI generic port * @coord: access coordinates (performance) for switch from CDAT * @link_latency: calculated PCIe downstream latency */ @@ -636,6 +637,7 @@ struct cxl_dport { resource_size_t rcrb; bool rch; struct cxl_port *port; + struct access_coordinate *genport_coord; struct access_coordinate coord; long link_latency; }; From patchwork Mon May 8 20:47:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8C68C77B7F for ; Mon, 8 May 2023 20:47:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232643AbjEHUrj (ORCPT ); Mon, 8 May 2023 16:47:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229492AbjEHUri (ORCPT ); Mon, 8 May 2023 16:47:38 -0400 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90F9655A9 for ; Mon, 8 May 2023 13:47:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578857; x=1715114857; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6X1f545mj1OVHPb/RNhze5zXFRbLTLVYEFvGxjtabeo=; b=ne28Im5RD5oo/iw4Kn7FUHxWV+4HbEZj7zBB4104jc6JLMWuI6mTA2Rq OJsr3fIcM/r6UrnWcarDp/q3w/heQ82ZsRwKmrSuAk5UuWdsTk0AQoIAU zBK33GqBNN+6iMcpIH3Jd7BtEWI4KGwm27hkhcMeeaVdz5VAZR/jhr+m2 REmFqLpacSjMCAtcFymjcTM0kNgVaDvqaWGYX759w5qMOWYM6wVSrkp/E aNpFYEsARJapRvtLK2IF1ZVbLT3He25Y2xtQT2f4UbYbMMef/7zDx7ItV Z7mA0SAI7gRzZnhen3X9Rzbt1s52oj4bSVeWQWnt+7SfhQnTEfWsL3Ml9 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="348580605" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="348580605" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="822826906" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="822826906" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:36 -0700 Subject: [PATCH v5 07/14] cxl: Add helper function that calculate performance data for downstream ports From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:36 -0700 Message-ID: <168357885604.2756219.14660517574518073528.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org The CDAT information from the switch, Switch Scoped Latency and Bandwidth Information Strucutre (SSLBIS), is parsed and stored under a cxl_dport based on the correlated downstream port id from the SSLBIS entry. Walk the entire CXL port paths and collect all the performance data. Also pick up the link latency number that's stored under the dports. The entire path PCIe bandwidth can be retrieved using the pcie_bandwidth_available() call. Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- drivers/cxl/core/port.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 3 ++ 2 files changed, 72 insertions(+) diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index e4f75847b851..1111a3cebc8e 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -1950,6 +1951,74 @@ bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd) } EXPORT_SYMBOL_NS_GPL(schedule_cxl_memdev_detach, CXL); +static void combine_coordinates(struct access_coordinate *c1, + struct access_coordinate *c2) +{ + if (c2->write_bandwidth) + c1->write_bandwidth = min_t(unsigned int, + c1->write_bandwidth, + c2->write_bandwidth); + c1->write_latency += c2->write_latency; + + if (c2->read_bandwidth) + c1->read_bandwidth = min_t(unsigned int, + c1->read_bandwidth, + c2->read_bandwidth); + c1->read_latency += c2->read_latency; +} + +/** + * cxl_port_get_perf_coordinates - Retrieve performance numbers stored in dports + * of CXL path + * @port: endpoint cxl_port + * @coord: output performance data + * + * Return: errno on failure, 0 on success. + */ +int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, + struct access_coordinate *coord) +{ + struct access_coordinate c = { + .read_bandwidth = UINT_MAX, + .write_bandwidth = UINT_MAX, + }; + struct cxl_port *iter = port; + struct cxl_dport *dport; + struct pci_dev *pdev; + unsigned int bw; + + if (!is_cxl_endpoint(port)) + return -EINVAL; + + dport = iter->parent_dport; + while (iter && !is_cxl_root(iter)) { + combine_coordinates(&c, &dport->coord); + c.write_latency += dport->link_latency; + c.read_latency += dport->link_latency; + + if (dport->genport_coord) + combine_coordinates(&c, dport->genport_coord); + + iter = to_cxl_port(iter->dev.parent); + dport = iter->parent_dport; + } + + /* Get the calculated PCI paths bandwidth */ + pdev = to_pci_dev(port->uport->parent); + bw = pcie_bandwidth_available(pdev, NULL, NULL, NULL); + if (bw == 0) + return -ENXIO; + bw /= BITS_PER_BYTE; + + c.write_bandwidth = min_t(unsigned int, c.write_bandwidth, bw); + c.read_bandwidth = min_t(unsigned int, c.read_bandwidth, bw); + + *coord = c; + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_endpoint_get_perf_coordinates, CXL); + /* for user tooling to ensure port disable work has completed */ static ssize_t flush_store(struct bus_type *bus, const char *buf, size_t count) { diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 033b822a20f2..0c8952e568cc 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -858,6 +858,9 @@ struct qtg_dsm_input { __le32 wr_bw; }; +int cxl_endpoint_get_perf_coordinates(struct cxl_port *port, + struct access_coordinate *coord); + /* * Unit test builds overrides this to __weak, find the 'strong' version * of these symbols in tools/testing/cxl/. From patchwork Mon May 8 20:47:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93B42C7EE22 for ; Mon, 8 May 2023 20:47:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232812AbjEHUrp (ORCPT ); Mon, 8 May 2023 16:47:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229492AbjEHUro (ORCPT ); Mon, 8 May 2023 16:47:44 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5623C59D3 for ; Mon, 8 May 2023 13:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578863; x=1715114863; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j2KTMOdcq34IFPybLPHZeWcZBlG57IRLduw7Te0ynVI=; b=VSUpj6sAbXv0uipahfuhB1ouNAgZ0Ji8SUqaPLsfy+Svh7d/vJI0fbac RfNYU63AWcMlDu2lwp/fjsKHEzsnmtBpxRPxxKBM45apcOX0LkYjqdtdB 44mr4/BmEEbFJl660qrF2OikM3LQVbuveBX1mNKwE9l8jAP+qPsgcdDoY smRkNlUb9LmQsi26VSZabMH6dXtlExcFxCdDn5eSJC0ds+cv+VSosvd+f aCSQechuR/pl8PDasXUV6Vs/QziQh/x7Dz8kNEz+IRHHkydXZKGkw7Biq FPbSypaHwQEwfGrA/ZslybnjIYSu7ldM+n8RjBo+DaDubpsiJ5W9KKpZl g==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="436072409" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="436072409" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="944996493" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="944996493" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:42 -0700 Subject: [PATCH v5 08/14] cxl: Compute the entire CXL path latency and bandwidth data From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:42 -0700 Message-ID: <168357886214.2756219.978696616095404572.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL Memory Device SW Guide rev1.0 2.11.2 provides instruction on how to caluclate latency and bandwidth for CXL memory device. Calculate minimum bandwidth and total latency for the path from the CXL device to the root port. The QTG id is retrieved by providing the performance data as input and calling the root port callback ->get_qos_class(). The retrieved id is stored with the cxl_port of the CXL device. For example for a device that is directly attached to a host bus: Total Latency = Device Latency (from CDAT) + Dev to Host Bus (HB) Link Latency + Generic Port Latency Min Bandwidth = Min bandwidth for link bandwidth between HB and CXL device, device CDAT bandwidth, and Generic Port Bandwidth For a device that has a switch in between host bus and CXL device: Total Latency = Device (CDAT) Latency + Dev to Switch Link Latency + Switch (CDAT) Latency + Switch to HB Link Latency + Generic Port Latency Min Bandwidth = Min bandwidth for link bandwidth between CXL device to CXL switch, CXL device CDAT bandwidth, CXL switch CDAT bandwidth, CXL switch to HB bandwidth, and Generic Port Bandwidth. Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v5: - Use new API call cxl_endpoint_get_perf_coordinates(). - Use root_port->get_qos_class() (Dan) - Add endieness handling to DSM input. --- drivers/cxl/cxl.h | 1 + drivers/cxl/port.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 54 insertions(+), 3 deletions(-) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 0c8952e568cc..7c94f07771c1 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -833,6 +833,7 @@ struct dsmas_entry { struct range dpa_range; u8 handle; struct access_coordinate coord; + int qos_class; }; #ifdef CONFIG_ACPI diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index c5a24b75bf03..b474997cc7ee 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -67,6 +67,52 @@ static void dsmas_list_destroy(struct list_head *dsmas_list) } } +static int cxl_port_perf_data_calculate(struct cxl_port *port, + struct list_head *dsmas_list) +{ + struct access_coordinate c; + struct qtg_dsm_input input; + struct cxl_port *root_port; + struct cxl_root *cxl_root; + struct dsmas_entry *dent; + int rc, qos_class; + + rc = cxl_endpoint_get_perf_coordinates(port, &c); + if (rc) { + dev_dbg(&port->dev, "Failed to retrieve perf coordinates.\n"); + return rc; + } + + root_port = find_cxl_root(port); + cxl_root = to_cxl_root(root_port); + if (!cxl_root->ops && !cxl_root->ops->get_qos_class) + return -EOPNOTSUPP; + + list_for_each_entry(dent, dsmas_list, list) { + dent->coord.read_latency = dent->coord.read_latency + + c.read_latency; + dent->coord.write_latency = dent->coord.write_latency + + c.write_latency; + dent->coord.read_bandwidth = min_t(int, c.read_bandwidth, + dent->coord.read_bandwidth); + dent->coord.write_bandwidth = min_t(int, c.write_bandwidth, + dent->coord.write_bandwidth); + + input.rd_lat = cpu_to_le32(dent->coord.read_latency); + input.wr_lat = cpu_to_le32(dent->coord.write_latency); + input.rd_bw = cpu_to_le32(dent->coord.read_bandwidth); + input.wr_bw = cpu_to_le32(dent->coord.write_bandwidth); + + qos_class = cxl_root->ops->get_qos_class(root_port, &input); + if (qos_class < 0) + continue; + + dent->qos_class = qos_class; + } + + return 0; +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -155,10 +201,14 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) LIST_HEAD(dsmas_list); rc = cxl_cdat_endpoint_process(port, &dsmas_list); - if (rc < 0) + if (rc < 0) { dev_dbg(&port->dev, "Failed to parse CDAT: %d\n", rc); - - /* Performance data processing */ + } else { + rc = cxl_port_perf_data_calculate(port, &dsmas_list); + if (rc) + dev_dbg(&port->dev, + "Failed to do perf coord calculations.\n"); + } dsmas_list_destroy(&dsmas_list); } From patchwork Mon May 8 20:47:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E432C7EE22 for ; Mon, 8 May 2023 20:47:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230027AbjEHUru (ORCPT ); Mon, 8 May 2023 16:47:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233333AbjEHUrt (ORCPT ); Mon, 8 May 2023 16:47:49 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D45C05595 for ; Mon, 8 May 2023 13:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578868; x=1715114868; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l4vAICEVugnj0cTgHH6YR4VLxq7drchAv/8YeNLDUcU=; b=QZcvfOxNsceYPMPYyj3HI/2hEIKotSVUXMzmPE7rLl5w89ewqnJA5FYZ aDy2FiTe3gGuFHAKifYpjTXmSSwaVnGFKZKDcSvHe+J99QX2jTXc/afPj 1haEIRV66VubzomYB2+U9qovlcehHC4NWc4cYlX2T8dzfDEzqwtkQWv9o F9zLhPUk/ruOptgNok+i5X41HEyKSzOclLNtpO7sShZRePsHVrzMxVyoL HZkh3sEyf2t2xFT2TEpmLywAnw8bmkn9GwUhQcGDhESATHM+LAh2UxG1j QDPSVSh88aDLtrbYmrhRAUtnrg+T8xTPWpvIoPE7fx7t/8XjvMtF5IZs5 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="436072417" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="436072417" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="944996580" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="944996580" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:48 -0700 Subject: [PATCH v5 09/14] cxl: Wait Memory_Info_Valid before access memory related info From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:47 -0700 Message-ID: <168357886796.2756219.4806167633587850772.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CXL rev3.0 8.1.3.8.2 Memory_Info_valid field The Memory_Info_Valid bit indicates that the CXL Range Size High and Size Low registers are valid. The bit must be set within 1 second of reset deassertion to the device. Check valid bit before we check the Memory_Active bit when waiting for cxl_await_media_ready() to ensure that the memory info is valid for consumption. Fixes: 2e4ba0ec9783 ("cxl/pci: Move cxl_await_media_ready() to the core") Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v2: - Check both ranges. (Jonathan) --- drivers/cxl/core/pci.c | 83 +++++++++++++++++++++++++++++++++++++++++++----- drivers/cxl/cxlpci.h | 2 + 2 files changed, 77 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 6c99a964eb54..536672d469a1 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -102,21 +102,55 @@ int devm_cxl_port_enumerate_dports(struct cxl_port *port) } EXPORT_SYMBOL_NS_GPL(devm_cxl_port_enumerate_dports, CXL); -/* - * Wait up to @media_ready_timeout for the device to report memory - * active. - */ -int cxl_await_media_ready(struct cxl_dev_state *cxlds) +static int cxl_dvsec_mem_range_valid(struct cxl_dev_state *cxlds, int id) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + int d = cxlds->cxl_dvsec; + bool valid = false; + int rc, i; + u32 temp; + + if (id > CXL_DVSEC_RANGE_MAX) + return -EINVAL; + + /* Check MEM INFO VALID bit first, give up after 1s */ + i = 1; + do { + rc = pci_read_config_dword(pdev, + d + CXL_DVSEC_RANGE_SIZE_LOW(id), + &temp); + if (rc) + return rc; + + valid = FIELD_GET(CXL_DVSEC_MEM_INFO_VALID, temp); + if (valid) + break; + msleep(1000); + } while (i--); + + if (!valid) { + dev_err(&pdev->dev, + "Timeout awaiting memory range %d valid after 1s.\n", + id); + return -ETIMEDOUT; + } + + return 0; +} + +static int cxl_dvsec_mem_range_active(struct cxl_dev_state *cxlds, int id) { struct pci_dev *pdev = to_pci_dev(cxlds->dev); int d = cxlds->cxl_dvsec; bool active = false; - u64 md_status; int rc, i; + u32 temp; - for (i = media_ready_timeout; i; i--) { - u32 temp; + if (id > CXL_DVSEC_RANGE_MAX) + return -EINVAL; + /* Check MEM ACTIVE bit, up to 60s timeout by default */ + for (i = media_ready_timeout; i; i--) { rc = pci_read_config_dword( pdev, d + CXL_DVSEC_RANGE_SIZE_LOW(0), &temp); if (rc) @@ -135,6 +169,39 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds) return -ETIMEDOUT; } + return 0; +} + +/* + * Wait up to @media_ready_timeout for the device to report memory + * active. + */ +int cxl_await_media_ready(struct cxl_dev_state *cxlds) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + int d = cxlds->cxl_dvsec; + int rc, i, hdm_count; + u64 md_status; + u16 cap; + + rc = pci_read_config_word(pdev, + d + CXL_DVSEC_CAP_OFFSET, &cap); + if (rc) + return rc; + + hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap); + for (i = 0; i < hdm_count; i++) { + rc = cxl_dvsec_mem_range_valid(cxlds, i); + if (rc) + return rc; + } + + for (i = 0; i < hdm_count; i++) { + rc = cxl_dvsec_mem_range_active(cxlds, i); + if (rc) + return rc; + } + md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET); if (!CXLMDEV_READY(md_status)) return -EIO; diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 84c9f73c7d92..1772cd226108 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -31,6 +31,8 @@ #define CXL_DVSEC_RANGE_BASE_LOW(i) (0x24 + (i * 0x10)) #define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) +#define CXL_DVSEC_RANGE_MAX 2 + /* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ #define CXL_DVSEC_FUNCTION_MAP 2 From patchwork Mon May 8 20:47:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E04DC77B7F for ; Mon, 8 May 2023 20:47:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232167AbjEHUr4 (ORCPT ); Mon, 8 May 2023 16:47:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbjEHUrz (ORCPT ); Mon, 8 May 2023 16:47:55 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 956D359CA for ; Mon, 8 May 2023 13:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578874; x=1715114874; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VQ+b8UCBNC8s+QuBVZHNtxr6Zzk35sxJV9Sy2icjV4c=; b=jxfpWQY0vrKqFwGnWDplpcoZfrfpAh2wONcFuvB22H7tqVor3JUp4t3P gUXiU5rhMzq6ZCNQZRcC2sZnfn9pydaR5fhMGt7bJWc3Aej2QWwqueEoK E7SUNYhiPHydHUiZY36fNwROHShq4K71jzmvmVlGKx85NcZeLI1IKPnjH C+Jja/lYTPyUgX7VAtbf5B7BhXIyi9kIWoI5fVrimwxmWChOmzhw4Tl2V keqmpGGk4zQAxGDrWjUm9alSY+KKO6VG+qq7cGT/1QeGnZaDg3qBVQVo8 4yDb2WamGdQzBvC9Xh5JE/Kbs4XBHLUt6Upl6ZTmkDjO/I0j480G7CvSW w==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="436072433" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="436072433" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="944996670" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="944996670" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:54 -0700 Subject: [PATCH v5 10/14] cxl: Move identify and partition query from pci probe to port probe From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:53 -0700 Message-ID: <168357887373.2756219.10680050253858860934.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Move the enumeration of device capacity to cxl_port_probe() from cxl_pci_probe(). The size and capacity information should be read after cxl_await_media_ready() so the data is valid. Fixes: 5e2411ae8071 ("cxl/memdev: Change cxl_mem to a more descriptive name") Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- drivers/cxl/pci.c | 8 -------- drivers/cxl/port.c | 8 ++++++++ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index ed39d133b70d..06324266eae8 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -707,14 +707,6 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; - rc = cxl_dev_state_identify(cxlds); - if (rc) - return rc; - - rc = cxl_mem_create_range_info(cxlds); - if (rc) - return rc; - rc = cxl_alloc_irq_vectors(pdev); if (rc) return rc; diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index b474997cc7ee..0bdb9d73a389 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -180,6 +180,14 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) return rc; } + rc = cxl_dev_state_identify(cxlds); + if (rc) + return rc; + + rc = cxl_mem_create_range_info(cxlds); + if (rc) + return rc; + rc = devm_cxl_enumerate_decoders(cxlhdm, &info); if (rc) return rc; From patchwork Mon May 8 20:47:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45102C77B7F for ; Mon, 8 May 2023 20:48:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233333AbjEHUsC (ORCPT ); Mon, 8 May 2023 16:48:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233361AbjEHUsB (ORCPT ); Mon, 8 May 2023 16:48:01 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57D82558B for ; Mon, 8 May 2023 13:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578880; x=1715114880; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V/wYL8+fOhlUEF9f0RZ8WOxU5SNImHABMY796EVKtuI=; b=m0b67sZI+8m3KuHZYjsLwCGlvwolMIwjUUrDl/Yksb28BWGI9j4bj6XN gWG6/dHqGf4KmdrIIq+O+9JwovMaSI9vFGVr/ZkGRwfoy0tdSBzbgJ5Bv q/L3MOKnsa0gjSzPmbmjgYjNEx+4KPtZTzHkige1C4VfjAVQmcIzJO2ZU ZKZ7VW0krqoyHd0UPOOnFiCqCmgS24x5BVeiRbYsTMMN4wnAl2DD8dAFf b3LgqOq5BMhrCR0/SksP6RUOJ6HZAdWHD/BMZVU39s1NBFW95B6NLZAPq FyNTXKcAxPRPPb2UeN1oCwyBX36bvICP7Y1R1bxka9PUqOr7bPqc2FVVe w==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="436072443" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="436072443" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="944996760" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="944996760" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:47:59 -0700 Subject: [PATCH v5 11/14] cxl: Move read_cdat_data() to after media is ready From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:47:59 -0700 Message-ID: <168357887947.2756219.6022306227580486103.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org CDAT data is only valid after the media is ready. Move read_cdat_data() to after cxl_awai_media_read() is successful. Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- drivers/cxl/port.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 0bdb9d73a389..1d55c460e1ab 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -162,9 +162,6 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) if (IS_ERR(cxlhdm)) return PTR_ERR(cxlhdm); - /* Cache the data early to ensure is_visible() works */ - read_cdat_data(port); - get_device(&cxlmd->dev); rc = devm_add_action_or_reset(&port->dev, schedule_detach, cxlmd); if (rc) @@ -180,6 +177,9 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) return rc; } + /* Cache the data early to ensure is_visible() works */ + read_cdat_data(port); + rc = cxl_dev_state_identify(cxlds); if (rc) return rc; From patchwork Mon May 8 20:48:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A5FCC7EE22 for ; Mon, 8 May 2023 20:48:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233361AbjEHUsH (ORCPT ); Mon, 8 May 2023 16:48:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbjEHUsH (ORCPT ); Mon, 8 May 2023 16:48:07 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E102558B for ; Mon, 8 May 2023 13:48:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578886; x=1715114886; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l+H8Xn5/n4gPpCIp2jHgUiu2YY0BIMBGrDMySvrhMbY=; b=cjyUNvg6cwmsPpNlETzCllGLkg5yK7iqkWTnwJPB8iNB8uCqJZ8G6u4a sqTiTzLz3rPRxNNVTuZSD0NTEMAsbGD/PqTm9B0Iv8dFNJoYKwo4rvfQz AyCU1777Y1VxKutgXxibChu9Cb2YMMcRy2gOjNVuEtwm/6vuLR5dwAHku J57QPwlwr8NVKmU8Teq3px+TB+T1l45TJ94HKERTCWkhWbJTdPOfsvWXn TBm5LH9z5YqCCbctC+W09VKJQlG66rGqLtf0wdNOjchq/7RCdvja6SMQP 6J4zUK+ZqbSR63gDkYQj7+WO1yeKsmtL3BO5bv0ECQGWF4UPn8mLgXyDd A==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="436072452" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="436072452" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="944996818" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="944996818" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:05 -0700 Subject: [PATCH v5 12/14] cxl: Store QTG IDs and related info to the CXL memory device context From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:48:05 -0700 Message-ID: <168357888523.2756219.11912716889990252182.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Once the QTG ID _DSM is executed successfully, the QTG ID is retrieved from the return package. Create a list of entries in the cxl_memdev context and store the QTG ID and the associated DPA range. This information can be exposed to user space via sysfs in order to help region setup for hot-plugged CXL memory devices. Signed-off-by: Dave Jiang --- v4: - Remove unused qos_list from cxl_md v3: - Move back to QTG ID per partition --- drivers/cxl/core/mbox.c | 3 +++ drivers/cxl/cxlmem.h | 21 +++++++++++++++++++++ drivers/cxl/port.c | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 60 insertions(+) diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index f2addb457172..9c363060e5c1 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1120,6 +1120,9 @@ struct cxl_dev_state *cxl_dev_state_create(struct device *dev) mutex_init(&cxlds->mbox_mutex); mutex_init(&cxlds->event.log_lock); cxlds->dev = dev; + INIT_LIST_HEAD(&cxlds->perf_list); + cxlds->ram_qos_class = CXL_QOS_CLASS_INVALID; + cxlds->pmem_qos_class = CXL_QOS_CLASS_INVALID; return cxlds; } diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 001dabf0231b..9d77b7e420ce 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -5,6 +5,7 @@ #include #include #include +#include #include "cxl.h" /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ @@ -215,6 +216,19 @@ struct cxl_event_state { struct mutex log_lock; }; +/** + * struct perf_prop - performance property entry + * @list - list entry + * @dpa_range - range for DPA address + * @qos_class - QoS Class cookie + */ +struct perf_prop_entry { + struct list_head list; + struct range dpa_range; + u16 qos_class; + struct access_coordinate coord; +}; + /** * struct cxl_dev_state - The driver device state * @@ -251,6 +265,9 @@ struct cxl_event_state { * @serial: PCIe Device Serial Number * @event: event log driver state * @mbox_send: @dev specific transport for transmitting mailbox commands + * @ram_qos_class: QTG ID for volatile region + * @pmem_qos_class: QTG ID for persistent region + * @perf_list: performance data entries list * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for * details on capacity parameters. @@ -283,6 +300,10 @@ struct cxl_dev_state { u64 next_volatile_bytes; u64 next_persistent_bytes; + int ram_qos_class; + int pmem_qos_class; + struct list_head perf_list; + resource_size_t component_reg_phys; u64 serial; diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 1d55c460e1ab..c8c37dd79ecc 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -113,6 +113,40 @@ static int cxl_port_perf_data_calculate(struct cxl_port *port, return 0; } +static void cxl_memdev_set_qtg(struct cxl_dev_state *cxlds, struct list_head *dsmas_list) +{ + struct range pmem_range = { + .start = cxlds->pmem_res.start, + .end = cxlds->pmem_res.end, + }; + struct range ram_range = { + .start = cxlds->ram_res.start, + .end = cxlds->ram_res.end, + }; + struct perf_prop_entry *perf; + struct dsmas_entry *dent; + + list_for_each_entry(dent, dsmas_list, list) { + perf = devm_kzalloc(cxlds->dev, sizeof(*perf), GFP_KERNEL); + if (!perf) + return; + + perf->dpa_range = dent->dpa_range; + perf->qos_class = dent->qos_class; + perf->coord = dent->coord; + list_add_tail(&perf->list, &cxlds->perf_list); + + if (resource_size(&cxlds->ram_res) && + range_contains(&ram_range, &dent->dpa_range) && + cxlds->ram_qos_class == CXL_QOS_CLASS_INVALID) + cxlds->ram_qos_class = dent->qos_class; + else if (resource_size(&cxlds->pmem_res) && + range_contains(&pmem_range, &dent->dpa_range) && + cxlds->pmem_qos_class == CXL_QOS_CLASS_INVALID) + cxlds->pmem_qos_class = dent->qos_class; + } +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -216,6 +250,8 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) if (rc) dev_dbg(&port->dev, "Failed to do perf coord calculations.\n"); + else + cxl_memdev_set_qtg(cxlds, &dsmas_list); } dsmas_list_destroy(&dsmas_list); From patchwork Mon May 8 20:48:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44785C7EE22 for ; Mon, 8 May 2023 20:48:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232084AbjEHUsN (ORCPT ); Mon, 8 May 2023 16:48:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229452AbjEHUsM (ORCPT ); Mon, 8 May 2023 16:48:12 -0400 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D62A659FF for ; Mon, 8 May 2023 13:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578891; x=1715114891; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Iz8HaHenMHkAaNw5LvBDDwYjz+eqlAY8w/hDmaiVtC0=; b=F52wKDQRHZ4z7W9b8TqW6/jwy5TeuIZfb/imtutaqP7wPlTkSyd1Jjoj ByTOjlzSg1ooio4RMXoKKjNgat3MNl4BmthuZlMc6bg54oprV6Ip9dOkD /ONBCV4rNyxF0j+1Yc9hipQ1vuEqtV5NcpoT40cBXre3Uc21tFPd4toHJ 4f5zZTjsD3lM1Y+3PINTIBTT1UjLMrxsEjURGYLLfMwLbzBtUvQHSJyuZ 7NOGqrqv1T5KxK31m4R0SF9dBd0ogy6KX5sCk/fAs4TyS1EjR+7M8NKKm 5qnkoPeMedTElETAAgOgEj7EL7IotwI9Ebm0vnRLNDwZWDE42Z+QNq0TM Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="436072464" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="436072464" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="944996826" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="944996826" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:11 -0700 Subject: [PATCH v5 13/14] cxl: Export sysfs attributes for memory device QoS class From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Dan Williams , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:48:10 -0700 Message-ID: <168357889098.2756219.3746540473463747491.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Export qos_class sysfs attributes for the CXL memory device. The QoS clas should show up as /sys/bus/cxl/devices/memX/ram/qos_class for the volatile partition and /sys/bus/cxl/devices/memX/pmem/qos_class for the persistent partition. The QTG ID is retrieved via _DSM after supplying the calculated bandwidth and latency for the entire CXL path from device to the CPU. This ID is used to match up to the root decoder QoS class to determine which CFMWS the memory range of a hotplugged CXL mem device should be assigned under. While there may be multiple DSMAS exported by the device CDAT, the driver will only expose the first QTG ID per partition in sysfs for now. In the future when multiple QTG IDs are necessary, they can be exposed. [1] [1]: https://lore.kernel.org/linux-cxl/167571650007.587790.10040913293130712882.stgit@djiang5-mobl3.local/T/#md2a47b1ead3e1ba08f50eab29a4af1aed1d215ab Suggested-by: Dan Williams Signed-off-by: Dave Jiang --- v5: - Change qtg_id to qos_class v4: - Change kernel version for documentation to v6.5 v3: - Expand description of qtg_id. (Alison) --- Documentation/ABI/testing/sysfs-bus-cxl | 32 +++++++++++++++++++++++++++++++ drivers/cxl/core/memdev.c | 26 +++++++++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 2f24e42ef36d..c13ae49dd9aa 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -28,6 +28,22 @@ Description: Payload in the CXL-2.0 specification. +What: /sys/bus/cxl/devices/memX/ram/qos_class +Date: May, 2023 +KernelVersion: v6.5 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) For CXL host platforms that support "QoS Telemmetry" + this attribute conveys a platform specific cookie that + identifies a QoS performance class for the volatile + partition of the CXL mem device. This class-id can be + compared against a similar "qos_class" published for + a root decoder. While it is not required that the endpoints + map their local memory-class to a matching platform class, + mismatches are not recommended and there are platform specific + side-effects that may result. + + What: /sys/bus/cxl/devices/memX/pmem/size Date: December, 2020 KernelVersion: v5.12 @@ -38,6 +54,22 @@ Description: Payload in the CXL-2.0 specification. +What: /sys/bus/cxl/devices/memX/pmem/qos_class +Date: May, 2023 +KernelVersion: v6.5 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) For CXL host platforms that support "QoS Telemmetry" + this attribute conveys a platform specific cookie that + identifies a QoS performance class for the persistent + partition of the CXL mem device. This class-id can be + compared against a similar "qos_class" published for + a root decoder. While it is not required that the endpoints + map their local memory-class to a matching platform class, + mismatches are not recommended and there are platform specific + side-effects that may result. + + What: /sys/bus/cxl/devices/memX/serial Date: January, 2022 KernelVersion: v5.18 diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 28a05f2fe32d..8e183bef33e8 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -76,6 +76,18 @@ static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL); +static ssize_t ram_qos_class_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + + return sysfs_emit(buf, "%d\n", cxlds->ram_qos_class); +} + +static struct device_attribute dev_attr_ram_qos_class = + __ATTR(qos_class, 0444, ram_qos_class_show, NULL); + static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -89,6 +101,18 @@ static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL); +static ssize_t pmem_qos_class_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + + return sysfs_emit(buf, "%d\n", cxlds->pmem_qos_class); +} + +static struct device_attribute dev_attr_pmem_qos_class = + __ATTR(qos_class, 0444, pmem_qos_class_show, NULL); + static ssize_t serial_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -117,11 +141,13 @@ static struct attribute *cxl_memdev_attributes[] = { static struct attribute *cxl_memdev_pmem_attributes[] = { &dev_attr_pmem_size.attr, + &dev_attr_pmem_qos_class.attr, NULL, }; static struct attribute *cxl_memdev_ram_attributes[] = { &dev_attr_ram_size.attr, + &dev_attr_ram_qos_class.attr, NULL, }; From patchwork Mon May 8 20:48:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 13235069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CD7CC7EE22 for ; Mon, 8 May 2023 20:48:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229492AbjEHUsY (ORCPT ); Mon, 8 May 2023 16:48:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229452AbjEHUsY (ORCPT ); Mon, 8 May 2023 16:48:24 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2768F5B9F for ; Mon, 8 May 2023 13:48:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683578903; x=1715114903; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hO89vfS+V/rKPbvFZXHUbz9FU6GF93xdkbPmxFcYC10=; b=eyYIxh0wxvS1X/0/nWtf28yVQ4DNYnDNKJS//NEM16oyynJCQ1/+m45s WP8h+AcCy10Ktc+sFkMs80dLi6tos1rTZtbImlxHGbipCkMmjRtdZODsV EkGuDM9pN3PZ28daku7NjgwRF4SaUWew+3yfceJLBLtGDmIOcNPPoWc+1 YRpZTc8vjZk09K2GrH2s6/FzQeGqPh7M2vp/fyJqDEPvgbgsI94zOCKaH jEt7LAMszqDYP9yDM6rX154qF2RVtGO5ctQ0+TPUKzdwULcvJei+QMtqJ UNeoLc/iT8RO//VnvsCfUR79ZkezBIEsVY4p0LsctEDHzGFa34Rtwz8Rq g==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="349786232" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="349786232" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="676195192" X-IronPort-AV: E=Sophos;i="5.99,259,1677571200"; d="scan'208";a="676195192" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.213.172.228]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 13:48:17 -0700 Subject: [PATCH v5 14/14] cxl/mem: Add debugfs output for QTG related data From: Dave Jiang To: linux-cxl@vger.kernel.org Cc: Dan Williams , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, Jonathan.Cameron@huawei.com Date: Mon, 08 May 2023 13:48:16 -0700 Message-ID: <168357889673.2756219.5112791678818283230.stgit@djiang5-mobl3> In-Reply-To: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> References: <168357873843.2756219.5839806150467356492.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Add debugfs output to /sys/kernel/debug/cxl/memX/qtgmap The debugfs attribute will dump out all the DSMAS ranges and the associated QTG ID exported by the CXL device CDAT. Suggested-by: Dan Williams Signed-off-by: Dave Jiang Reviewed-by: Jonathan Cameron --- v4: - Use cxlds->qos_list instead of the stray cxlmd->qos_list --- Documentation/ABI/testing/debugfs-cxl | 11 +++++++++++ MAINTAINERS | 1 + drivers/cxl/mem.c | 17 +++++++++++++++++ 3 files changed, 29 insertions(+) create mode 100644 Documentation/ABI/testing/debugfs-cxl diff --git a/Documentation/ABI/testing/debugfs-cxl b/Documentation/ABI/testing/debugfs-cxl new file mode 100644 index 000000000000..0f36eeb7e59b --- /dev/null +++ b/Documentation/ABI/testing/debugfs-cxl @@ -0,0 +1,11 @@ +What: /sys/kernel/debug/cxl/memX/qtg_map +Date: Mar, 2023 +KernelVersion: v6.4 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) Entries of all Device Physical Address (DPA) ranges + provided by the device Coherent Device Attributes Table (CDAT) + Device Scoped Memory Affinity Structure (DSMAS) entries with + the matching QoS Throttling Group (QTG) id calculated from the + latency and bandwidth of the CXL path from the memory device + to the CPU. diff --git a/MAINTAINERS b/MAINTAINERS index fd8c4c560f8d..256e4e57017c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5171,6 +5171,7 @@ M: Ben Widawsky M: Dan Williams L: linux-cxl@vger.kernel.org S: Maintained +F: Documentation/ABI/testing/debugfs-cxl F: drivers/cxl/ F: include/uapi/linux/cxl_mem.h diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 39c4b54f0715..587e261a7f76 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -45,6 +45,22 @@ static int cxl_mem_dpa_show(struct seq_file *file, void *data) return 0; } +static int cxl_mem_qtg_show(struct seq_file *file, void *data) +{ + struct device *dev = file->private; + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + struct perf_prop_entry *perf; + + list_for_each_entry(perf, &cxlds->perf_list, list) { + seq_printf(file, "%08llx-%08llx : QoS Class: %u\n", + perf->dpa_range.start, perf->dpa_range.end, + perf->qos_class); + } + + return 0; +} + static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, struct cxl_dport *parent_dport) { @@ -117,6 +133,7 @@ static int cxl_mem_probe(struct device *dev) dentry = cxl_debugfs_create_dir(dev_name(dev)); debugfs_create_devm_seqfile(dev, "dpamem", dentry, cxl_mem_dpa_show); + debugfs_create_devm_seqfile(dev, "qtgmap", dentry, cxl_mem_qtg_show); rc = devm_add_action_or_reset(dev, remove_debugfs, dentry); if (rc) return rc;