From patchwork Thu Jun 10 18:57:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 12313785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D499C48BD1 for ; Thu, 10 Jun 2021 18:57:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6CE00613FF for ; Thu, 10 Jun 2021 18:57:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230425AbhFJS7f (ORCPT ); Thu, 10 Jun 2021 14:59:35 -0400 Received: from mga06.intel.com ([134.134.136.31]:35899 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230291AbhFJS7e (ORCPT ); Thu, 10 Jun 2021 14:59:34 -0400 IronPort-SDR: 1KZKOShtFi2sbh1XtycmyBFeQ9tV4nRKZnlQzSpaf+a1muMeOKWrg68tsrKng0DLQXur2VUqsA E7waL5ZEmmzw== X-IronPort-AV: E=McAfee;i="6200,9189,10011"; a="266527354" X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="266527354" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:37 -0700 IronPort-SDR: vXpfAiHa39+30CYHg51K444enUBXUWB0JB5iuBps8Aej7et7V2XXkcUC9lLngVe38TBWy7RymA AmQ+4lbsXUvw== X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="482963769" Received: from millers-mobl.amr.corp.intel.com (HELO bad-guy.kumite) ([10.252.140.70]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:37 -0700 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , Alison Schofield , Dan Williams , Ira Weiny , Jonathan Cameron , Vishal Verma Subject: [RFC PATCH 1/4] cxl/region: Add region creation ABI Date: Thu, 10 Jun 2021 11:57:22 -0700 Message-Id: <20210610185725.897541-2-ben.widawsky@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210610185725.897541-1-ben.widawsky@intel.com> References: <20210610185725.897541-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Regions are created as a child of the decoder that encompasses an address space with constraints. Regions only exist for persistent capacities. As an example, creating a region with: echo 1 > /sys/bus/cxl/devices/decoder1.0/create_region Will yield /sys/bus/cxl/devices/decoder1.0/region1.0:0 That region may then be deleted with: echo "region1.0:0 > /sys/bus/cxl/devices/decoder1.0/delete_region Signed-off-by: Ben Widawsky --- Documentation/ABI/testing/sysfs-bus-cxl | 19 +++ .../driver-api/cxl/memory-devices.rst | 8 + drivers/cxl/Makefile | 2 +- drivers/cxl/core.c | 71 ++++++++ drivers/cxl/cxl.h | 11 ++ drivers/cxl/mem.h | 19 +++ drivers/cxl/region.c | 155 ++++++++++++++++++ 7 files changed, 284 insertions(+), 1 deletion(-) create mode 100644 drivers/cxl/region.c diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 0b6a2e6e8fbb..5bcbefd4ea38 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -127,3 +127,22 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + Writing a value of '1' will create a new uninitialized region + that will be mapped by the CXL decoderX.Y. Reading from this + node will return the last created region. Regions must be + subsequently configured and bound to a region driver before they + can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + Deletes the named region. A region must be unbound from the + region driver before being deleted. diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentation/driver-api/cxl/memory-devices.rst index 487ce4f41d77..3066638b087f 100644 --- a/Documentation/driver-api/cxl/memory-devices.rst +++ b/Documentation/driver-api/cxl/memory-devices.rst @@ -39,6 +39,14 @@ CXL Core .. kernel-doc:: drivers/cxl/core.c :doc: cxl core +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.c + :doc: cxl region + +.. kernel-doc:: drivers/cxl/region.c + :identifiers: + External Interfaces =================== diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile index a29efb3e8ad2..2b9dd9187788 100644 --- a/drivers/cxl/Makefile +++ b/drivers/cxl/Makefile @@ -4,6 +4,6 @@ obj-$(CONFIG_CXL_MEM) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -cxl_core-y := core.o +cxl_core-y := core.o region.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o diff --git a/drivers/cxl/core.c b/drivers/cxl/core.c index 1b9ee0b08384..cda09a9cd98e 100644 --- a/drivers/cxl/core.c +++ b/drivers/cxl/core.c @@ -7,6 +7,7 @@ #include #include #include "cxl.h" +#include "mem.h" /** * DOC: cxl core @@ -119,7 +120,68 @@ static ssize_t target_list_show(struct device *dev, } static DEVICE_ATTR_RO(target_list); +static ssize_t create_region_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + int rc; + + device_lock(dev); + rc = sprintf(buf, "%s\n", + cxld->youngest ? dev_name(&cxld->youngest->dev) : ""); + device_unlock(dev); + + return rc; +} + +static ssize_t create_region_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + struct cxl_region *region; + ssize_t rc; + int val; + + rc = kstrtoint(buf, 0, &val); + if (rc) + return rc; + if (val != 1) + return -EINVAL; + + region = cxl_alloc_region(cxld); + if (IS_ERR(region)) + return PTR_ERR(region); + + rc = cxl_add_region(cxld, region); + if (rc) { + cxl_free_region(cxld, region); + return rc; + } + + cxld->youngest = region; + return len; +} +static DEVICE_ATTR_RW(create_region); + +static ssize_t delete_region_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + int rc; + + rc = cxl_delete_region(cxld, buf); + if (rc) + return rc; + + return len; +} +static DEVICE_ATTR_WO(delete_region); + static struct attribute *cxl_decoder_base_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_start.attr, &dev_attr_size.attr, &dev_attr_locked.attr, @@ -170,7 +232,13 @@ static void cxl_decoder_release(struct device *dev) { struct cxl_decoder *cxld = to_cxl_decoder(dev); struct cxl_port *port = to_cxl_port(dev->parent); + struct cxl_region *region; + list_for_each_entry(region, &cxld->regions, list) { + cxl_delete_region(cxld, dev_name(®ion->dev)); + } + dev_WARN_ONCE(dev, !ida_is_empty(&cxld->region_ida), + "Lost track of a region"); ida_free(&port->decoder_ida, cxld->id); kfree(cxld); } @@ -475,8 +543,11 @@ cxl_decoder_alloc(struct cxl_port *port, int nr_targets, resource_size_t base, .interleave_ways = interleave_ways, .interleave_granularity = interleave_granularity, .target_type = type, + .regions = LIST_HEAD_INIT(cxld->regions), }; + ida_init(&cxld->region_ida); + /* handle implied target_list */ if (interleave_ways == 1) cxld->target[0] = diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index b988ea288f53..1ffc5e07e24d 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -7,6 +7,7 @@ #include #include #include +#include /** * DOC: cxl objects @@ -182,6 +183,9 @@ enum cxl_decoder_type { * @interleave_granularity: data stride per dport * @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking + * @region_ida: allocator for region ids. + * @regions: List of regions mapped (may be disabled) by this decoder. + * @youngest: Last region created for this decoder. * @target: active ordered target list in current decoder configuration */ struct cxl_decoder { @@ -192,6 +196,9 @@ struct cxl_decoder { int interleave_granularity; enum cxl_decoder_type target_type; unsigned long flags; + struct ida region_ida; + struct list_head regions; + struct cxl_region *youngest; struct cxl_dport *target[]; }; @@ -231,6 +238,10 @@ struct cxl_dport { struct list_head list; }; +struct cxl_region *cxl_alloc_region(struct cxl_decoder *cxld); +void cxl_free_region(struct cxl_decoder *cxld, struct cxl_region *region); +int cxl_add_region(struct cxl_decoder *cxld, struct cxl_region *region); +int cxl_delete_region(struct cxl_decoder *cxld, const char *region); struct cxl_port *to_cxl_port(struct device *dev); struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport, resource_size_t component_reg_phys, diff --git a/drivers/cxl/mem.h b/drivers/cxl/mem.h index 13868ff7cadf..4c4eb06a966b 100644 --- a/drivers/cxl/mem.h +++ b/drivers/cxl/mem.h @@ -3,6 +3,8 @@ #ifndef __CXL_MEM_H__ #define __CXL_MEM_H__ +#include + /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ #define CXLMDEV_STATUS_OFFSET 0x0 #define CXLMDEV_DEV_FATAL BIT(0) @@ -46,6 +48,23 @@ struct cxl_memdev { int id; }; +/** + * struct cxl_region - CXL region + * @dev: This region's device. + * @id: This regions id. Id is globally unique across all regions. + * @res: Address space consumed by this region. + * @list: Node in decoders region list. + * @targets: The memory devices comprising the region. + */ +struct cxl_region { + struct device dev; + int id; + struct resource res; + struct list_head list; + struct cxl_memdev *targets[]; +}; + + /** * struct cxl_mem - A CXL memory device * @pdev: The PCI device associated with this CXL device. diff --git a/drivers/cxl/region.c b/drivers/cxl/region.c new file mode 100644 index 000000000000..1f47bc17bd50 --- /dev/null +++ b/drivers/cxl/region.c @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include +#include +#include +#include +#include +#include "cxl.h" +#include "mem.h" + +/** + * DOC: cxl region + * + * A CXL region encompasses a chunk of host physical address space that may be + * consumed by a single device (x1 interleave aka linear) or across multiple + * devices (xN interleaved). A region is a child device of a &struct + * cxl_decoder. There may be multiple active regions under a single &struct + * cxl_decoder. The common case for multiple regions would be several linear, + * contiguous regions under a single decoder. Generally, there will be a 1:1 + * relationship between decoder and region when the region is interleaved. + */ + +static void cxl_region_release(struct device *dev); + +static const struct device_type cxl_region_type = { + .name = "cxl_region", + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region(struct device *dev) +{ + if (dev_WARN_ONCE(dev, dev->type != &cxl_region_type, + "not a cxl_region device\n")) + return NULL; + + return container_of(dev, struct cxl_region, dev); +} + +void cxl_free_region(struct cxl_decoder *cxld, struct cxl_region *region) +{ + ida_free(&cxld->region_ida, region->id); + kfree(region->targets); + kfree(region); +} + +static void cxl_region_release(struct device *dev) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev->parent); + struct cxl_region *region; + + region = to_cxl_region(dev); + cxl_free_region(cxld, region); +} + +struct cxl_region *cxl_alloc_region(struct cxl_decoder *cxld) +{ + struct cxl_region *region; + int rc; + + region = kzalloc(struct_size(region, targets, cxld->interleave_ways), + GFP_KERNEL); + if (!region) { + pr_err("No memory\n"); + return ERR_PTR(-ENOMEM); + } + + rc = ida_alloc(&cxld->region_ida, GFP_KERNEL); + if (rc < 0) { + dev_err(&cxld->dev, "Couldn't get a new id\n"); + kfree(region); + return ERR_PTR(rc); + } + region->id = rc; + + return region; +} + +/** + * cxl_add_region - Adds a region to a decoder + * @cxld: Parent decoder. + * @region: Region to be added to the decoder. + * + * This is the second step of region initialization. Regions exist within an + * address space which is mapped by a @cxld, and that @cxld enforces constraints + * upon the region as it is configured. Regions may be added to a @cxld but not + * activated and therefore it is possible to have more regions in a @cxld than + * there are interleave ways in the @cxld. Regions exist only for persistent + * capacities. + * + * Return: zero if the region was added to the @cxld, else returns negative + * error code. + */ +int cxl_add_region(struct cxl_decoder *cxld, struct cxl_region *region) +{ + struct cxl_port *port = to_cxl_port(cxld->dev.parent); + struct device *dev = ®ion->dev; + struct resource *res; + int rc; + + device_initialize(dev); + dev->parent = &cxld->dev; + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type; + rc = dev_set_name(dev, "region%d.%d:%d", port->id, cxld->id, region->id); + if (rc) + goto err; + + rc = device_add(dev); + if (rc) + goto err; + + res = ®ion->res; + res->name = "Persistent Memory"; + res->start = 0; + res->end = -1; + res->flags = IORESOURCE_MEM; + res->desc = IORES_DESC_PERSISTENT_MEMORY; + + dev_dbg(dev, "Added %s to %s\n", dev_name(dev), dev_name(&cxld->dev)); + + return 0; + +err: + put_device(dev); + return rc; +} + +static struct cxl_region * +cxl_find_region_by_name(struct cxl_decoder *cxld, const char *name) +{ + struct device *region_dev; + + region_dev = device_find_child_by_name(&cxld->dev, name); + if (!region_dev) + return ERR_PTR(-ENOENT); + + return to_cxl_region(region_dev); +} + +int cxl_delete_region(struct cxl_decoder *cxld, const char *region_name) +{ + struct cxl_region *region; + + region = cxl_find_region_by_name(cxld, region_name); + if (IS_ERR(region)) + return PTR_ERR(region); + + dev_dbg(&cxld->dev, "Requested removal of %s from %s\n", + dev_name(®ion->dev), dev_name(&cxld->dev)); + + device_unregister(®ion->dev); + put_device(®ion->dev); + + return 0; +} From patchwork Thu Jun 10 18:57:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 12313789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E899FC48BE0 for ; Thu, 10 Jun 2021 18:57:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D236D61403 for ; Thu, 10 Jun 2021 18:57:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230457AbhFJS7f (ORCPT ); Thu, 10 Jun 2021 14:59:35 -0400 Received: from mga06.intel.com ([134.134.136.31]:35899 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230265AbhFJS7f (ORCPT ); Thu, 10 Jun 2021 14:59:35 -0400 IronPort-SDR: kckhwmCeDjLifwtItOjix9gKMcGhZTYWbNLIy9EwkxbSkcRTBINn4WA0xe+SgoCO2+AYI6gNcO kVxs/OQi58xA== X-IronPort-AV: E=McAfee;i="6200,9189,10011"; a="266527356" X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="266527356" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:37 -0700 IronPort-SDR: saHShBUgcYetj/bTvykOn98mOGXoO7QvUK78tDn2AE6s3z9BpdTmtT6Z0qiXWmH58efrlWdLAm 3KG9oB3sw1Kw== X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="482963775" Received: from millers-mobl.amr.corp.intel.com (HELO bad-guy.kumite) ([10.252.140.70]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:37 -0700 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , Alison Schofield , Dan Williams , Ira Weiny , Jonathan Cameron , Vishal Verma Subject: [RFC PATCH 2/4] cxl/region: Create attribute structure / verify Date: Thu, 10 Jun 2021 11:57:23 -0700 Message-Id: <20210610185725.897541-3-ben.widawsky@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210610185725.897541-1-ben.widawsky@intel.com> References: <20210610185725.897541-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Introduce a verification mechanism for a region. Regions have complex configuration requirements and it is beneficial to provide a way to verify the constraints are met before trying to bind. Primarily it adds ABI to inform userspace of more detailed information about what failed rather than the limited choices of errno at bind time. It's important to point out that a verified region can still fail to bind, but the first step in binding will be to run the same verification algorithm. Signed-off-by: Ben Widawsky --- Functionally it might make sense to squash this patch in with other patches adding attributes. From a discussion standpoint however, it's nice to have this broken out as I suspect there might be some debate about it. --- Documentation/ABI/testing/sysfs-bus-cxl | 13 +++++++++++++ drivers/cxl/region.c | 22 +++++++++++++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 5bcbefd4ea38..699c8514fd7b 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -146,3 +146,16 @@ Contact: linux-cxl@vger.kernel.org Description: Deletes the named region. A region must be unbound from the region driver before being deleted. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/verify +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + Instructs the kernel to verify that the regionX.Y:Z is properly + configured and provide more detailed information about + configuration errors. A value of 0 indicates the region is + properly configured and ready to bind, otherwise a negative + integer is returned describing the first error found in the + configuration. A verified region can still fail binding due to + lack of resources. diff --git a/drivers/cxl/region.c b/drivers/cxl/region.c index 1f47bc17bd50..ea1ac848c713 100644 --- a/drivers/cxl/region.c +++ b/drivers/cxl/region.c @@ -20,11 +20,31 @@ * relationship between decoder and region when the region is interleaved. */ -static void cxl_region_release(struct device *dev); +static ssize_t verify_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "0"); +} + +static DEVICE_ATTR_RO(verify); + +static struct attribute *region_attrs[] = { + &dev_attr_verify.attr, + NULL, +}; +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, +}; + +static void cxl_region_release(struct device *dev); static const struct device_type cxl_region_type = { .name = "cxl_region", .release = cxl_region_release, + .groups = region_groups, }; static struct cxl_region *to_cxl_region(struct device *dev) From patchwork Thu Jun 10 18:57:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 12313787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D5D3C48BE5 for ; Thu, 10 Jun 2021 18:57:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 461AB613F5 for ; Thu, 10 Jun 2021 18:57:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230265AbhFJS7f (ORCPT ); Thu, 10 Jun 2021 14:59:35 -0400 Received: from mga06.intel.com ([134.134.136.31]:35902 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230366AbhFJS7f (ORCPT ); Thu, 10 Jun 2021 14:59:35 -0400 IronPort-SDR: gs6TkBXxZDPMN3FCOXj86N0NFmB4XYH3Skd17U/NwuI4uU7x6ircPbhPOQ/wgkN1lY0enD1jDX S5On8CPfTx5Q== X-IronPort-AV: E=McAfee;i="6200,9189,10011"; a="266527360" X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="266527360" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:38 -0700 IronPort-SDR: cGtR8Zoqv60kAIEFkNQBCVmEyMOrqflVMD0KhAlXFvQCj4S9TCf0QUdDjMgWjXKKmW3rrfK72m mh5jQYM7atCw== X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="482963778" Received: from millers-mobl.amr.corp.intel.com (HELO bad-guy.kumite) ([10.252.140.70]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:37 -0700 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , Alison Schofield , Dan Williams , Ira Weiny , Jonathan Cameron , Vishal Verma Subject: [RFC PATCH 3/4] cxl: Move cxl_memdev conversion helper to mem.h Date: Thu, 10 Jun 2021 11:57:24 -0700 Message-Id: <20210610185725.897541-4-ben.widawsky@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210610185725.897541-1-ben.widawsky@intel.com> References: <20210610185725.897541-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Signed-off-by: Ben Widawsky --- drivers/cxl/mem.h | 5 +++++ drivers/cxl/pci.c | 5 ----- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/cxl/mem.h b/drivers/cxl/mem.h index 4c4eb06a966b..9795aa924035 100644 --- a/drivers/cxl/mem.h +++ b/drivers/cxl/mem.h @@ -48,6 +48,11 @@ struct cxl_memdev { int id; }; +static inline struct cxl_memdev *to_cxl_memdev(struct device *dev) +{ + return container_of(dev, struct cxl_memdev, dev); +} + /** * struct cxl_region - CXL region * @dev: This region's device. diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 5a1705b52278..be158f8f995e 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -1142,11 +1142,6 @@ static int cxl_mem_setup_regs(struct cxl_mem *cxlm) return ret; } -static struct cxl_memdev *to_cxl_memdev(struct device *dev) -{ - return container_of(dev, struct cxl_memdev, dev); -} - static void cxl_memdev_release(struct device *dev) { struct cxl_memdev *cxlmd = to_cxl_memdev(dev); From patchwork Thu Jun 10 18:57:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 12313791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5480AC48BE6 for ; Thu, 10 Jun 2021 18:57:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3B8D2613FF for ; Thu, 10 Jun 2021 18:57:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230366AbhFJS7g (ORCPT ); Thu, 10 Jun 2021 14:59:36 -0400 Received: from mga06.intel.com ([134.134.136.31]:35899 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230291AbhFJS7f (ORCPT ); Thu, 10 Jun 2021 14:59:35 -0400 IronPort-SDR: wKSeCN5Wwmp/avNoSCq/Fo7tS3//qiDAPEUFF/TG35VvuptWK20MddAp0XO2nfh/DPun18MtLn nVvGmgYKavig== X-IronPort-AV: E=McAfee;i="6200,9189,10011"; a="266527362" X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="266527362" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:38 -0700 IronPort-SDR: STed1BdFswKKvwfvCZqbv4Fj2gIca2nB2fWATAflHIfW4QCOUKTYPWfNs4K3jSTpiS8y01fL/+ 6RU2KVaabdyA== X-IronPort-AV: E=Sophos;i="5.83,264,1616482800"; d="scan'208";a="482963782" Received: from millers-mobl.amr.corp.intel.com (HELO bad-guy.kumite) ([10.252.140.70]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2021 11:57:38 -0700 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , Alison Schofield , Dan Williams , Ira Weiny , Jonathan Cameron , Vishal Verma Subject: [RFC PATCH 4/4] cxl/region: Introduce concept of region configuration Date: Thu, 10 Jun 2021 11:57:25 -0700 Message-Id: <20210610185725.897541-5-ben.widawsky@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210610185725.897541-1-ben.widawsky@intel.com> References: <20210610185725.897541-1-ben.widawsky@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org The region creation APIs leave a region unconfigured. Configuring the region will work in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may "commit" the config. What the kernel decides to do after a config is committed is out of scope at this point. Introduced here are the most basic attributes needed to configure a region. A x1 interleave example is provided below: decoder1.0 ├── create_region ├── delete_region ├── devtype ├── locked ├── region1.0:0 │   ├── offset │   ├── size │   ├── subsystem -> ../../../../../../../bus/cxl │   ├── target0 │   ├── uevent │   ├── uuid │   └── verify ├── size ├── start ├── subsystem -> ../../../../../../bus/cxl ├── target_list ├── target_type └── uevent Signed-off-by: Ben Widawsky --- Documentation/ABI/testing/sysfs-bus-cxl | 27 +++ drivers/cxl/mem.h | 2 + drivers/cxl/region.c | 227 +++++++++++++++++++++++- 3 files changed, 255 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 699c8514fd7b..d7174a84f70d 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -159,3 +159,30 @@ Description: integer is returned describing the first error found in the configuration. A verified region can still fail binding due to lack of resources. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{offset,size} +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + A region resides within an address space that is claimed by a + decoder. The region will be of some size within the address + space and at some offset that must also reside within the + address space. The size and position of the region is specified + by these attributes. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/uuid +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + The unique identifier for the region. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/target[0-15] +Date: June, 2021 +KernelVersion: v5.14 +Contact: linux-cxl@vger.kernel.org +Description: + Memory devices are the backing storage for a region. Each target + must be populated with a memdev in order for the region to be + eligible to be activated. diff --git a/drivers/cxl/mem.h b/drivers/cxl/mem.h index 9795aa924035..059fbf084fa1 100644 --- a/drivers/cxl/mem.h +++ b/drivers/cxl/mem.h @@ -58,6 +58,7 @@ static inline struct cxl_memdev *to_cxl_memdev(struct device *dev) * @dev: This region's device. * @id: This regions id. Id is globally unique across all regions. * @res: Address space consumed by this region. + * @uuid: The UUID for this region. * @list: Node in decoders region list. * @targets: The memory devices comprising the region. */ @@ -65,6 +66,7 @@ struct cxl_region { struct device dev; int id; struct resource res; + uuid_t uuid; struct list_head list; struct cxl_memdev *targets[]; }; diff --git a/drivers/cxl/region.c b/drivers/cxl/region.c index ea1ac848c713..a69ee00514cb 100644 --- a/drivers/cxl/region.c +++ b/drivers/cxl/region.c @@ -3,7 +3,9 @@ #include #include #include +#include #include +#include #include #include "cxl.h" #include "mem.h" @@ -20,15 +22,130 @@ * relationship between decoder and region when the region is interleaved. */ +static struct cxl_region *to_cxl_region(struct device *dev); + +#define cxl_region_ways(region) \ + to_cxl_decoder((region)->dev.parent)->interleave_ways + static ssize_t verify_show(struct device *dev, struct device_attribute *attr, char *buf) { + struct cxl_decoder *cxld = to_cxl_decoder(dev->parent); + struct cxl_region *region = to_cxl_region(dev); + struct resource decode_res; + int i; + + decode_res = (struct resource)DEFINE_RES_MEM(cxld->range.start, + range_len(&cxld->range)); + + /* Invalid region size */ + if (!resource_contains(&decode_res, ®ion->res)) + return sysfs_emit(buf, "size"); + + if (resource_size(®ion->res) % (SZ_256M * cxld->interleave_ways)) + return sysfs_emit(buf, "alignment"); + + /* Missing target memory device */ + for (i = 0; i < cxld->interleave_ways; i++) + if (!region->targets[i]) + return sysfs_emit(buf, "memdev"); + return sysfs_emit(buf, "0"); } static DEVICE_ATTR_RO(verify); +static bool is_region_active(struct cxl_region *region) +{ + /* TODO: Regions can't be activated yet. */ + return false; +} + +static ssize_t offset_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_region *region = to_cxl_region(dev); + + return sysfs_emit(buf, "%#llx\n", region->res.start); +} + +static ssize_t offset_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *region = to_cxl_region(dev); + unsigned long long val; + ssize_t rc; + + rc = kstrtoull(buf, 0, &val); + if (rc) + return rc; + + if (is_region_active(region)) { + /* TODO: */ + } else { + region->res.start = val; + } + + return len; +} + +static DEVICE_ATTR_RW(offset); + +static ssize_t size_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_region *region = to_cxl_region(dev); + + return sysfs_emit(buf, "%llu\n", resource_size(®ion->res)); +} + +static ssize_t size_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *region = to_cxl_region(dev); + unsigned long long val; + ssize_t rc; + + rc = kstrtoull(buf, 0, &val); + if (rc) + return rc; + + if (is_region_active(region)) { + /* TODO: */ + } else { + region->res.end = region->res.start + val - 1; + } + + return len; +} +static DEVICE_ATTR_RW(size); + +static ssize_t uuid_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_region *region = to_cxl_region(dev); + + return sysfs_emit(buf, "%pUb\n", ®ion->uuid); +} +static ssize_t uuid_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *region = to_cxl_region(dev); + ssize_t rc; + + if (len != UUID_STRING_LEN + 1) + return -EINVAL; + + rc = uuid_parse(buf, ®ion->uuid); + + return rc ? rc : len; +} +static DEVICE_ATTR_RW(uuid); + static struct attribute *region_attrs[] = { &dev_attr_verify.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, NULL, }; @@ -36,8 +153,111 @@ static const struct attribute_group region_group = { .attrs = region_attrs, }; +static size_t show_targetN(struct cxl_region *region, char *buf, int n) +{ + if (region->targets[n]) + return sysfs_emit(buf, "%s\n", dev_name(®ion->targets[n]->dev)); + else + return sysfs_emit(buf, "nil\n"); +} + +static size_t set_targetN(struct cxl_region *region, const char *buf, int n, size_t len) +{ + struct device *memdev_dev; + struct cxl_memdev *cxlmd; + ssize_t rc; + int val; + + rc = kstrtoint(buf, 0, &val); + if (!rc && val == 0) { + cxlmd = region->targets[n] = cxlmd; + if (cxlmd) + put_device(&cxlmd->dev); + region->targets[n] = NULL; + return len; + } + + memdev_dev = bus_find_device_by_name(&cxl_bus_type, NULL, buf); + if (!memdev_dev) + return -ENOENT; + + cxlmd = to_cxl_memdev(memdev_dev); + get_device(&cxlmd->dev); + region->targets[n] = cxlmd; + + return len; +} + +#define TARGET_ATTR_RW(n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, n); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, n, len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW(0); +TARGET_ATTR_RW(1); +TARGET_ATTR_RW(2); +TARGET_ATTR_RW(3); +TARGET_ATTR_RW(4); +TARGET_ATTR_RW(5); +TARGET_ATTR_RW(6); +TARGET_ATTR_RW(7); +TARGET_ATTR_RW(8); +TARGET_ATTR_RW(9); +TARGET_ATTR_RW(10); +TARGET_ATTR_RW(11); +TARGET_ATTR_RW(12); +TARGET_ATTR_RW(13); +TARGET_ATTR_RW(14); +TARGET_ATTR_RW(15); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, + NULL, +}; + +static umode_t visible_targets(struct kobject *kobj, struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + struct cxl_region *region = to_cxl_region(dev); + + if (n < cxl_region_ways(region)) + return a->mode; + return 0; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + static const struct attribute_group *region_groups[] = { ®ion_group, + ®ion_interleave_group, + NULL, }; static void cxl_region_release(struct device *dev); @@ -58,8 +278,13 @@ static struct cxl_region *to_cxl_region(struct device *dev) void cxl_free_region(struct cxl_decoder *cxld, struct cxl_region *region) { + int i; + ida_free(&cxld->region_ida, region->id); - kfree(region->targets); + for (i = 0; i < cxld->interleave_ways; i++) { + if (region->targets[i]) + put_device(®ion->targets[i]->dev); + } kfree(region); }