From patchwork Mon Aug 21 18:14:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alison Schofield X-Patchwork-Id: 13359731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14F6CEE49A6 for ; Mon, 21 Aug 2023 18:14:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237123AbjHUSOu (ORCPT ); Mon, 21 Aug 2023 14:14:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237119AbjHUSOt (ORCPT ); Mon, 21 Aug 2023 14:14:49 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 656AE10E for ; Mon, 21 Aug 2023 11:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692641688; x=1724177688; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Fk5ispeQuoAGFTua/Q7cqH/TXSFcqw6a+J1sZnrPrb0=; b=lLEy/qMxCzxZoeQBHrJFazMdyFlsEmHSWeCgbkWau5Zz3ur7FipkLa3S hdde/uxNDH53ubsDmeKxBMUP7S+v0u7EKYazFiD5BZGTnieE9zVszqv2m GdcGzb8k+6fRcVG5oQqjzBx43c1T2BJixTVqNbYp5iD0SWL90UCwIJsI8 8RXNryvC1BCFBnye/+JC/W6N+yYXxhz9VUtexW0Z7nMAq9jPCiezGF2go 1xTK2GHWWDcnHX5JlJEmMQS02CUQ0DpVDEOoMXCtQ3r4DvxN1mK8CQJBK 5j/H9fP1OyFGd5ik/LoTEho9vp5egKNUkE6CHCME44Z2sASDhoNU3IxVx A==; X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="353978360" X-IronPort-AV: E=Sophos;i="6.01,190,1684825200"; d="scan'208";a="353978360" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Aug 2023 11:14:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="859569973" X-IronPort-AV: E=Sophos;i="6.01,190,1684825200"; d="scan'208";a="859569973" Received: from aschofie-mobl2.amr.corp.intel.com (HELO localhost) ([10.209.106.94]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Aug 2023 11:14:41 -0700 From: alison.schofield@intel.com To: Davidlohr Bueso , Jonathan Cameron , Dave Jiang , Alison Schofield , Vishal Verma , Ira Weiny , Dan Williams Cc: linux-cxl@vger.kernel.org Subject: [PATCH v3 1/2] cxl/region: Try to add a region resource to a soft reserved parent Date: Mon, 21 Aug 2023 11:14:36 -0700 Message-Id: <4cf018843ee9846c5b9907b8fc7140ab66b929b6.1692638817.git.alison.schofield@intel.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Alison Schofield During region autodiscovery, the region driver always inserts the region resource as a child of the root decoder, a CXL WINDOW. It has the effect of making a soft reserved resource, with an exactly matching address range, a child of the region resource. It looks like this in /proc/iomem: 2080000000-29dbfffffff : CXL Window 0 2080000000-247fffffff : region0 2080000000-247fffffff : Soft Reserved Search for soft reserved resources that include the region resource and add the new region resource as a child of that found resource. If a soft reserved resource is not found, insert to the root decoder as usual. With this change, it looks like this: 2080000000-29dbfffffff : CXL Window 0 2080000000-247fffffff : Soft Reserved 2080000000-247fffffff : region0 This odd parenting only occurs when the resources are an exact match. When the region resource only uses part of a soft reserved resource, the parenting appears more logical like this: 2080000000-29dbfffffff : CXL Window 0 2080000000-287fffffff : Soft Reserved 2080000000-247fffffff : region0 Aside from the more logical appearance, this change is in preparation for further cleanup in region teardown. A follow-on patch intends to find and free soft reserved resources upon region teardown. Signed-off-by: Alison Schofield Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron --- drivers/cxl/core/region.c | 57 ++++++++++++++++++++++++++++++++------- 1 file changed, 47 insertions(+), 10 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index e115ba382e04..5c487aab15ad 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -2709,6 +2709,31 @@ static int match_region_by_range(struct device *dev, void *data) return rc; } +static int insert_resource_soft_reserved(struct resource *soft_res, void *arg) +{ + struct resource *parent, *new, *res = arg; + bool found = false; + int rc = 0; + + parent = soft_res->parent; + if (!parent) + return 0; + + /* Caller provides a copy of soft_res. Find the actual resource. */ + for (new = parent->child; new; new = new->sibling) { + if (resource_contains(new, soft_res)) { + rc = insert_resource(new, res); + found = true; + break; + } + } + /* Caller handles failure to find or insert resource */ + if (!found || rc) + return rc; + + return 1; +} + /* Establish an empty region covering the given HPA range */ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, struct cxl_endpoint_decoder *cxled) @@ -2755,16 +2780,28 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, *res = DEFINE_RES_MEM_NAMED(hpa->start, range_len(hpa), dev_name(&cxlr->dev)); - rc = insert_resource(cxlrd->res, res); - if (rc) { - /* - * Platform-firmware may not have split resources like "System - * RAM" on CXL window boundaries see cxl_region_iomem_release() - */ - dev_warn(cxlmd->dev.parent, - "%s:%s: %s %s cannot insert resource\n", - dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), - __func__, dev_name(&cxlr->dev)); + + /* Try inserting to a Soft Reserved parent. Fallback to root decoder */ + rc = walk_iomem_res_desc(IORES_DESC_SOFT_RESERVED, 0, res->start, + res->end, res, insert_resource_soft_reserved); + if (!rc || rc == -EBUSY) + dev_dbg(&cxlmd->dev, + "insert %pr to soft reserved parent failed rc:%d\n", + res, rc); + if (rc != 1) { + rc = insert_resource(cxlrd->res, res); + if (rc) { + /* + * Platform-firmware may not have split resources + * like "System RAM" on CXL window boundaries see + * cxl_region_iomem_release() + */ + dev_warn(cxlmd->dev.parent, + "%s:%s: %s %s cannot insert resource\n", + dev_name(&cxlmd->dev), + dev_name(&cxled->cxld.dev), __func__, + dev_name(&cxlr->dev)); + } } p->res = res; From patchwork Mon Aug 21 18:14:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alison Schofield X-Patchwork-Id: 13359730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B66F3EE4993 for ; Mon, 21 Aug 2023 18:14:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237077AbjHUSOt (ORCPT ); Mon, 21 Aug 2023 14:14:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236158AbjHUSOs (ORCPT ); Mon, 21 Aug 2023 14:14:48 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D2B710E for ; Mon, 21 Aug 2023 11:14:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692641684; x=1724177684; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dBhUPfL41O3d3+5y359VALWO/si9HvuvMds6LKMnTIs=; b=gydhm+0DLUdNiKEutV87CHibkvegU4ecdaQR7TZwrkOz13e/VaTdWLmp 3O/1Qf1qJkMLoOoeZkgkQSECf35MUYuQP8HmA3XduDX/I6Du1YL/OcW6L 9E6WjuM8H6O4yIspyakaBGTIzCjfsiZ+RH6vTLQ9wdOQsSC5akmxZnOYe RNXgxf1mz0QAhXwNp4r2fts4NYCCaADhkZFs1ttVLJd8NTpxrxHvPdQeM qELP0OVRJI3OMdV5lbCvSJISRJkUwBRhYTk8Bdt/J/L5VmSwomfBYiIvG 13wNH1+XQ5Jox6jeh9W8cnSKsJJHtqaF5vl5t/5Hvttpg2/sjHcsLrXpx w==; X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="353978361" X-IronPort-AV: E=Sophos;i="6.01,190,1684825200"; d="scan'208";a="353978361" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Aug 2023 11:14:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="859569977" X-IronPort-AV: E=Sophos;i="6.01,190,1684825200"; d="scan'208";a="859569977" Received: from aschofie-mobl2.amr.corp.intel.com (HELO localhost) ([10.209.106.94]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Aug 2023 11:14:41 -0700 From: alison.schofield@intel.com To: Davidlohr Bueso , Jonathan Cameron , Dave Jiang , Alison Schofield , Vishal Verma , Ira Weiny , Dan Williams Cc: linux-cxl@vger.kernel.org, Derick Marks Subject: [PATCH v3 2/2] cxl/region: Remove a soft reserved resource at region teardown Date: Mon, 21 Aug 2023 11:14:37 -0700 Message-Id: <29312c0765224ae76862d59a17748c8188fb95f1.1692638817.git.alison.schofield@intel.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org From: Alison Schofield When CXL regions are created through autodiscovery their resource may be a child of a soft reserved resource. Currently, when such a region is destroyed, its soft reserved resource remains in place. That means that the HPA range that the region could release is actually unavailable for reuse. Free the soft reserved resource on region teardown by examining the alignment of the resources, and handling accordingly. The two resources will be exactly aligned, partially aligned, or not aligned at all. |----------- "Soft Reserved" -----------| |-------------- "Region #" -------------| Exactly aligned. Any dangling children move up to a parent on removal. The removal of this soft reserved seems guaranteed, however the availability of the address range for reuse depends on complete cleanup of the region child resources also. |----------- "Soft Reserved" -----------| |-------- "Region #" -------| or |----------- "Soft Reserved" -----------| |-------- "Region #" -------| Either start or end aligns. Unlike removing a resource, which simply moves child resources up the resource tree, adjustments fail if any child resources map the range being truncated. So this one will fail for dangling child resources of the region. |---------- "Soft Reserved" ----------| |---- "Region #" ----| No alignment. Freeing the resource of a region in the middle succeeds if no child resources map the leading or trailing address space. Reported-by: Derick Marks Signed-off-by: Alison Schofield Reviewed-by: Dave Jiang --- drivers/cxl/core/region.c | 130 +++++++++++++++++++++++++++++++++++--- 1 file changed, 121 insertions(+), 9 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 5c487aab15ad..dcf8d4ad2cf4 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -569,22 +569,134 @@ static int alloc_hpa(struct cxl_region *cxlr, resource_size_t size) return 0; } +static int add_soft_reserved(struct resource *parent, resource_size_t start, + resource_size_t len, unsigned long flags) +{ + struct resource *res __free(kfree) = kmalloc(sizeof(*res), GFP_KERNEL); + int rc; + + if (!res) + return -ENOMEM; + + *res = DEFINE_RES_MEM_NAMED(start, len, "Soft Reserved"); + + res->desc = IORES_DESC_SOFT_RESERVED; + res->flags = res->flags | flags; + rc = insert_resource(parent, res); + if (rc) + return rc; + + no_free_ptr(res); + return 0; +} + +static void remove_soft_reserved(struct cxl_region *cxlr, struct resource *soft, + resource_size_t start, resource_size_t end) +{ + struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent); + resource_size_t new_start, new_end; + struct resource *parent; + unsigned long flags; + int rc; + + /* Prevent new usage while removing or adjusting the resource */ + guard(mutex)(&cxlrd->range_lock); + + /* Aligns at both resource start and end */ + if (soft->start == start && soft->end == end) { + rc = remove_resource(soft); + if (rc) + dev_dbg(&cxlr->dev, + "cannot remove soft reserved resource %pr\n", + soft); + else + kfree(soft); + + return; + } + + /* Aligns at either resource start or end */ + if (soft->start == start || soft->end == end) { + if (soft->start == start) { + new_start = end + 1; + new_end = soft->end; + } else { + new_start = soft->start; + new_end = start + 1; + } + rc = adjust_resource(soft, new_start, new_end - new_start + 1); + if (rc) + dev_dbg(&cxlr->dev, + "cannot adjust soft reserved resource %pr\n", + soft); + return; + } + + /* + * No alignment. Attempt a 3-way split that removes the part of + * the resource the region occupied, and then creates new soft + * reserved resources for the leading and trailing addr space. + * adjust_resource() will stop the attempt if there are any + * child resources. + */ + + /* Save the original soft reserved resource params before adjusting */ + new_start = soft->start; + new_end = soft->end; + parent = soft->parent; + flags = soft->flags; + + rc = adjust_resource(soft, start, end - start); + if (rc) { + dev_dbg(&cxlr->dev, + "cannot adjust soft reserved resource %pr\n", soft); + return; + } + rc = remove_resource(soft); + if (rc) + dev_warn(&cxlr->dev, + "cannot remove soft reserved resource %pr\n", soft); + + rc = add_soft_reserved(parent, new_start, start - new_start, flags); + if (rc) + dev_warn(&cxlr->dev, + "cannot add new soft reserved resource at %pa\n", + &new_start); + + rc = add_soft_reserved(parent, end + 1, new_end - end, flags); + if (rc) + dev_warn(&cxlr->dev, + "cannot add new soft reserved resource at %pa + 1\n", + &end); +} + static void cxl_region_iomem_release(struct cxl_region *cxlr) { struct cxl_region_params *p = &cxlr->params; + struct resource *parent, *res = p->res; + resource_size_t start, end; if (device_is_registered(&cxlr->dev)) lockdep_assert_held_write(&cxl_region_rwsem); - if (p->res) { - /* - * Autodiscovered regions may not have been able to insert their - * resource. - */ - if (p->res->parent) - remove_resource(p->res); - kfree(p->res); - p->res = NULL; + if (!res) + return; + /* + * Autodiscovered regions may not have been able to insert their + * resource. If a Soft Reserved parent resource exists, try to + * remove that also. + */ + if (p->res->parent) { + parent = p->res->parent; + start = p->res->start; + end = p->res->end; + remove_resource(p->res); + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags) && + parent->desc == IORES_DESC_SOFT_RESERVED) + remove_soft_reserved(cxlr, parent, start, end); } + + kfree(p->res); + p->res = NULL; } static int free_hpa(struct cxl_region *cxlr)