From patchwork Wed Jan 31 21:59:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alison Schofield X-Patchwork-Id: 13540028 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2475139AEE for ; Wed, 31 Jan 2024 21:59:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706738379; cv=none; b=Eqlsk+Pa/q1ZdOBZMHKmgCVuiBtqpBaGesyi3QbTvsLG5eVor4v46db7u9/TFY5tGQxkg2FPHV3VzGSkIW5hBfAxWAiOZrCVvTK8PGVYzT/lD1MrV0SBHKcfhbkqyV3E/Yu9bufS/il1OEvjs7Q9BIoZeMf3UzXIM3LZmCxWrXY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706738379; c=relaxed/simple; bh=1j/Hm44zXG32we2KNQio/SvUYluJazkTrdmhImuD8iU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AM0nKh1boUVlqqHo00e4Vsj8pmko4bf4x39PYviNcWlCEDLUJtQ+AY9Pvc8H6qnygE5hreqjwVHDwZY196G//VsCHK38DesEbvemk+xWvuXS3tjOtQG1MXgCn8ikMLdDDVsHop2AF+1zuBQTonvkUPxFdIhDDBs9Z/9MH8kpigY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=dmhvvmgA; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="dmhvvmgA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706738377; x=1738274377; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1j/Hm44zXG32we2KNQio/SvUYluJazkTrdmhImuD8iU=; b=dmhvvmgAn+XZRhoCPlJ98O3Uj/aXvSQQgTCRDeTYy62TSAGE0+jtVepU 20nxaHcpz3R14abWTwRb5A3Ovu3jV7+ERo/ZAeOqQRhtPdHtOgaLkBMf9 IcdD8jdFmaiiwWNxpaRbjnApobv9BxFHl4sl4Y2hO9pVt4z8Gl0AiAVWq MusseelM//MDDCHbYyjTs6pFI6cuFW5zhii8YpUkrUX3x176IT8uf1id8 rBftaUKmtNreAg50IfVFEAzysdJYxN2dXgs9aSYw9NS9u7QXBs9Vn0XMi jX5DTopYP1RqTq8MD6TEI+SzFOYmix0KyDGV/HaB0kRYXaab5Iwd72G1e w==; X-IronPort-AV: E=McAfee;i="6600,9927,10969"; a="2675445" X-IronPort-AV: E=Sophos;i="6.05,233,1701158400"; d="scan'208";a="2675445" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jan 2024 13:59:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10969"; a="931989693" X-IronPort-AV: E=Sophos;i="6.05,233,1701158400"; d="scan'208";a="931989693" Received: from aschofie-mobl2.amr.corp.intel.com (HELO localhost) ([10.209.39.51]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jan 2024 13:59:35 -0800 From: alison.schofield@intel.com To: Davidlohr Bueso , Jonathan Cameron , Dave Jiang , Alison Schofield , Vishal Verma , Ira Weiny , Dan Williams Cc: linux-cxl@vger.kernel.org, Wonjae Lee Subject: [PATCH v3 1/2] cxl/region: Handle endpoint decoders in cxl_region_find_decoder() Date: Wed, 31 Jan 2024 13:59:30 -0800 Message-Id: <79ae6d72978ef9f3ceec9722e1cb793820553c8e.1706736863.git.alison.schofield@intel.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Alison Schofield In preparation for adding a new caller of cxl_region_find_decoders() teach it to find a decoder from a cxl_endpoint_decoder structure. Combining switch and endpoint decoder lookup in one function prevents code duplication in call sites. Update the existing caller. Signed-off-by: Alison Schofield --- drivers/cxl/core/region.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 0f05692bfec3..7af40d9cf80a 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -730,12 +730,17 @@ static int match_auto_decoder(struct device *dev, void *data) return 0; } -static struct cxl_decoder *cxl_region_find_decoder(struct cxl_port *port, - struct cxl_region *cxlr) +static struct cxl_decoder * +cxl_region_find_decoder(struct cxl_port *port, + struct cxl_endpoint_decoder *cxled, + struct cxl_region *cxlr) { struct device *dev; int id = 0; + if (port == cxled_to_port(cxled)) + return &cxled->cxld; + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) dev = device_find_child(&port->dev, &cxlr->params, match_auto_decoder); @@ -853,10 +858,7 @@ static int cxl_rr_alloc_decoder(struct cxl_port *port, struct cxl_region *cxlr, { struct cxl_decoder *cxld; - if (port == cxled_to_port(cxled)) - cxld = &cxled->cxld; - else - cxld = cxl_region_find_decoder(port, cxlr); + cxld = cxl_region_find_decoder(port, cxled, cxlr); if (!cxld) { dev_dbg(&cxlr->dev, "%s: no decoder available\n", dev_name(&port->dev)); From patchwork Wed Jan 31 21:59:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alison Schofield X-Patchwork-Id: 13540029 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBB9339AE6 for ; Wed, 31 Jan 2024 21:59:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706738379; cv=none; b=hFGUzEOvUhSXMPOi6tWT8zg2YdYyswohmEI5Pi2GmPCQRA5qK6W5+0aIJnXE/wKhhrvlbVK2E3vk3UX8EZXwth3h1UJh5IzvVxxaqBUT1y6qmq65bj3OeerJY0t8dh60jZyVBOqjYxb8GuHyGXdRrI8tMeSttLcYwo+gMefIC84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706738379; c=relaxed/simple; bh=02lEaTuW3XQSk1sKz7B8hwGtvjKcA5GXD6le+asivjw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hRmSslcpUxIKuPh8CJ0cbA6p2brNvcXsKap3rHxWYYV4fTgZwQZqj9Dw+VC118MnLIIy54nL1IioQUhb+Di4vblWrkmxNha2F7iIu04LU71CmT+zOoM0rFVvs0H26OloTABeRWi6oXQ8BauA4k4qO3+34pM5RQeaSJ11Y3Rx+4k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=I53zyK8F; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="I53zyK8F" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1706738378; x=1738274378; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=02lEaTuW3XQSk1sKz7B8hwGtvjKcA5GXD6le+asivjw=; b=I53zyK8F5+YN+R/F38DxvM4nItfjMT751dI6hWN6tuiaW7nhg/hZYE14 dceYV0rEh4rRumsEanMhecAmbmJv/aBQ06XZex7WBm27hDEnUiLc2z2rL cZJ/fsEemlyDEWlHGL+Xp0TuEcP+/dlUsiKSOAO8euy2PMCEXP3SJ0pRF 5McPZiXgJ8MYLy99X1gBAsU0OWdmp9cf6+VaawJyCWv1DZ/FvWqH/4EY+ Y/pxYbNnGgYionJhCA+pYEmyu5pR5nOPOW4n4KyLYv4xREMVSxfEa8ZMi XSv8uEPJHO7wLEBtobJey/HO+PCDT907iUoR11xoQRlxR0vqtHAYbmmnT g==; X-IronPort-AV: E=McAfee;i="6600,9927,10969"; a="2675453" X-IronPort-AV: E=Sophos;i="6.05,233,1701158400"; d="scan'208";a="2675453" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jan 2024 13:59:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10969"; a="931989698" X-IronPort-AV: E=Sophos;i="6.05,233,1701158400"; d="scan'208";a="931989698" Received: from aschofie-mobl2.amr.corp.intel.com (HELO localhost) ([10.209.39.51]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jan 2024 13:59:36 -0800 From: alison.schofield@intel.com To: Davidlohr Bueso , Jonathan Cameron , Dave Jiang , Alison Schofield , Vishal Verma , Ira Weiny , Dan Williams Cc: linux-cxl@vger.kernel.org, Wonjae Lee Subject: [PATCH v3 2/2] cxl/region: Allow out of order assembly of autodiscovered regions Date: Wed, 31 Jan 2024 13:59:31 -0800 Message-Id: <3dec69ee97524ab229a20c6739272c3000b18408.1706736863.git.alison.schofield@intel.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Alison Schofield Autodiscovered regions can fail to assemble if they are not discovered in HPA decode order. The user will see failure messages like: [] cxl region0: endpoint5: HPA order violation region1 [] cxl region0: endpoint5: failed to allocate region reference The check that is causing the failure helps the CXL driver enforce a CXL spec mandate that decoders be committed in HPA order. The check is needless for autodiscovered regions since their decoders are already programmed. Trying to enforce order in the assembly of these regions is useless because they are assembled once all their member endpoints arrive, and there is no guarantee on the order in which endpoints are discovered during probe. Keep the existing check, but for autodiscovered regions, allow the out of order assembly after a sanity check that the lesser numbered decoder has the lesser HPA starting address. Signed-off-by: Alison Schofield --- Changes in v3: - Get correct endpoint or switch decoders (Dan) drivers/cxl/core/region.c | 48 +++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 7af40d9cf80a..0b37e2e674e5 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -758,8 +758,31 @@ cxl_region_find_decoder(struct cxl_port *port, return to_cxl_decoder(dev); } -static struct cxl_region_ref *alloc_region_ref(struct cxl_port *port, - struct cxl_region *cxlr) +static bool auto_order_ok(struct cxl_port *port, struct cxl_region *cxlr_iter, + struct cxl_decoder *cxld) +{ + struct cxl_region_ref *rr = cxl_rr_load(port, cxlr_iter); + struct cxl_decoder *cxld_iter = rr->decoder; + + /* + * Allow the out of order assembly of auto-discovered regions. + * Per CXL Spec 3.1 8.2.4.20.12 software must commit decoders + * in HPA order. Confirm that the decoder with the lesser HPA + * starting address has the lesser id. + */ + dev_dbg(&cxld->dev, "check for HPA violation %s:%d < %s:%d\n", + dev_name(&cxld->dev), cxld->id, + dev_name(&cxld_iter->dev), cxld_iter->id); + + if (cxld_iter->id > cxld->id) + return true; + + return false; +} + +static struct cxl_region_ref * +alloc_region_ref(struct cxl_port *port, struct cxl_region *cxlr, + struct cxl_endpoint_decoder *cxled) { struct cxl_region_params *p = &cxlr->params; struct cxl_region_ref *cxl_rr, *iter; @@ -769,16 +792,21 @@ static struct cxl_region_ref *alloc_region_ref(struct cxl_port *port, xa_for_each(&port->regions, index, iter) { struct cxl_region_params *ip = &iter->region->params; - if (!ip->res) + if (!ip->res || ip->res->start < p->res->start) continue; - if (ip->res->start > p->res->start) { - dev_dbg(&cxlr->dev, - "%s: HPA order violation %s:%pr vs %pr\n", - dev_name(&port->dev), - dev_name(&iter->region->dev), ip->res, p->res); - return ERR_PTR(-EBUSY); + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) { + struct cxl_decoder *cxld; + + cxld = cxl_region_find_decoder(port, cxled, cxlr); + if (auto_order_ok(port, iter->region, cxld)) + continue; } + dev_dbg(&cxlr->dev, "%s: HPA order violation %s:%pr vs %pr\n", + dev_name(&port->dev), + dev_name(&iter->region->dev), ip->res, p->res); + + return ERR_PTR(-EBUSY); } cxl_rr = kzalloc(sizeof(*cxl_rr), GFP_KERNEL); @@ -955,7 +983,7 @@ static int cxl_port_attach_region(struct cxl_port *port, nr_targets_inc = true; } } else { - cxl_rr = alloc_region_ref(port, cxlr); + cxl_rr = alloc_region_ref(port, cxlr, cxled); if (IS_ERR(cxl_rr)) { dev_dbg(&cxlr->dev, "%s: failed to allocate region reference\n",