From patchwork Fri Nov 15 18:46:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876696 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE2F818787A for ; Fri, 15 Nov 2024 18:46:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696389; cv=none; b=ezD8m6iCTbZykZv5BWV2X32Cp+luGX9vEwU/WNJxlmErzEciqyN/rl1debLEkHeggqvKOcITNdSE36cZM3PzXeOXxNo0veAwMnXoPjKWGGREycA+SFdWDjnqb3Z/YRSigUvfGqcZW/70Pi5vG81I0HOwcRYPyWTvPLKP1m6m2do= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696389; c=relaxed/simple; bh=wD+XIhKFNUIAbTLMKPo9wsnzKTJTpx4Wre2NG4iL7dw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Sncl1K6LKkSihZE1j27fhEjgwhij/oMrSngewXKx7QeMMaKS08hOQUNMhrE3R473oSXLoFpuEcq1/DFsRpwRWN402wmGDnA6r4YCNNK+7RmIqML1WdozePDg3XZk7RpMIsXPou7HwUKEsKab8Sge2rU1DJlykJV2pj0G9ik3p5U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=K0lD1Kgz; arc=none smtp.client-ip=198.175.65.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="K0lD1Kgz" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696388; x=1763232388; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=wD+XIhKFNUIAbTLMKPo9wsnzKTJTpx4Wre2NG4iL7dw=; b=K0lD1KgzDerkQSpWQgjoQTmqiNgLDqivdUOF+V+DJG6um2mHK9jMtrE5 pzJmdnchnia6BiT+vgdLdMej0ah/gyV4b3wTPDf34QCawtkhc/YUA/F0B hsBu2QHQ1XgYn0PjONnLD71kgovE/GmCcpdzvINrJrnX7VktstD2FJRk2 feQ2mwTpZNjVwFTyA1oQ6B9rmEWmmxyb9DRYLoOOeENV+x8mSQ9+8ignw sDnV//lQ6toz4Q7phlnjqXy36YqapMn70+iBhfT+5RuZXP//eQuKhiGEE OPVV1jRB4+Nwo0sXVXbcTqY8xy683LWoLbECPb9yO/MLE0ek5POOg43sn Q==; X-CSE-ConnectionGUID: 1W4rANzTSPmO9/8KnFa9eA== X-CSE-MsgGUID: 4iauAU/wSZuhpdXuPvIM+Q== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="42794865" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="42794865" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by orvoesa105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:28 -0800 X-CSE-ConnectionGUID: jn1jpH9IQ0CC47raUqiM+Q== X-CSE-MsgGUID: FqH+eF6pQieQOMjCb9HOGA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="92715690" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:25 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:19 -0600 Subject: [ndctl PATCH v3 1/9] ndctl/cxl-events: Don't fail test until event counts are reported Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-1-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=1494; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=wD+XIhKFNUIAbTLMKPo9wsnzKTJTpx4Wre2NG4iL7dw=; b=h5rXdxx2f6NLIG13P8WgFf76OcfOZw8o6MvAoLFyhPcOvI60Texz+qsoJ0ZjEEvsBuB/+3Qp9 5jR/Q04Ki2pDYn+2F6+ZCrjvcTnkaHDoWHrPEH1CZs/w3fH6rkWkw/S X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= In testing DCD event modifications a failed cxl-event test lacked details on the event counts. This was because the greps were failing the test rather than the check against the counts. Suppress the grep failure and rely on event count checks for pass/fail of the test. Reviewed-by: Dave Jiang Signed-off-by: Ira Weiny --- test/cxl-events.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/cxl-events.sh b/test/cxl-events.sh index ff4f3fdff1d8f6fd80f093126a27bf14b52d167f..c216d6aa9148c938a649cb22656127b3df440039 100644 --- a/test/cxl-events.sh +++ b/test/cxl-events.sh @@ -71,10 +71,10 @@ echo 0 > /sys/kernel/tracing/tracing_on echo "TEST: Events seen" trace_out=$(cat /sys/kernel/tracing/trace) -num_overflow=$(grep -c "cxl_overflow" <<< "${trace_out}") -num_fatal=$(grep -c "log=Fatal" <<< "${trace_out}") -num_failure=$(grep -c "log=Failure" <<< "${trace_out}") -num_info=$(grep -c "log=Informational" <<< "${trace_out}") +num_overflow=$(grep -c "cxl_overflow" <<< "${trace_out}" || true) +num_fatal=$(grep -c "log=Fatal" <<< "${trace_out}" || true) +num_failure=$(grep -c "log=Failure" <<< "${trace_out}" || true) +num_info=$(grep -c "log=Informational" <<< "${trace_out}" || true) echo " LOG (Expected) : (Found)" echo " overflow ($num_overflow_expected) : $num_overflow" echo " Fatal ($num_fatal_expected) : $num_fatal" From patchwork Fri Nov 15 18:46:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876697 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E09661D5ABE for ; Fri, 15 Nov 2024 18:46:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696394; cv=none; b=E7w6oW6nN2rFnbyjs592h+MXjkQAygu4FpImXjc8xHtTq71XVBdTNz/+aeaorkRq+qaiqkQCdqBvoT7HJGAnQNHaQ2yVZwdBvBgUEzr6L8v6sFnDPzdrejl1y6dv5o2m5ELwvphK6pURylr2P0eUs2nurjE3x+P+r3hfFuuCC/A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696394; c=relaxed/simple; bh=YMdKA2RGM+4rycwcwrUUs1tZXNNeH8g15UUiSnncD/0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=TE02agfB1sUthpEkfPSSfsOwRdE/6Ii0HUAZoYxI38MtQFb9o5tEyd4gDrDfoH0LiCijel7mdDexjkwXf/2nuzKBeOi8jWhc6eWU/XConO2I39wEBRCH6M+OVsbP4PJl6OERZMovP+6/inYmWwJDFmc4vpee/ccug8cYqKhfrQw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZaggTBsV; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZaggTBsV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696393; x=1763232393; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=YMdKA2RGM+4rycwcwrUUs1tZXNNeH8g15UUiSnncD/0=; b=ZaggTBsV2fVrpGq4QfTSWibrgxdJ2QFcljN+vvWxocfh7ci2vxnxBabl SIGigXEupjEowHODFTQmQKubJjoCJkAXjJkV6iQVkzehpbgHIB87xYzQr b+MNTl1SdokJNyvQAS8txgnBw6USePV0erpzU9SkEYN6IRGVIyLQG9/BR u1i186+gFVa9kZY8pC0eR0NiWARPhQjaWuLfVAoY4Nt+Pd6z9qgjgolDG 0GBxNvzs/M46szSljVZbuh3gz4jfJZ7jSoDTIStDIU87Eb2oPMP+UBxk3 EKIGkmF4fOXl34179d9juRQobsSQoVnFtgs8Z2iQyBxpaZHWuKx+Hzte4 A==; X-CSE-ConnectionGUID: cX6hASuHRyOPS4joSD/ykA== X-CSE-MsgGUID: 0OWXlEdqQKW6UAJu4SBreg== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848430" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848430" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:29 -0800 X-CSE-ConnectionGUID: DnSobLYEQOS2HchTwCZRMQ== X-CSE-MsgGUID: xpOkvP3lQrOHBBAQriFQeQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392835" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:28 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:20 -0600 Subject: [ndctl PATCH v3 2/9] ndctl/cxl/region: Report max size for region creation Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-2-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=1059; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=YMdKA2RGM+4rycwcwrUUs1tZXNNeH8g15UUiSnncD/0=; b=xlhrpYAXv38PnwN9ztjfoZwGqOdXWArEcghreRSvqgyKMJl05RKr8QfOztRvYHcp+Md5VRzxg RYJwXeK9fsZDjdSnNBxKgkUCjc6SSzTcoXHXl3iEffRqLIylht+zmqJ X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= When creating a region if the size exceeds the max an error is printed. However, the max available space is not reported which makes it harder to determine what is wrong. Add the max size available to the output error. Reviewed-by: Dave Jiang Reviewed-by: Fan Ni Signed-off-by: Ira Weiny --- cxl/region.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cxl/region.c b/cxl/region.c index 96aa5931d2281c7577679b7f6165218964fa0425..207cf2d003148992255c715f286bc0f38de2ca84 100644 --- a/cxl/region.c +++ b/cxl/region.c @@ -677,8 +677,8 @@ static int create_region(struct cxl_ctx *ctx, int *count, } if (!default_size && size > max_extent) { log_err(&rl, - "%s: region size %#lx exceeds max available space\n", - cxl_decoder_get_devname(p->root_decoder), size); + "%s: region size %#lx exceeds max available space (%#lx)\n", + cxl_decoder_get_devname(p->root_decoder), size, max_extent); return -ENOSPC; } From patchwork Fri Nov 15 18:46:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876698 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4AD21CEE8D for ; Fri, 15 Nov 2024 18:46:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696395; cv=none; b=m7DosQuXPk6Rg+MKJEbUNiNC/NN+rq4+/8vUUnd2lC1Yt5BNPuGdiZFY9VgllZ9kFv33WbgKMMLSzDf17s+ownuvEZuAYNrDgGcPS3Me99ryER5Iui5Um3igs+TMqB3kOh31rbQyUnXWDVWP1tM30uPLyqVZU5Nj8oSSGFvNG7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696395; c=relaxed/simple; bh=nX4+TBmzT/8S/6xSSe7TLFYuuRvqvwN/md0J6OgzWuI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=GcqEZIqDU+B6nhQoLrg//BI6XK++Kza0RLwXIm1dBkfdAK+ROgvmvnrnsOlcN7W+RwyMirzeIxddPxGUuzXCaoohYeQZ2sQYr1YnSGlNmRcBz/dKhH6Vy6NIf0x/zbS9EaGBlu/WE6VfTuFV3qdtoBz5fzinczvmGwEpmQVMgG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=UbX/X1uG; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="UbX/X1uG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696394; x=1763232394; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=nX4+TBmzT/8S/6xSSe7TLFYuuRvqvwN/md0J6OgzWuI=; b=UbX/X1uGnTKJabloPTSzfpO9VGrVvBtxZ/tBpt/b++IyyF+BC1rwei8j Alcb9JhByD5f4ELMe/sgdBFuPGRYJR0HBJfJd2LD/PsMKKZ3k0W7YwQ9m tBcJf/1qClRiMxB1eyrBUcg85A0EoC9ikO0Zu3ehFK7k6yDoUwtPmCUKc T7RcUJT+cUf33Fu3xroeo7ztD8iIeeBYfMGJUUMo//8n80qmdSMDtH3kn s0+IceUnTwkuIUVxRCqAhz8um2JYLjTgSieUqi0fjIL7ph3Ds487VVO/q 7l5VhGVZL9Q48208pzDfGf3G7R1EZB7XjSJ5FWNvcg6RLa2TekWzC1F8G Q==; X-CSE-ConnectionGUID: 3g38gRPoTFODE/RU0iVWsw== X-CSE-MsgGUID: JBWf3kMTRouVzMxhCvLStA== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848440" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848440" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:31 -0800 X-CSE-ConnectionGUID: RHh4eDWdTQSclNexuGG/xg== X-CSE-MsgGUID: D6auNIb+SQimGJlyTRMjfQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392849" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:30 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:21 -0600 Subject: [ndctl PATCH v3 3/9] libcxl: Separate region mode from decoder mode Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-3-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=9745; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=nX4+TBmzT/8S/6xSSe7TLFYuuRvqvwN/md0J6OgzWuI=; b=hUwPv21EkGSjKG2rN1+f7kmqnd9OZZO4NLU2CV3k1aUo8ggqCJlmwvBm75fZJVEFHxeNKVQMQ yb62hN1H5rMD+vF4UcgdGh75bZ/5QynBdfW62Cm2I+rjk2Xr7k1qP/C X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= With the introduction of DCD, region mode and decoder mode no longer remain a 1:1 relation. An interleaved region may be made up of Dynamic Capacity partitions with different indexes on each of the target devices. Introduce a new region mode enumeration and access function to libcxl. To maintain compatibility with existing software the region mode values are defined the same as the current decoder mode. In addition cxl_region_get_mode() is retained. Signed-off-by: Ira Weiny --- Changes: [Alison/iweiny: Ensure correct libcxl symbol export version] [Alison/iweiny: Add new libcxl interfaces to documentation] [iweiny: Add deprecated cxl_region_get_mode() documentation] --- Documentation/cxl/lib/libcxl.txt | 20 ++++++++++++++++++-- cxl/lib/libcxl.c | 25 +++++++++++++++---------- cxl/lib/libcxl.sym | 5 +++++ cxl/lib/private.h | 2 +- cxl/libcxl.h | 35 +++++++++++++++++++++++++++++++++++ 5 files changed, 74 insertions(+), 13 deletions(-) diff --git a/Documentation/cxl/lib/libcxl.txt b/Documentation/cxl/lib/libcxl.txt index 40598a08b9f4840f79e3ab43f62f412d8b2136ed..d5c3558aacecb08d7f5754fdcc77d6e743560601 100644 --- a/Documentation/cxl/lib/libcxl.txt +++ b/Documentation/cxl/lib/libcxl.txt @@ -553,11 +553,20 @@ struct cxl_region *cxl_region_get_next(struct cxl_region *region); ===== REGION: Attributes ---- +enum cxl_region_mode { + CXL_REGION_MODE_NONE = CXL_DECODER_MODE_NONE, + CXL_REGION_MODE_MIXED = CXL_DECODER_MODE_MIXED, + CXL_REGION_MODE_PMEM = CXL_DECODER_MODE_PMEM, + CXL_REGION_MODE_RAM = CXL_DECODER_MODE_RAM, +}; +const char *cxl_region_mode_name(enum cxl_region_mode mode); +enum cxl_region_mode cxl_region_mode_from_ident(const char *ident); +enum cxl_region_mode cxl_region_get_region_mode(struct cxl_region *region); + int cxl_region_get_id(struct cxl_region *region); const char *cxl_region_get_devname(struct cxl_region *region); void cxl_region_get_uuid(struct cxl_region *region, uuid_t uu); unsigned long long cxl_region_get_size(struct cxl_region *region); -enum cxl_decoder_mode cxl_region_get_mode(struct cxl_region *region); unsigned long long cxl_region_get_resource(struct cxl_region *region); unsigned int cxl_region_get_interleave_ways(struct cxl_region *region); unsigned int cxl_region_get_interleave_granularity(struct cxl_region *region); @@ -576,8 +585,12 @@ int cxl_region_clear_all_targets(struct cxl_region *region); int cxl_region_decode_commit(struct cxl_region *region); int cxl_region_decode_reset(struct cxl_region *region); struct daxctl_region *cxl_region_get_daxctl_region(struct cxl_region *region); ----- +DEPRECATED: + +enum cxl_decoder_mode cxl_region_get_mode(struct cxl_region *region); + +---- A region's resource attribute is the Host Physical Address at which the region's address space starts. The region's address space is a subset of the parent root decoder's address space. @@ -601,6 +614,9 @@ where its properties can be interrogated by daxctl. The helper cxl_region_get_daxctl_region() returns an 'struct daxctl_region *' that can be used with other libdaxctl APIs. +Regions now have a mode distinct from decoders. cxl_region_get_mode() is +deprecated in favor of cxl_region_get_region_mode(). + include::../../copyright.txt[] SEE ALSO diff --git a/cxl/lib/libcxl.c b/cxl/lib/libcxl.c index 63aa4ef3acdc2fb3c4ec6c13be5feb802e817d0d..35a40091e8f5813c1b3ef2ffb931c9ec584b02ad 100644 --- a/cxl/lib/libcxl.c +++ b/cxl/lib/libcxl.c @@ -431,10 +431,10 @@ CXL_EXPORT bool cxl_region_qos_class_mismatch(struct cxl_region *region) if (!memdev) continue; - if (region->mode == CXL_DECODER_MODE_RAM) { + if (region->mode == CXL_REGION_MODE_RAM) { if (root_decoder->qos_class != memdev->ram_qos_class) return true; - } else if (region->mode == CXL_DECODER_MODE_PMEM) { + } else if (region->mode == CXL_REGION_MODE_PMEM) { if (root_decoder->qos_class != memdev->pmem_qos_class) return true; } @@ -619,9 +619,9 @@ static void *add_cxl_region(void *parent, int id, const char *cxlregion_base) sprintf(path, "%s/mode", cxlregion_base); if (sysfs_read_attr(ctx, path, buf) < 0) - region->mode = CXL_DECODER_MODE_NONE; + region->mode = CXL_REGION_MODE_NONE; else - region->mode = cxl_decoder_mode_from_ident(buf); + region->mode = cxl_region_mode_from_ident(buf); sprintf(path, "%s/modalias", cxlregion_base); if (sysfs_read_attr(ctx, path, buf) == 0) @@ -748,11 +748,16 @@ CXL_EXPORT unsigned long long cxl_region_get_resource(struct cxl_region *region) return region->start; } -CXL_EXPORT enum cxl_decoder_mode cxl_region_get_mode(struct cxl_region *region) +CXL_EXPORT enum cxl_region_mode cxl_region_get_region_mode(struct cxl_region *region) { return region->mode; } +CXL_EXPORT enum cxl_decoder_mode cxl_region_get_mode(struct cxl_region *region) +{ + return (enum cxl_decoder_mode)cxl_region_get_region_mode(region); +} + CXL_EXPORT unsigned int cxl_region_get_interleave_ways(struct cxl_region *region) { @@ -2700,7 +2705,7 @@ cxl_decoder_get_region(struct cxl_decoder *decoder) } static struct cxl_region *cxl_decoder_create_region(struct cxl_decoder *decoder, - enum cxl_decoder_mode mode) + enum cxl_region_mode mode) { struct cxl_ctx *ctx = cxl_decoder_get_ctx(decoder); char *path = decoder->dev_buf; @@ -2708,9 +2713,9 @@ static struct cxl_region *cxl_decoder_create_region(struct cxl_decoder *decoder, struct cxl_region *region; int rc; - if (mode == CXL_DECODER_MODE_PMEM) + if (mode == CXL_REGION_MODE_PMEM) sprintf(path, "%s/create_pmem_region", decoder->dev_path); - else if (mode == CXL_DECODER_MODE_RAM) + else if (mode == CXL_REGION_MODE_RAM) sprintf(path, "%s/create_ram_region", decoder->dev_path); rc = sysfs_read_attr(ctx, path, buf); @@ -2754,13 +2759,13 @@ static struct cxl_region *cxl_decoder_create_region(struct cxl_decoder *decoder, CXL_EXPORT struct cxl_region * cxl_decoder_create_pmem_region(struct cxl_decoder *decoder) { - return cxl_decoder_create_region(decoder, CXL_DECODER_MODE_PMEM); + return cxl_decoder_create_region(decoder, CXL_REGION_MODE_PMEM); } CXL_EXPORT struct cxl_region * cxl_decoder_create_ram_region(struct cxl_decoder *decoder) { - return cxl_decoder_create_region(decoder, CXL_DECODER_MODE_RAM); + return cxl_decoder_create_region(decoder, CXL_REGION_MODE_RAM); } CXL_EXPORT int cxl_decoder_get_nr_targets(struct cxl_decoder *decoder) diff --git a/cxl/lib/libcxl.sym b/cxl/lib/libcxl.sym index 0c155a40ad4765106f0eab1745281d462af782fe..17a660f508ad1e053af2992824535ccf7ce877b2 100644 --- a/cxl/lib/libcxl.sym +++ b/cxl/lib/libcxl.sym @@ -288,3 +288,8 @@ global: cxl_memdev_trigger_poison_list; cxl_region_trigger_poison_list; } LIBCXL_7; + +LIBCXL_9 { +global: + cxl_region_get_region_mode; +} LIBECXL_8; diff --git a/cxl/lib/private.h b/cxl/lib/private.h index b6cd910e93359b53cac34427acfe84c7abcb78b0..0f45be89b6a00477d13fb6d7f1906213a3073c48 100644 --- a/cxl/lib/private.h +++ b/cxl/lib/private.h @@ -171,7 +171,7 @@ struct cxl_region { unsigned int interleave_ways; unsigned int interleave_granularity; enum cxl_decode_state decode_state; - enum cxl_decoder_mode mode; + enum cxl_region_mode mode; struct daxctl_region *dax_region; struct kmod_module *module; struct list_head mappings; diff --git a/cxl/libcxl.h b/cxl/libcxl.h index 0a5fd0e13cc24e0032d4a83d780278fbe0038d32..06b87a0924faafec6c80eca83ea7551d4e117256 100644 --- a/cxl/libcxl.h +++ b/cxl/libcxl.h @@ -303,6 +303,39 @@ int cxl_memdev_is_enabled(struct cxl_memdev *memdev); for (endpoint = cxl_endpoint_get_first(port); endpoint != NULL; \ endpoint = cxl_endpoint_get_next(endpoint)) +enum cxl_region_mode { + CXL_REGION_MODE_NONE = CXL_DECODER_MODE_NONE, + CXL_REGION_MODE_MIXED = CXL_DECODER_MODE_MIXED, + CXL_REGION_MODE_PMEM = CXL_DECODER_MODE_PMEM, + CXL_REGION_MODE_RAM = CXL_DECODER_MODE_RAM, +}; + +static inline const char *cxl_region_mode_name(enum cxl_region_mode mode) +{ + static const char *names[] = { + [CXL_REGION_MODE_NONE] = "none", + [CXL_REGION_MODE_MIXED] = "mixed", + [CXL_REGION_MODE_PMEM] = "pmem", + [CXL_REGION_MODE_RAM] = "ram", + }; + + if (mode < CXL_REGION_MODE_NONE || mode > CXL_REGION_MODE_RAM) + mode = CXL_REGION_MODE_NONE; + return names[mode]; +} + +static inline enum cxl_region_mode +cxl_region_mode_from_ident(const char *ident) +{ + if (strcmp(ident, "ram") == 0) + return CXL_REGION_MODE_RAM; + else if (strcmp(ident, "volatile") == 0) + return CXL_REGION_MODE_RAM; + else if (strcmp(ident, "pmem") == 0) + return CXL_REGION_MODE_PMEM; + return CXL_REGION_MODE_NONE; +} + struct cxl_region; struct cxl_region *cxl_region_get_first(struct cxl_decoder *decoder); struct cxl_region *cxl_region_get_next(struct cxl_region *region); @@ -318,6 +351,8 @@ const char *cxl_region_get_devname(struct cxl_region *region); void cxl_region_get_uuid(struct cxl_region *region, uuid_t uu); unsigned long long cxl_region_get_size(struct cxl_region *region); unsigned long long cxl_region_get_resource(struct cxl_region *region); +enum cxl_region_mode cxl_region_get_region_mode(struct cxl_region *region); +/* Deprecated: use cxl_region_get_region_mode() */ enum cxl_decoder_mode cxl_region_get_mode(struct cxl_region *region); unsigned int cxl_region_get_interleave_ways(struct cxl_region *region); unsigned int cxl_region_get_interleave_granularity(struct cxl_region *region); From patchwork Fri Nov 15 18:46:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876699 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1371C1684AC for ; Fri, 15 Nov 2024 18:46:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696396; cv=none; b=O1S7dWySiFRA8e6pbJ9NkvS52lQepS8T6X13bGK3+CH1NCQuDbnW/ltP30/1nPDP7tBimV99jG5HVnqPYc/vDbHDaX6j6epHCpiD9JfXuBkHWyo23GQOnNvKV5DvZBXaan6JlN7BNRnRUzPFCjGskvaQbOrdUHEJSXtigfZ1rxk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696396; c=relaxed/simple; bh=HgvcFJJlwID2rYxZWQTHetcFY5nTyif5z/Ba07c9JXA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=eizuKmGrTjA2OTLllHEbgOE5xJGP0sSViB1+DNuI8hszxwN62a/Il23npRxsyWlCZiXHyBQd64mfQEhBP4W52czn/ooZdh03HLE1Qbayl7wbKsAEOK8MNQmnDT6NtWpetWcetnYkI7M/yBveKcYSpXReyuI5DkSt43EzLo1AW2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=JZXqaora; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="JZXqaora" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696395; x=1763232395; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=HgvcFJJlwID2rYxZWQTHetcFY5nTyif5z/Ba07c9JXA=; b=JZXqaoragNDFnBPWIMCfetf1JgrYleROlDnwK6M0azOwH2Ku7QzLu+Yw +xGequEHdrQMyS7r7K6A4aNq2jIclL6wikQV94BEQlO0wv7cxy/ln/MlI fzI2NgkHkVGsdtfstMuQqUWGeytTpIhWdbd/EI2vgbax45VZewv8Cdyrd hucchU+GYNveaqGZnNoMPv6xrf9pDeV4n8DMezhiT5R/2VZ/smCxkk9+h HZFUoFNkW2JE9p2UO8myXl2PHYWqI4OFqKZIBWI7qZsuNGUhaPrRmXCSf wZkVrxBGIUJ3ODQeAqQEpvtqMUwuPlKwHhf5A0y7g0YX6+4dUS4IQayHS Q==; X-CSE-ConnectionGUID: XHmpNREjTlWoK2cV1WdT2A== X-CSE-MsgGUID: i6gIPO5rSuO+aJ7H58OYgw== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848458" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848458" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:33 -0800 X-CSE-ConnectionGUID: lV48peE/QmyecOsi2ZBz8A== X-CSE-MsgGUID: Ya5hG+9TQs6THbp/Em89Rw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392874" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:31 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:22 -0600 Subject: [ndctl PATCH v3 4/9] cxl/region: Use new region mode in cxl-cli Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-4-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=6905; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=HgvcFJJlwID2rYxZWQTHetcFY5nTyif5z/Ba07c9JXA=; b=fOVM9MYIAfZqC+3+j9iQRET4pTFr2zr7OAG/dtGeTEJfvi/au9mC8IB/e8z2fQgTFgz5A3Fgu 9aZppdOm7/tDaooOISEGmOvnXe6BpdLWqG0MKiSAVwNJNUXkEusrpsj X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= With the introduction of DCD, region mode and decoder mode no longer remain a 1:1 relation. An interleaved region may be made up of Dynamic Capacity partitions with different indexes on each of the target devices. Modify cxl-cli to use the new region mode interface from libcxl. Modify parameter processing and variable name changes for clarity in the future. Functionality remains the same. Signed-off-by: Ira Weiny --- Changes: [Alison: split libcxl and cxl-cli changes] [iweiny: Process region mode based on --type] [iweiny: s/mode/decoder_mode for clarity] --- cxl/json.c | 6 +++--- cxl/region.c | 50 ++++++++++++++++++++++++++++++++------------------ 2 files changed, 35 insertions(+), 21 deletions(-) diff --git a/cxl/json.c b/cxl/json.c index 5066d3bed13f8fcc36ab8f0ea127685c246d94d7..dcd3cc28393faf7e8adf299a857531ecdeaac50a 100644 --- a/cxl/json.c +++ b/cxl/json.c @@ -1147,7 +1147,7 @@ void util_cxl_mappings_append_json(struct json_object *jregion, struct json_object *util_cxl_region_to_json(struct cxl_region *region, unsigned long flags) { - enum cxl_decoder_mode mode = cxl_region_get_mode(region); + enum cxl_region_mode mode = cxl_region_get_region_mode(region); const char *devname = cxl_region_get_devname(region); struct json_object *jregion, *jobj; u64 val; @@ -1174,8 +1174,8 @@ struct json_object *util_cxl_region_to_json(struct cxl_region *region, json_object_object_add(jregion, "size", jobj); } - if (mode != CXL_DECODER_MODE_NONE) { - jobj = json_object_new_string(cxl_decoder_mode_name(mode)); + if (mode != CXL_REGION_MODE_NONE) { + jobj = json_object_new_string(cxl_region_mode_name(mode)); if (jobj) json_object_object_add(jregion, "type", jobj); } diff --git a/cxl/region.c b/cxl/region.c index 207cf2d003148992255c715f286bc0f38de2ca84..527bd6708b162815068a95ddb360fce3914347de 100644 --- a/cxl/region.c +++ b/cxl/region.c @@ -49,7 +49,8 @@ struct parsed_params { int argc; const char **argv; struct cxl_decoder *root_decoder; - enum cxl_decoder_mode mode; + enum cxl_decoder_mode decoder_mode; + enum cxl_region_mode region_mode; bool enforce_qos; }; @@ -301,19 +302,28 @@ static int parse_create_options(struct cxl_ctx *ctx, int count, return -ENXIO; p->num_memdevs = json_object_array_length(p->memdevs); + p->region_mode = CXL_REGION_MODE_NONE; if (param.type) { - p->mode = cxl_decoder_mode_from_ident(param.type); - if (p->mode == CXL_DECODER_MODE_RAM && param.uuid) { + p->region_mode = cxl_region_mode_from_ident(param.type); + if (p->region_mode == CXL_REGION_MODE_RAM && param.uuid) { log_err(&rl, "can't set UUID for ram / volatile regions"); goto err; } - if (p->mode == CXL_DECODER_MODE_NONE) { + if (p->region_mode == CXL_REGION_MODE_NONE) { log_err(&rl, "unsupported type: %s\n", param.type); goto err; } - } else { - p->mode = CXL_DECODER_MODE_PMEM; + } + + switch (p->region_mode) { + case CXL_REGION_MODE_RAM: + p->decoder_mode = CXL_DECODER_MODE_RAM; + break; + case CXL_REGION_MODE_PMEM: + default: + p->decoder_mode = CXL_DECODER_MODE_PMEM; + break; } if (param.size) { @@ -410,7 +420,7 @@ static void collect_minsize(struct cxl_ctx *ctx, struct parsed_params *p) struct cxl_memdev *memdev = json_object_get_userdata(jobj); u64 size = 0; - switch(p->mode) { + switch(p->decoder_mode) { case CXL_DECODER_MODE_RAM: size = cxl_memdev_get_ram_size(memdev); break; @@ -446,7 +456,7 @@ static int create_region_validate_qos_class(struct parsed_params *p) json_object_array_get_idx(p->memdevs, i); struct cxl_memdev *memdev = json_object_get_userdata(jobj); - if (p->mode == CXL_DECODER_MODE_RAM) + if (p->decoder_mode == CXL_DECODER_MODE_RAM) qos_class = cxl_memdev_get_ram_qos_class(memdev); else qos_class = cxl_memdev_get_pmem_qos_class(memdev); @@ -475,7 +485,7 @@ static int validate_decoder(struct cxl_decoder *decoder, const char *devname = cxl_decoder_get_devname(decoder); int rc; - switch(p->mode) { + switch(p->decoder_mode) { case CXL_DECODER_MODE_RAM: if (!cxl_decoder_is_volatile_capable(decoder)) { log_err(&rl, "%s is not volatile capable\n", devname); @@ -512,10 +522,14 @@ static void set_type_from_decoder(struct cxl_ctx *ctx, struct parsed_params *p) * default to pmem if both types are set, otherwise the single * capability dominates. */ - if (cxl_decoder_is_volatile_capable(p->root_decoder)) - p->mode = CXL_DECODER_MODE_RAM; - if (cxl_decoder_is_pmem_capable(p->root_decoder)) - p->mode = CXL_DECODER_MODE_PMEM; + if (cxl_decoder_is_volatile_capable(p->root_decoder)) { + p->decoder_mode = CXL_DECODER_MODE_RAM; + p->region_mode = CXL_REGION_MODE_RAM; + } + if (cxl_decoder_is_pmem_capable(p->root_decoder)) { + p->decoder_mode = CXL_DECODER_MODE_PMEM; + p->region_mode = CXL_REGION_MODE_PMEM; + } } static int create_region_validate_config(struct cxl_ctx *ctx, @@ -685,14 +699,14 @@ static int create_region(struct cxl_ctx *ctx, int *count, if (size > max_extent) size = ALIGN_DOWN(max_extent, SZ_256M * p->ways); - if (p->mode == CXL_DECODER_MODE_PMEM) { + if (p->region_mode == CXL_REGION_MODE_PMEM) { region = cxl_decoder_create_pmem_region(p->root_decoder); if (!region) { log_err(&rl, "failed to create region under %s\n", param.root_decoder); return -ENXIO; } - } else if (p->mode == CXL_DECODER_MODE_RAM) { + } else if (p->region_mode == CXL_REGION_MODE_RAM) { region = cxl_decoder_create_ram_region(p->root_decoder); if (!region) { log_err(&rl, "failed to create region under %s\n", @@ -714,7 +728,7 @@ static int create_region(struct cxl_ctx *ctx, int *count, try(cxl_region, set_interleave_granularity, region, granularity); try(cxl_region, set_interleave_ways, region, p->ways); - if (p->mode == CXL_DECODER_MODE_PMEM) { + if (p->region_mode == CXL_REGION_MODE_PMEM) { if (!param.uuid) uuid_generate(p->uuid); try(cxl_region, set_uuid, region, p->uuid); @@ -732,14 +746,14 @@ static int create_region(struct cxl_ctx *ctx, int *count, rc = -ENXIO; goto out; } - if (cxl_decoder_get_mode(ep_decoder) != p->mode) { + if (cxl_decoder_get_mode(ep_decoder) != p->decoder_mode) { /* * The cxl_memdev_find_decoder() helper returns a free * decoder whose size has been checked for 0. * Thus it is safe to change the mode here if needed. */ try(cxl_decoder, set_dpa_size, ep_decoder, 0); - try(cxl_decoder, set_mode, ep_decoder, p->mode); + try(cxl_decoder, set_mode, ep_decoder, p->decoder_mode); } try(cxl_decoder, set_dpa_size, ep_decoder, size/p->ways); rc = cxl_region_set_target(region, i, ep_decoder); From patchwork Fri Nov 15 18:46:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876700 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D8391D90CB for ; Fri, 15 Nov 2024 18:46:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696398; cv=none; b=KvyB9wzPdL4hRnZBOtPLUbrM9Npv14gCsDgTJ//oQlUqYapZPjNgscbYIsXWS4nhmLjRz85N+yXG7N6GAcFFX0/gKcYzVwKHHieCz/f7kAxjcdfWYqRwzo2CvRAyZYSVONOyGPEpYeGyPCIODPPhHMvAyukYGm5uLsjpGzeme+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696398; c=relaxed/simple; bh=YmJ3GPJfG2ZcPPrwjtNyeIKNB8tHqC8nBIvqPi44NMc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ZQuYamCSZSZDOYPtOYScm51kTx8HT/0GWxxNxNvJ1p/lVF8mopgFZCti3vFSwBjQJgVUEt9t32nAgULkzEssQE0IobQ1JxpyAus42hQPPsbfZv6k51/x0ORddyM5BXYogiB52B/+TBpR7jRJI7m4o/NGNGIQOHBZPUq5vpGYqno= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=BcACeeEA; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="BcACeeEA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696397; x=1763232397; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=YmJ3GPJfG2ZcPPrwjtNyeIKNB8tHqC8nBIvqPi44NMc=; b=BcACeeEAO/wjdR3zaK3zkU7nGlOvXDjTepPxv3HfRcp+nare3EC8EO+y q1SxWQkBemofQlN0AOAOxAu65/FEYxslytWaJcwjdIIglfm4+qIsRCfu8 a+E4ZdA4t7beaC7N3keQwBiUiE96kgbjp+VxynKNa5uAr9huq8xq80a00 WCjgxGh5YWkrvrXJ2YJNkfO44MV1+ve1U7HMyJ/5BZDfpsH5nMS4EpzCu hMMtNywY0a6pCWrBnP3TfjlvhV3GuIgWizTXm+H73SH1X7VDZccwxRrTl pNt7v1rl3iuxG1HNi4phcojXeZqvNef18qT/HXTRfjMkgil1In4QpdnEP w==; X-CSE-ConnectionGUID: CB1VM1qhRcamCCG9/qA4Ww== X-CSE-MsgGUID: k5Thiid/TfmgwLPC9KvQwA== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848465" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848465" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:35 -0800 X-CSE-ConnectionGUID: bIu2oi79T3yqUwSn06abSA== X-CSE-MsgGUID: JLWALOFjS76dlVuwyyNxtA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392887" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:34 -0800 From: ira.weiny@intel.com Date: Fri, 15 Nov 2024 12:46:23 -0600 Subject: [ndctl PATCH v3 5/9] libcxl: Add Dynamic Capacity region support Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-5-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny , Sushant1 Kumar X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=16640; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=k4ri6zLhmakcgPtxLGqpICruyufD9Ijx1p8zKJmXfkk=; b=U34EsI+XNd8QUJQJn33SCSAw24Ca3LRRxMB5gBh9A8Ej8rG3BgCl9nRV9EFH2ncgMk4oG69nr XgJy3NFFy/XDdeRshAmhhm+zbr7VhvzzXrwRuIkbysgtuVL9zEYsbFS X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= From: Navneet Singh CXL Dynamic Capacity Devices (DCDs) optionally support dynamic capacity with up to eight partitions (Regions) (dc0-dc7). CXL regions can now be sparse and defined as dynamic capacity (dc). Add support for DCD devices and regions to libcxl. Add documentation for the new interfaces. Signed-off-by: Navneet Singh Co-developed-by: Sushant1 Kumar Signed-off-by: Sushant1 Kumar Co-developed-by: Ira Weiny Signed-off-by: Ira Weiny --- Changes: [djiang: remove errant white space changes] [djiang: use for loop for dc capability sets] [djiang: change cxl_memdev_get_dc_size() to take unsigned int] [djiang: change cxl_decoder_is_dc_capable() to take unsigned int] [Alison: s/Co-Authored-by/Co-developed-by/] [Alison: document new libcxl calls] [Alison: reduce indentation] [Alison: Fix libcxl export symbols] [iweiny: Add get mode documentation] [iweiny: add docs for DC support] [iweiny: use decoder_mode rather than index in libcxl calls] --- Documentation/cxl/lib/libcxl.txt | 17 ++++++- cxl/lib/libcxl.c | 98 ++++++++++++++++++++++++++++++++++++++++ cxl/lib/libcxl.sym | 3 ++ cxl/lib/private.h | 4 ++ cxl/libcxl.h | 52 ++++++++++++++++++++- 5 files changed, 171 insertions(+), 3 deletions(-) diff --git a/Documentation/cxl/lib/libcxl.txt b/Documentation/cxl/lib/libcxl.txt index d5c3558aacecb08d7f5754fdcc77d6e743560601..abca08fc81e6e84d176facafad6decae2f875880 100644 --- a/Documentation/cxl/lib/libcxl.txt +++ b/Documentation/cxl/lib/libcxl.txt @@ -68,6 +68,7 @@ int cxl_memdev_get_major(struct cxl_memdev *memdev); int cxl_memdev_get_minor(struct cxl_memdev *memdev); unsigned long long cxl_memdev_get_pmem_size(struct cxl_memdev *memdev); unsigned long long cxl_memdev_get_ram_size(struct cxl_memdev *memdev); +unsigned long long cxl_memdev_get_dc_size(struct cxl_memdev *memdev, enum cxl_decoder_mode mode); const char *cxl_memdev_get_firmware_version(struct cxl_memdev *memdev); size_t cxl_memdev_get_label_size(struct cxl_memdev *memdev); int cxl_memdev_nvdimm_bridge_active(struct cxl_memdev *memdev); @@ -422,11 +423,23 @@ enum cxl_decoder_mode { CXL_DECODER_MODE_MIXED, CXL_DECODER_MODE_PMEM, CXL_DECODER_MODE_RAM, + CXL_DECODER_MODE_DC0, + CXL_DECODER_MODE_DC1, + CXL_DECODER_MODE_DC2, + CXL_DECODER_MODE_DC3, + CXL_DECODER_MODE_DC4, + CXL_DECODER_MODE_DC5, + CXL_DECODER_MODE_DC6, + CXL_DECODER_MODE_DC7, }; enum cxl_decoder_mode cxl_decoder_get_mode(struct cxl_decoder *decoder); +const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode); +enum cxl_decoder_mode cxl_decoder_mode_from_ident(const char *ident); +bool cxl_decoder_mode_is_dc(enum cxl_decoder_mode mode); int cxl_decoder_set_mode(struct cxl_decoder *decoder, enum cxl_decoder_mode mode); bool cxl_decoder_is_pmem_capable(struct cxl_decoder *decoder); +bool cxl_decoder_is_dc_capable(struct cxl_decoder *decoder, enum cxl_decoder_mode mode); bool cxl_decoder_is_volatile_capable(struct cxl_decoder *decoder); bool cxl_decoder_is_mem_capable(struct cxl_decoder *decoder); bool cxl_decoder_is_accelmem_capable(struct cxl_decoder *decoder); @@ -558,6 +571,7 @@ enum cxl_region_mode { CXL_REGION_MODE_MIXED = CXL_DECODER_MODE_MIXED, CXL_REGION_MODE_PMEM = CXL_DECODER_MODE_PMEM, CXL_REGION_MODE_RAM = CXL_DECODER_MODE_RAM, + CXL_REGION_MODE_DC, }; const char *cxl_region_mode_name(enum cxl_region_mode mode); enum cxl_region_mode cxl_region_mode_from_ident(const char *ident); @@ -615,7 +629,8 @@ cxl_region_get_daxctl_region() returns an 'struct daxctl_region *' that can be used with other libdaxctl APIs. Regions now have a mode distinct from decoders. cxl_region_get_mode() is -deprecated in favor of cxl_region_get_region_mode(). +deprecated in favor of cxl_region_get_region_mode(). Dynamic capacity regions +require the use of cxl_region_get_region_mode(). include::../../copyright.txt[] diff --git a/cxl/lib/libcxl.c b/cxl/lib/libcxl.c index 35a40091e8f5813c1b3ef2ffb931c9ec584b02ad..df250db9dbacb2f0f34e8a592ce194159584fe4f 100644 --- a/cxl/lib/libcxl.c +++ b/cxl/lib/libcxl.c @@ -1304,6 +1304,19 @@ static void *add_cxl_memdev(void *parent, int id, const char *cxlmem_base) else memdev->ram_qos_class = atoi(buf); + memdev->dc_partition_count = 0; + for (int partition = 0; partition < MAX_NUM_DC_REGIONS; partition++) { + sprintf(path, "%s/dc%d/size", cxlmem_base, partition); + if (sysfs_read_attr(ctx, path, buf) < 0) + continue; + memdev->dc_size[partition] = strtoull(buf, NULL, 0); + memdev->dc_partition_count++; + sprintf(path, "%s/dc%d/qos_class", cxlmem_base, partition); + if (sysfs_read_attr(ctx, path, buf) < 0) + continue; + memdev->dc_qos_class[partition] = strtoull(buf, NULL, 0); + } + sprintf(path, "%s/payload_max", cxlmem_base); if (sysfs_read_attr(ctx, path, buf) == 0) { memdev->payload_max = strtoull(buf, NULL, 0); @@ -1540,6 +1553,23 @@ CXL_EXPORT int cxl_memdev_get_ram_qos_class(struct cxl_memdev *memdev) return memdev->ram_qos_class; } +static int cxl_decoder_dc_mode_to_index(enum cxl_decoder_mode mode) +{ + if (mode < CXL_DECODER_MODE_DC0 || mode > CXL_DECODER_MODE_DC7) + return -EINVAL; + return mode - CXL_DECODER_MODE_DC0; +} + +CXL_EXPORT unsigned long long cxl_memdev_get_dc_size(struct cxl_memdev *memdev, + enum cxl_decoder_mode mode) +{ + int index = cxl_decoder_dc_mode_to_index(mode); + + if (index < 0) + return 0; + return memdev->dc_size[index]; +} + CXL_EXPORT const char *cxl_memdev_get_firmware_verison(struct cxl_memdev *memdev) { return memdev->firmware_version; @@ -2275,6 +2305,22 @@ static void *add_cxl_decoder(void *parent, int id, const char *cxldecoder_base) decoder->mode = CXL_DECODER_MODE_RAM; else if (strcmp(buf, "pmem") == 0) decoder->mode = CXL_DECODER_MODE_PMEM; + else if (strcmp(buf, "dc0") == 0) + decoder->mode = CXL_DECODER_MODE_DC0; + else if (strcmp(buf, "dc1") == 0) + decoder->mode = CXL_DECODER_MODE_DC1; + else if (strcmp(buf, "dc2") == 0) + decoder->mode = CXL_DECODER_MODE_DC2; + else if (strcmp(buf, "dc3") == 0) + decoder->mode = CXL_DECODER_MODE_DC3; + else if (strcmp(buf, "dc4") == 0) + decoder->mode = CXL_DECODER_MODE_DC4; + else if (strcmp(buf, "dc5") == 0) + decoder->mode = CXL_DECODER_MODE_DC5; + else if (strcmp(buf, "dc6") == 0) + decoder->mode = CXL_DECODER_MODE_DC6; + else if (strcmp(buf, "dc7") == 0) + decoder->mode = CXL_DECODER_MODE_DC7; else if (strcmp(buf, "mixed") == 0) decoder->mode = CXL_DECODER_MODE_MIXED; else if (strcmp(buf, "none") == 0) @@ -2318,6 +2364,8 @@ static void *add_cxl_decoder(void *parent, int id, const char *cxldecoder_base) case CXL_PORT_SWITCH: decoder->pmem_capable = true; decoder->volatile_capable = true; + for (unsigned index = 0; index < MAX_NUM_DC_REGIONS; index++) + decoder->dc_capable[index] = true; decoder->mem_capable = true; decoder->accelmem_capable = true; sprintf(path, "%s/locked", cxldecoder_base); @@ -2341,6 +2389,14 @@ static void *add_cxl_decoder(void *parent, int id, const char *cxldecoder_base) { "cap_type2", &decoder->accelmem_capable }, { "cap_type3", &decoder->mem_capable }, { "cap_ram", &decoder->volatile_capable }, + { "cap_ram", &decoder->dc_capable[0] }, + { "cap_ram", &decoder->dc_capable[1] }, + { "cap_ram", &decoder->dc_capable[2] }, + { "cap_ram", &decoder->dc_capable[3] }, + { "cap_ram", &decoder->dc_capable[4] }, + { "cap_ram", &decoder->dc_capable[5] }, + { "cap_ram", &decoder->dc_capable[6] }, + { "cap_ram", &decoder->dc_capable[7] }, { "cap_pmem", &decoder->pmem_capable }, { "locked", &decoder->locked }, }; @@ -2592,6 +2648,30 @@ CXL_EXPORT int cxl_decoder_set_mode(struct cxl_decoder *decoder, case CXL_DECODER_MODE_RAM: sprintf(buf, "ram"); break; + case CXL_DECODER_MODE_DC0: + sprintf(buf, "dc0"); + break; + case CXL_DECODER_MODE_DC1: + sprintf(buf, "dc1"); + break; + case CXL_DECODER_MODE_DC2: + sprintf(buf, "dc2"); + break; + case CXL_DECODER_MODE_DC3: + sprintf(buf, "dc3"); + break; + case CXL_DECODER_MODE_DC4: + sprintf(buf, "dc4"); + break; + case CXL_DECODER_MODE_DC5: + sprintf(buf, "dc5"); + break; + case CXL_DECODER_MODE_DC6: + sprintf(buf, "dc6"); + break; + case CXL_DECODER_MODE_DC7: + sprintf(buf, "dc7"); + break; default: err(ctx, "%s: unsupported mode: %d\n", cxl_decoder_get_devname(decoder), mode); @@ -2648,6 +2728,16 @@ CXL_EXPORT bool cxl_decoder_is_mem_capable(struct cxl_decoder *decoder) return decoder->mem_capable; } +CXL_EXPORT bool cxl_decoder_is_dc_capable(struct cxl_decoder *decoder, + enum cxl_decoder_mode mode) +{ + int index = cxl_decoder_dc_mode_to_index(mode); + + if (index < 0) + return false; + return decoder->dc_capable[index]; +} + CXL_EXPORT bool cxl_decoder_is_accelmem_capable(struct cxl_decoder *decoder) { return decoder->accelmem_capable; @@ -2717,6 +2807,8 @@ static struct cxl_region *cxl_decoder_create_region(struct cxl_decoder *decoder, sprintf(path, "%s/create_pmem_region", decoder->dev_path); else if (mode == CXL_REGION_MODE_RAM) sprintf(path, "%s/create_ram_region", decoder->dev_path); + else if (mode == CXL_REGION_MODE_DC) + sprintf(path, "%s/create_dc_region", decoder->dev_path); rc = sysfs_read_attr(ctx, path, buf); if (rc < 0) { @@ -2768,6 +2860,12 @@ cxl_decoder_create_ram_region(struct cxl_decoder *decoder) return cxl_decoder_create_region(decoder, CXL_REGION_MODE_RAM); } +CXL_EXPORT struct cxl_region * +cxl_decoder_create_dc_region(struct cxl_decoder *decoder) +{ + return cxl_decoder_create_region(decoder, CXL_REGION_MODE_DC); +} + CXL_EXPORT int cxl_decoder_get_nr_targets(struct cxl_decoder *decoder) { return decoder->nr_targets; diff --git a/cxl/lib/libcxl.sym b/cxl/lib/libcxl.sym index 17a660f508ad1e053af2992824535ccf7ce877b2..fdb227789985443a13c72751bbd42ab383db5f97 100644 --- a/cxl/lib/libcxl.sym +++ b/cxl/lib/libcxl.sym @@ -292,4 +292,7 @@ global: LIBCXL_9 { global: cxl_region_get_region_mode; + cxl_memdev_get_dc_size; + cxl_decoder_is_dc_capable; + cxl_decoder_create_dc_region; } LIBECXL_8; diff --git a/cxl/lib/private.h b/cxl/lib/private.h index 0f45be89b6a00477d13fb6d7f1906213a3073c48..3efa230bfb632e6c6048aadd18f799b07d4bdfd3 100644 --- a/cxl/lib/private.h +++ b/cxl/lib/private.h @@ -47,6 +47,9 @@ struct cxl_memdev { struct list_node list; unsigned long long pmem_size; unsigned long long ram_size; + unsigned long long dc_size[MAX_NUM_DC_REGIONS]; + unsigned long long dc_qos_class[MAX_NUM_DC_REGIONS]; + int dc_partition_count; int ram_qos_class; int pmem_qos_class; int payload_max; @@ -140,6 +143,7 @@ struct cxl_decoder { bool pmem_capable; bool volatile_capable; bool mem_capable; + bool dc_capable[MAX_NUM_DC_REGIONS]; bool accelmem_capable; bool locked; enum cxl_decoder_target_type target_type; diff --git a/cxl/libcxl.h b/cxl/libcxl.h index 06b87a0924faafec6c80eca83ea7551d4e117256..d7f8a37816f236acd71fc834eae70a7a17a2721a 100644 --- a/cxl/libcxl.h +++ b/cxl/libcxl.h @@ -72,6 +72,9 @@ int cxl_memdev_get_minor(struct cxl_memdev *memdev); struct cxl_ctx *cxl_memdev_get_ctx(struct cxl_memdev *memdev); unsigned long long cxl_memdev_get_pmem_size(struct cxl_memdev *memdev); unsigned long long cxl_memdev_get_ram_size(struct cxl_memdev *memdev); +enum cxl_decoder_mode; +unsigned long long cxl_memdev_get_dc_size(struct cxl_memdev *memdev, + enum cxl_decoder_mode mode); int cxl_memdev_get_pmem_qos_class(struct cxl_memdev *memdev); int cxl_memdev_get_ram_qos_class(struct cxl_memdev *memdev); const char *cxl_memdev_get_firmware_verison(struct cxl_memdev *memdev); @@ -191,11 +194,20 @@ unsigned long long cxl_decoder_get_max_available_extent(struct cxl_decoder *decoder); int cxl_root_decoder_get_qos_class(struct cxl_decoder *decoder); +#define MAX_NUM_DC_REGIONS 8 enum cxl_decoder_mode { CXL_DECODER_MODE_NONE, CXL_DECODER_MODE_MIXED, CXL_DECODER_MODE_PMEM, CXL_DECODER_MODE_RAM, + CXL_DECODER_MODE_DC0, + CXL_DECODER_MODE_DC1, + CXL_DECODER_MODE_DC2, + CXL_DECODER_MODE_DC3, + CXL_DECODER_MODE_DC4, + CXL_DECODER_MODE_DC5, + CXL_DECODER_MODE_DC6, + CXL_DECODER_MODE_DC7, }; static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode) @@ -205,9 +217,17 @@ static inline const char *cxl_decoder_mode_name(enum cxl_decoder_mode mode) [CXL_DECODER_MODE_MIXED] = "mixed", [CXL_DECODER_MODE_PMEM] = "pmem", [CXL_DECODER_MODE_RAM] = "ram", + [CXL_DECODER_MODE_DC0] = "dc0", + [CXL_DECODER_MODE_DC1] = "dc1", + [CXL_DECODER_MODE_DC2] = "dc2", + [CXL_DECODER_MODE_DC3] = "dc3", + [CXL_DECODER_MODE_DC4] = "dc4", + [CXL_DECODER_MODE_DC5] = "dc5", + [CXL_DECODER_MODE_DC6] = "dc6", + [CXL_DECODER_MODE_DC7] = "dc7", }; - if (mode < CXL_DECODER_MODE_NONE || mode > CXL_DECODER_MODE_RAM) + if (mode < CXL_DECODER_MODE_NONE || mode > CXL_DECODER_MODE_DC7) mode = CXL_DECODER_MODE_NONE; return names[mode]; } @@ -221,9 +241,30 @@ cxl_decoder_mode_from_ident(const char *ident) return CXL_DECODER_MODE_RAM; else if (strcmp(ident, "pmem") == 0) return CXL_DECODER_MODE_PMEM; + else if (strcmp(ident, "dc0") == 0) + return CXL_DECODER_MODE_DC0; + else if (strcmp(ident, "dc1") == 0) + return CXL_DECODER_MODE_DC1; + else if (strcmp(ident, "dc2") == 0) + return CXL_DECODER_MODE_DC2; + else if (strcmp(ident, "dc3") == 0) + return CXL_DECODER_MODE_DC3; + else if (strcmp(ident, "dc4") == 0) + return CXL_DECODER_MODE_DC4; + else if (strcmp(ident, "dc5") == 0) + return CXL_DECODER_MODE_DC5; + else if (strcmp(ident, "dc6") == 0) + return CXL_DECODER_MODE_DC6; + else if (strcmp(ident, "dc7") == 0) + return CXL_DECODER_MODE_DC7; return CXL_DECODER_MODE_NONE; } +static inline bool cxl_decoder_mode_is_dc(enum cxl_decoder_mode mode) +{ + return (mode >= CXL_DECODER_MODE_DC0 && mode <= CXL_DECODER_MODE_DC7); +} + enum cxl_decoder_mode cxl_decoder_get_mode(struct cxl_decoder *decoder); int cxl_decoder_set_mode(struct cxl_decoder *decoder, enum cxl_decoder_mode mode); @@ -248,6 +289,8 @@ enum cxl_decoder_target_type { enum cxl_decoder_target_type cxl_decoder_get_target_type(struct cxl_decoder *decoder); bool cxl_decoder_is_pmem_capable(struct cxl_decoder *decoder); +bool cxl_decoder_is_dc_capable(struct cxl_decoder *decoder, + enum cxl_decoder_mode mode); bool cxl_decoder_is_volatile_capable(struct cxl_decoder *decoder); bool cxl_decoder_is_mem_capable(struct cxl_decoder *decoder); bool cxl_decoder_is_accelmem_capable(struct cxl_decoder *decoder); @@ -258,6 +301,7 @@ unsigned int cxl_decoder_get_interleave_ways(struct cxl_decoder *decoder); struct cxl_region *cxl_decoder_get_region(struct cxl_decoder *decoder); struct cxl_region *cxl_decoder_create_pmem_region(struct cxl_decoder *decoder); struct cxl_region *cxl_decoder_create_ram_region(struct cxl_decoder *decoder); +struct cxl_region *cxl_decoder_create_dc_region(struct cxl_decoder *decoder); struct cxl_decoder *cxl_decoder_get_by_name(struct cxl_ctx *ctx, const char *ident); struct cxl_memdev *cxl_decoder_get_memdev(struct cxl_decoder *decoder); @@ -308,6 +352,7 @@ enum cxl_region_mode { CXL_REGION_MODE_MIXED = CXL_DECODER_MODE_MIXED, CXL_REGION_MODE_PMEM = CXL_DECODER_MODE_PMEM, CXL_REGION_MODE_RAM = CXL_DECODER_MODE_RAM, + CXL_REGION_MODE_DC, }; static inline const char *cxl_region_mode_name(enum cxl_region_mode mode) @@ -317,9 +362,10 @@ static inline const char *cxl_region_mode_name(enum cxl_region_mode mode) [CXL_REGION_MODE_MIXED] = "mixed", [CXL_REGION_MODE_PMEM] = "pmem", [CXL_REGION_MODE_RAM] = "ram", + [CXL_REGION_MODE_DC] = "dc", }; - if (mode < CXL_REGION_MODE_NONE || mode > CXL_REGION_MODE_RAM) + if (mode < CXL_REGION_MODE_NONE || mode > CXL_REGION_MODE_DC) mode = CXL_REGION_MODE_NONE; return names[mode]; } @@ -333,6 +379,8 @@ cxl_region_mode_from_ident(const char *ident) return CXL_REGION_MODE_RAM; else if (strcmp(ident, "pmem") == 0) return CXL_REGION_MODE_PMEM; + else if (strcmp(ident, "dc") == 0) + return CXL_REGION_MODE_DC; return CXL_REGION_MODE_NONE; } From patchwork Fri Nov 15 18:46:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876701 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9C1E1CF7A1 for ; Fri, 15 Nov 2024 18:46:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696399; cv=none; b=NH1sp7snUx1giFYr5g7CsOBhraRAzx2AqzUrDTl5rKq6PcURlyvWricCL9p/yZwD6t5awE3G6a8jgpJglZBCf7eOnFzshX6+1HJe6UfjAn02c4fgzpmMZKpfIBzED2XPXqpmqsuTg3XVOzqpQ6WShWCpsMyAtdk4U0KeSkJg34w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696399; c=relaxed/simple; bh=t6GbvHI3ED9i2ASzhEVWbtZHqnu1k8IUNk7QpOyXA30=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hNN+KYHh55TcNRoKpN8bdLv5QKtMs0GKKd0bRsdvmqUFJoxH6XFstEgR8XRt1L4Nhq9NOsoi0cy6LeRuPuJluPqhIpZ8EIFNOU8t5H7fyIBUjbSn/2JBBFZVynsUo05V7zRGW7NtND8OksGjuvDfuqZiZZNL9Ch5kLtZu8NZddM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=KwqoCsiS; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="KwqoCsiS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696398; x=1763232398; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=t6GbvHI3ED9i2ASzhEVWbtZHqnu1k8IUNk7QpOyXA30=; b=KwqoCsiSwBgIWwsqqb0RNgXKkeEycJEHOrUsIJ8eklT86YB1Ty23dubV lJvSbKsnIQgsLldbaQI7qa0XJ+1aL3viYi/punxYktFzQJVgh5tt8v5UZ 2+l5vAT/B8MWuwM0uCMp37Egq4oqmm0f2jWAE6crxtQ2+wNzQpo7DRfB6 D8F1ZullkGVGTAXzEokSjHuILDhP2F8XCNBLLSjwIa2vIXm0t2xOfbGfR ZfZToD6LAK4H1PRalOJ1LoDwqVsG2uxfR+pLYFVWxZM5MUDua8AQ+6V5t 88cBOnHDxijXoL5eFODE+K3nXLD4u3wznQ20f6Ha/318otokYJZ6C4TGN g==; X-CSE-ConnectionGUID: c1ATo6soTBiGNzaUWjJpeA== X-CSE-MsgGUID: Mhxm+douQ0+HRj9fbNF7SA== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848472" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848472" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:38 -0800 X-CSE-ConnectionGUID: Q/CsOw7hRaC1SlFyv2onjg== X-CSE-MsgGUID: VIg71eHLRBa34xMaViv/nw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392907" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:36 -0800 From: ira.weiny@intel.com Date: Fri, 15 Nov 2024 12:46:24 -0600 Subject: [ndctl PATCH v3 6/9] cxl/region: Add cxl-cli support for DCD regions Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-6-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny , Sushant1 Kumar X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=9701; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=6kbMCnC7Z99b+GpG47OCrOAiIaFJjRZF6qdh4ygC3Hg=; b=W3j80PsL0dpp47qUm4VKbtudyCTqgBD1wIFn2/xdK2WwXbg/x9+KDVfU1Ymk0Aru2IWTsm6nX xnU9YF2gyPfBnjilnvQPG3PvE2twb9sC8cTAapvlYx+UwTJql9R0sf3 X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= From: Navneet Singh CXL Dynamic Capacity Devices (DCDs) optionally support dynamic capacity with up to eight partitions (Regions) (dc0-dc7). CXL regions can now be sparse and defined as dynamic capacity (dc). DCD region creation requires a specific partition, or decoder mode, to be supplied. Introduce a required option for dc regions to specify the decoder mode. Add support for dynamic capacity region creation. Signed-off-by: Navneet Singh Co-developed-by: Sushant1 Kumar Signed-off-by: Sushant1 Kumar Co-developed-by: Ira Weiny Signed-off-by: Ira Weiny --- Changes: [djiang: remove errant white space changes] [djiang: use for loop for dc capability sets] [djiang/iweiny: change cxl_memdev_get_dc_size() to take mode] [djiang/iweiny: change cxl_decoder_is_dc_capable() to take mode] [Alison: s/Co-Authored-by/Co-developed-by/] [Alison: document new libcxl calls] [Alison: reduce indentation] [Alison: Fix libcxl export symbols] [iweiny: Add get mode documentation] [iweiny: add docs for DC support] [iweiny: Add -M option to clarify region vs decoder mode] --- Documentation/cxl/cxl-create-region.txt | 11 ++++++++-- cxl/json.c | 27 ++++++++++++++++++++++- cxl/memdev.c | 4 +++- cxl/region.c | 39 ++++++++++++++++++++++++++++++++- 4 files changed, 76 insertions(+), 5 deletions(-) diff --git a/Documentation/cxl/cxl-create-region.txt b/Documentation/cxl/cxl-create-region.txt index b244af60b8a63281ed63d0d6f4027ea729ad51b0..a12cc8d3f19fa582376599ecc8512640f15ce42c 100644 --- a/Documentation/cxl/cxl-create-region.txt +++ b/Documentation/cxl/cxl-create-region.txt @@ -75,8 +75,9 @@ include::bus-option.txt[] -t:: --type=:: - Specify the region type - 'pmem' or 'ram'. Default to root decoder - capability, and if that is ambiguous, default to 'pmem'. + Specify the region type - 'pmem', 'ram', or 'dc'. Default to root + decoder capability including the first of any DC partition found. If + the decoder capability is ambiguous, default to 'pmem'. -U:: --uuid=:: @@ -105,6 +106,12 @@ include::bus-option.txt[] supplied, the first cross-host bridge (if available), decoder that supports the largest interleave will be chosen. +-M:: +--decoder-mode=:: + For a 'dc' region type, specify the dynamic capacity partition to be + used on each device, 'dcX' [where X is 0-7]. Ignored for region type + of 'ram' or 'pmem'. + -Q:: --enforce-qos:: Parameter to enforce qos_class mismatch failure. Region create operation diff --git a/cxl/json.c b/cxl/json.c index dcd3cc28393faf7e8adf299a857531ecdeaac50a..c5391be84fba51da57fc15ece7c1f94cce139276 100644 --- a/cxl/json.c +++ b/cxl/json.c @@ -754,12 +754,15 @@ err_free: return jpoison; } +#define DC_SIZE_NAME_LEN 64 struct json_object *util_cxl_memdev_to_json(struct cxl_memdev *memdev, unsigned long flags) { const char *devname = cxl_memdev_get_devname(memdev); + char size_name[DC_SIZE_NAME_LEN]; struct json_object *jdev, *jobj; unsigned long long serial, size; + enum cxl_decoder_mode mode; const char *fw_version; int numa_node; int qos_class; @@ -800,6 +803,16 @@ struct json_object *util_cxl_memdev_to_json(struct cxl_memdev *memdev, } } + for (mode = CXL_DECODER_MODE_DC0; mode <= CXL_DECODER_MODE_DC7; mode++) { + size = cxl_memdev_get_dc_size(memdev, mode); + if (!size) + continue; + jobj = util_json_object_size(size, flags); + if (!jobj) + continue; + sprintf(size_name, "%s_size", cxl_decoder_mode_name(mode)); + json_object_object_add(jdev, size_name, jobj); + } if (flags & UTIL_JSON_HEALTH) { jobj = util_cxl_memdev_health_to_json(memdev, flags); if (jobj) @@ -948,12 +961,15 @@ struct json_object *util_cxl_bus_to_json(struct cxl_bus *bus, return jbus; } +#define DC_CAPABILITY_NAME_LEN 16 struct json_object *util_cxl_decoder_to_json(struct cxl_decoder *decoder, unsigned long flags) { const char *devname = cxl_decoder_get_devname(decoder); struct cxl_port *port = cxl_decoder_get_port(decoder); + char dc_capable_name[DC_CAPABILITY_NAME_LEN]; struct json_object *jdecoder, *jobj; + enum cxl_decoder_mode mode; struct cxl_region *region; u64 val, size; @@ -1013,7 +1029,7 @@ struct json_object *util_cxl_decoder_to_json(struct cxl_decoder *decoder, } if (cxl_port_is_endpoint(port)) { - enum cxl_decoder_mode mode = cxl_decoder_get_mode(decoder); + mode = cxl_decoder_get_mode(decoder); size = cxl_decoder_get_dpa_size(decoder); val = cxl_decoder_get_dpa_resource(decoder); @@ -1059,6 +1075,15 @@ struct json_object *util_cxl_decoder_to_json(struct cxl_decoder *decoder, json_object_object_add( jdecoder, "volatile_capable", jobj); } + for (mode = CXL_DECODER_MODE_DC0; mode <= CXL_DECODER_MODE_DC7; mode++) { + if (!cxl_decoder_is_dc_capable(decoder, mode)) + continue; + jobj = json_object_new_boolean(true); + if (!jobj) + continue; + sprintf(dc_capable_name, "%s_capable", cxl_decoder_mode_name(mode)); + json_object_object_add(jdecoder, dc_capable_name, jobj); + } } if (cxl_port_is_root(port) && diff --git a/cxl/memdev.c b/cxl/memdev.c index 6e44d1578d03b6af998502e54714635b8f31b556..b132527d7c21e99c9da75ea7cbf1497fd924a142 100644 --- a/cxl/memdev.c +++ b/cxl/memdev.c @@ -269,8 +269,10 @@ static int __reserve_dpa(struct cxl_memdev *memdev, if (mode == CXL_DECODER_MODE_RAM) avail_dpa = cxl_memdev_get_ram_size(memdev); - else + else if (mode == CXL_DECODER_MODE_PMEM) avail_dpa = cxl_memdev_get_pmem_size(memdev); + else if (cxl_decoder_mode_is_dc(mode)) + avail_dpa = cxl_memdev_get_dc_size(memdev, mode); cxl_decoder_foreach(port, decoder) { size = cxl_decoder_get_dpa_size(decoder); diff --git a/cxl/region.c b/cxl/region.c index 527bd6708b162815068a95ddb360fce3914347de..79f434b0c99545523f3b8209f90ff2f02111881c 100644 --- a/cxl/region.c +++ b/cxl/region.c @@ -26,6 +26,7 @@ static struct region_params { const char *uuid; const char *root_decoder; const char *region; + const char *decoder_mode; int ways; int granularity; bool memdevs; @@ -79,9 +80,11 @@ OPT_INTEGER('w', "ways", ¶m.ways, \ OPT_INTEGER('g', "granularity", ¶m.granularity, \ "granularity of the interleave set"), \ OPT_STRING('t', "type", ¶m.type, \ - "region type", "region type - 'pmem' or 'ram'"), \ + "region type", "region type - 'pmem', 'ram', or 'dc'"), \ OPT_STRING('U', "uuid", ¶m.uuid, \ "region uuid", "uuid for the new region (default: autogenerate)"), \ +OPT_STRING('M', "decoder-mode", ¶m.decoder_mode, "decoder mode", \ + "decoder mode for dc regions - 'dcX' [where X is 0-7]"), \ OPT_BOOLEAN('m', "memdevs", ¶m.memdevs, \ "non-option arguments are memdevs"), \ OPT_BOOLEAN('u', "human", ¶m.human, "use human friendly number formats"), \ @@ -314,12 +317,19 @@ static int parse_create_options(struct cxl_ctx *ctx, int count, log_err(&rl, "unsupported type: %s\n", param.type); goto err; } + if (p->region_mode == CXL_REGION_MODE_DC && !param.decoder_mode) { + log_err(&rl, "dc type requires a decoder mode\n"); + goto err; + } } switch (p->region_mode) { case CXL_REGION_MODE_RAM: p->decoder_mode = CXL_DECODER_MODE_RAM; break; + case CXL_REGION_MODE_DC: + p->decoder_mode = cxl_decoder_mode_from_ident(param.decoder_mode); + break; case CXL_REGION_MODE_PMEM: default: p->decoder_mode = CXL_DECODER_MODE_PMEM; @@ -427,6 +437,9 @@ static void collect_minsize(struct cxl_ctx *ctx, struct parsed_params *p) case CXL_DECODER_MODE_PMEM: size = cxl_memdev_get_pmem_size(memdev); break; + case CXL_DECODER_MODE_DC0 ... CXL_DECODER_MODE_DC7: + size = cxl_memdev_get_dc_size(memdev, p->decoder_mode); + break; default: /* Shouldn't ever get here */ ; } @@ -498,6 +511,13 @@ static int validate_decoder(struct cxl_decoder *decoder, return -EINVAL; } break; + case CXL_DECODER_MODE_DC0 ... CXL_DECODER_MODE_DC7: + if (!cxl_decoder_is_dc_capable(decoder, p->decoder_mode)) { + log_err(&rl, "%s is not %s capable\n", devname, + cxl_decoder_mode_name(p->decoder_mode)); + return -EINVAL; + } + break; default: log_err(&rl, "unknown type: %s\n", param.type); return -EINVAL; @@ -514,10 +534,20 @@ static int validate_decoder(struct cxl_decoder *decoder, static void set_type_from_decoder(struct cxl_ctx *ctx, struct parsed_params *p) { + enum cxl_decoder_mode mode; + /* if param.type was explicitly specified, nothing to do here */ if (param.type) return; + for (mode = CXL_DECODER_MODE_DC0; mode <= CXL_DECODER_MODE_DC7; mode++) { + if (cxl_decoder_is_dc_capable(p->root_decoder, mode)) { + p->decoder_mode = mode; + p->region_mode = CXL_REGION_MODE_DC; + break; + } + } + /* * default to pmem if both types are set, otherwise the single * capability dominates. @@ -713,6 +743,13 @@ static int create_region(struct cxl_ctx *ctx, int *count, param.root_decoder); return -ENXIO; } + } else if (p->region_mode == CXL_REGION_MODE_DC) { + region = cxl_decoder_create_dc_region(p->root_decoder); + if (!region) { + log_err(&rl, "failed to create region under %s\n", + param.root_decoder); + return -ENXIO; + } } else { log_err(&rl, "region type '%s' is not supported\n", param.type); From patchwork Fri Nov 15 18:46:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876702 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61E4218787A for ; Fri, 15 Nov 2024 18:46:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696407; cv=none; b=IB2E6La4cgzs16inA1y0i6eDm9XirIKvDOVQH7/gYeNhQJfsAexxQTPh4PhAe8GLYZioKt3R7LrlOnrdpjxNCVmZweosho+JYLJv9VmEXGqtY/gFzfF6c+Z62AU71q8DyI7FM95SmDFqvcxpdZA6GZzgQYF7lKQbIcqIBODLhL8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696407; c=relaxed/simple; bh=bee6YOxiEEbvEMLIRmtuV/fOI+H/2Oms2YLawlgg7js=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dk6Tzu5Uw7isESjYCHduagTojO01t4p0RRCju0avxoNSt4Z1WMjM+zJE6Cz5ocEfGsZVB9cHG3S6CGyzwY85tUErGXWtIsamZACG5q27GaHmpZQ1A5TOJO6yvQeYTLIe/9G/EpJ3QfchJvCFPDy8Y5P6X4ratofm8zvVJ+1PIeA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=mG+NEU5u; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="mG+NEU5u" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696406; x=1763232406; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=bee6YOxiEEbvEMLIRmtuV/fOI+H/2Oms2YLawlgg7js=; b=mG+NEU5ur9JllL2c0++evqliDvWNt9d2VeuNexJ3Uob0Uwj6nPk/7d9y XifXcHg2k8aTSeyhqOKs/rVWPlICyAN77EwrBPiCoehvXFtezW2S6aG6O UEAEY9VKaitWttjumTbYQYC0OyUTGDOvxkEXMnQWzsoilM+BhLjGTEyOz J1zbH+EZtXfaP41CqC/QzDjpaaqoPSK24AITTEItA7qMJvVeMfHil4PvL fpyjlejeVSiPFoMv5cftBMBbYWWcwR04jK2BzJMCB7hR0hp+YLJKrufvb Pp8PsLOmPKbP1/ubHrQByQI8Ar+gylwfOKQyjY9CDxNLISfBG9gdHDvF3 Q==; X-CSE-ConnectionGUID: quaP1mT4QASQViItEuLjJw== X-CSE-MsgGUID: Q66pveZgRLeyTwNphCv2VQ== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848484" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848484" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:39 -0800 X-CSE-ConnectionGUID: CVEQM3PJTP6ybiwVqegBkg== X-CSE-MsgGUID: eUJlIw0lQcmyblV6t+pfKg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392921" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:38 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:25 -0600 Subject: [ndctl PATCH v3 7/9] libcxl: Add extent functionality to DC regions Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-7-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=9208; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=bee6YOxiEEbvEMLIRmtuV/fOI+H/2Oms2YLawlgg7js=; b=BdfY5YPJHPLrIlELQrItexXj0kOodXdTeU4l/qW0B9R2A54Sxz6YZyxQc5GkJvuWEuHqAwEnT m7Mo9QDw8+yDQ/tLXYsuMUj7rjyVBk6B5NYHxOXz/DePLVUdnOy5ETy X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= DCD regions have 0 or more extents. The ability to list those and their properties is useful to end users. Add extent scanning and reporting functionality to libcxl. Signed-off-by: Ira Weiny --- Changes: [djiang: report strerror() on opendir() error] [djiang: Fix up strtoull() error checking] [Alison: Enhance man page] [Alison: Enhance extent processing debug] [Alison: Fix up libcxl export symbols] [Alison/iweiny: Add documentation] --- Documentation/cxl/lib/libcxl.txt | 27 ++++++++ cxl/lib/libcxl.c | 138 +++++++++++++++++++++++++++++++++++++++ cxl/lib/libcxl.sym | 5 ++ cxl/lib/private.h | 11 ++++ cxl/libcxl.h | 11 ++++ 5 files changed, 192 insertions(+) diff --git a/Documentation/cxl/lib/libcxl.txt b/Documentation/cxl/lib/libcxl.txt index abca08fc81e6e84d176facafad6decae2f875880..0b53cf9a3a09a3e8c9059f796823b52d22d1077f 100644 --- a/Documentation/cxl/lib/libcxl.txt +++ b/Documentation/cxl/lib/libcxl.txt @@ -632,6 +632,33 @@ Regions now have a mode distinct from decoders. cxl_region_get_mode() is deprecated in favor of cxl_region_get_region_mode(). Dynamic capacity regions require the use of cxl_region_get_region_mode(). +EXTENTS +------- + +=== EXTENT: Enumeration +---- +struct cxl_region_extent; +struct cxl_region_extent *cxl_extent_get_first(struct cxl_region *region); +struct cxl_region_extent *cxl_extent_get_next(struct cxl_region_extent *extent); +#define cxl_extent_foreach(region, extent) \ + for (extent = cxl_extent_get_first(region); \ + extent != NULL; \ + extent = cxl_extent_get_next(extent)) + +---- + +=== EXTENT: Attributes +---- +unsigned long long cxl_extent_get_offset(struct cxl_region_extent *extent); +unsigned long long cxl_extent_get_length(struct cxl_region_extent *extent); +void cxl_extent_get_tag(struct cxl_region_extent *extent, uuid_t tag); +---- + +Extents represent available memory within a dynamic capacity region. Extent +objects are available for informational purposes to aid in allocation of +memory. + + include::../../copyright.txt[] SEE ALSO diff --git a/cxl/lib/libcxl.c b/cxl/lib/libcxl.c index df250db9dbacb2f0f34e8a592ce194159584fe4f..a029b14dcccf038b02b28d05df6f0dc71557df5e 100644 --- a/cxl/lib/libcxl.c +++ b/cxl/lib/libcxl.c @@ -568,6 +568,7 @@ static void *add_cxl_region(void *parent, int id, const char *cxlregion_base) region->ctx = ctx; region->decoder = decoder; list_head_init(®ion->mappings); + list_head_init(®ion->extents); region->dev_path = strdup(cxlregion_base); if (!region->dev_path) @@ -1178,6 +1179,143 @@ cxl_mapping_get_next(struct cxl_memdev_mapping *mapping) return list_next(®ion->mappings, mapping, list); } +static void cxl_extents_init(struct cxl_region *region) +{ + const char *devname = cxl_region_get_devname(region); + struct cxl_ctx *ctx = cxl_region_get_ctx(region); + char *extent_path, *dax_region_path; + struct dirent *de; + DIR *dir = NULL; + + if (region->extents_init) + return; + region->extents_init = 1; + + dax_region_path = calloc(1, strlen(region->dev_path) + 64); + if (!dax_region_path) { + err(ctx, "%s: allocation failure\n", devname); + return; + } + + extent_path = calloc(1, strlen(region->dev_path) + 100); + if (!extent_path) { + err(ctx, "%s: allocation failure\n", devname); + free(dax_region_path); + return; + } + + sprintf(dax_region_path, "%s/dax_region%d", + region->dev_path, region->id); + dir = opendir(dax_region_path); + if (!dir) { + err(ctx, "no extents found (%s): %s\n", + strerror(errno), dax_region_path); + free(extent_path); + free(dax_region_path); + return; + } + + while ((de = readdir(dir)) != NULL) { + struct cxl_region_extent *extent; + char buf[SYSFS_ATTR_SIZE]; + u64 offset, length; + int id, region_id; + + if (sscanf(de->d_name, "extent%d.%d", ®ion_id, &id) != 2) + continue; + + sprintf(extent_path, "%s/extent%d.%d/offset", + dax_region_path, region_id, id); + if (sysfs_read_attr(ctx, extent_path, buf) < 0) { + err(ctx, "%s: failed to read extent%d.%d/offset\n", + devname, region_id, id); + continue; + } + + offset = strtoull(buf, NULL, 0); + if (offset == ULLONG_MAX) { + err(ctx, "%s extent%d.%d: failed to read offset\n", + devname, region_id, id); + continue; + } + + sprintf(extent_path, "%s/extent%d.%d/length", + dax_region_path, region_id, id); + if (sysfs_read_attr(ctx, extent_path, buf) < 0) { + err(ctx, "%s: failed to read extent%d.%d/length\n", + devname, region_id, id); + continue; + } + + length = strtoull(buf, NULL, 0); + if (length == ULLONG_MAX) { + err(ctx, "%s extent%d.%d: failed to read length\n", + devname, region_id, id); + continue; + } + + sprintf(extent_path, "%s/extent%d.%d/tag", + dax_region_path, region_id, id); + buf[0] = '\0'; + if (sysfs_read_attr(ctx, extent_path, buf) != 0) + dbg(ctx, "%s extent%d.%d: failed to read tag\n", + devname, region_id, id); + + extent = calloc(1, sizeof(*extent)); + if (!extent) { + err(ctx, "%s extent%d.%d: allocation failure\n", + devname, region_id, id); + continue; + } + if (strlen(buf) && uuid_parse(buf, extent->tag) < 0) + err(ctx, "%s:%s\n", extent_path, buf); + extent->region = region; + extent->offset = offset; + extent->length = length; + + list_node_init(&extent->list); + list_add(®ion->extents, &extent->list); + dbg(ctx, "%s added extent%d.%d\n", devname, region_id, id); + } + free(dax_region_path); + free(extent_path); + closedir(dir); +} + +CXL_EXPORT struct cxl_region_extent * +cxl_extent_get_first(struct cxl_region *region) +{ + cxl_extents_init(region); + + return list_top(®ion->extents, struct cxl_region_extent, list); +} + +CXL_EXPORT struct cxl_region_extent * +cxl_extent_get_next(struct cxl_region_extent *extent) +{ + struct cxl_region *region = extent->region; + + return list_next(®ion->extents, extent, list); +} + +CXL_EXPORT unsigned long long +cxl_extent_get_offset(struct cxl_region_extent *extent) +{ + return extent->offset; +} + +CXL_EXPORT unsigned long long +cxl_extent_get_length(struct cxl_region_extent *extent) +{ + return extent->length; +} + +CXL_EXPORT void +cxl_extent_get_tag(struct cxl_region_extent *extent, uuid_t tag) +{ + memcpy(tag, extent->tag, sizeof(uuid_t)); +} + CXL_EXPORT struct cxl_decoder * cxl_mapping_get_decoder(struct cxl_memdev_mapping *mapping) { diff --git a/cxl/lib/libcxl.sym b/cxl/lib/libcxl.sym index fdb227789985443a13c72751bbd42ab383db5f97..d8e8dbc7e091792fe48faa4657ab7cf1d795efdd 100644 --- a/cxl/lib/libcxl.sym +++ b/cxl/lib/libcxl.sym @@ -295,4 +295,9 @@ global: cxl_memdev_get_dc_size; cxl_decoder_is_dc_capable; cxl_decoder_create_dc_region; + cxl_extent_get_first; + cxl_extent_get_next; + cxl_extent_get_offset; + cxl_extent_get_length; + cxl_extent_get_tag; } LIBECXL_8; diff --git a/cxl/lib/private.h b/cxl/lib/private.h index 3efa230bfb632e6c6048aadd18f799b07d4bdfd3..62278ec79963c198dcca490015e4c3f7621109b2 100644 --- a/cxl/lib/private.h +++ b/cxl/lib/private.h @@ -164,6 +164,7 @@ struct cxl_region { struct cxl_decoder *decoder; struct list_node list; int mappings_init; + int extents_init; struct cxl_ctx *ctx; void *dev_buf; size_t buf_len; @@ -179,6 +180,7 @@ struct cxl_region { struct daxctl_region *dax_region; struct kmod_module *module; struct list_head mappings; + struct list_head extents; }; struct cxl_memdev_mapping { @@ -188,6 +190,15 @@ struct cxl_memdev_mapping { struct list_node list; }; +#define CXL_REGION_EXTENT_TAG 0x10 +struct cxl_region_extent { + struct cxl_region *region; + u64 offset; + u64 length; + uuid_t tag; + struct list_node list; +}; + enum cxl_cmd_query_status { CXL_CMD_QUERY_NOT_RUN = 0, CXL_CMD_QUERY_OK, diff --git a/cxl/libcxl.h b/cxl/libcxl.h index d7f8a37816f236acd71fc834eae70a7a17a2721a..1d294ac0278d798214acb2f62e98aaaccaf60ea5 100644 --- a/cxl/libcxl.h +++ b/cxl/libcxl.h @@ -445,6 +445,17 @@ unsigned int cxl_mapping_get_position(struct cxl_memdev_mapping *mapping); mapping != NULL; \ mapping = cxl_mapping_get_next(mapping)) +struct cxl_region_extent; +struct cxl_region_extent *cxl_extent_get_first(struct cxl_region *region); +struct cxl_region_extent *cxl_extent_get_next(struct cxl_region_extent *extent); +#define cxl_extent_foreach(region, extent) \ + for (extent = cxl_extent_get_first(region); \ + extent != NULL; \ + extent = cxl_extent_get_next(extent)) +unsigned long long cxl_extent_get_offset(struct cxl_region_extent *extent); +unsigned long long cxl_extent_get_length(struct cxl_region_extent *extent); +void cxl_extent_get_tag(struct cxl_region_extent *extent, uuid_t tag); + struct cxl_cmd; const char *cxl_cmd_get_devname(struct cxl_cmd *cmd); struct cxl_cmd *cxl_cmd_new_raw(struct cxl_memdev *memdev, int opcode); From patchwork Fri Nov 15 18:46:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876703 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 388AE1DDA0E for ; Fri, 15 Nov 2024 18:46:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696408; cv=none; b=YIBBSMSs2+Kztfd9CPiTlnIReT3n7KD3hIIstMaSXbltvn7uKnNpv/ncXnu1gCMh7a+xy3lOUZB6DjYfttBmEyuec8otXo8I6DjrM9O3Br5Pbq8k8Hv8zyw3xiIzC8oKvHDynNvP6DnLGpGiJAFRdqf44S0/Mz4mjC4lNLjX8+g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696408; c=relaxed/simple; bh=z4Q7yx0px99G72dCFd63VRlaEhBEeLBZbinW+hnIdJA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=K/KXUiUopXUMeZIY40rMlClrKS/Ft27SLBD8j89sPd4K/BR+Tf4pUs0g9txT6XrpOKRQB19YmdkTfl4cQD3vZcS5mDAXAaclu5FhyHMfC7FVdDLXa5OLthpQ8OsGIfJQBRZpaOe8xQsNHkZcuAi1M4laKyg5YGkYCDchiIzGQcg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=BNXQOOWy; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="BNXQOOWy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696407; x=1763232407; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=z4Q7yx0px99G72dCFd63VRlaEhBEeLBZbinW+hnIdJA=; b=BNXQOOWy/6pB+4vhhEODmvb+wkrYEbBfPT6GkoGEC2iYvIwHNvbwOcf8 y1VYA95z2teVaIDfN/HavmtE9lZPRboZYl+SQK9FW79NqSsfuzvzoonHY 3fwBYsXlFR/cDb24Y2YF5PcLDIbbcXl46ofCSubBaH36jLlk7m/+PghHt aywmhzVTukqFjUbUkotm7X8cPd6nsXGjVQoKiyxxMv5nvXygNF8uIoeIx l9LvMo437z48Kv3C3FTpFd0PlxYbW8vZ/9idm985f5IIvvmPabyb579Pf gJ/lZjQOhYi981F6BxYQl+Y/Z5fc9HI26J3Lu727yiksRtNbqGayl/g1p A==; X-CSE-ConnectionGUID: 2sRlOMWlQxucBv+Dw6O+xA== X-CSE-MsgGUID: lCLuXzMeTuW0dQSFOBO/Mg== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848488" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848488" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:41 -0800 X-CSE-ConnectionGUID: 99zl8i2zQ+u8MiqAGVZ9qA== X-CSE-MsgGUID: KbyOlwNdQLyi5W4EINZuXQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392932" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:40 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:26 -0600 Subject: [ndctl PATCH v3 8/9] cxl/region: Add extent output to region query Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-8-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=6809; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=z4Q7yx0px99G72dCFd63VRlaEhBEeLBZbinW+hnIdJA=; b=EKFBAT/lFfNI0tcpMvPSjjsf8m8/j5XJocP6PxvetkMtHlBp2e0UEh5m2gfDKaED2lk8ihbar wHjNBvqP6gkCUN0/NNckfLrOjXwsiyv6O4eU5VdM+RGOHorGtST66QU X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= DCD regions have 0 or more extents. The ability to list those and their properties is useful to end users. Add an option for extent output to region queries. An example of this is: $ ./build/cxl/cxl list -r 8 -Nu { "region":"region8", ... "type":"dc", ... "extents":[ { "offset":"0x10000000", "length":"64.00 MiB (67.11 MB)", "tag":"00000000-0000-0000-0000-000000000000" }, { "offset":"0x8000000", "length":"64.00 MiB (67.11 MB)", "tag":"00000000-0000-0000-0000-000000000000" } ] } Signed-off-by: Ira Weiny --- Changes: [djiang: report strerror() on opendir() error] [djiang: Fix up strtoull() error checking] [Alison: Enhance man page] [Alison: Enhance extent processing debug] [Alison: Fix up libcxl export symbols] --- Documentation/cxl/cxl-list.txt | 29 ++++++++++++++++++++++++++ cxl/filter.h | 3 +++ cxl/json.c | 47 ++++++++++++++++++++++++++++++++++++++++++ cxl/json.h | 3 +++ cxl/list.c | 3 +++ util/json.h | 1 + 6 files changed, 86 insertions(+) diff --git a/Documentation/cxl/cxl-list.txt b/Documentation/cxl/cxl-list.txt index 9a9911e7dd9bba561c6202784017db1bb4b9f4bd..43453cc72245f586070f8c4f31b3ee475e3c6cd2 100644 --- a/Documentation/cxl/cxl-list.txt +++ b/Documentation/cxl/cxl-list.txt @@ -411,6 +411,35 @@ OPTIONS } ---- +-N:: +--extents:: + Append Dynamic Capacity extent information. +---- +13:34:28 > ./build/cxl/cxl list -r 8 -Nu +{ + "region":"region8", + "resource":"0xf030000000", + "size":"512.00 MiB (536.87 MB)", + "type":"dc", + "interleave_ways":1, + "interleave_granularity":256, + "decode_state":"commit", + "extents":[ + { + "offset":"0x10000000", + "length":"64.00 MiB (67.11 MB)", + "tag":"00000000-0000-0000-0000-000000000000" + }, + { + "offset":"0x8000000", + "length":"64.00 MiB (67.11 MB)", + "tag":"00000000-0000-0000-0000-000000000000" + } + ] +} +---- + + -r:: --region:: Specify CXL region device name(s), or device id(s), to filter the listing. diff --git a/cxl/filter.h b/cxl/filter.h index 956a46e0c7a9f05abf696cce97a365164e95e50d..a31b80c87ccac407bd4ff98b302a23b33cbe413c 100644 --- a/cxl/filter.h +++ b/cxl/filter.h @@ -31,6 +31,7 @@ struct cxl_filter_params { bool alert_config; bool dax; bool media_errors; + bool extents; int verbose; struct log_ctx ctx; }; @@ -91,6 +92,8 @@ static inline unsigned long cxl_filter_to_flags(struct cxl_filter_params *param) flags |= UTIL_JSON_DAX | UTIL_JSON_DAX_DEVS; if (param->media_errors) flags |= UTIL_JSON_MEDIA_ERRORS; + if (param->extents) + flags |= UTIL_JSON_EXTENTS; return flags; } diff --git a/cxl/json.c b/cxl/json.c index c5391be84fba51da57fc15ece7c1f94cce139276..450e62243ecdfec1aa011241ff7257ac3b37196f 100644 --- a/cxl/json.c +++ b/cxl/json.c @@ -1169,6 +1169,50 @@ void util_cxl_mappings_append_json(struct json_object *jregion, json_object_object_add(jregion, "mappings", jmappings); } +void util_cxl_extents_append_json(struct json_object *jregion, + struct cxl_region *region, + unsigned long flags) +{ + struct json_object *jextents; + struct cxl_region_extent *extent; + + jextents = json_object_new_array(); + if (!jextents) + return; + + cxl_extent_foreach(region, extent) { + struct json_object *jextent, *jobj; + unsigned long long val; + char tag_str[40]; + uuid_t tag; + + jextent = json_object_new_object(); + if (!jextent) + continue; + + val = cxl_extent_get_offset(extent); + jobj = util_json_object_hex(val, flags); + if (jobj) + json_object_object_add(jextent, "offset", jobj); + + val = cxl_extent_get_length(extent); + jobj = util_json_object_size(val, flags); + if (jobj) + json_object_object_add(jextent, "length", jobj); + + cxl_extent_get_tag(extent, tag); + uuid_unparse(tag, tag_str); + jobj = json_object_new_string(tag_str); + if (jobj) + json_object_object_add(jextent, "tag", jobj); + + json_object_array_add(jextents, jextent); + json_object_set_userdata(jextent, extent, NULL); + } + + json_object_object_add(jregion, "extents", jextents); +} + struct json_object *util_cxl_region_to_json(struct cxl_region *region, unsigned long flags) { @@ -1255,6 +1299,9 @@ struct json_object *util_cxl_region_to_json(struct cxl_region *region, } } + if (flags & UTIL_JSON_EXTENTS) + util_cxl_extents_append_json(jregion, region, flags); + if (cxl_region_qos_class_mismatch(region)) { jobj = json_object_new_boolean(true); if (jobj) diff --git a/cxl/json.h b/cxl/json.h index eb7572be4106baf0469ba9243a9a767d07df8882..f9c07ab41a337838b75ffee4486f6c48ddc99863 100644 --- a/cxl/json.h +++ b/cxl/json.h @@ -20,6 +20,9 @@ struct json_object *util_cxl_region_to_json(struct cxl_region *region, void util_cxl_mappings_append_json(struct json_object *jregion, struct cxl_region *region, unsigned long flags); +void util_cxl_extents_append_json(struct json_object *jregion, + struct cxl_region *region, + unsigned long flags); void util_cxl_targets_append_json(struct json_object *jdecoder, struct cxl_decoder *decoder, const char *ident, const char *serial, diff --git a/cxl/list.c b/cxl/list.c index 0b25d78248d5f4f529fd2c2e073e43895c722568..47d135166212b87449f960e94ee75657f7040ca9 100644 --- a/cxl/list.c +++ b/cxl/list.c @@ -59,6 +59,8 @@ static const struct option options[] = { "include alert configuration information"), OPT_BOOLEAN('L', "media-errors", ¶m.media_errors, "include media-error information "), + OPT_BOOLEAN('N', "extents", ¶m.extents, + "include extent information (Dynamic Capacity regions only)"), OPT_INCR('v', "verbose", ¶m.verbose, "increase output detail"), #ifdef ENABLE_DEBUG OPT_BOOLEAN(0, "debug", &debug, "debug list walk"), @@ -135,6 +137,7 @@ int cmd_list(int argc, const char **argv, struct cxl_ctx *ctx) param.decoders = true; param.targets = true; param.regions = true; + param.extents = true; /*fallthrough*/ case 0: break; diff --git a/util/json.h b/util/json.h index 560f845c6753ee176f7c64b4310fe1f9b1ce6d39..79ae3240e7ce151be75f6666fcaba0ba90aba7fc 100644 --- a/util/json.h +++ b/util/json.h @@ -21,6 +21,7 @@ enum util_json_flags { UTIL_JSON_TARGETS = (1 << 11), UTIL_JSON_PARTITION = (1 << 12), UTIL_JSON_ALERT_CONFIG = (1 << 13), + UTIL_JSON_EXTENTS = (1 << 14), }; void util_display_json_array(FILE *f_out, struct json_object *jarray, From patchwork Fri Nov 15 18:46:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 13876704 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EBED1E0DD1 for ; Fri, 15 Nov 2024 18:46:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696409; cv=none; b=Wt4MxcUtDuwGUjqlvcZL3+Iq9DFjvMgZYFJigwAWLiZhkSRbJ7sPBBHCN3E/onJ8HYppfHb9xA4aG3eTjWZGssp3fr7ovtt3/wu6X0sk1/EM1MRK3mBYWHinPmQeRe0hu9AbQK25xGUvVdWnLrj5hNcNfhYdFJw5Vxxs07AqCQc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731696409; c=relaxed/simple; bh=XTnqjwo8m9M3KT6jsKsvMfK/JinxzMfjLxv/ZdTU4/0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=iP/cCmnwaIAFb1PkDFqkR/TqbhGSzST9q2WOGpsr+Ip0BcQRwVCZA1myCaCBoep4ZfofnOy2wt6TnFUAJHw7TYE342p5HeoU/P4oNWQA1B8B5uiOw5hUZpLpThrYTshmlkZHJax1nSKaZ+aoqU8JZ0W/hVp2PFvJAv7uqTxg5qQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=FJQwjw/7; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="FJQwjw/7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731696408; x=1763232408; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=XTnqjwo8m9M3KT6jsKsvMfK/JinxzMfjLxv/ZdTU4/0=; b=FJQwjw/7yJi+GyFd13lXRnlGpBQvBqnKpNgH7BLrGXCgsWCmaNj8daoj M0YBj+B8eKVs7c7HCoYk0P2es7PDPV9iPXCE7/Xwi9mAqy51pMaxDSMUy 1MGdfjUyjQkjnrB0h4RRJ9SW1rPNkRz95EP2yBzJPXGndiCUGHrMoFRa1 paqPjfPGhcjX1JSkXitB6vvbaSbu3NCAW19L45/nQOYTwwDbLo0deMD8V F/wYzlZXyZEZywRwx83F2XbrLF8stIvle9uLkjeNizoysmvkQKFeea9PU Ju8f68Cgqm/5s+PBt4j4Q+mtZHxfgP0kw4Ah5GWrhDLmqdbkGbZl+3D8w w==; X-CSE-ConnectionGUID: YNhPjd1gQF+pB55LseVLOQ== X-CSE-MsgGUID: IfXiuON7TQmDYaBE6qV5dg== X-IronPort-AV: E=McAfee;i="6700,10204,11257"; a="31848495" X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="31848495" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:43 -0800 X-CSE-ConnectionGUID: U1GCEyr3Sm2Wg7RAYCWdNA== X-CSE-MsgGUID: 9YCWHCyyRj+3C1dHj+N99g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,157,1728975600"; d="scan'208";a="89392941" Received: from ehanks-mobl1.amr.corp.intel.com (HELO localhost) ([10.125.108.112]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Nov 2024 10:46:41 -0800 From: Ira Weiny Date: Fri, 15 Nov 2024 12:46:27 -0600 Subject: [ndctl PATCH v3 9/9] cxl/test: Add Dynamic Capacity tests Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241115-dcd-region2-v3-9-585d480ccdab@intel.com> References: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> In-Reply-To: <20241115-dcd-region2-v3-0-585d480ccdab@intel.com> To: Alison Schofield Cc: Vishal Verma , Jonathan Cameron , Fan Ni , Navneet Singh , Dan Williams , Dave Jiang , linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, Ira Weiny X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731696382; l=30045; i=ira.weiny@intel.com; s=20221211; h=from:subject:message-id; bh=XTnqjwo8m9M3KT6jsKsvMfK/JinxzMfjLxv/ZdTU4/0=; b=ydZSgQ+KntqMqB/co0b+cGFTypQnU1MTQWfW/wOmtj65R9nf241s+dVHFc/a/DVqW4p4eHBKg b2FCwHgTdOZApSFOVWWQozZ+yV6I6Eyq1PAtN9Pqi35+RqIjr/n8qu0 X-Developer-Key: i=ira.weiny@intel.com; a=ed25519; pk=noldbkG+Wp1qXRrrkfY1QJpDf7QsOEthbOT7vm0PqsE= cxl_test provides a good way to ensure quick smoke and regression testing. The complexity of DCD and the new sparse DAX regions required to use them benefits greatly with a series of smoke tests. The only part of the kernel stack which must be bypassed is the actual irq of DCD events. However, the event processing itself can be tested via cxl_test calling directly into the event processing. In this way the rest of the stack; management of sparse regions, the extent device lifetimes, and the dax device operations can be tested. Add Dynamic Capacity Device tests for kernels which have DCD support. Signed-off-by: Ira Weiny --- Changes: [djiang: split up tests into sub functions for readability] [iweiny: reduce coupling of tests and call tests which are dependant from each other] [iweiny: Reduce some test redundancy] [iweiny: Add test to reject an extent with a tag] [iweiny: Adjust to new -M option in create-region] [iweiny: Properly use DC partition 1] --- test/cxl-dcd.sh | 879 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ test/meson.build | 2 + 2 files changed, 881 insertions(+) diff --git a/test/cxl-dcd.sh b/test/cxl-dcd.sh new file mode 100644 index 0000000000000000000000000000000000000000..e9bc78eca1b09766979f2f36dcd2b9be98eb58e1 --- /dev/null +++ b/test/cxl-dcd.sh @@ -0,0 +1,879 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2024 Intel Corporation. All rights reserved. + +. "$(dirname "$0")/common" + +rc=77 +set -ex + +trap 'err $LINENO' ERR + +check_prereq "jq" + +rc=1 + +dev_path="/sys/bus/platform/devices" +cxl_path="/sys/bus/cxl/devices" + +# a test extent tag +test_tag=dc-test-tag + +# +# The test devices have 2G of non DC capacity. A single DC reagion of 1G is +# added beyond that. +# +# The testing centers around 3 extents. Two are "pre-existing" on test load +# called pre-ext and pre2-ext. The other is created within this script alone +# called base. + +# +# | 2G non- | DC region (1G) | +# | DC cap | | +# | ... |-------------------------------------------------------| +# | |--------| |----------| |----------| | +# | | (base) | |(pre-ext) | |(pre2-ext)| | + +dc0_size="" +dc1_size="" + +base_dpa=0x80000000 + +# base extent at dpa 2G - 64M long +base_ext_offset=0x0 +base_ext_dpa=$(($base_dpa + $base_ext_offset)) +base_ext_length=0x4000000 + +# pre existing extent base + 128M, 64M length +# 0x00000088000000-0x0000008bffffff +pre_ext_offset=0x8000000 +pre_ext_dpa=$(($base_dpa + $pre_ext_offset)) +pre_ext_length=0x4000000 + +# pre2 existing extent base + 256M, 64M length +# 0x00000090000000-0x00000093ffffff +pre2_ext_offset=0x10000000 +pre2_ext_dpa=$(($base_dpa + $pre2_ext_offset)) +pre2_ext_length=0x4000000 + +mem="" +bus="" +device="" +decoder="" + +# ======================================================================== +# Support functions +# ======================================================================== + +create_dcd_region() +{ + mem="$1" + decoder="$2" + reg_size_string="" + if [ "$3" != "" ]; then + reg_size_string="-s $3" + fi + + dcd_partition="dc0" + if [ "$4" != "" ]; then + dcd_partition="$4" + fi + + # create region + rc=$($CXL create-region -t dc -M ${dcd_partition} -d "$decoder" -m "$mem" ${reg_size_string} | jq -r ".region") + + if [[ ! $rc ]]; then + echo "create-region failed for $decoder / $mem" + err "$LINENO" + fi + + echo ${rc} +} + +check_region() +{ + search=$1 + region_size=$2 + + result=$($CXL list -r "$search" | jq -r ".[].region") + if [ "$result" != "$search" ]; then + echo "check region failed to find $search" + err "$LINENO" + fi + + result=$($CXL list -r "$search" | jq -r ".[].size") + if [ "$result" != "$region_size" ]; then + echo "check region failed invalid size $result != $region_size" + err "$LINENO" + fi +} + +check_not_region() +{ + search=$1 + + result=$($CXL list -r "$search" | jq -r ".[].region") + if [ "$result" == "$search" ]; then + echo "check not region failed; $search found" + err "$LINENO" + fi +} + +destroy_region() +{ + local region=$1 + $CXL disable-region $region + $CXL destroy-region $region +} + +inject_extent() +{ + device="$1" + dpa="$2" + length="$3" + tag="$4" + + more="0" + if [ "$5" != "" ]; then + more="1" + fi + + echo ${dpa}:${length}:${tag}:${more} > "${dev_path}/${device}/dc_inject_extent" +} + +remove_extent() +{ + device="$1" + dpa="$2" + length="$3" + + echo ${dpa}:${length} > "${dev_path}/${device}/dc_del_extent" +} + +create_dax_dev() +{ + reg="$1" + + dax_dev=$($DAXCTL create-device -r $reg | jq -er '.[].chardev') + + echo ${dax_dev} +} + +fail_create_dax_dev() +{ + reg="$1" + + set +e + result=$($DAXCTL create-device -r $reg) + set -e + if [ "$result" == "0" ]; then + echo "FAIL device created" + err "$LINENO" + fi +} + +shrink_dax_dev() +{ + dev="$1" + new_size="$2" + + $DAXCTL disable-device $dev + $DAXCTL reconfigure-device $dev -s $new_size + $DAXCTL enable-device $dev +} + +destroy_dax_dev() +{ + dev="$1" + + $DAXCTL disable-device $dev + $DAXCTL destroy-device $dev +} + +check_dax_dev() +{ + search="$1" + size="$2" + + result=$($DAXCTL list -d $search | jq -er '.[].chardev') + if [ "$result" != "$search" ]; then + echo "check dax device failed to find $search" + err "$LINENO" + fi + result=$($DAXCTL list -d $search | jq -er '.[].size') + if [ "$result" -ne "$size" ]; then + echo "check dax device failed incorrect size $result; exp $size" + err "$LINENO" + fi +} + +# check that the dax device is not there. +check_not_dax_dev() +{ + reg="$1" + search="$2" + result=$($DAXCTL list -r $reg -D | jq -er '.[].chardev') + if [ "$result" == "$search" ]; then + echo "FAIL found $search" + err "$LINENO" + fi +} + +check_extent() +{ + region=$1 + offset=$(($2)) + length=$(($3)) + + result=$($CXL list -r "$region" -N | jq -r ".[].extents[] | select(.offset == ${offset}) | .length") + if [[ $result != $length ]]; then + echo "FAIL region $1 could not find extent @ $offset ($length)" + err "$LINENO" + fi +} + +check_extent_cnt() +{ + region=$1 + count=$(($2)) + + result=$($CXL list -r $region -N | jq -r '.[].extents[].offset' | wc -l) + if [[ $result != $count ]]; then + echo "FAIL region $1: found wrong number of extents $result; expect $count" + err "$LINENO" + fi +} + + +# ======================================================================== +# Tests +# ======================================================================== + +# testing pre existing extents must be called first as the extents were created +# by cxl-test being loaded +test_pre_existing_extents() +{ + echo "" + echo "Test: pre-existing extent" + echo "" + region=$(create_dcd_region ${mem} ${decoder}) + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | | |----------| |----------| | + # | | |(pre-ext) | |(pre2-ext)| | + check_region ${region} ${dc0_size} + # should contain pre-created extents + check_extent ${region} ${pre_ext_offset} ${pre_ext_length} + check_extent ${region} ${pre2_ext_offset} ${pre2_ext_length} + + remove_extent ${device} $pre_ext_dpa $pre_ext_length + # | | |----------| | + # | | |(pre2-ext)| | + remove_extent ${device} $pre2_ext_dpa $pre2_ext_length + # | | | + # | | | + check_extent_cnt ${region} 0 + destroy_region ${region} + check_not_region ${region} +} + +test_remove_extent_under_dax_device() +{ + # Remove the pre-created test extent out from under dax device + # stack should hold ref until dax device deleted + echo "" + echo "Test: Remove extent from under DAX dev" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | | | + # | | | + + + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + # | | |----------| | + # | | |(pre-ext) | | + check_extent ${region} ${pre_ext_offset} ${pre_ext_length} + + dax_dev=$(create_dax_dev ${region}) + # | | |----------| | + # | | |(pre-ext) | | + # | | | dax0.1 | | + check_extent_cnt ${region} 1 + remove_extent ${device} $pre_ext_dpa $pre_ext_length + # In-use extents are not released. + check_dax_dev ${dax_dev} $pre_ext_length + + check_extent_cnt ${region} 1 + destroy_dax_dev ${dax_dev} + # | | |----------| | + # | | |(pre-ext) | | + check_not_dax_dev ${region} ${dax_dev} + + check_extent_cnt ${region} 1 + # Remove after use + remove_extent ${device} $pre_ext_dpa $pre_ext_length + # | | | + # | | | + check_extent_cnt ${region} 0 + destroy_region ${region} + check_not_region ${region} +} + +test_remove_extents_in_use() +{ + echo "" + echo "Test: Remove extents under sparse dax device" + echo "" + remove_extent ${device} $base_ext_dpa $base_ext_length + check_extent_cnt ${region} 2 + remove_extent ${device} $pre_ext_dpa $pre_ext_length + check_extent_cnt ${region} 2 +} + +test_create_dax_dev_spanning_two_extents() +{ + echo "" + echo "Test: Create dax device spanning 2 extents" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + check_extent ${region} ${pre_ext_offset} ${pre_ext_length} + # | |--------| | + # | | (base) | | + inject_extent ${device} $base_ext_dpa $base_ext_length "" + check_extent ${region} ${base_ext_offset} ${base_ext_length} + # | |--------| |----------| | + # | | (base) | |(pre-ext) | | + + check_extent_cnt ${region} 2 + dax_dev=$(create_dax_dev ${region}) + # | |--------| |----------| | + # | | (base) | |(pre-ext) | | + # | | dax0.1 | | dax0.1 | | + + echo "Checking if dev dax is spanning sparse extents" + ext_sum_length="$(($base_ext_length + $pre_ext_length))" + check_dax_dev ${dax_dev} $ext_sum_length + + test_remove_extents_in_use + + destroy_dax_dev ${dax_dev} + check_not_dax_dev ${region} ${dax_dev} + + # In-use extents were not released. Check they can be removed after the + # dax device is removed. + check_extent_cnt ${region} 2 + remove_extent ${device} $base_ext_dpa $base_ext_length + remove_extent ${device} $pre_ext_dpa $pre_ext_length + check_extent_cnt ${region} 0 + destroy_region ${region} + check_not_region ${region} +} + +test_inject_tag_support() +{ + echo "" + echo "Test: inject without/with tag" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + check_extent ${region} ${pre_ext_offset} ${pre_ext_length} + inject_extent ${device} $base_ext_dpa $base_ext_length "ira" + + # extent with tag should be rejected + check_extent_cnt ${region} 1 + remove_extent ${device} $pre_ext_dpa $pre_ext_length + check_extent_cnt ${region} 0 + + destroy_region ${region} + check_not_region ${region} +} + +test_partial_extent_remove () +{ + echo "" + echo "Test: partial extent remove" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + + inject_extent ${device} $base_ext_dpa $base_ext_length "" + # | ... |-------------------------------------------------------| + # | |--------| | + # | | (base) | | + + dax_dev=$(create_dax_dev ${region}) + + # | ... |-------------------------------------------------------| + # | |--------| | + # | | (base) | | + # | | dax0.1 | | + + partial_ext_dpa="$(($base_ext_dpa + ($base_ext_length / 2)))" + partial_ext_length="$(($base_ext_length / 2))" + echo "Removing Partial : $partial_ext_dpa $partial_ext_length" + + # | | |---| | + # Partial + + remove_extent ${device} $partial_ext_dpa $partial_ext_length + # In-use extents are not released. + check_extent_cnt ${region} 1 + + # | ... |-------------------------------------------------------| + # | |--------| | + # | | (base) | | + # | | dax0.1 | | + + destroy_dax_dev ${dax_dev} + check_not_dax_dev ${region} ${dax_dev} + + # | ... |-------------------------------------------------------| + # | |--------| | + # | | (base) | | + + # Partial results in whole extent removal + check_extent_cnt ${region} 1 + remove_extent ${device} $partial_ext_dpa $partial_ext_length + # | | |---| | + # Partial + check_extent_cnt ${region} 0 + + # | ... |-------------------------------------------------------| + destroy_region ${region} + check_not_region ${region} +} + +test_multiple_extent_remove () +{ + # Test multiple extent remove + echo "" + echo "Test: multiple extent remove with single extent remove command" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + inject_extent ${device} $base_ext_dpa $base_ext_length "" + # | ... |-------------------------------------------------------| + # | |--------| |-------------------| | + # | | (base) | | (pre)-existing | | + + check_extent_cnt ${region} 2 + dax_dev=$(create_dax_dev ${region}) + + # | ... |-------------------------------------------------------| + # | |--------| |-------------------| | + # | | (base) | | (pre)-existing | | + # | | dax0.1 | | dax0.1 | | + + partial_ext_dpa="$(($base_ext_dpa + ($base_ext_length / 2)))" + partial_ext_length="$(($pre_ext_dpa - $base_ext_dpa))" + echo "Removing multiple in span : $partial_ext_dpa $partial_ext_length" + # |------------------| + # Partial + remove_extent ${device} $partial_ext_dpa $partial_ext_length + + # | ... |-------------------------------------------------------| + # | |--------| |-------------------| | + # | | (base) | | (pre)-existing | | + # | | dax0.1 | | dax0.1 | | + + # In-use extents are not released. + check_extent_cnt ${region} 2 + + destroy_dax_dev ${dax_dev} + check_not_dax_dev ${region} ${dax_dev} + + # | ... |-------------------------------------------------------| + # | |--------| |-------------------| | + # | | (base) | | (pre)-existing | | + + # Remove both extents + check_extent_cnt ${region} 2 + # |------------------| + # Partial + remove_extent ${device} $partial_ext_dpa $partial_ext_length + # | ... |-------------------------------------------------------| + check_extent_cnt ${region} 0 + + destroy_region ${region} + check_not_region ${region} +} + +test_destroy_region_without_extent_removal () +{ + echo "" + echo "Test: Destroy region without extent removal" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + inject_extent ${device} $base_ext_dpa $base_ext_length "" + check_extent_cnt ${region} 2 + destroy_region ${region} + check_not_region ${region} +} + +test_destroy_with_extent_and_dax () +{ + echo "" + echo "Test: Destroy region with extents and dax devices" + echo "" + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + check_extent_cnt ${region} 0 + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | | |----------| | + # | | |(pre-ext) | | + + check_extent_cnt ${region} 1 + dax_dev=$(create_dax_dev ${region}) + # | | | | | + check_dax_dev ${dax_dev} ${pre_ext_length} + destroy_region ${region} + check_not_region ${region} + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | | | + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + check_extent_cnt ${region} 0 + destroy_region ${region} + check_not_region ${region} +} + +test_dax_device_ops () +{ + echo "" + echo "Test: Fail sparse dax dev creation without space" + echo "" + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | | |-------------------| | + # | | | (pre)-existing | | + + check_extent_cnt ${region} 1 + + # | | | dax0.1 | | + + dax_dev=$(create_dax_dev ${region}) + check_dax_dev ${dax_dev} $pre_ext_length + fail_create_dax_dev ${region} + + echo "" + echo "Test: Resize sparse dax device" + echo "" + + # Shrink + # | | | dax0.1 | | + resize_ext_length=$(($pre_ext_length / 2)) + shrink_dax_dev ${dax_dev} $resize_ext_length + check_dax_dev ${dax_dev} $resize_ext_length + + # Fill + # | | | dax0.1 | dax0.2 | | + dax_dev=$(create_dax_dev ${region}) + check_dax_dev ${dax_dev} $resize_ext_length + destroy_region ${region} + check_not_region ${region} + + + # 2 extent + # create dax dev + # resize into 1st extent + # create dev on rest of 1st and all of second + # Ensure both devices are correct + + echo "" + echo "Test: Resize sparse dax device across extents" + echo "" + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + inject_extent ${device} $base_ext_dpa $base_ext_length "" + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | |--------| |-------------------| | + # | | (base) | | (pre)-existing | | + + check_extent_cnt ${region} 2 + dax_dev=$(create_dax_dev ${region}) + ext_sum_length="$(($base_ext_length + $pre_ext_length))" + + # | | dax0.1 | | dax0.1 | | + + check_dax_dev ${dax_dev} $ext_sum_length + resize_ext_length=33554432 # 32MB + + # | | D1 | | + + shrink_dax_dev ${dax_dev} $resize_ext_length + check_dax_dev ${dax_dev} $resize_ext_length + + # | | D1 | D2| | dax0.2 | | + + dax_dev=$(create_dax_dev ${region}) + remainder_length=$((ext_sum_length - $resize_ext_length)) + check_dax_dev ${dax_dev} $remainder_length + + # | | D1 | D2| | dax0.2 | | + + remainder_length=$((remainder_length / 2)) + shrink_dax_dev ${dax_dev} $remainder_length + check_dax_dev ${dax_dev} $remainder_length + + # | | D1 | D2| | dax0.2 | dax0.3 | | + + dax_dev=$(create_dax_dev ${region}) + check_dax_dev ${dax_dev} $remainder_length + destroy_region ${region} + check_not_region ${region} +} + +test_reject_overlapping () +{ + echo "" + echo "Test: Rejecting overlapping extents" + echo "" + + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + + # | 2G non- | DC region | + # | DC cap | | + # | ... |-------------------------------------------------------| + # | | |-------------------| | + # | | | (pre)-existing | | + + check_extent_cnt ${region} 1 + + # Attempt overlapping extent + # + # | | |-----------------| | + # | | | overlapping | | + + partial_ext_dpa="$(($base_ext_dpa + ($pre_ext_dpa / 2)))" + partial_ext_length=$pre_ext_length + inject_extent ${device} $partial_ext_dpa $partial_ext_length "" + + # Should only see the original ext + check_extent_cnt ${region} 1 + destroy_region ${region} + check_not_region ${region} +} + +test_two_regions() +{ + echo "" + echo "Test: create 2 regions in the same DC partition" + echo "" + region_size=$(($dc1_size / 2)) + region=$(create_dcd_region ${mem} ${decoder_dc1} ${region_size} dc1) + check_region ${region} ${region_size} + + region_two=$(create_dcd_region ${mem} ${decoder_dc1} ${region_size} dc1) + check_region ${region_two} ${region_size} + + destroy_region ${region_two} + check_not_region ${region_two} + destroy_region ${region} + check_not_region ${region} +} + +test_more_bit() +{ + echo "" + echo "Test: More bit" + echo "" + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" 1 + # More bit should hold off surfacing extent until the more bit is 0 + check_extent_cnt ${region} 0 + inject_extent ${device} $base_ext_dpa $base_ext_length "" + check_extent_cnt ${region} 2 + destroy_region ${region} + check_not_region ${region} +} + +test_driver_tear_down() +{ + echo "" + echo "Test: driver remove tear down" + echo "" + region=$(create_dcd_region ${mem} ${decoder}) + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + check_extent ${region} ${pre_ext_offset} ${pre_ext_length} + dax_dev=$(create_dax_dev ${region}) + # remove driver releases extents + modprobe -r dax_cxl + check_extent_cnt ${region} 0 +} + +test_driver_bring_up() +{ + # leave region up, driver removed. + echo "" + echo "Test: no driver inject ok" + echo "" + check_region ${region} ${dc0_size} + inject_extent ${device} $pre_ext_dpa $pre_ext_length "" + check_extent_cnt ${region} 1 + + modprobe dax_cxl + check_extent_cnt ${region} 1 + + destroy_region ${region} + check_not_region ${region} +} + +test_driver_reload() +{ + test_driver_tear_down + test_driver_bring_up +} + +test_event_reporting() +{ + # Test event reporting + # results expected + num_dcd_events_expected=2 + + echo "Test: Prep event trace" + echo "" > /sys/kernel/tracing/trace + echo 1 > /sys/kernel/tracing/events/cxl/enable + echo 1 > /sys/kernel/tracing/tracing_on + + inject_extent ${device} $base_ext_dpa $base_ext_length "" + remove_extent ${device} $base_ext_dpa $base_ext_length + + echo 0 > /sys/kernel/tracing/tracing_on + + echo "Test: Events seen" + trace_out=$(cat /sys/kernel/tracing/trace) + + # Look for DCD events + num_dcd_events=$(grep -c "cxl_dynamic_capacity" <<< "${trace_out}") + echo " LOG (Expected) : (Found)" + echo " DCD events ($num_dcd_events_expected) : $num_dcd_events" + + if [ "$num_dcd_events" -ne $num_dcd_events_expected ]; then + err "$LINENO" + fi +} + + +# ======================================================================== +# main() +# ======================================================================== + +modprobe -r cxl_test +modprobe cxl_test + +readarray -t memdevs < <("$CXL" list -b cxl_test -Mi | jq -r '.[].memdev') + +for mem in ${memdevs[@]}; do + dc0_size=$($CXL list -m $mem | jq -r '.[].dc0_size') + if [ "$dc0_size" == "null" ]; then + continue + fi + dc1_size=$($CXL list -m $mem | jq -r '.[].dc1_size') + if [ "$dc1_size" == "null" ]; then + continue + fi + decoder=$($CXL list -b cxl_test -D -d root -m "$mem" | + jq -r ".[] | + select(.dc0_capable == true) | + select(.nr_targets == 1) | + select(.max_available_extent >= ${dc0_size}) | + .decoder") + decoder_dc1=$($CXL list -b cxl_test -D -d root -m "$mem" | + jq -r ".[] | + select(.dc1_capable == true) | + select(.nr_targets == 1) | + select(.max_available_extent >= ${dc1_size}) | + .decoder") + if [[ $decoder ]]; then + bus=`"$CXL" list -b cxl_test -m ${mem} | jq -r '.[].bus'` + device=$($CXL list -m $mem | jq -r '.[].host') + break + fi +done + +echo "TEST: DCD test device bus:${bus} decoder:${decoder} mem:${mem} device:${device} size:${dc0_size}" + +if [ "$decoder" == "" ] || [ "$device" == "" ] || [ "$dc0_size" == "" ]; then + echo "No mem device/decoder found with DCD support" + exit 77 +fi + +if [ "$decoder_dc1" == "" ]; then + echo "insufficient DC capability for ${mem}/${device}" + exit 77 +fi + +# testing pre existing extents must be called first as the extents were created +# by cxl-test being loaded +test_pre_existing_extents +test_remove_extent_under_dax_device +test_create_dax_dev_spanning_two_extents +test_inject_tag_support +test_partial_extent_remove +test_multiple_extent_remove +test_destroy_region_without_extent_removal +test_destroy_with_extent_and_dax +test_dax_device_ops +test_reject_overlapping +test_two_regions +test_more_bit +test_driver_reload +test_event_reporting + +modprobe -r cxl_test + +check_dmesg "$LINENO" + +exit 0 diff --git a/test/meson.build b/test/meson.build index d871e28e17ce512cd1e7b43f3ec081729fe5e03a..1cfcb60d16e05272893ae1c67820aa8614281505 100644 --- a/test/meson.build +++ b/test/meson.build @@ -161,6 +161,7 @@ cxl_sanitize = find_program('cxl-sanitize.sh') cxl_destroy_region = find_program('cxl-destroy-region.sh') cxl_qos_class = find_program('cxl-qos-class.sh') cxl_poison = find_program('cxl-poison.sh') +cxl_dcd = find_program('cxl-dcd.sh') tests = [ [ 'libndctl', libndctl, 'ndctl' ], @@ -194,6 +195,7 @@ tests = [ [ 'cxl-destroy-region.sh', cxl_destroy_region, 'cxl' ], [ 'cxl-qos-class.sh', cxl_qos_class, 'cxl' ], [ 'cxl-poison.sh', cxl_poison, 'cxl' ], + [ 'cxl-dcd.sh', cxl_dcd, 'cxl' ], ] if get_option('destructive').enabled()