From patchwork Wed Mar 6 10:39:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13583844 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D43A5FDDC; Wed, 6 Mar 2024 10:40:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.15 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721617; cv=none; b=hU01BQa4fmOko1yWPMxU7rmbBf0VVt39sZ04IG5dadyOSsRuBZG8sPhwWLiDF3yPZLNoPtXVOtimhBfPkz1VV2hrXsP6FzjBPawHhjeqUhT+m1pKs/ttuqUiksCS9GWub+cp2vgVZiV6cVNzY3xpDVhyy32kH9/sq4/9rwdyAEU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721617; c=relaxed/simple; bh=ytJVUPhnNOpLuteR3P9HW/TY0UWzPBgOZqqHRWr+b7U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IYgFXfHJ3tGHpW8nF0g+GC+Kt8SK6GdZdqMsfnMVGDAqzy+27Qo5sxfNAgFHycav3/N9hJMNIJAM16SccDJAquKl5JExQXM3dXHjy7Tj1OELC43sE13Lef/p2ZEJe1QGsFUGcLLhJk0q5uuxE2Gemyg3u7YAaIRmEA9Crp1Epsc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LOvf1xpv; arc=none smtp.client-ip=192.198.163.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LOvf1xpv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709721616; x=1741257616; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ytJVUPhnNOpLuteR3P9HW/TY0UWzPBgOZqqHRWr+b7U=; b=LOvf1xpvBPnKtxkj93sGzS2u/48wszrZsVOc/Kwae74cgL0CXb06/oDC Ay/cIYt06Z9JntQaRuqW1g76NZk2GZdFHvvjLc7IHtInMpjE7r6VErPQ0 Xr/MRbqyabUEJ1uk10WhTPGHtnzxrdzJj6SISLrvhPCL1+rymj3loX4KV rux4YryGaWF6cb6L6jPl0HoO75Lm3D6GTtKG1mB7OedlPtJ4KA3l/uRmk j5UjqNPxkArgE+M2ZKl1nRF6Hx/NUCHO15C+2PYIMNpqSsWkyPanzAoam EUvDIhNdVo7ffHLwNqb2Y5rMpOOe3mEvhBBHXTaFtSgvudrndHwlCwtA0 w==; X-IronPort-AV: E=McAfee;i="6600,9927,11004"; a="4497564" X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="4497564" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:39:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="9803667" Received: from ksznyce-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.213.25.14]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:39:45 -0800 From: Maciej Wieczor-Retman To: Fenghua Yu , Reinette Chatre , Shuah Khan Cc: ilpo.jarvinen@linux.intel.com, tony.luck@intel.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH 1/4] selftests/resctrl: Adjust effective L3 cache size with SNC enabled Date: Wed, 6 Mar 2024 11:39:18 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Sub-NUMA Cluster divides CPUs sharing an L3 cache into separate NUMA nodes. Systems may support splitting into either two or four nodes. When SNC mode is enabled the effective amount of L3 cache available for allocation is divided by the number of nodes per L3. Detect which SNC mode is active by comparing the number of CPUs that share a cache with CPU0, with the number of CPUs on node0. Signed-off-by: Tony Luck Co-developed-by: Maciej Wieczor-Retman Signed-off-by: Maciej Wieczor-Retman --- tools/testing/selftests/resctrl/resctrl.h | 4 ++ tools/testing/selftests/resctrl/resctrlfs.c | 59 +++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/tools/testing/selftests/resctrl/resctrl.h b/tools/testing/selftests/resctrl/resctrl.h index 2051bd135e0d..41811e87f81c 100644 --- a/tools/testing/selftests/resctrl/resctrl.h +++ b/tools/testing/selftests/resctrl/resctrl.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -43,6 +44,8 @@ #define DEFAULT_SPAN (250 * MB) +#define MAX_SNC 4 + #define PARENT_EXIT() \ do { \ kill(ppid, SIGKILL); \ @@ -129,6 +132,7 @@ extern pid_t bm_pid, ppid; extern char llc_occup_path[1024]; +int snc_ways(void); int get_vendor(void); bool check_resctrlfs_support(void); int filter_dmesg(void); diff --git a/tools/testing/selftests/resctrl/resctrlfs.c b/tools/testing/selftests/resctrl/resctrlfs.c index 1cade75176eb..e4d3624a8817 100644 --- a/tools/testing/selftests/resctrl/resctrlfs.c +++ b/tools/testing/selftests/resctrl/resctrlfs.c @@ -156,6 +156,63 @@ int get_domain_id(const char *resource, int cpu_no, int *domain_id) return 0; } +/* + * Count number of CPUs in a /sys bit map + */ +static unsigned int count_sys_bitmap_bits(char *name) +{ + FILE *fp = fopen(name, "r"); + int count = 0, c; + + if (!fp) + return 0; + + while ((c = fgetc(fp)) != EOF) { + if (!isxdigit(c)) + continue; + switch (c) { + case 'f': + count++; + case '7': case 'b': case 'd': case 'e': + count++; + case '3': case '5': case '6': case '9': case 'a': case 'c': + count++; + case '1': case '2': case '4': case '8': + count++; + } + } + fclose(fp); + + return count; +} + +/* + * Detect SNC by comparing #CPUs in node0 with #CPUs sharing LLC with CPU0. + * If some CPUs are offline the numbers may not be exact multiples of each + * other. Any offline CPUs on node0 will be also gone from shared_cpu_map of + * CPU0 but offline CPUs from other nodes will only make the cache_cpus value + * lower. Still try to get the ratio right by preventing the second possibility. + */ +int snc_ways(void) +{ + int node_cpus, cache_cpus, i; + + node_cpus = count_sys_bitmap_bits("/sys/devices/system/node/node0/cpumap"); + cache_cpus = count_sys_bitmap_bits("/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_map"); + + if (!node_cpus || !cache_cpus) { + fprintf(stderr, "Warning could not determine Sub-NUMA Cluster mode\n"); + return 1; + } + + for (i = 1; i <= MAX_SNC ; i++) { + if (i * node_cpus >= cache_cpus) + return i; + } + + return 1; +} + /* * get_cache_size - Get cache size for a specified CPU * @cpu_no: CPU number @@ -211,6 +268,8 @@ int get_cache_size(int cpu_no, const char *cache_type, unsigned long *cache_size break; } + if (cache_num == 3) + *cache_size /= snc_ways(); return 0; } From patchwork Wed Mar 6 10:39:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13583845 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48CBA605A6; Wed, 6 Mar 2024 10:40:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.15 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721621; cv=none; b=HkMJT1agczzy5cu7xFcipostMxOyY+gO4MfKLnV57zFvPL//tfrSxl7nQLPMUAY+xItogWtYs+oABrQYA7Mf+SZOTlHc3T9OscOCOURBbDF/6j8hjKXfnBkfE2HXXvcNMlZewhw7kJ+w53ZTp0YyvJsdhC0C/bWKp6MX54Z4/rA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721621; c=relaxed/simple; bh=OeMpkQvXEIjf/HvoRVZNwJ8BVyXuhxxDqh/xCdOmCks=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CE6WNKeVGGmLzwU/xEFrjyFYeX4W+5e0C4fuWNjQ/t3JhdmLsUNoDAhY29J7bWWMlTRF3q8SoQpnG5+QhRO/EdHqDL5ItCExENh5z0fzsdDk5oiBUvUeqp+gCn3ulIzb9Xio22U/pylicMxdpl2/ETI03GdymAHG/oEljuYFJGY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=VWpRpgmS; arc=none smtp.client-ip=192.198.163.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="VWpRpgmS" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709721618; x=1741257618; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OeMpkQvXEIjf/HvoRVZNwJ8BVyXuhxxDqh/xCdOmCks=; b=VWpRpgmSwCsfKxzK9+JJlndVFw7ZWwrnknaBdjAZAon1MbJ3G7HUwLnR NcU8BqwvbB6GrCM3tm4dzj8ksOp/YYpl/LxyrsHe5/FeARwoHqa9o/hta ubChqfN3/QjADAg6f6wdc0aGBylcEiAxK/GleWkSXRANJ5Pvx6/226V/d jHp2SBB+J7XHQhGP+sT1kNj5QKnsjXTThT/jKikgGIu2RlUJSKLN3j0Ey W8VKS0LB3AT+cRBYweeNJXyD+TU0auY2uasmn/g9mmUdEJMSFGVYNj8JC mGTIToDyeEc9BAJ8GvJlCinG6dytL7gvrHH8x/F5ooABqUguhKqiAFp1V g==; X-IronPort-AV: E=McAfee;i="6600,9927,11004"; a="4497573" X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="4497573" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:39:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="9803718" Received: from ksznyce-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.213.25.14]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:39:52 -0800 From: Maciej Wieczor-Retman To: Fenghua Yu , Reinette Chatre , Shuah Khan Cc: ilpo.jarvinen@linux.intel.com, tony.luck@intel.com, "Shaopeng Tan (Fujitsu)" , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH 2/4] selftests/resctrl: SNC support for CMT Date: Wed, 6 Mar 2024 11:39:19 +0100 Message-ID: <75849cb145429798b21c23b6be4abd7ece9df57b.1709721159.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Cache Monitoring Technology (CMT) works by measuring how much data in L3 cache is occupied by a given process identified by its Resource Monitoring ID (RMID). On systems with Sub-Numa Clusters (SNC) enabled, a process can occupy not only the cache that belongs to its own NUMA node but also pieces of other NUMA nodes' caches that lie on the same socket. A simple correction to make the CMT selftest NUMA-aware is to sum values reported by all nodes on the same socket for a given RMID. Reported-by: "Shaopeng Tan (Fujitsu)" Closes: https://lore.kernel.org/all/TYAPR01MB6330B9B17686EF426D2C3F308B25A@TYAPR01MB6330.jpnprd01.prod.outlook.com/ Signed-off-by: Maciej Wieczor-Retman --- tools/testing/selftests/resctrl/cache.c | 17 +++++++++++------ tools/testing/selftests/resctrl/resctrl.h | 4 +++- tools/testing/selftests/resctrl/resctrl_val.c | 9 ++++++--- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/tools/testing/selftests/resctrl/cache.c b/tools/testing/selftests/resctrl/cache.c index 1b339d6bbff1..dab81920033b 100644 --- a/tools/testing/selftests/resctrl/cache.c +++ b/tools/testing/selftests/resctrl/cache.c @@ -161,16 +161,21 @@ int perf_event_measure(int pe_fd, struct perf_event_read *pe_read, * * Return: =0 on success. <0 on failure. */ -int measure_llc_resctrl(const char *filename, int bm_pid) +int measure_llc_resctrl(const char *filename, int bm_pid, const char *ctrlgrp, + const char *mongrp, int res_id) { - unsigned long llc_occu_resc = 0; + unsigned long sum = 0, llc_occu_resc = 0; int ret; - ret = get_llc_occu_resctrl(&llc_occu_resc); - if (ret < 0) - return ret; + for (int i = 0 ; i < snc_ways() ; i++) { + set_cmt_path(ctrlgrp, mongrp, res_id + i); + ret = get_llc_occu_resctrl(&llc_occu_resc); + if (ret < 0) + return ret; + sum += llc_occu_resc; + } - return print_results_cache(filename, bm_pid, llc_occu_resc); + return print_results_cache(filename, bm_pid, sum); } /* diff --git a/tools/testing/selftests/resctrl/resctrl.h b/tools/testing/selftests/resctrl/resctrl.h index 41811e87f81c..178fb2eab13a 100644 --- a/tools/testing/selftests/resctrl/resctrl.h +++ b/tools/testing/selftests/resctrl/resctrl.h @@ -133,6 +133,7 @@ extern pid_t bm_pid, ppid; extern char llc_occup_path[1024]; int snc_ways(void); +void set_cmt_path(const char *ctrlgrp, const char *mongrp, char sock_num); int get_vendor(void); bool check_resctrlfs_support(void); int filter_dmesg(void); @@ -182,7 +183,8 @@ int perf_open(struct perf_event_attr *pea, pid_t pid, int cpu_no); int perf_event_reset_enable(int pe_fd); int perf_event_measure(int pe_fd, struct perf_event_read *pe_read, const char *filename, int bm_pid); -int measure_llc_resctrl(const char *filename, int bm_pid); +int measure_llc_resctrl(const char *filename, int bm_pid, const char *ctrlgrp, + const char *mongrp, int res_id); void show_cache_info(int no_of_bits, __u64 avg_llc_val, size_t cache_span, bool lines); /* diff --git a/tools/testing/selftests/resctrl/resctrl_val.c b/tools/testing/selftests/resctrl/resctrl_val.c index 5a49f07a6c85..e75e3923ebe2 100644 --- a/tools/testing/selftests/resctrl/resctrl_val.c +++ b/tools/testing/selftests/resctrl/resctrl_val.c @@ -557,7 +557,7 @@ static int print_results_bw(char *filename, int bm_pid, float bw_imc, return 0; } -static void set_cmt_path(const char *ctrlgrp, const char *mongrp, char sock_num) +void set_cmt_path(const char *ctrlgrp, const char *mongrp, char sock_num) { if (strlen(ctrlgrp) && strlen(mongrp)) sprintf(llc_occup_path, CON_MON_LCC_OCCUP_PATH, RESCTRL_PATH, @@ -698,8 +698,8 @@ int resctrl_val(const struct resctrl_test *test, { char *resctrl_val = param->resctrl_val; unsigned long bw_resc_start = 0; + int res_id, ret = 0, pipefd[2]; struct sigaction sigact; - int ret = 0, pipefd[2]; char pipe_message = 0; union sigval value; @@ -828,6 +828,8 @@ int resctrl_val(const struct resctrl_test *test, sleep(1); /* Test runs until the callback setup() tells the test to stop. */ + get_domain_id("L3", uparams->cpu, &res_id); + res_id *= snc_ways(); while (1) { ret = param->setup(test, uparams, param); if (ret == END_OF_TESTS) { @@ -844,7 +846,8 @@ int resctrl_val(const struct resctrl_test *test, break; } else if (!strncmp(resctrl_val, CMT_STR, sizeof(CMT_STR))) { sleep(1); - ret = measure_llc_resctrl(param->filename, bm_pid); + ret = measure_llc_resctrl(param->filename, bm_pid, param->ctrlgrp, + param->mongrp, res_id); if (ret) break; } From patchwork Wed Mar 6 10:39:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13583846 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C62A605B6; Wed, 6 Mar 2024 10:40:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.15 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721621; cv=none; b=IFWrcomKh4lDfvqzcx6CD0uc0Hkdd2nHrdZwsjj6PVpBBcSXy9oUMdAvyQZ7vxS9ib7HL/e92NOmZvR6PRlO+tAsPFdxD/AzxoUoE2Ymrm4xdnfnIZq74A2FXUKV/lX/EaUHPshPRyqtOAixupg2jVfk1JpKUO8KmJagRJFmEPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721621; c=relaxed/simple; bh=s+ut/76woukc2cIw6pdftfYqrtPuS0qGolRJE53TM8Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HbR9aLzlPm6rP97JXZzVQz/V9+T3a2rGVwJI+JUCJuoC+0iOXgjaQPQtRf8mXWzQtzOEMTyLILw6X7UnanBJI89y6BPM7r1JpDgYpYbizlIkF8EyEN+S3VqXvxpBCkFCKullXH41w4T9GKRY441CMuToO8mR/tpF9uP+fGXK4Gs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bOZqnaV3; arc=none smtp.client-ip=192.198.163.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bOZqnaV3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709721618; x=1741257618; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=s+ut/76woukc2cIw6pdftfYqrtPuS0qGolRJE53TM8Q=; b=bOZqnaV3LuX5zHVNzACjo2KYVFSFeKOD8NPyqgfE3QSzPt53TKfCbIjJ DsqL+QyH92dnmmAW6YcxXTrheFzdTqnkBsAbngVjsq8nItNXVE0reHNO+ mtb/bza8sQYywIOhlZDBFj6AWuAXrjFP4U2QjCq7xdCtM0sQgVkbK7a22 XQS7ncdHFJVBJOex76XBUF31E/427oVfsZfcOQbYNf6/KFvK5vXPAjkG+ 6wO3tnqbKKj3j7B7vPCQRN4D7mzctEp1Q8dzvw02oHyp3wxRTaU9Wye8S i7pZcx1wgojRBJzyeNT0kTDRCXG611KMXhUjXhB3hI0LVWdpV33/cvDI4 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11004"; a="4497591" X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="4497591" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:40:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="9803753" Received: from ksznyce-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.213.25.14]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:39:59 -0800 From: Maciej Wieczor-Retman To: Fenghua Yu , Reinette Chatre , Shuah Khan Cc: ilpo.jarvinen@linux.intel.com, tony.luck@intel.com, "Shaopeng Tan (Fujitsu)" , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH 3/4] selftests/resctrl: SNC support for MBM Date: Wed, 6 Mar 2024 11:39:20 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Memory Bandwidth Monitoring (MBM) measures how much data flows between cache levels. Only the flow for a process specified with a Resource Monitoring ID (RMID) is measured. Sub-Numa Clustering (SNC) causes MBM selftest to fail because the increased amount of NUMA nodes per socket is not taken into account. That in turn makes the test use incorrect values for the start and end of the measurement by tracking the wrong node. For the MBM selftest to be NUMA-aware it needs to track the start and end values of a measurement not for a single node but for all nodes on the same socket. Then summing all measured values comes out as the real bandwidth used by the process. Reported-by: "Shaopeng Tan (Fujitsu)" Closes: https://lore.kernel.org/lkml/TYAPR01MB6330A4EB3633B791939EA45E8B39A@TYAPR01MB6330.jpnprd01.prod.outlook.com/ Signed-off-by: Maciej Wieczor-Retman --- tools/testing/selftests/resctrl/mba_test.c | 1 - tools/testing/selftests/resctrl/resctrl_val.c | 37 ++++++++++++------- 2 files changed, 23 insertions(+), 15 deletions(-) diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c index 7946e32e85c8..fc31a61dab0c 100644 --- a/tools/testing/selftests/resctrl/mba_test.c +++ b/tools/testing/selftests/resctrl/mba_test.c @@ -147,7 +147,6 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param struct resctrl_val_param param = { .resctrl_val = MBA_STR, .ctrlgrp = "c1", - .mongrp = "m1", .filename = RESULT_FILE_NAME, .bw_report = "reads", .setup = mba_setup diff --git a/tools/testing/selftests/resctrl/resctrl_val.c b/tools/testing/selftests/resctrl/resctrl_val.c index e75e3923ebe2..2fe9f8bb4a45 100644 --- a/tools/testing/selftests/resctrl/resctrl_val.c +++ b/tools/testing/selftests/resctrl/resctrl_val.c @@ -595,9 +595,10 @@ static void initialize_llc_occu_resctrl(const char *ctrlgrp, const char *mongrp, static int measure_vals(const struct user_params *uparams, struct resctrl_val_param *param, - unsigned long *bw_resc_start) + unsigned long *bw_resc_start, + int res_id) { - unsigned long bw_resc, bw_resc_end; + unsigned long bw_resc = 0, bw_resc_sum = 0, bw_resc_end; float bw_imc; int ret; @@ -612,17 +613,19 @@ static int measure_vals(const struct user_params *uparams, if (ret < 0) return ret; - ret = get_mem_bw_resctrl(&bw_resc_end); - if (ret < 0) - return ret; + for (int i = 0 ; i < snc_ways() ; i++) { + set_mbm_path(param->ctrlgrp, strlen(param->mongrp) > 0 ? param->mongrp : NULL, + res_id + i); + ret = get_mem_bw_resctrl(&bw_resc_end); + bw_resc = (bw_resc_end - bw_resc_start[i]) / MB; + bw_resc_sum += bw_resc; + bw_resc_start[i] = bw_resc_end; + } - bw_resc = (bw_resc_end - *bw_resc_start) / MB; - ret = print_results_bw(param->filename, bm_pid, bw_imc, bw_resc); + ret = print_results_bw(param->filename, bm_pid, bw_imc, bw_resc_sum); if (ret) return ret; - *bw_resc_start = bw_resc_end; - return 0; } @@ -697,12 +700,16 @@ int resctrl_val(const struct resctrl_test *test, struct resctrl_val_param *param) { char *resctrl_val = param->resctrl_val; - unsigned long bw_resc_start = 0; int res_id, ret = 0, pipefd[2]; + unsigned long *bw_resc_start; struct sigaction sigact; char pipe_message = 0; union sigval value; + bw_resc_start = calloc(snc_ways(), sizeof(unsigned long)); + if (!bw_resc_start) + return -1; + if (strcmp(param->filename, "") == 0) sprintf(param->filename, "stdio"); @@ -710,7 +717,7 @@ int resctrl_val(const struct resctrl_test *test, !strncmp(resctrl_val, MBM_STR, sizeof(MBM_STR))) { ret = validate_bw_report_request(param->bw_report); if (ret) - return ret; + goto out_free; } /* @@ -721,7 +728,7 @@ int resctrl_val(const struct resctrl_test *test, if (pipe(pipefd)) { ksft_perror("Unable to create pipe"); - + free(bw_resc_start); return -1; } @@ -733,7 +740,7 @@ int resctrl_val(const struct resctrl_test *test, bm_pid = fork(); if (bm_pid == -1) { ksft_perror("Unable to fork"); - + free(bw_resc_start); return -1; } @@ -841,7 +848,7 @@ int resctrl_val(const struct resctrl_test *test, if (!strncmp(resctrl_val, MBM_STR, sizeof(MBM_STR)) || !strncmp(resctrl_val, MBA_STR, sizeof(MBA_STR))) { - ret = measure_vals(uparams, param, &bw_resc_start); + ret = measure_vals(uparams, param, bw_resc_start, res_id); if (ret) break; } else if (!strncmp(resctrl_val, CMT_STR, sizeof(CMT_STR))) { @@ -855,6 +862,8 @@ int resctrl_val(const struct resctrl_test *test, out: kill(bm_pid, SIGKILL); +out_free: + free(bw_resc_start); return ret; } From patchwork Wed Mar 6 10:39:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13583847 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4278E60884; Wed, 6 Mar 2024 10:40:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.15 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721623; cv=none; b=CmCfXgbSE7rXoubYey5HbSQPFGEhPuZv8SO2vp5QxsWKdopkw1litETUx0Y5vwPyJA32lKwmepHw5+pzUijNry27KiqnrKctmo4Rhf181mz8dqnNxI6gX79UAd6hTh0uWO+g5uR4jkHy7dqjnP8KBeDOi4rtNIYU27FS22iJjX8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709721623; c=relaxed/simple; bh=l02LUWZ0uZyyld4Sn9bjwcmqKToiN7adxC+/gtNEed8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fWoBXkiF63pUcZ3/zuXX21z4cqBVqLYXTMakVq5aYX3ZFUsZr38RhY728nbLt4jDasEsqpKVMxqyjtGqET3s5EC6siu4jZutmqU14AW59mbA2Q3pOGq7GCiX+Zctz9Y1XKFeZICmVPvbQMcOWohodZiakK/gdFinCVgYQrJs4ow= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Y8M6s2aK; arc=none smtp.client-ip=192.198.163.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Y8M6s2aK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709721621; x=1741257621; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l02LUWZ0uZyyld4Sn9bjwcmqKToiN7adxC+/gtNEed8=; b=Y8M6s2aKNQNcYXbzqgjcxECufv6pKGIIyJrqkkkRvfOQkfNdo3hnehlA jhnb622en5R3QEfu5/M7Vgz/hxBrP2lwONY0ZR68fXi2CkqVoMugwB+gk i7T5LhM99v93CmJERbhQR2cdXZtH/1GASA+s2lAMjKKQZvgK0Rc5F6BtL tCHRUoJrJrrviffjggFvPVX5xglOw4502CBUzxdTsvAbjoqDvbfxsWHKG OLIlNEePl700OVUw1LRzawktWnnlDAPxpOo5XGADDTtKkpjr6+/aqqTz7 ls9dS3h2r86mZzHwyqsk0NIbVN1apmuvKVRoZsgIspPhcsUsG25vA8rhz A==; X-IronPort-AV: E=McAfee;i="6600,9927,11004"; a="4497610" X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="4497610" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:40:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,208,1705392000"; d="scan'208";a="9803790" Received: from ksznyce-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.213.25.14]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2024 02:40:10 -0800 From: Maciej Wieczor-Retman To: Fenghua Yu , Reinette Chatre , Shuah Khan Cc: ilpo.jarvinen@linux.intel.com, tony.luck@intel.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: [PATCH 4/4] selftests/resctrl: Adjust SNC support messages Date: Wed, 6 Mar 2024 11:39:21 +0100 Message-ID: <8a158ada92f06b97a4679721e84e787e94b94647.1709721159.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Resctrl selftest prints a message on test failure that Sub-Numa Clustering (SNC) could be enabled and points the user to check theirs BIOS settings. No actual check is performed before printing that message so it is not very accurate in pinpointing a problem. Figuring out if SNC is enabled is only one part of the problem, the other being whether the kernel supports it. As there is no easy interface that simply states SNC support in the kernel one can find that information by comparing L3 cache sizes from different sources. Cache size reported by /sys/devices/system/node/node0/cpu0/cache/index3/size will always show the full cache size even if it's split by enabled SNC. On the other hand /sys/fs/resctrl/size has information about L3 size, that with kernel support is adjusted for enabled SNC. Add a function to find a cache size from /sys/fs/resctrl/size since finding that information from the other source is already implemented. Add a function that compares the two cache sizes and use it to make the SNC support message more meaningful. Add the SNC support message just after MBA's check_results() since MBA shares code with MBM and also can suffer from enabled SNC if there is no support in the kernel. Signed-off-by: Maciej Wieczor-Retman --- tools/testing/selftests/resctrl/cat_test.c | 2 +- tools/testing/selftests/resctrl/cmt_test.c | 6 +- tools/testing/selftests/resctrl/mba_test.c | 2 + tools/testing/selftests/resctrl/mbm_test.c | 4 +- tools/testing/selftests/resctrl/resctrl.h | 5 +- tools/testing/selftests/resctrl/resctrlfs.c | 69 ++++++++++++++++++++- 6 files changed, 79 insertions(+), 9 deletions(-) diff --git a/tools/testing/selftests/resctrl/cat_test.c b/tools/testing/selftests/resctrl/cat_test.c index 4cb991be8e31..1cdaadf35f03 100644 --- a/tools/testing/selftests/resctrl/cat_test.c +++ b/tools/testing/selftests/resctrl/cat_test.c @@ -253,7 +253,7 @@ static int cat_run_test(const struct resctrl_test *test, const struct user_param return ret; /* Get L3/L2 cache size */ - ret = get_cache_size(uparams->cpu, test->resource, &cache_total_size); + ret = get_sys_cache_size(uparams->cpu, test->resource, &cache_total_size); if (ret) return ret; ksft_print_msg("Cache size :%lu\n", cache_total_size); diff --git a/tools/testing/selftests/resctrl/cmt_test.c b/tools/testing/selftests/resctrl/cmt_test.c index a81f91222a89..b7cada602484 100644 --- a/tools/testing/selftests/resctrl/cmt_test.c +++ b/tools/testing/selftests/resctrl/cmt_test.c @@ -112,7 +112,7 @@ static int cmt_run_test(const struct resctrl_test *test, const struct user_param if (ret) return ret; - ret = get_cache_size(uparams->cpu, "L3", &cache_total_size); + ret = get_sys_cache_size(uparams->cpu, "L3", &cache_total_size); if (ret) return ret; ksft_print_msg("Cache size :%lu\n", cache_total_size); @@ -157,8 +157,8 @@ static int cmt_run_test(const struct resctrl_test *test, const struct user_param goto out; ret = check_results(¶m, span, n); - if (ret && (get_vendor() == ARCH_INTEL)) - ksft_print_msg("Intel CMT may be inaccurate when Sub-NUMA Clustering is enabled. Check BIOS configuration.\n"); + if (ret && (get_vendor() == ARCH_INTEL) && snc_ways() > 1 && !snc_kernel_support()) + ksft_print_msg("Kernel doesn't support Sub-NUMA Clustering but it is enabled. Check BIOS configuration.\n"); out: cmt_test_cleanup(); diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c index fc31a61dab0c..89fe3ecbf497 100644 --- a/tools/testing/selftests/resctrl/mba_test.c +++ b/tools/testing/selftests/resctrl/mba_test.c @@ -160,6 +160,8 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param goto out; ret = check_results(); + if (ret && (get_vendor() == ARCH_INTEL) && snc_ways() > 1 && !snc_kernel_support()) + ksft_print_msg("Kernel doesn't support Sub-NUMA Clustering but it is enabled. Check BIOS configuration.\n"); out: mba_test_cleanup(); diff --git a/tools/testing/selftests/resctrl/mbm_test.c b/tools/testing/selftests/resctrl/mbm_test.c index d67ffa3ec63a..e12b4b06f6d5 100644 --- a/tools/testing/selftests/resctrl/mbm_test.c +++ b/tools/testing/selftests/resctrl/mbm_test.c @@ -129,8 +129,8 @@ static int mbm_run_test(const struct resctrl_test *test, const struct user_param goto out; ret = check_results(DEFAULT_SPAN); - if (ret && (get_vendor() == ARCH_INTEL)) - ksft_print_msg("Intel MBM may be inaccurate when Sub-NUMA Clustering is enabled. Check BIOS configuration.\n"); + if (ret && (get_vendor() == ARCH_INTEL) && snc_ways() > 1 && !snc_kernel_support()) + ksft_print_msg("Kernel doesn't support Sub-NUMA Clustering but it is enabled. Check BIOS configuration.\n"); out: mbm_test_cleanup(); diff --git a/tools/testing/selftests/resctrl/resctrl.h b/tools/testing/selftests/resctrl/resctrl.h index 178fb2eab13a..038e1269a3fc 100644 --- a/tools/testing/selftests/resctrl/resctrl.h +++ b/tools/testing/selftests/resctrl/resctrl.h @@ -28,6 +28,7 @@ #define RESCTRL_PATH "/sys/fs/resctrl" #define PHYS_ID_PATH "/sys/devices/system/cpu/cpu" #define INFO_PATH "/sys/fs/resctrl/info" +#define SIZE_PATH "/sys/fs/resctrl/size" /* * CPU vendor IDs @@ -168,14 +169,16 @@ unsigned long create_bit_mask(unsigned int start, unsigned int len); unsigned int count_contiguous_bits(unsigned long val, unsigned int *start); int get_full_cbm(const char *cache_type, unsigned long *mask); int get_mask_no_shareable(const char *cache_type, unsigned long *mask); -int get_cache_size(int cpu_no, const char *cache_type, unsigned long *cache_size); int resource_info_unsigned_get(const char *resource, const char *filename, unsigned int *val); +int get_sys_cache_size(int cpu_no, const char *cache_type, unsigned long *cache_size); +int get_resctrl_cache_size(const char *cache_type, unsigned long *cache_size); void ctrlc_handler(int signum, siginfo_t *info, void *ptr); int signal_handler_register(void); void signal_handler_unregister(void); void cat_test_cleanup(void); unsigned int count_bits(unsigned long n); void cmt_test_cleanup(void); +int snc_kernel_support(void); void perf_event_attr_initialize(struct perf_event_attr *pea, __u64 config); void perf_event_initialize_read_format(struct perf_event_read *pe_read); diff --git a/tools/testing/selftests/resctrl/resctrlfs.c b/tools/testing/selftests/resctrl/resctrlfs.c index e4d3624a8817..dbd10cb7abf5 100644 --- a/tools/testing/selftests/resctrl/resctrlfs.c +++ b/tools/testing/selftests/resctrl/resctrlfs.c @@ -214,14 +214,14 @@ int snc_ways(void) } /* - * get_cache_size - Get cache size for a specified CPU + * get_sys_cache_size - Get cache size for a specified CPU * @cpu_no: CPU number * @cache_type: Cache level L2/L3 * @cache_size: pointer to cache_size * * Return: = 0 on success, < 0 on failure. */ -int get_cache_size(int cpu_no, const char *cache_type, unsigned long *cache_size) +int get_sys_cache_size(int cpu_no, const char *cache_type, unsigned long *cache_size) { char cache_path[1024], cache_str[64]; int length, i, cache_num; @@ -273,6 +273,44 @@ int get_cache_size(int cpu_no, const char *cache_type, unsigned long *cache_size return 0; } +/* + * get_resctrl_cache_size - Get cache size as reported by resctrl + * @cache_type: Cache level L2/L3 + * @cache_size: pointer to cache_size + * + * Return: = 0 on success, < 0 on failure. + */ +int get_resctrl_cache_size(const char *cache_type, unsigned long *cache_size) +{ + char line[256], cache_prefix[16], *stripped_line, *token; + size_t len; + FILE *fp; + + strcpy(cache_prefix, cache_type); + strncat(cache_prefix, ":", 1); + + fp = fopen(SIZE_PATH, "r"); + if (!fp) { + ksft_print_msg("Failed to open %s : '%s'\n", + SIZE_PATH, strerror(errno)); + return -1; + } + + while (fgets(line, sizeof(line), fp)) { + stripped_line = strstr(line, cache_prefix); + + if (stripped_line) { + len = strlen(cache_prefix); + stripped_line += len; + token = strtok(stripped_line, ";"); + if (sscanf(token, "0=%lu", cache_size) <= 0) + return -1; + } + } + fclose(fp); + return 0; +} + #define CORE_SIBLINGS_PATH "/sys/bus/cpu/devices/cpu" /* @@ -935,3 +973,30 @@ unsigned int count_bits(unsigned long n) return count; } + +/** + * snc_kernel_support - Compare system reported cache size and resctrl + * reported cache size to get an idea if SNC is supported on the kernel side. + * If SNC is enabled and the kernel does support it the value should be equal. + * If the kernel doesn't support SNC the. + * + * Return: 0 if not supported, 1 if supported, < 0 on failure. + */ +int snc_kernel_support(void) +{ + unsigned long resctrl_cache_size, node_cache_size; + int ret; + + ret = get_sys_cache_size(0, "L3", &node_cache_size); + if (ret < 0) + return ret; + + ret = get_resctrl_cache_size("L3", &resctrl_cache_size); + if (ret < 0) + return ret; + + if (resctrl_cache_size == node_cache_size) + return 1; + + return 0; +}