From patchwork Fri Oct 25 09:50:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Carlo Nonato X-Patchwork-Id: 13850382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3093BD0C5F8 for ; Fri, 25 Oct 2024 09:51:43 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.825716.1240101 (Exim 4.92) (envelope-from ) id 1t4GyI-0000XQ-0U; Fri, 25 Oct 2024 09:51:06 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 825716.1240101; Fri, 25 Oct 2024 09:51:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t4GyH-0000VD-Bg; Fri, 25 Oct 2024 09:51:05 +0000 Received: by outflank-mailman (input) for mailman id 825716; Fri, 25 Oct 2024 09:51:03 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1t4GyF-0006YL-Qb for xen-devel@lists.xenproject.org; Fri, 25 Oct 2024 09:51:03 +0000 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [2a00:1450:4864:20::534]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id a4eaf4e4-92b6-11ef-99a3-01e77a169b0f; Fri, 25 Oct 2024 11:51:02 +0200 (CEST) Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-5cbb0900c86so666255a12.0 for ; Fri, 25 Oct 2024 02:51:02 -0700 (PDT) Received: from carlo-ubuntu.home (dynamic-adsl-94-34-131-227.clienti.tiscali.it. [94.34.131.227]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5cbb629e17dsm446938a12.34.2024.10.25.02.51.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Oct 2024 02:51:01 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: a4eaf4e4-92b6-11ef-99a3-01e77a169b0f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=minervasys-tech.20230601.gappssmtp.com; s=20230601; t=1729849861; x=1730454661; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QFXHxRV0utQpqNK90yX6wgQYF4czrji82Ch4kfIut88=; b=1WgujvS+QY5zbx/rlD22qvGmcXIotsXnZfp/23ldmZXTIIdNgsXtNLCHvhhp9qeXi6 dH76F2kKP3s27Bi+brn09D9c1pWv7bTSakdnAiwBSQwu8zrFdTnw3ZBSfh7Y9d/YK6kA jyPEOEx7+GL64qSpuTe+lDzUKLbVDugZKdBMU32DrT/VRWvyM8tsPfCLOuhraDP7585J LIYtMtV0WaV8grJ/eA0VWVEA4jXgENBoNXPm+GsUUyGADumd1eLAjgVwvMETF4r2vwH1 9c09X3SKlqjae/YljEyEhkaF96cN69EEwWMif3BGIlgs79XhmhuIoNv6DIxEgD4Hc09S l5vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729849861; x=1730454661; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QFXHxRV0utQpqNK90yX6wgQYF4czrji82Ch4kfIut88=; b=aEyEkq/AoJY2o/CqxQ5IrBUCvmWbgg1vVCnRfQm//nLzE3PwhOcSlGS8vfq3vdwBDr 5hJShfzBl7YfEiJzfD1cjUIjeeFDEEKm5EIoVG+Qj5ZJZbPvdHJ6u4e2MiPVKMHm69Dz a9yVLrxCCe2aLTGAgm+fNjb2ks4TW3xmiZg7WyXZvzUeleZaRG7nzmLOSGOs9PhJTdrV NembayN6FYUuvyRcfzjA8nfhxZmcwoyiYUdDcTcLA4yVzMoDG0A6ZgsIMsxtcx93kgQd U4WDnVptj5MCNWAoqri4FvNlTPXd/dUra8hPMmMbeqPo3dutQ9SEyfHDUiz+PCFmbKoX cuvg== X-Gm-Message-State: AOJu0YxTPzsYpXrhfk0NPE71V/3hGN9yPCx5wMcKXh6y63YW6cJV0SCU P13HpIgfunmadfAg/KlKqEprGtbsTRQWtij6/u5vWhz+KwBFEq3iQ7awYqxQ3Cz9chZBWxpeedq S0E0= X-Google-Smtp-Source: AGHT+IEf5Dy9LuV7O8Nwt4q7bRLq+NfUTkCA2fnl7SCGvjA/9wPdpBM8RxZi80aIV1XDcgOFBdtavA== X-Received: by 2002:a05:6402:3513:b0:5c5:b90a:5b78 with SMTP id 4fb4d7f45d1cf-5cba242810dmr4539027a12.5.1729849861367; Fri, 25 Oct 2024 02:51:01 -0700 (PDT) From: Carlo Nonato To: xen-devel@lists.xenproject.org Cc: andrea.bastoni@minervasys.tech, Luca Miccio , Andrew Cooper , Jan Beulich , Julien Grall , Stefano Stabellini , Marco Solieri , Carlo Nonato Subject: [PATCH v9 11/13] xen/arm: add Xen cache colors command line parameter Date: Fri, 25 Oct 2024 11:50:12 +0200 Message-ID: <20241025095014.42376-12-carlo.nonato@minervasys.tech> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241025095014.42376-1-carlo.nonato@minervasys.tech> References: <20241025095014.42376-1-carlo.nonato@minervasys.tech> MIME-Version: 1.0 From: Luca Miccio Add a new command line parameter to configure Xen cache colors. These colors are dumped together with other coloring info. Benchmarking the VM interrupt response time provides an estimation of LLC usage by Xen's most latency-critical runtime task. Results on Arm Cortex-A53 on Xilinx Zynq UltraScale+ XCZU9EG show that one color, which reserves 64 KiB of L2, is enough to attain best responsiveness: - Xen 1 color latency: 3.1 us - Xen 2 color latency: 3.1 us Since this is the most common target for Arm cache coloring, the default amount of Xen colors is set to one. More colors are instead very likely to be needed on processors whose L1 cache is physically-indexed and physically-tagged, such as Cortex-A57. In such cases, coloring applies to L1 also, and there typically are two distinct L1-colors. Therefore, reserving only one color for Xen would senselessly partitions a cache memory that is already private, i.e. underutilize it. Signed-off-by: Luca Miccio Signed-off-by: Marco Solieri Signed-off-by: Carlo Nonato Reviewed-by: Jan Beulich --- v9: - no changes v8: - added bound check on xen_colors in llc_coloring_init() v7: - removed XEN_DEFAULT_COLOR - XEN_DEFAULT_NUM_COLORS is now used in a for loop to set xen default colors --- docs/misc/cache-coloring.rst | 2 ++ docs/misc/xen-command-line.pandoc | 10 ++++++++++ xen/common/llc-coloring.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/docs/misc/cache-coloring.rst b/docs/misc/cache-coloring.rst index 25f015c2de..f967df7360 100644 --- a/docs/misc/cache-coloring.rst +++ b/docs/misc/cache-coloring.rst @@ -113,6 +113,8 @@ Specific documentation is available at `docs/misc/xen-command-line.pandoc`. +----------------------+-------------------------------+ | ``buddy-alloc-size`` | Buddy allocator reserved size | +----------------------+-------------------------------+ +| ``xen-llc-colors`` | Xen color configuration | ++----------------------+-------------------------------+ Colors selection format *********************** diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index 21a7c48ad1..02ed2e0aea 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -2923,6 +2923,16 @@ mode. **WARNING: `x2apic_phys` is deprecated and superseded by `x2apic-mode`. The latter takes precedence if both are set.** +### xen-llc-colors (arm64) +> `= List of [ | - ]` + +> Default: `0: the lowermost color` + +Specify Xen LLC color configuration. This options is available only when +`CONFIG_LLC_COLORING` is enabled. +Two colors are most likely needed on platforms where private caches are +physically indexed, e.g. the L1 instruction cache of the Arm Cortex-A57. + ### xenheap_megabytes (arm32) > `= ` diff --git a/xen/common/llc-coloring.c b/xen/common/llc-coloring.c index c7f5378bbb..3b4b1a8a4f 100644 --- a/xen/common/llc-coloring.c +++ b/xen/common/llc-coloring.c @@ -11,6 +11,7 @@ #include #define NR_LLC_COLORS (1U << CONFIG_LLC_COLORS_ORDER) +#define XEN_DEFAULT_NUM_COLORS 1 bool __ro_after_init llc_coloring_enabled; boolean_param("llc-coloring", llc_coloring_enabled); @@ -27,6 +28,9 @@ static unsigned int __ro_after_init default_colors[NR_LLC_COLORS]; static unsigned int __initdata dom0_colors[NR_LLC_COLORS]; static unsigned int __initdata dom0_num_colors; +static unsigned int __ro_after_init xen_colors[NR_LLC_COLORS]; +static unsigned int __ro_after_init xen_num_colors; + #define mfn_color_mask (max_nr_colors - 1) #define mfn_to_color(mfn) (mfn_x(mfn) & mfn_color_mask) @@ -85,6 +89,13 @@ static int __init parse_dom0_colors(const char *s) } custom_param("dom0-llc-colors", parse_dom0_colors); +static int __init parse_xen_colors(const char *s) +{ + return parse_color_config(s, xen_colors, ARRAY_SIZE(xen_colors), + &xen_num_colors); +} +custom_param("xen-llc-colors", parse_xen_colors); + static void print_colors(const unsigned int *colors, unsigned int num_colors) { unsigned int i; @@ -163,6 +174,22 @@ void __init llc_coloring_init(void) for ( i = 0; i < max_nr_colors; i++ ) default_colors[i] = i; + if ( !xen_num_colors ) + { + unsigned int i; + + xen_num_colors = MIN(XEN_DEFAULT_NUM_COLORS, max_nr_colors); + + printk(XENLOG_WARNING + "Xen LLC color config not found. Using first %u colors\n", + xen_num_colors); + for ( i = 0; i < xen_num_colors; i++ ) + xen_colors[i] = i; + } + else if ( xen_num_colors > max_nr_colors || + !check_colors(xen_colors, xen_num_colors) ) + panic("Bad LLC color config for Xen\n"); + arch_llc_coloring_init(); } @@ -173,6 +200,8 @@ void dump_llc_coloring_info(void) printk("LLC coloring info:\n"); printk(" Number of LLC colors supported: %u\n", max_nr_colors); + printk(" Xen LLC colors (%u): ", xen_num_colors); + print_colors(xen_colors, xen_num_colors); } void domain_dump_llc_colors(const struct domain *d)