From patchwork Wed Jan 8 10:11:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13930480 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6AFEE77188 for ; Wed, 8 Jan 2025 10:11:53 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867072.1278486 (Exim 4.92) (envelope-from ) id 1tVT2N-0004zG-Hk; Wed, 08 Jan 2025 10:11:43 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867072.1278486; Wed, 08 Jan 2025 10:11:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT2N-0004z9-EZ; Wed, 08 Jan 2025 10:11:43 +0000 Received: by outflank-mailman (input) for mailman id 867072; Wed, 08 Jan 2025 10:11:42 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT2M-0004yz-He for xen-devel@lists.xenproject.org; Wed, 08 Jan 2025 10:11:42 +0000 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [2a00:1450:4864:20::333]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f4171666-cda8-11ef-99a4-01e77a169b0f; Wed, 08 Jan 2025 11:11:40 +0100 (CET) Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-4361e89b6daso113925375e9.3 for ; Wed, 08 Jan 2025 02:11:40 -0800 (PST) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436e2dc1763sm15465765e9.12.2025.01.08.02.11.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jan 2025 02:11:39 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f4171666-cda8-11ef-99a4-01e77a169b0f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1736331100; x=1736935900; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=jiq6XWslhxdWPp8jdeQUdb+1Oh9S+Y/2AWgC7Fw0yEI=; b=PTm/g8gO4fYfyn6Qm6kOieCnGoxjGEY5MORxCrffXqExy/vnl9adwDKzwnkhbokMQL 6y8QU02OxSy0zPYIcK8xzARuij1UX8axA7715T5o4upNHZZOn/lNsrGUOvsJFWG8oi2+ T5CdQWSf3AcP7fgHQIW1PvhaHvX4dkK7U2ftpyVjplo/3Hqf0uvtRvQ8Al30H8ka//Ei W8OqNtAOmFyNPozshUWUd6cK2oS15IOwt8iZCMEtPJ3ecxJBL4CyiHlQ6Cumy4piyuGb hm/5QqGCspC1M16clU5pr8KZxGR306iuSkMFDtLJT2wAgQ8pwKUhVZ0Dbu3gecyNk8gD KG/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736331100; x=1736935900; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jiq6XWslhxdWPp8jdeQUdb+1Oh9S+Y/2AWgC7Fw0yEI=; b=CAmydWSMiwaO2XyDKKiRozSCbE7An4LqUz1zKCEoP98l2pcSFIYOZJWyfNkJSSdyOE gN+uCZFKNHKQwScYGccrK4f8BT7is6/wwLX2PWKD1U7IB31hznxFsetLQYqHFlb+1TSN dsWAq3Uj8LdpieGjjVcNzOEgD/YSEJyLhi6wqnNvYhe1Ov15bYuBo/Vvgb2J/lG9ljLF YBeNx40G5IUI5vuBGu3ex7BSi4eDStPiRytDNhBfEGPZaiQyFL9F4Ic9GGfZaciE0Z2b eMLUKqsdTipMxt7+nb/OeKnW2ULU1AX4j4V+GKcfdK7zaofPNEcVS9Tvx66nlSj/IvR4 n57g== X-Gm-Message-State: AOJu0Yx/wq9JnjGpzAAWsekhX348JPm+TQwD1zmSd6kXISdqxdEWiOwY DSO1KZH6l0qxJGmT2E6BXodXz1VuM07dSTmkqeo/ErLxj4nDj9EGrHZqJg9XJsnxHVPTkhKZPWw = X-Gm-Gg: ASbGnctuKTbpqKGuFv5A+jwDxQ/qGX4WPjmlSFIiJs3oczZatPa8zd/RYoeMJCqttM5 B+nx8BhF52cNBBZRIfGbyIc5EZCUDADWi5Aq7Ln5FLTZtbuEHZyk+9NEL2lmmBBoTZS0WOAPHnd 52KufjC6quLQ1QRY5VjsuTEINqJGR6KQxR2IgGBKzqbZt55S4xr4zPhXdB1dXmS0wD7ioYdjZgo ukytm8q4rB550wyb703D7ImS+Lf1UzBkXxlfqY+OuzG3vEYPkUFdmoim46tSEn/1Zq0BU1rmUjW W8CBT3ZjtBNaqeHIb8vsdh6mepWZmcZD9RPEjTTPXg== X-Google-Smtp-Source: AGHT+IHxkzldPvFinqhqJsbizyW1OS7RrjlG9Mzgl6WpJKyH8mvMAFUwuvncRARsZ++CFHhtD9WYNA== X-Received: by 2002:a05:600c:a09:b0:434:f1e9:afb3 with SMTP id 5b1f17b1804b1-436e267863emr15417995e9.3.1736331100027; Wed, 08 Jan 2025 02:11:40 -0800 (PST) Message-ID: <2e61ad8d-3b1f-49bb-bc93-61c820d6caf3@suse.com> Date: Wed, 8 Jan 2025 11:11:38 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v4 1/6] x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= References: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Before we start actually adjusting behavior when ERMS is available, follow Linux commit 161ec53c702c ("x86, mem, intel: Initialize Enhanced REP MOVSB/STOSB") and zap the CPUID-derived feature flag when the MSR bit is clear. Don't extend the artificial clearing to guest view, though: Guests can take their own decision in this regard, as they can read (most of) MISC_ENABLE. Signed-off-by: Jan Beulich --- TBD: Would be nice if "cpuid=no-erms" propagated to guest view (for "cpuid=" generally meaning to affect guests as well as Xen), but since both disabling paths use setup_clear_cpu_cap() they're indistinguishable in guest_common_feature_adjustments(). A separate boolean could take care of this, but would look clumsy to me. --- v4: Also adjust guest_common_max_feature_adjustments(). v3: New. --- a/xen/arch/x86/cpu/intel.c +++ b/xen/arch/x86/cpu/intel.c @@ -337,8 +337,18 @@ static void cf_check early_init_intel(st paddr_bits = 36; if (c == &boot_cpu_data) { + uint64_t misc_enable; + check_memory_type_self_snoop_errata(); + /* + * If fast string is not enabled in IA32_MISC_ENABLE for any reason, + * clear the enhanced fast string CPU capability. + */ + rdmsrl(MSR_IA32_MISC_ENABLE, misc_enable); + if (!(misc_enable & MSR_IA32_MISC_ENABLE_FAST_STRING)) + setup_clear_cpu_cap(X86_FEATURE_ERMS); + intel_init_levelling(); } --- a/xen/arch/x86/cpu-policy.c +++ b/xen/arch/x86/cpu-policy.c @@ -487,6 +487,12 @@ static void __init guest_common_max_feat */ if ( test_bit(X86_FEATURE_RTM, fs) ) __set_bit(X86_FEATURE_RTM_ALWAYS_ABORT, fs); + + /* + * We expose MISC_ENABLE to guests, so our internal clearing of ERMS when + * FAST_STRING is not set should not affect the view of migrating-in guests. + */ + __set_bit(X86_FEATURE_ERMS, fs); } static void __init guest_common_default_feature_adjustments(uint32_t *fs) @@ -591,6 +597,15 @@ static void __init guest_common_feature_ */ if ( host_cpu_policy.feat.ibrsb ) __set_bit(X86_FEATURE_IBPB, fs); + + /* + * We expose MISC_ENABLE to guests, so our internal clearing of ERMS when + * FAST_STRING is not set should not propagate to guest view. Guests can + * judge on their own whether to ignore the CPUID bit when the MSR bit is + * clear. + */ + if ( raw_cpu_policy.feat.erms ) + __set_bit(X86_FEATURE_ERMS, fs); } static void __init calculate_pv_max_policy(void) --- a/xen/arch/x86/include/asm/msr-index.h +++ b/xen/arch/x86/include/asm/msr-index.h @@ -490,6 +490,7 @@ #define MSR_IA32_THERM_INTERRUPT 0x0000019b #define MSR_IA32_THERM_STATUS 0x0000019c #define MSR_IA32_MISC_ENABLE 0x000001a0 +#define MSR_IA32_MISC_ENABLE_FAST_STRING (1<<0) #define MSR_IA32_MISC_ENABLE_PERF_AVAIL (1<<7) #define MSR_IA32_MISC_ENABLE_BTS_UNAVAIL (1<<11) #define MSR_IA32_MISC_ENABLE_PEBS_UNAVAIL (1<<12) From patchwork Wed Jan 8 10:12:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13930481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 61610E77188 for ; Wed, 8 Jan 2025 10:12:45 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867081.1278496 (Exim 4.92) (envelope-from ) id 1tVT3D-0005ZT-SP; Wed, 08 Jan 2025 10:12:35 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867081.1278496; Wed, 08 Jan 2025 10:12:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT3D-0005ZM-PV; Wed, 08 Jan 2025 10:12:35 +0000 Received: by outflank-mailman (input) for mailman id 867081; Wed, 08 Jan 2025 10:12:35 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT3C-0005HN-W9 for xen-devel@lists.xenproject.org; Wed, 08 Jan 2025 10:12:34 +0000 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [2a00:1450:4864:20::32b]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 1426599a-cda9-11ef-a0df-8be0dac302b0; Wed, 08 Jan 2025 11:12:34 +0100 (CET) Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-436281c8a38so118367475e9.3 for ; Wed, 08 Jan 2025 02:12:34 -0800 (PST) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436e2e2275bsm15384075e9.40.2025.01.08.02.12.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jan 2025 02:12:33 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1426599a-cda9-11ef-a0df-8be0dac302b0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1736331154; x=1736935954; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=zXvyugoFvnrztoDgpMNB+n4/oYBCMLezGO47d4RFcqU=; b=fjfIQI/cnumDqT+WSZKABaamHRqvX6UjGpbx96t36LPTrVsa7IvVpIkBu6+7i9pX4P 1ykbz4uFHJdayztqxd3TjI+YWJ5YYeFf/h0Gzcir1ePIOQg4qSZtQXUHdzI3+gFhhRck N+OpizFpXGasmxJYkndLYSa8KwnGgOM6B38tzlWn69rgoRnADJnh0qZgHrs5BXL9V+1G Gmyno3fqlgGL5InQFyVB/NO/OXZyuN33E8tb1k+JQDK0ep4m7ZkOPqohfx5gVdc5qLUx YjowV0HsbEO5B6MhOV3d6gsv6Y1EdcC/1vZiXAG5/d940HBBwI5MKFgo99CUQZvJgiNj uIgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736331154; x=1736935954; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zXvyugoFvnrztoDgpMNB+n4/oYBCMLezGO47d4RFcqU=; b=fT2ZU612SwOGgdJnnlUsAZOPxf/YOhGgoPbhlX7JBySYGs7AUB+oib4/AXjXw/sh0B S4eBTWOEx78WKhNPUG5zgiY7/h4XIBG7LMks8QZ6vwQtri59K2HcgTuBoFyrpkIQfOG0 +79Po3fYciI1ncPXr7c+JwQII2a6Vv1I6WFiW4sqynHXw8+85kwbGXgKS5UXldVEpiqb 3Ztw2n8CbFFX15m9n9ANhl2vfcXa0AC5Ja2hHFceb7cM39DAeZPtqVD4eCeWgLMW7Ah8 y9QH22zvCcOVBwwak4hurdbSGehP/h2jvSl6HOjq5eb+csUKYEaPSvV4TGlIA6nhp2FD 6XXw== X-Gm-Message-State: AOJu0Yzw8BqOsHIrtpP2K6sICKJCvqHJ0Mw7aCu0AIta7LUWoeKN3T+Y wIAj7+L/QvHryvCxHukadt1ASDd3aPjZzVjZ+gxI3mTjnnTRPjD8Ue9GxxtwdUj8JyhqhsMJ5p4 = X-Gm-Gg: ASbGncsRhztJNn3IZfA+J54KzYssMe3qXZ4stfsYEyEdiJGgQeRv2E9JJ4DPiTjpe4J o8kxnpwmpub6PnCZCL2eyySgmo7tIw6GR04agKmm0AMhkKJ5ieqSrPLrieub/dr5RS8gO7VBwNs hsfdQlGo6BwcJZSY2yLY0CItus911Pt7Zu6b0gJam1MFH4Gg1LHaRRDJJxJ9ByeJSgM8rN6IuD8 eSpIJtQv/Kt6tbl2/7vAW57a21lqTQTvZO2SXhkKZhu+75UbwJvt+Eg7EbrRIbQx/5p7xen7BcD UQHkacizyLopZ21tYziwb6PGkXw361k1usM4sjIy3Q== X-Google-Smtp-Source: AGHT+IGpICQp2GOtzcPok35p9XGlX2+7mlNFLT5fzZUxMkTA6mcxQ3KXOuSr1lc6ZaZkjJjN0i72NA== X-Received: by 2002:a05:600c:198c:b0:435:306:e5dd with SMTP id 5b1f17b1804b1-436e26f47e0mr14638435e9.22.1736331153856; Wed, 08 Jan 2025 02:12:33 -0800 (PST) Message-ID: Date: Wed, 8 Jan 2025 11:12:32 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v4 2/6] x86: re-work memset() From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= References: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors: A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching. Signed-off-by: Jan Beulich --- We may want to consider branching over the REP STOSQ as well, if the number of qwords turns out to be zero. We may also want to consider using non-REP STOS{L,W,B} for the tail. --- v4: Use %r8 instead of %rsi in a few places. v3: Re-base. --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -48,6 +48,7 @@ obj-$(CONFIG_INDIRECT_THUNK) += indirect obj-$(CONFIG_PV) += ioport_emulate.o obj-y += irq.o obj-$(CONFIG_KEXEC) += machine_kexec.o +obj-y += memset.o obj-y += mm.o x86_64/mm.o obj-$(CONFIG_HVM) += monitor.o obj-y += mpparse.o --- /dev/null +++ b/xen/arch/x86/memset.S @@ -0,0 +1,30 @@ +#include + +.macro memset + and $7, %edx + shr $3, %rcx + movzbl %sil, %esi + mov $0x0101010101010101, %rax + imul %rsi, %rax + mov %rdi, %r8 + rep stosq + or %edx, %ecx + jz 0f + rep stosb +0: + mov %r8, %rax + ret +.endm + +.macro memset_erms + mov %esi, %eax + mov %rdi, %r8 + rep stosb + mov %r8, %rax + ret +.endm + +FUNC(memset) + mov %rdx, %rcx + ALTERNATIVE memset, memset_erms, X86_FEATURE_ERMS +END(memset) --- a/xen/arch/x86/string.c +++ b/xen/arch/x86/string.c @@ -22,19 +22,6 @@ void *(memcpy)(void *dest, const void *s return dest; } -void *(memset)(void *s, int c, size_t n) -{ - long d0, d1; - - asm volatile ( - "rep stosb" - : "=&c" (d0), "=&D" (d1) - : "a" (c), "1" (s), "0" (n) - : "memory"); - - return s; -} - void *(memmove)(void *dest, const void *src, size_t n) { long d0, d1, d2; From patchwork Wed Jan 8 10:12:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13930482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B73D8E7719A for ; Wed, 8 Jan 2025 10:13:09 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867084.1278506 (Exim 4.92) (envelope-from ) id 1tVT3b-000618-4E; Wed, 08 Jan 2025 10:12:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867084.1278506; Wed, 08 Jan 2025 10:12:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT3b-00060z-1R; Wed, 08 Jan 2025 10:12:59 +0000 Received: by outflank-mailman (input) for mailman id 867084; Wed, 08 Jan 2025 10:12:57 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT3Z-00060Z-Ja for xen-devel@lists.xenproject.org; Wed, 08 Jan 2025 10:12:57 +0000 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [2a00:1450:4864:20::32b]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 20d6701e-cda9-11ef-99a4-01e77a169b0f; Wed, 08 Jan 2025 11:12:55 +0100 (CET) Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-43635796b48so4144095e9.0 for ; Wed, 08 Jan 2025 02:12:55 -0800 (PST) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a1c89e2dfsm53176882f8f.74.2025.01.08.02.12.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jan 2025 02:12:54 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 20d6701e-cda9-11ef-99a4-01e77a169b0f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1736331175; x=1736935975; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=SJd+n7gK2V+zAKy/YovEFCBG4RViNaVaa+G5JynvQ0Y=; b=SESG5Vs/5FZmznDi7R5LHCl3gImG7lF2Ag1GCvBE4uO2x9+qPwT0THxKRjQ6ZgI2Ln 12kzoGzgF2en5gBeBmkTPrS8j0QLJTueM/PYbfPOS1njF7z82BXTI7DqBdCCT4DFDRER moISej7JbftmYfwfYWqcNwDL+jzdCadKPcDtYfsKOdkfG3Kkv5XmOHKgaSp05aIFjzzx Hg/5cA21yAjSsrCcnT6kbk6/IxfJZHb3sk9ZSkEIGFDbF9oJn36rongCgjRuvQff/IYm WGLWQWueaANDLHSp6f39pRShlIS4fU6Ii81T/nokVO0s3EusheFkPpzOoRLyyTCPGVBx MblQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736331175; x=1736935975; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SJd+n7gK2V+zAKy/YovEFCBG4RViNaVaa+G5JynvQ0Y=; b=fQyRJWgyWabYRkUo14bPJhk+Kh5sz++VLbJwS7iRLu0/Ek+Wc5kWF+kAzsOzu3rCki F7SaZc2552ns1PdkoMbLg9ZUq2CAthYZSKWU3JEdtS/WjBPdL6MR/Tnv+B/RVQ37PCit ZWOA7pQMLT7D7oYeuScWq6MUKgqFxZsQ8/GsbJcb7zz6Oi+vazvraxALc+3nYfs9wZYe hZy35yGzKHXaesvSV9ABp4VtZKRjmULUph4mHaTBi9JpHPLBq0Vw3F8jPnhS1/kCudhw hqg9jrSECQRuwl4Fp2ADaMjthW5rN8vKSMUao8PCxMF4PQwiA+Uz8G3Uam9UyHdjU8F2 bTNg== X-Gm-Message-State: AOJu0YzSLZTbAJ6TR5PYmPAij/XoqnVT1l8r8xfe1k2efHzJCYzgRWda o+DS8A8u4HpkIC/HvrCk1aqT+jBLTXvRpm67XjSK0WFP14FEgjGqKpSJjL9UL316IcQqP+wQgJc = X-Gm-Gg: ASbGncty0pdkf2VqvgC2HHHJ0fPQubkI9mYJgRdvxPPglniwNlPF/rdaJDGlmmuQ9Zm ACyvzHYwVdE+Dqe89WWycWOdFcR+Z8S2v2GmotLDP0zZldFzp5iOSDJBOh80l2Rr5iP0rY+/0po C8sGWQo4Br3hpK5yiP4P+JU7xyo880+nztGl8qLdXm4Cpr4Ic0R1eXHyg9w8qh5viVW275jqQEL YgrcDQ9iwOSitand9D96pWbE4ihATyXKXZr5hlH+A0w4DtpmxG1nMnCN7q4O306g/ZlHoMNs7qA Hx15yiodB0OSbxjj0+PGq418qVXpR52ag+BxfH0uLA== X-Google-Smtp-Source: AGHT+IE8V4ACYrMlmEfpW17iXFZZTNXBv8g9V7pBQrCMzefmxeej1rCQP71NCdCYerf/ji2U05v0fQ== X-Received: by 2002:a05:600c:3b94:b0:436:185f:dfae with SMTP id 5b1f17b1804b1-436dc1c213emr54693925e9.6.1736331175138; Wed, 08 Jan 2025 02:12:55 -0800 (PST) Message-ID: <4dfc23f8-3fd5-4814-9d0d-5fbbe4c3e9c8@suse.com> Date: Wed, 8 Jan 2025 11:12:53 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v4 3/6] x86: re-work memcpy() From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= References: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors: A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching. Alternatives patching, however, requires an extra precaution: It uses memcpy() itself, and hence the function may patch itself. Luckily the patched-in code only replaces the prolog of the original function. Make sure this remains this way. Additionally alternatives patching, while supposedly safe via enforcing a control flow change when modifying already prefetched code, may not really be. Afaict a request is pending to drop the first of the two options in the SDM's "Handling Self- and Cross-Modifying Code" section. Insert a serializing instruction there. Signed-off-by: Jan Beulich --- We may want to consider branching over the REP MOVSQ as well, if the number of qwords turns out to be zero. We may also want to consider using non-REP MOVS{L,W,B} for the tail. TBD: We may further need a workaround similar to Linux'es 8ca97812c3c8 ("x86/mce: Work around an erratum on fast string copy instructions"). --- v4: Use CR2 write as serializing insn, and limit its use to boot time. v3: Re-base. --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -48,6 +48,7 @@ obj-$(CONFIG_INDIRECT_THUNK) += indirect obj-$(CONFIG_PV) += ioport_emulate.o obj-y += irq.o obj-$(CONFIG_KEXEC) += machine_kexec.o +obj-y += memcpy.o obj-y += memset.o obj-y += mm.o x86_64/mm.o obj-$(CONFIG_HVM) += monitor.o --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -153,12 +153,16 @@ void init_or_livepatch add_nops(void *in * executing. * * "noinline" to cause control flow change and thus invalidate I$ and - * cause refetch after modification. + * cause refetch after modification. While the SDM continues to suggest this + * is sufficient, it may not be - issue a serializing insn afterwards as well, + * unless this is for live-patching. */ static void init_or_livepatch noinline text_poke(void *addr, const void *opcode, size_t len) { memcpy(addr, opcode, len); + if ( system_state < SYS_STATE_active ) + asm volatile ( "mov %%rax, %%cr2" ::: "memory" ); } extern void *const __initdata_cf_clobber_start[]; --- /dev/null +++ b/xen/arch/x86/memcpy.S @@ -0,0 +1,20 @@ +#include + +FUNC(memcpy) + mov %rdx, %rcx + mov %rdi, %rax + /* + * We need to be careful here: memcpy() is involved in alternatives + * patching, so the code doing the actual copying (i.e. past setting + * up registers) may not be subject to patching (unless further + * precautions were taken). + */ + ALTERNATIVE "and $7, %edx; shr $3, %rcx", \ + "rep movsb; ret", X86_FEATURE_ERMS + rep movsq + or %edx, %ecx + jz 1f + rep movsb +1: + ret +END(memcpy) --- a/xen/arch/x86/string.c +++ b/xen/arch/x86/string.c @@ -7,21 +7,6 @@ #include -void *(memcpy)(void *dest, const void *src, size_t n) -{ - long d0, d1, d2; - - asm volatile ( - " rep ; movs"__OS" ; " - " mov %k4,%k3 ; " - " rep ; movsb " - : "=&c" (d0), "=&D" (d1), "=&S" (d2) - : "0" (n/BYTES_PER_LONG), "r" (n%BYTES_PER_LONG), "1" (dest), "2" (src) - : "memory" ); - - return dest; -} - void *(memmove)(void *dest, const void *src, size_t n) { long d0, d1, d2; From patchwork Wed Jan 8 10:13:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13930496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82836E77188 for ; Wed, 8 Jan 2025 10:22:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867123.1278545 (Exim 4.92) (envelope-from ) id 1tVTCZ-0001LM-NV; Wed, 08 Jan 2025 10:22:15 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867123.1278545; Wed, 08 Jan 2025 10:22:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVTCZ-0001LF-Ku; Wed, 08 Jan 2025 10:22:15 +0000 Received: by outflank-mailman (input) for mailman id 867123; Wed, 08 Jan 2025 10:22:14 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT3x-0005HN-Gg for xen-devel@lists.xenproject.org; Wed, 08 Jan 2025 10:13:21 +0000 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [2a00:1450:4864:20::32b]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 2fe0419d-cda9-11ef-a0df-8be0dac302b0; Wed, 08 Jan 2025 11:13:20 +0100 (CET) Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-4361b6f9faeso4076785e9.1 for ; Wed, 08 Jan 2025 02:13:20 -0800 (PST) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a1c8acafesm52401179f8f.98.2025.01.08.02.13.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jan 2025 02:13:20 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2fe0419d-cda9-11ef-a0df-8be0dac302b0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1736331200; x=1736936000; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=Ik+kdwvsn5sI13Sqvs0LoMAnkH/bBuVd5/sAq0CDmGU=; b=OzfZ1q1uDOWOLVe/M1r/qLXLQSICiOUO67YSoKNGO36gkSFIhFr7YA9y+tdK2Gyhz7 QpRy4ASsLT/Kp28PNd4ZcrSC+QCF/Y7QvXV17yk6tp5KF9Vc7xMReEde+iejz76Sr+4n xzPHvY5UCt29ZrRPQsJ7vQ9Vmoo1qeDy4dUOgniwl/U3+IQJRmSbJD/xSoEUEc2/31T9 JpiZHf/nNil3UB7ARVDPpjqufVvqcf/em5DWdjBwgCVtd9zMCOOWpK0tdYfS/wS5YSNq vDzVekApxhuU7hgnYLZGJ7DQf/ZV2PzVDBDfYBGc9q2a5fwo6aLqHUe93aT/ESTUN1+z WcHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736331200; x=1736936000; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ik+kdwvsn5sI13Sqvs0LoMAnkH/bBuVd5/sAq0CDmGU=; b=KrgBq70RBsxaH8CUF62oqBIHWtFi+5BogzezHNzG5Si1O0qX930flMEQO9gPp8F0V5 zBZnm5e54wcAIT9tHpumxguy7afudWDMvZ2AlWV38mvp12saMqwzlIGapxcVZXe9nWjb XQXTPoQvGU6wIMylfmXqeZ5eBWL9mLZcokJJc38ay8D9lUE0BG/HtYq0Cznv5WP6ONgi VWZKQD/ABcZgbUKLWhuOWypV+f81ni2wbdtaDIyVkOwLN90O56atgigSXlAJ3x9CfCBb G6zxU1n0I5LrMO71MXrsHHD+vvgTw4TNseFxaib+5jZsHdR3HCHjaxulGQQRlK0HVY0b AmCw== X-Gm-Message-State: AOJu0YxjkpiAlmnTw+R6huD/j/b9D7kSdw4P9wDBBv3imXjtsQWlKgZE ftL0CE1hODnsLuW3piyquF79JY7XHfcNHCLlJRI3gwy+0faWNkkYtU6gjbiut2Yieh3rPVGaWWE = X-Gm-Gg: ASbGncvBOhvpVuH/7h6EYruLTiFgKkzYgckSGecReO4ZlbebwvbgtyPwQqsxnUGXLpz 7b89xalvLiPAVS9noHxahiUR/3CvmRlmF+9pB7KMcbXI4r2hmc1qnMjY1P9RHNLYm1FLTw9YIkd pDyj/ovuCMtc/yacv/IXKjO9ako8aKLT9z8UjFam0YijCVybaAd/NAFabHLEgHfvxgTF3bYaq6i EDggpiLOhFid3rG0t6m3cSvECKCrn/mWGg1LeDPuXSNw1Kd8ZzBUpaT9P18VHikcRDumn+KfHtI 2dHSgJ+mjKbBLNHdlrld4lI2kvxXCLseFlFDJCfJlw== X-Google-Smtp-Source: AGHT+IHaGFaaqLP4jo8cZOgpMUC9M5m87pxwlFQdAZx7tezjS6+Wa1QPSoUExOwxvVruxJ1a4gdKsA== X-Received: by 2002:a05:600c:1d0d:b0:434:fe3c:c662 with SMTP id 5b1f17b1804b1-436e1e301e4mr19196365e9.12.1736331200437; Wed, 08 Jan 2025 02:13:20 -0800 (PST) Message-ID: Date: Wed, 8 Jan 2025 11:13:19 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v4 4/6] x86: control memset() and memcpy() inlining From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= References: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Stop the compiler from inlining non-trivial memset() and memcpy() (for memset() see e.g. map_vcpu_info() or kimage_load_segments() for examples). This way we even keep the compiler from using REP STOSQ / REP MOVSQ when we'd prefer REP STOSB / REP MOVSB (when ERMS is available). With gcc10 this yields a modest .text size reduction (release build) of around 2k. Unfortunately these options aren't understood by the clang versions I have readily available for testing with; I'm unaware of equivalents. Note also that using cc-option-add is not an option here, or at least I couldn't make things work with it (in case the option was not supported by the compiler): The embedded comma in the option looks to be getting in the way. Requested-by: Andrew Cooper Signed-off-by: Jan Beulich --- v3: Re-base. v2: New. --- The boundary values are of course up for discussion - I wasn't really certain whether to use 16 or 32; I'd be less certain about using yet larger values. Similarly whether to permit the compiler to emit REP STOSQ / REP MOVSQ for known size, properly aligned blocks is up for discussion. --- a/xen/arch/x86/arch.mk +++ b/xen/arch/x86/arch.mk @@ -65,6 +65,9 @@ endif $(call cc-option-add,CFLAGS_stack_boundary,CC,-mpreferred-stack-boundary=3) export CFLAGS_stack_boundary +CFLAGS += $(call cc-option,$(CC),-mmemcpy-strategy=unrolled_loop:16:noalign$(comma)libcall:-1:noalign) +CFLAGS += $(call cc-option,$(CC),-mmemset-strategy=unrolled_loop:16:noalign$(comma)libcall:-1:noalign) + ifeq ($(CONFIG_UBSAN),y) # Don't enable alignment sanitisation. x86 has efficient unaligned accesses, # and various things (ACPI tables, hypercall pages, stubs, etc) are wont-fix. From patchwork Wed Jan 8 10:13:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13930483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 98DB5E77199 for ; Wed, 8 Jan 2025 10:14:04 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867093.1278515 (Exim 4.92) (envelope-from ) id 1tVT4X-0006dC-E4; Wed, 08 Jan 2025 10:13:57 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867093.1278515; Wed, 08 Jan 2025 10:13:57 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT4X-0006d5-AW; Wed, 08 Jan 2025 10:13:57 +0000 Received: by outflank-mailman (input) for mailman id 867093; Wed, 08 Jan 2025 10:13:55 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT4V-0006cy-Ko for xen-devel@lists.xenproject.org; Wed, 08 Jan 2025 10:13:55 +0000 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [2a00:1450:4864:20::32d]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 43f3fb00-cda9-11ef-a0df-8be0dac302b0; Wed, 08 Jan 2025 11:13:54 +0100 (CET) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4361c705434so119545695e9.3 for ; Wed, 08 Jan 2025 02:13:54 -0800 (PST) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a7f1c26a6sm3900751f8f.53.2025.01.08.02.13.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jan 2025 02:13:53 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 43f3fb00-cda9-11ef-a0df-8be0dac302b0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1736331234; x=1736936034; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=Bo1b/k71KJVmsVTKGv+xBBoI5XLp0eOf8O7yEx73nPg=; b=a/v8XjZVeeOopyxRPQd15eA3furRxb+N610aBOMr5VahhQWVAD22+ZTjC4lQQi6e71 AwyWNiIl49fMddIgT2f/Op7lZDY2m4AwKgRvKNLuw98STdpNA7/hy7nVQit8sCvz4DoH z2BFLNL9hl9xxsf1CniPNbqOKo7L+n/P13gtmIOTMi/f6ivVIgV7ntJFchGAEXyDMsWz 7yraZ4kN8Yd2rKRf4Lek0TXMDZikytkq2MeDwxL0AkGJJkfcdexrkIq9P+yL2A/3dzSE 1Ha9JpcvXzGzlUZLbEcSFIddw7xr/RZubgqgEtCQRmjJrBuWxhXLdSrm6I7Zh7YCl4q2 KB5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736331234; x=1736936034; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Bo1b/k71KJVmsVTKGv+xBBoI5XLp0eOf8O7yEx73nPg=; b=VHBTp8MgbeNvqawpLkFrXYa2YT7yV32rl4eSeRlB8NzwlLZ343aS1k1sKtFkGsOnmA jlklm5fH/H9IJvR+nSshVEwhw39KpkJ/eyqa1c/OqLcXP4PHvWQBOPES7tkwzvq0xj09 dooMUypbUgVMJS4kTKkoXhzk49o9ITZ1nhSLFp0v9jafPQE1mi1wDJbpXgegdpb6nl7Q TgUObU4BqiN0xvKJ2QKLZB5TogMYJ5EFsRt0o3hjB7flqSboRvbSe4meNHGNxxHLKFKp a6mAtbuLy7U1Lc5AdIREDGzhKVMNAyKkiz6JHHIDJcIMSTgiP7pnqE+PYqG0q/y13eNy oO1g== X-Gm-Message-State: AOJu0YzVFV5Aw7tWlom+2GjvxileYNBcK/O5VkVRBFux/gQK85yNC5nt s9cxaFb+gbps5ak7UctTEBNwZikooUjBChbb0Ud5Vs6RIv3BlVxVho0zRU3bg8RNqIOPgME93Sw = X-Gm-Gg: ASbGncvMmp66ng5VrNb2o0kA+wpH3dadEyTsNW69zDhobk6Gby26pZCYPkHZHaXwp85 QFwLl+bQ4U5NdHr7iqyEOSGVVVIR1svokchlF58ebdQccFxg5LPi/lPLX719Iomgibc2bfq0qdM kFJ0ryfDx1BXKd6JoT+/1/bDrwlc+LG9GvK631HKlil0bOIgn1uqH9kCHHmrzZ9882SpbwypQnN JC07pxVkYKVLWgKROTpBVz9ZnUAYdjAWp2ts/z2wjMnpjRzf+8adDZs2iYqacxam1Wbd9H8/647 EqUivfzNHmX+IHRtOTzQkRbGb4Mn3tJ1nJl/r7phMg== X-Google-Smtp-Source: AGHT+IEfbeIpnncbii9/ulkoYjr5/qnJQ+KeF5tDhIvUc40vwREXBgMLneFJI50PQ4zdbR4RW2eg2A== X-Received: by 2002:a5d:648a:0:b0:38a:4b8a:e47d with SMTP id ffacd0b85a97d-38a8730ac0emr1557290f8f.26.1736331234049; Wed, 08 Jan 2025 02:13:54 -0800 (PST) Message-ID: <8f3e1256-6ec2-410b-94b7-23295f569f9f@suse.com> Date: Wed, 8 Jan 2025 11:13:52 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v4 5/6] x86: introduce "hot" and "cold" page clearing functions From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= References: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> The present clear_page_sse2() is useful in case a page isn't going to get touched again soon, or if we want to limit churn on the caches. Amend it by alternatively using CLZERO, which has been found to be quite a bit faster on Zen2 hardware at least. Note that to use CLZERO, we need to know the cache line size, and hence a feature dependency on CLFLUSH gets introduced. For cases where latency is the most important aspect, or when it is expected that sufficiently large parts of a page will get accessed again soon after the clearing, introduce a "hot" alternative. Again use alternatives patching to select between a "legacy" and an ERMS variant. Don't switch any callers just yet - this will be the subject of subsequent changes. Signed-off-by: Jan Beulich --- v3: Re-base. v2: New. --- Note: Ankur indicates that for ~L3-size or larger regions MOVNT/CLZERO is better even latency-wise. --- a/xen/arch/x86/clear_page.S +++ b/xen/arch/x86/clear_page.S @@ -1,9 +1,9 @@ .file __FILE__ -#include -#include +#include +#include -FUNC(clear_page_sse2) + .macro clear_page_sse2 mov $PAGE_SIZE/32, %ecx xor %eax,%eax @@ -17,4 +17,43 @@ FUNC(clear_page_sse2) sfence ret -END(clear_page_sse2) + .endm + + .macro clear_page_clzero + mov %rdi, %rax + mov $PAGE_SIZE/64, %ecx + .globl clear_page_clzero_post_count +clear_page_clzero_post_count: + +0: clzero + sub $-64, %rax + .globl clear_page_clzero_post_neg_size +clear_page_clzero_post_neg_size: + sub $1, %ecx + jnz 0b + + sfence + ret + .endm + +FUNC(clear_page_cold) + ALTERNATIVE clear_page_sse2, clear_page_clzero, X86_FEATURE_CLZERO +END(clear_page_cold) + + .macro clear_page_stosb + mov $PAGE_SIZE, %ecx + xor %eax,%eax + rep stosb + ret + .endm + + .macro clear_page_stosq + mov $PAGE_SIZE/8, %ecx + xor %eax, %eax + rep stosq + ret + .endm + +FUNC(clear_page_hot) + ALTERNATIVE clear_page_stosq, clear_page_stosb, X86_FEATURE_ERMS +END(clear_page_hot) --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -58,6 +58,9 @@ DEFINE_PER_CPU(bool, full_gdt_loaded); DEFINE_PER_CPU(uint32_t, pkrs); +extern uint32_t clear_page_clzero_post_count[]; +extern int8_t clear_page_clzero_post_neg_size[]; + void __init setup_clear_cpu_cap(unsigned int cap) { const uint32_t *dfs; @@ -355,8 +358,38 @@ void __init early_cpu_init(bool verbose) edx &= ~cleared_caps[FEATURESET_1d]; ecx &= ~cleared_caps[FEATURESET_1c]; - if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) - c->x86_cache_alignment = ((ebx >> 8) & 0xff) * 8; + if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) { + unsigned int size = ((ebx >> 8) & 0xff) * 8; + + c->x86_cache_alignment = size; + + /* + * Patch in parameters of clear_page_cold()'s CLZERO + * alternative. Note that for now we cap this at 128 bytes. + * Larger cache line sizes would still be dealt with + * correctly, but would cause redundant work done. + */ + if (size > 128) + size = 128; + if (size && !(size & (size - 1))) { + /* + * Need to play some games to keep the compiler from + * recognizing the negative array index as being out + * of bounds. The labels in assembler code really are + * _after_ the locations to be patched, so the + * negative index is intentional. + */ + uint32_t *pcount = clear_page_clzero_post_count; + int8_t *neg_size = clear_page_clzero_post_neg_size; + + OPTIMIZER_HIDE_VAR(pcount); + OPTIMIZER_HIDE_VAR(neg_size); + pcount[-1] = PAGE_SIZE / size; + neg_size[-1] = -size; + } + else + setup_clear_cpu_cap(X86_FEATURE_CLZERO); + } /* Leaf 0x1 capabilities filled in early for Xen. */ c->x86_capability[FEATURESET_1d] = edx; c->x86_capability[FEATURESET_1c] = ecx; --- a/xen/arch/x86/include/asm/asm-defns.h +++ b/xen/arch/x86/include/asm/asm-defns.h @@ -20,6 +20,10 @@ .byte 0x0f, 0x01, 0xdd .endm +.macro clzero + .byte 0x0f, 0x01, 0xfc +.endm + /* * Call a noreturn function. This could be JMP, but CALL results in a more * helpful backtrace. BUG is to catch functions which do decide to return... --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -219,10 +219,11 @@ typedef struct { u64 pfn; } pagetable_t; #define pagetable_from_paddr(p) pagetable_from_pfn((p)>>PAGE_SHIFT) #define pagetable_null() pagetable_from_pfn(0) -void clear_page_sse2(void *pg); +void clear_page_hot(void *pg); +void clear_page_cold(void *pg); void copy_page_sse2(void *to, const void *from); -#define clear_page(_p) clear_page_sse2(_p) +#define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) /* Convert between Xen-heap virtual addresses and machine addresses. */ --- a/xen/tools/gen-cpuid.py +++ b/xen/tools/gen-cpuid.py @@ -212,6 +212,10 @@ def crunch_numbers(state): # the first place. APIC: [X2APIC, TSC_DEADLINE, EXTAPIC], + # The CLZERO insn requires a means to determine the cache line size, + # which is tied to the CLFLUSH insn. + CLFLUSH: [CLZERO], + # AMD built MMXExtentions and 3DNow as extentions to MMX. MMX: [MMXEXT, _3DNOW], From patchwork Wed Jan 8 10:15:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13930489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4053E7719A for ; Wed, 8 Jan 2025 10:15:49 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.867107.1278535 (Exim 4.92) (envelope-from ) id 1tVT6E-0007fl-2O; Wed, 08 Jan 2025 10:15:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 867107.1278535; Wed, 08 Jan 2025 10:15:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT6D-0007fe-Vz; Wed, 08 Jan 2025 10:15:41 +0000 Received: by outflank-mailman (input) for mailman id 867107; Wed, 08 Jan 2025 10:15:41 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tVT6D-0007AP-AE for xen-devel@lists.xenproject.org; Wed, 08 Jan 2025 10:15:41 +0000 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [2a00:1450:4864:20::329]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 827ace7d-cda9-11ef-99a4-01e77a169b0f; Wed, 08 Jan 2025 11:15:39 +0100 (CET) Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-436249df846so114545285e9.3 for ; Wed, 08 Jan 2025 02:15:39 -0800 (PST) Received: from [10.156.60.236] (ip-037-024-206-209.um08.pools.vodafone-ip.de. [37.24.206.209]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436e2e22c6dsm15489755e9.41.2025.01.08.02.15.37 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 08 Jan 2025 02:15:38 -0800 (PST) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 827ace7d-cda9-11ef-99a4-01e77a169b0f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1736331339; x=1736936139; darn=lists.xenproject.org; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=P1MIbiccam+0w1GrD/h7pQdOgwuOnzTZkVsUXNrVWs0=; b=Sz3trCHFmI23NmQD04ypMOJpFXitG1AqMrQSXU+h/y6HC93nRzx0uWR/fg0LQi2BhY iuN0riM4FrkRdqJM+hZ3XEN5PjHyw2XoiNBkgJ2OKK38b+ptKzsMNhIRheceQOVAi13E 9Qs/AvHYJnn6tIiEGN4oguuCKi3xaViZ1vW423wxN9rMCME5Ma/4wHXtfoxcXtmFVD2T Fyq/+6FJ5z0s3rjsC1Pe8kHScV2azQnav6ulNSp3xdC5+0rCgDI8AE3ke62oEtcSUeI4 VF2wiD2Qgste4ecc3Ggii3jOZekhWXArv1aW4EZR+Ekshvg7ddMIYy3oEAKOUtoallQX 3eMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736331339; x=1736936139; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :references:cc:to:from:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=P1MIbiccam+0w1GrD/h7pQdOgwuOnzTZkVsUXNrVWs0=; b=STBgvjpo5qlS5d42zuIKBchGtgrpp3UAFnP0asDkrAYHAoudrQoytBLNPuTzLFIHcJ ZHQSw/tcBxC5VQLumMRQJHdnzlxKgFDTOvLSTvMPebMypRbraIz/FVbOlX4QaHcHd6jz xwa+slsYXxHbwoFWJeBaXA5EODe547j8IvTmzSSOyDNUNVsnBvtoLPaBK/ojna9BOupG 5D9SV0GwPHqCnKc6DzW0jW3joLlzeBT17vdK1v9ycueyitvax/d5IoVzWszX2OqfM15d i+1EHD47e6w1LwNa8LlmmhF/Z9x3OKzH9MZ99v1SlI6Q0ni1ubcsekTOpHbmzx8sH0yv 2zKQ== X-Gm-Message-State: AOJu0YzUAICLGLochZi8ysUstyJ7ho51GX4jIMdgz9YXVACfCe21y7sw /CXoR9VHEZOICbdpNlOap2NRo7Xu4i659hBa4hpH1BIwwaFwnWCyGH6xA/KDNT2V3YhsJL+yE48 = X-Gm-Gg: ASbGncvw0WS/03i95F7AStR1BtqM/4cNbi5YxzEno7OaF+YlrXQ/LRsORy4mvTvOWFl sp65gXuBt65aqF7935CbZuP65BGnuBLa4Xv82KaRbPZAVyhDpwHYM5u8MHUaNT8eOruGxB6xtUv OE+k96g+VEc6mksS77cEJOuIEbDpbLueRCOStf0cbHwY+sdHLhEK8J2GisI6IBR1r29jWUVrVRG PxYpVSPG2vTfEsYshN+/HcGt7Mlu6AM8xsqPPqDpoC5556mV31YlNcUCELs3Stn6USxwfg0+lr6 pM6Czg22LPD3ojk6QaG7G2bnbNlDdCugtrNnPY3HrQ== X-Google-Smtp-Source: AGHT+IFeve6rmgNKMBreSEhP0nNBBk0KZo5lJVNQCRQQxaQPU6Oh4TiQR9KKQ1AY3LJtNX9D29QQvg== X-Received: by 2002:a05:600c:5491:b0:434:f7f0:1880 with SMTP id 5b1f17b1804b1-436e26ec32amr14641945e9.32.1736331338903; Wed, 08 Jan 2025 02:15:38 -0800 (PST) Message-ID: <4afef39b-ff01-40f8-9bf1-68202f3a8b60@suse.com> Date: Wed, 8 Jan 2025 11:15:37 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: [PATCH v4 6/6] mm: allow page scrubbing routine(s) to be arch controlled From: Jan Beulich To: "xen-devel@lists.xenproject.org" Cc: Andrew Cooper , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= , Shawn Anastasio , Oleksii Kurochko , Julien Grall , Stefano Stabellini , Volodymyr Babchuk , Bertrand Marquis , Michal Orzel , Anthony PERARD References: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Content-Language: en-US Autocrypt: addr=jbeulich@suse.com; keydata= xsDiBFk3nEQRBADAEaSw6zC/EJkiwGPXbWtPxl2xCdSoeepS07jW8UgcHNurfHvUzogEq5xk hu507c3BarVjyWCJOylMNR98Yd8VqD9UfmX0Hb8/BrA+Hl6/DB/eqGptrf4BSRwcZQM32aZK 7Pj2XbGWIUrZrd70x1eAP9QE3P79Y2oLrsCgbZJfEwCgvz9JjGmQqQkRiTVzlZVCJYcyGGsD /0tbFCzD2h20ahe8rC1gbb3K3qk+LpBtvjBu1RY9drYk0NymiGbJWZgab6t1jM7sk2vuf0Py O9Hf9XBmK0uE9IgMaiCpc32XV9oASz6UJebwkX+zF2jG5I1BfnO9g7KlotcA/v5ClMjgo6Gl MDY4HxoSRu3i1cqqSDtVlt+AOVBJBACrZcnHAUSuCXBPy0jOlBhxPqRWv6ND4c9PH1xjQ3NP nxJuMBS8rnNg22uyfAgmBKNLpLgAGVRMZGaGoJObGf72s6TeIqKJo/LtggAS9qAUiuKVnygo 3wjfkS9A3DRO+SpU7JqWdsveeIQyeyEJ/8PTowmSQLakF+3fote9ybzd880fSmFuIEJldWxp Y2ggPGpiZXVsaWNoQHN1c2UuY29tPsJgBBMRAgAgBQJZN5xEAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQoDSui/t3IH4J+wCfQ5jHdEjCRHj23O/5ttg9r9OIruwAn3103WUITZee e7Sbg12UgcQ5lv7SzsFNBFk3nEQQCACCuTjCjFOUdi5Nm244F+78kLghRcin/awv+IrTcIWF hUpSs1Y91iQQ7KItirz5uwCPlwejSJDQJLIS+QtJHaXDXeV6NI0Uef1hP20+y8qydDiVkv6l IreXjTb7DvksRgJNvCkWtYnlS3mYvQ9NzS9PhyALWbXnH6sIJd2O9lKS1Mrfq+y0IXCP10eS FFGg+Av3IQeFatkJAyju0PPthyTqxSI4lZYuJVPknzgaeuJv/2NccrPvmeDg6Coe7ZIeQ8Yj t0ARxu2xytAkkLCel1Lz1WLmwLstV30g80nkgZf/wr+/BXJW/oIvRlonUkxv+IbBM3dX2OV8 AmRv1ySWPTP7AAMFB/9PQK/VtlNUJvg8GXj9ootzrteGfVZVVT4XBJkfwBcpC/XcPzldjv+3 HYudvpdNK3lLujXeA5fLOH+Z/G9WBc5pFVSMocI71I8bT8lIAzreg0WvkWg5V2WZsUMlnDL9 mpwIGFhlbM3gfDMs7MPMu8YQRFVdUvtSpaAs8OFfGQ0ia3LGZcjA6Ik2+xcqscEJzNH+qh8V m5jjp28yZgaqTaRbg3M/+MTbMpicpZuqF4rnB0AQD12/3BNWDR6bmh+EkYSMcEIpQmBM51qM EKYTQGybRCjpnKHGOxG0rfFY1085mBDZCH5Kx0cl0HVJuQKC+dV2ZY5AqjcKwAxpE75MLFkr wkkEGBECAAkFAlk3nEQCGwwACgkQoDSui/t3IH7nnwCfcJWUDUFKdCsBH/E5d+0ZnMQi+G0A nAuWpQkjM1ASeQwSHEeAWPgskBQL In-Reply-To: <14b65231-b83b-43fb-bbcf-dec5c07d285b@suse.com> Especially when dealing with large amounts of memory, memset() may not be very efficient; this can be bad enough that even for debug builds a custom function is warranted. We additionally want to distinguish "hot" and "cold" cases (with, as initial heuristic, "hot" being for any allocations a domain does for itself, assuming that in all other cases the page wouldn't be accessed [again] soon). The goal is for accesses of "cold" pages to not disturb caches (albeit finding a good balance between this and the higher latency looks to be difficult). Keep the default fallback to clear_page_*() in common code; this may want to be revisited down the road. Signed-off-by: Jan Beulich Acked-by: Julien Grall --- v4: Re-base. v3: Re-base. v2: New. --- The choice between hot and cold in scrub_one_page()'s callers is certainly up for discussion / improvement. --- a/xen/arch/arm/include/asm/page.h +++ b/xen/arch/arm/include/asm/page.h @@ -144,6 +144,12 @@ extern size_t dcache_line_bytes; #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + static inline size_t read_dcache_line_bytes(void) { register_t ctr; --- a/xen/arch/ppc/include/asm/page.h +++ b/xen/arch/ppc/include/asm/page.h @@ -190,6 +190,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset(page, 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + /* TODO: Flush the dcache for an entire page. */ static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache) { --- a/xen/arch/riscv/include/asm/page.h +++ b/xen/arch/riscv/include/asm/page.h @@ -177,6 +177,12 @@ static inline void invalidate_icache(voi #define clear_page(page) memset((void *)(page), 0, PAGE_SIZE) #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE) +#define clear_page_hot clear_page +#define clear_page_cold clear_page + +#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE) +#define scrub_page_cold scrub_page_hot + static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache) { const void *v = map_domain_page(_mfn(mfn)); --- a/xen/arch/x86/Makefile +++ b/xen/arch/x86/Makefile @@ -59,6 +59,7 @@ obj-y += pci.o obj-y += physdev.o obj-$(CONFIG_COMPAT) += x86_64/physdev.o obj-$(CONFIG_X86_PSR) += psr.o +obj-bin-$(CONFIG_DEBUG) += scrub_page.o obj-y += setup.o obj-y += shutdown.o obj-y += smp.o --- a/xen/arch/x86/include/asm/page.h +++ b/xen/arch/x86/include/asm/page.h @@ -226,6 +226,11 @@ void copy_page_sse2(void *to, const void #define clear_page(_p) clear_page_cold(_p) #define copy_page(_t, _f) copy_page_sse2(_t, _f) +#ifdef CONFIG_DEBUG +void scrub_page_hot(void *); +void scrub_page_cold(void *); +#endif + /* Convert between Xen-heap virtual addresses and machine addresses. */ #define __pa(x) (virt_to_maddr(x)) #define __va(x) (maddr_to_virt(x)) --- /dev/null +++ b/xen/arch/x86/scrub_page.S @@ -0,0 +1,39 @@ + .file __FILE__ + +#include +#include +#include + +FUNC(scrub_page_cold) + mov $PAGE_SIZE/32, %ecx + mov $SCRUB_PATTERN, %rax + +0: movnti %rax, (%rdi) + movnti %rax, 8(%rdi) + movnti %rax, 16(%rdi) + movnti %rax, 24(%rdi) + add $32, %rdi + sub $1, %ecx + jnz 0b + + sfence + ret +END(scrub_page_cold) + + .macro scrub_page_stosb + mov $PAGE_SIZE, %ecx + mov $SCRUB_BYTE_PATTERN, %eax + rep stosb + ret + .endm + + .macro scrub_page_stosq + mov $PAGE_SIZE/8, %ecx + mov $SCRUB_PATTERN, %rax + rep stosq + ret + .endm + +FUNC(scrub_page_hot) + ALTERNATIVE scrub_page_stosq, scrub_page_stosb, X86_FEATURE_ERMS +END(scrub_page_hot) --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -135,6 +135,7 @@ #include #include #include +#include #include #include #include @@ -780,27 +781,31 @@ static void page_list_add_scrub(struct p page_list_add(pg, &heap(node, zone, order)); } -/* SCRUB_PATTERN needs to be a repeating series of bytes. */ -#ifndef NDEBUG -#define SCRUB_PATTERN 0xc2c2c2c2c2c2c2c2ULL -#else -#define SCRUB_PATTERN 0ULL +/* + * While in debug builds we want callers to avoid relying on allocations + * returning zeroed pages, for a production build, clear_page_*() is the + * fastest way to scrub. + */ +#ifndef CONFIG_DEBUG +# undef scrub_page_hot +# define scrub_page_hot clear_page_hot +# undef scrub_page_cold +# define scrub_page_cold clear_page_cold #endif -#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) -static void scrub_one_page(const struct page_info *pg) +static void scrub_one_page(const struct page_info *pg, bool cold) { + void *ptr; + if ( unlikely(pg->count_info & PGC_broken) ) return; -#ifndef NDEBUG - /* Avoid callers relying on allocations returning zeroed pages. */ - unmap_domain_page(memset(__map_domain_page(pg), - SCRUB_BYTE_PATTERN, PAGE_SIZE)); -#else - /* For a production build, clear_page() is the fastest way to scrub. */ - clear_domain_page(_mfn(page_to_mfn(pg))); -#endif + ptr = __map_domain_page(pg); + if ( cold ) + scrub_page_cold(ptr); + else + scrub_page_hot(ptr); + unmap_domain_page(ptr); } static void poison_one_page(struct page_info *pg) @@ -1080,12 +1085,14 @@ static struct page_info *alloc_heap_page if ( first_dirty != INVALID_DIRTY_IDX || (scrub_debug && !(memflags & MEMF_no_scrub)) ) { + bool cold = d && d != current->domain; + for ( i = 0; i < (1U << order); i++ ) { if ( test_and_clear_bit(_PGC_need_scrub, &pg[i].count_info) ) { if ( !(memflags & MEMF_no_scrub) ) - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], cold); dirty_cnt++; } @@ -1350,7 +1357,7 @@ bool scrub_free_pages(void) { if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) { - scrub_one_page(&pg[i]); + scrub_one_page(&pg[i], true); /* * We can modify count_info without holding heap * lock since we effectively locked this buddy by @@ -2072,7 +2079,7 @@ static struct page_info *alloc_color_hea if ( !(memflags & MEMF_no_scrub) ) { if ( need_scrub ) - scrub_one_page(pg); + scrub_one_page(pg, d != current->domain); else check_one_page(pg); } @@ -2223,7 +2230,7 @@ static void __init cf_check smp_scrub_he if ( !mfn_valid(_mfn(mfn)) || !page_state_is(pg, free) ) continue; - scrub_one_page(pg); + scrub_one_page(pg, true); } } @@ -2928,7 +2935,7 @@ void unprepare_staticmem_pages(struct pa if ( need_scrub ) { /* TODO: asynchronous scrubbing for pages of static memory. */ - scrub_one_page(pg); + scrub_one_page(pg, true); } pg[i].count_info |= PGC_static; --- /dev/null +++ b/xen/include/xen/scrub.h @@ -0,0 +1,24 @@ +#ifndef __XEN_SCRUB_H__ +#define __XEN_SCRUB_H__ + +#include + +/* SCRUB_PATTERN needs to be a repeating series of bytes. */ +#ifdef CONFIG_DEBUG +# define SCRUB_PATTERN _AC(0xc2c2c2c2c2c2c2c2,ULL) +#else +# define SCRUB_PATTERN _AC(0,ULL) +#endif +#define SCRUB_BYTE_PATTERN (SCRUB_PATTERN & 0xff) + +#endif /* __XEN_SCRUB_H__ */ + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */