From patchwork Fri May 31 09:33:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E25F8C27C44 for ; Fri, 31 May 2024 09:33:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2ACB26B0095; Fri, 31 May 2024 05:33:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 08EB66B00A0; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEDA16B0099; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8EF126B009C for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4B8B2805E3 for ; Fri, 31 May 2024 09:33:44 +0000 (UTC) X-FDA: 82178178768.27.17C6CD6 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf29.hostedemail.com (Postfix) with ESMTP id F1316120003 for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=RJ44r3mEV2SgE/KoVtmnKWZ2IGDv1l1PYxRL2TZKNuqnTs2c1ZIBmfEu7O3GZ3w9DjFH9H hlY6JjblYzaVweg7MWH/5kZJLtt7XdumuQyREYnalWwrLmCJz0Oq6F37vsO6qzj/ngN8/2 4UVnOihLJ9YHlo5UaOmJFw8IvX9gyX0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=nvv4gm6++P5wbF2mCqw2xnoLu6qjly35SHirIFmsuTRC0UaOnH8IUuY962O9549PsZEc5j o+fhB/je5d0tAOXDy7iLMSMvz1VmAWj+P1ftLjQrZ349EEZvd9u1DSm03+Uutg4pp3j5h9 55VpHXVrIRgNqtetr8yFcA/thhyifF4= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6322421B2C; Fri, 31 May 2024 09:33:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=cq3ZS3BbflQ7N2K6lBu8bapgCyqq54EbZLCUHI5AOJr2zGvzJwjgIv1pOlAsS/4VJ+HD// DqP8gM6quCsk/Rf4szRlG7kaJ5KOQrpnHm+OrwzFSPVsOugt54MfF1PraU/+2ZdMSbMJUb WfFBo7wBG5E4quordFfd4kyyCdsRO1E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=WQK9iHyhgXJjz5UKPeIRCrBHBTFp99Hwb64bZURii7O8xDLTgt/3LGZGt8RkE0fHxqXxKb zTsFKRohlCJOaEDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=cq3ZS3BbflQ7N2K6lBu8bapgCyqq54EbZLCUHI5AOJr2zGvzJwjgIv1pOlAsS/4VJ+HD// DqP8gM6quCsk/Rf4szRlG7kaJ5KOQrpnHm+OrwzFSPVsOugt54MfF1PraU/+2ZdMSbMJUb WfFBo7wBG5E4quordFfd4kyyCdsRO1E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=WQK9iHyhgXJjz5UKPeIRCrBHBTFp99Hwb64bZURii7O8xDLTgt/3LGZGt8RkE0fHxqXxKb zTsFKRohlCJOaEDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 45159132C2; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eAqVEHSZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:40 +0000 From: Vlastimil Babka Date: Fri, 31 May 2024 11:33:35 +0200 Subject: [PATCH RFC 4/4] mm, page_alloc: add static key for should_fail_alloc_page() MIME-Version: 1.0 Message-Id: <20240531-fault-injection-statickeys-v1-4-a513fd0a9614@suse.cz> References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Rspamd-Queue-Id: F1316120003 X-Stat-Signature: 5tfmk9yzcgc5wy9w3psxd6u9getdzz8i X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717148021-93072 X-HE-Meta: U2FsdGVkX18M8BhFO8MiWmHN/n6mPwLiGblOsy5ltKd5JVyMC8LluRcfmBhvMK/S/HdrQyfbAYH2tSQwMSjzoSCP+B6H3F2P81d/progZWIXGb4qSKqkSmElP2AZt9AgvpBQgL3pSuSiCm237+5YN+VhP2AnvteYSIN1Swg9M3IYfh+qo5j+1CKGR2rFkEZk+ah9rhaTIo8bWMik6WiqnTxiiB9sI4sogEot0aW+Tdv7L2qWwJZZyW7cqdSmssRUqZ9vyUHdNh/hJq2lyhO22LKRezmfFbMV7hwnFuYoHAYjz/Qr15kjaMVwaQh4b1Vh4vx5hgBtcgqHLCKukbMJKFyk/RKoK1YWD6NpWpyx1+ho16TS1t9LZ9Oz/ndWFDxSBtc7FwYU6hyDwgler2BlQkISQEZraoA75bMICpWPIK89SQdck14UmjwTxrg0v/+TOhNUpDAcXpyRQ03I8FEJaUQwKXoynfZeiU89vXgiwfdgGOc7sfbGJPreCdmsFrIBXq9F/c3yoVZNKb0Ga7lC+DiuMB0w94uc1oMcq0lCpZY4tXDf2+TSovE2hp8Lct2DPv0pYQilbKxyf5tQ1WJHJnigU3nzDpxRKx35P+lf7SpM6vjAUh9VVbxymeW6sXeNef7N5XqY5UWeK6/3XHg5ErEkQ/barXD7qSHAz6fGkV/DmVUelpRlrI8Vrudwid4fMXNDy7/5nJUhlJIx58zZ3g/rW0jTv1QN20gV1jsfMnht23CK2UKcnFVodPXyxgjWCp6XQC6L4q2hTzcEM/giuwVsEqhYcA806sTNjSfuio+ZignGkSpIAIUFQHrLgte+ub4YX691y6i4h/IdZaGPmMrpxHkBgxnOetRCUT/N0+NblSleYALYap0PToYbCDJqjOQ+f/z7E2yahf3hQlYsYKD80j74eksRuF4FPObbmKSTlV7gqDVvUjOHPBZU3Z6U9lUYth3I1gtoPSsLCSE jBzTy6YB /PbY8LuERryGtSISiBxSZv6I17TL2xtAjCtBzrapCNTIQmS4HRvskTr+GC1gzogERmre0ID8LCVgB+HamCNg+E/bcEY6RCxYks4iUeX4+blHCsPCB2sP9LkQxO027rSshIOcFAqTiVWRAEvk6bVTbZsIe2WKf3HqSvBwJGHDjElMr97G7nSDSkVNcjiPJ7JGcvHVaYQ9akpRrHck3BNsC49oGbYgYtKFkCKYY1GNxPgCayBpOzcHmd/sAlfBTCqQB/YQtkTqpo8fuaHdLgmaX+ibpYbnz2sEWYpzfDsTI109XT9B+qgMJmuXLnot95UrYJsHdlI0qEPp1als9U/2OSMcc2A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similarly to should_failslab(), remove the overhead of calling the noinline function should_fail_alloc_page() with a static key that guards the allocation hotpath callsite and is controlled by the fault and error injection frameworks. Signed-off-by: Vlastimil Babka Reviewed-by: Roman Gushchin --- mm/fail_page_alloc.c | 3 ++- mm/internal.h | 2 ++ mm/page_alloc.c | 11 ++++++++--- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/fail_page_alloc.c b/mm/fail_page_alloc.c index b1b09cce9394..0906b76d78e8 100644 --- a/mm/fail_page_alloc.c +++ b/mm/fail_page_alloc.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include "internal.h" static struct { struct fault_attr attr; @@ -9,7 +10,7 @@ static struct { bool ignore_gfp_reclaim; u32 min_order; } fail_page_alloc = { - .attr = FAULT_ATTR_INITIALIZER, + .attr = FAULT_ATTR_INITIALIZER_KEY(&should_fail_alloc_page_active.key), .ignore_gfp_reclaim = true, .ignore_gfp_highmem = true, .min_order = 1, diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..8539e39b02e6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -410,6 +410,8 @@ extern char * const zone_names[MAX_NR_ZONES]; /* perform sanity checks on struct pages being allocated or freed */ DECLARE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); +DECLARE_STATIC_KEY_FALSE(should_fail_alloc_page_active); + extern int min_free_kbytes; void setup_per_zone_wmarks(void); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2e22ce5675ca..e5dc3bafa549 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -274,6 +274,8 @@ int user_min_free_kbytes = -1; static int watermark_boost_factor __read_mostly = 15000; static int watermark_scale_factor = 10; +DEFINE_STATIC_KEY_FALSE(should_fail_alloc_page_active); + /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */ int movable_zone; EXPORT_SYMBOL(movable_zone); @@ -3012,7 +3014,7 @@ noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) { return __should_fail_alloc_page(gfp_mask, order); } -ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE); +ALLOW_ERROR_INJECTION_KEY(should_fail_alloc_page, TRUE, &should_fail_alloc_page_active); static inline long __zone_watermark_unusable_free(struct zone *z, unsigned int order, unsigned int alloc_flags) @@ -4430,8 +4432,11 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) - return false; + if (static_branch_unlikely(&should_fail_alloc_page_active)) { + if (should_fail_alloc_page(gfp_mask, order)) { + return false; + } + } *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags);