From patchwork Fri May 31 09:33:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 983C9C25B75 for ; Fri, 31 May 2024 09:33:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01A976B009B; Fri, 31 May 2024 05:33:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E224B6B0098; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC6BC6B0095; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 65BB86B0099 for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EEEFD160740 for ; Fri, 31 May 2024 09:33:43 +0000 (UTC) X-FDA: 82178178726.06.42E954B Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf15.hostedemail.com (Postfix) with ESMTP id AB7D0A001C for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JssftoR6; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="/5hT1B4N"; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JssftoR6; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="/5hT1B4N"; spf=pass (imf15.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u2635+TpURoaMWiOVxYkY4gXrnKDk2yUsxy6mghZqBU=; b=rF8+vc6sF7tAw9TOaUi99Ia57rKwXDjw04BnrUB+0chUBH/7u0EftB4W7IDKpbkbUzc+yG xUwnKyaUXe92SMuRDebZ3M948yuI5ybEtQek9y41AZI+3RnQ9HHQczMorKj6//gl6HxVg5 5ChDdh8u8mxwEXc01skTXmbG47deNjY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JssftoR6; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="/5hT1B4N"; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JssftoR6; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="/5hT1B4N"; spf=pass (imf15.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=bZ3BqVybjn92rlqb9trdsNERKcXtkcmqunFBRZcnP7xGH02a5ZVBOw41kEYZm0ufoeaPOT KUG/bk8OsVQWaIaIxH1KKmXabKWyoG0qSV5wHiw2yXSlBYVJ6GEbMK9oJFYKRM2SBp32YJ mh3nTPq1To6tkU0rraVDNuRVXS90Wg0= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 10AFD21B25; Fri, 31 May 2024 09:33:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u2635+TpURoaMWiOVxYkY4gXrnKDk2yUsxy6mghZqBU=; b=JssftoR6B7pLW9rYT+Tuv/kXK/+g2BXda/wOv7dsChTZxN/xGrJlQ1Jc6cmei7hmQSqNqP 6rqvF2BKD8laRmpvrMaXDERqb2Lu+k87pA+l3hoXVTU+jtV4440OtAWW1jJpaGRb1oLD33 LjQbmi/lydv2YxzuDTeIk5RFh3UnqbE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u2635+TpURoaMWiOVxYkY4gXrnKDk2yUsxy6mghZqBU=; b=/5hT1B4N5thhpu2CPFQv8Dbg+mgJCzLOUi8MarugqWlJa+NX1EDAVtQEkknmpwp0aRhQxi itSLEgs1gbsRQ2DQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u2635+TpURoaMWiOVxYkY4gXrnKDk2yUsxy6mghZqBU=; b=JssftoR6B7pLW9rYT+Tuv/kXK/+g2BXda/wOv7dsChTZxN/xGrJlQ1Jc6cmei7hmQSqNqP 6rqvF2BKD8laRmpvrMaXDERqb2Lu+k87pA+l3hoXVTU+jtV4440OtAWW1jJpaGRb1oLD33 LjQbmi/lydv2YxzuDTeIk5RFh3UnqbE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u2635+TpURoaMWiOVxYkY4gXrnKDk2yUsxy6mghZqBU=; b=/5hT1B4N5thhpu2CPFQv8Dbg+mgJCzLOUi8MarugqWlJa+NX1EDAVtQEkknmpwp0aRhQxi itSLEgs1gbsRQ2DQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id E3E9B13A64; Fri, 31 May 2024 09:33:39 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eMBdN3OZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:39 +0000 From: Vlastimil Babka Date: Fri, 31 May 2024 11:33:32 +0200 Subject: [PATCH RFC 1/4] fault-inject: add support for static keys around fault injection sites MIME-Version: 1.0 Message-Id: <20240531-fault-injection-statickeys-v1-1-a513fd0a9614@suse.cz> References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AB7D0A001C X-Stat-Signature: m8wixysif16413tmtrpjnnik7w3pgdq5 X-Rspam-User: X-HE-Tag: 1717148021-299478 X-HE-Meta: U2FsdGVkX1/31Xphb0TmrxPM1img/mLGKBpr5eeQlrF4+OA58iNYxq/SQiWKVRJ9+mS7cjXALQKt+M+lhwzMQkZbhY9qLn7Qyac+stmVMMpXBE7P9yeJx+Xp1INb9Jn86dYpX0QYgRdGs8C5YoQ47BXctzvhjsMDNZwqplxWGQQETkJ1uM6ulJN+o4zm1AdPN6XXan8YafSHuFgevonB10zYiYSgjW/ML3mQjBImQqr+OfhB7CY9rH8VORsXUmo0ftNnIAmYN7C99X+ZnWvP0SvvJlSt3ftY/Na3p+VR7MJHJdnKIfI7DTjPFubVNQqepVTuN3fLYV9FuwA9hDMc75e4auoH2NtriC+h/aTmhHQvL4ShzzWKFO76Cp5NL+W//61x9XU9XmJhBMoPdaVj0LSW5vHuO4UHkg3r1Lqp+yL0lXn51f9jTOfsf1keC1wYjY3feEj+sBn+BbnrlVeAcYfqrX51aiwws9/Goj9n46TiWEe40srlzY98tLi0MQzaZquhSzSFkKjHoXuAVF6VpC1biSw8XD/EPa0gOh3bSqqtff0IxpEwF5gz/5ZS38w+w/vvqhoudjyVmBcP1gQvy/6Txc/Mw6lgb28MwZKsCEbPYWtTFqkOW2Rbv7gUAd7gNSnN0FrCLwgEzOj8KQ9owDQUiJeBzgwWqzkY1tG4gSDCtARdp9NPV9lf1dJpDKl9thv3rfouK8REWuXKsFfv2O4ecx6dkM16DK00/UUyhByTewx22NK97oqEcc6zlB+XrVX6m/Y+a3BWud2KFdoZoFZK199cnFxFhUHvesKiT1rxbBYwCk9SaNGm2jKJuuHXVy2WKZSjrptW6GfX32e/o+DrqAPZAy9ckWco82ZYKbic7bsW0jts/GbfZxx8YMv8eo6sTcsym+jC8ooWbbjcsY7ZYvjvv1iOM3KIGlWvVXTcYf4PfpU66OYPG9IHsKYssNqCGy6Fq9Iq4oHoTT3 JwJ/Vp4s S7PT8Bip1VkmQ1gfCf50AvQutobDLTpbRKvL3/SEuEu8PIokQRzIUxIpWrttcvfteWcNeHQxaNWDc6Q0QXU9lu9ETfL0uHq7jad8xf1LnXgdldkiaHcigC6sG57YUmIZM4HsZm37QoFa2QA8IKacC0DOnYEDbKWtUOlkurJnoYvLG1YpoGGplfQJzqpj+7jjg89/bL+Qf6NYeJb+p5DWerR5PYQspRCP6+uj68yoTkmGfmdI3mOwhEeSEJG3SEeUqzqxGZqYSccc1W45JRFwN431wKu9wtdDsv1Ae4WW1XKEWPv8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some fault injection sites are placed in hotpaths and incur overhead even if not enabled, due to one or more function calls leading up to should_fail_ex() that returns false due to attr->probability == 0. This overhead can be eliminated if the outermost call into the checks is guarded with a static key, so add support for that. The framework should be told that such static key exist for a fault_attr, by initializing fault_attr->active with the static key address. When it's not NULL, enable the static key from setup_fault_attr() when the fault probability is non-zero. Also wire up writing into debugfs "probability" file to enable or disable the static key when transitioning between zero and non-zero probability. For now, do not add configfs interface support as the immediate plan is to leverage this for should_failslab() and should_fail_alloc_page() after other necessary preparatory changes, and none of the configfs based fault injection users. Signed-off-by: Vlastimil Babka --- include/linux/fault-inject.h | 7 ++++++- lib/fault-inject.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/include/linux/fault-inject.h b/include/linux/fault-inject.h index 6d5edef09d45..cfe75cc1bac4 100644 --- a/include/linux/fault-inject.h +++ b/include/linux/fault-inject.h @@ -9,6 +9,7 @@ #include #include #include +#include /* * For explanation of the elements of this struct, see @@ -30,13 +31,14 @@ struct fault_attr { unsigned long count; struct ratelimit_state ratelimit_state; struct dentry *dname; + struct static_key *active; }; enum fault_flags { FAULT_NOWARN = 1 << 0, }; -#define FAULT_ATTR_INITIALIZER { \ +#define FAULT_ATTR_INITIALIZER_KEY(_key) { \ .interval = 1, \ .times = ATOMIC_INIT(1), \ .require_end = ULONG_MAX, \ @@ -44,8 +46,11 @@ enum fault_flags { .ratelimit_state = RATELIMIT_STATE_INIT_DISABLED, \ .verbose = 2, \ .dname = NULL, \ + .active = (_key), \ } +#define FAULT_ATTR_INITIALIZER FAULT_ATTR_INITIALIZER_KEY(NULL) + #define DECLARE_FAULT_ATTR(name) struct fault_attr name = FAULT_ATTR_INITIALIZER int setup_fault_attr(struct fault_attr *attr, char *str); bool should_fail_ex(struct fault_attr *attr, ssize_t size, int flags); diff --git a/lib/fault-inject.c b/lib/fault-inject.c index d608f9b48c10..93c46d2ec106 100644 --- a/lib/fault-inject.c +++ b/lib/fault-inject.c @@ -35,6 +35,9 @@ int setup_fault_attr(struct fault_attr *attr, char *str) atomic_set(&attr->times, times); atomic_set(&attr->space, space); + if (probability != 0 && attr->active) + static_key_slow_inc(attr->active); + return 1; } EXPORT_SYMBOL_GPL(setup_fault_attr); @@ -166,6 +169,12 @@ EXPORT_SYMBOL_GPL(should_fail); #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS +/* + * Protect updating probability from debugfs as that may trigger static key + * changes when changing between zero and non-zero. + */ +static DEFINE_MUTEX(probability_mutex); + static int debugfs_ul_set(void *data, u64 val) { *(unsigned long *)data = val; @@ -186,6 +195,38 @@ static void debugfs_create_ul(const char *name, umode_t mode, debugfs_create_file(name, mode, parent, value, &fops_ul); } +static int debugfs_prob_set(void *data, u64 val) +{ + struct fault_attr *attr = data; + + mutex_lock(&probability_mutex); + + if (attr->active) { + if (attr->probability != 0 && val == 0) { + static_key_slow_dec(attr->active); + } else if (attr->probability == 0 && val != 0) { + static_key_slow_inc(attr->active); + } + } + + attr->probability = val; + + mutex_unlock(&probability_mutex); + + return 0; +} + +static int debugfs_prob_get(void *data, u64 *val) +{ + struct fault_attr *attr = data; + + *val = attr->probability; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_prob, debugfs_prob_get, debugfs_prob_set, "%llu\n"); + #ifdef CONFIG_FAULT_INJECTION_STACKTRACE_FILTER static int debugfs_stacktrace_depth_set(void *data, u64 val) @@ -218,7 +259,7 @@ struct dentry *fault_create_debugfs_attr(const char *name, if (IS_ERR(dir)) return dir; - debugfs_create_ul("probability", mode, dir, &attr->probability); + debugfs_create_file("probability", mode, dir, &attr, &fops_prob); debugfs_create_ul("interval", mode, dir, &attr->interval); debugfs_create_atomic_t("times", mode, dir, &attr->times); debugfs_create_atomic_t("space", mode, dir, &attr->space); From patchwork Fri May 31 09:33:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CB74C27C4F for ; Fri, 31 May 2024 09:33:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A3866B0098; Fri, 31 May 2024 05:33:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 262B06B009C; Fri, 31 May 2024 05:33:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE83B6B0095; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A41856B00A0 for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4B39314018A for ; Fri, 31 May 2024 09:33:44 +0000 (UTC) X-FDA: 82178178768.29.6502DE9 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf01.hostedemail.com (Postfix) with ESMTP id 0156F4001D for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=NLOye7sb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=egjcXQtp; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=NLOye7sb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=egjcXQtp; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t8Mu1INP306CXVJLbtkEYc46ROCJCSbEjjUhioLXt6I=; b=fg/mcdmUrzjwikVzD4czr23joUeEzH9QeIsF9YIZgS0YiA6XIneaE8MhGLKpOK3LGmhcEk plnB9fw6sCTsZPTeIqMfareb0fEhDUbCSYb94DL53l1MlBHWktp77w3SLCGsRVIlY6vMZp f1aO1E/IBhECdZTilILW1K88jgOfeXQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=NLOye7sb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=egjcXQtp; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=NLOye7sb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=egjcXQtp; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=WDZ2NX21JqPyF+jIgniC6A1N1doutynwI6/9Iv2Q5I6GpJwGhGwbGR0Yt5g80ZD/J/CZdS DQgeSpLIY+nf3bYXj1jS9A6dfOR3GvUUoXXBunP5f/d+zzvGWgfiD6bD/y45FgdEftpanR zSehqxFLa4qXxNdi0qKg8gT6VvTnKFg= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 306CE21B29; Fri, 31 May 2024 09:33:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8Mu1INP306CXVJLbtkEYc46ROCJCSbEjjUhioLXt6I=; b=NLOye7sbpB3u1184zrlaGLIfhjypv/L8wYQ7PttPtS19iOcn1heYNY/JhSZeqtmHmbGxHu OUeoUpvWM/WYj4FpaEZxsX90CaKbPaXZNHqaRpvzqIkQkwAcdTgH89ltA29ELTOr3hXNSA lCa6qtULc7LPWZeIOKoGPqhNDLgZNC8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8Mu1INP306CXVJLbtkEYc46ROCJCSbEjjUhioLXt6I=; b=egjcXQtpYNPE0TganvgfvmKJWiJQ/5fLjMR5R9KMtaSX1BeJ1Ih1v57ryPctvr1Dv1VIM8 kBCA6IwOGoLvXhAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8Mu1INP306CXVJLbtkEYc46ROCJCSbEjjUhioLXt6I=; b=NLOye7sbpB3u1184zrlaGLIfhjypv/L8wYQ7PttPtS19iOcn1heYNY/JhSZeqtmHmbGxHu OUeoUpvWM/WYj4FpaEZxsX90CaKbPaXZNHqaRpvzqIkQkwAcdTgH89ltA29ELTOr3hXNSA lCa6qtULc7LPWZeIOKoGPqhNDLgZNC8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8Mu1INP306CXVJLbtkEYc46ROCJCSbEjjUhioLXt6I=; b=egjcXQtpYNPE0TganvgfvmKJWiJQ/5fLjMR5R9KMtaSX1BeJ1Ih1v57ryPctvr1Dv1VIM8 kBCA6IwOGoLvXhAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 0B90913A90; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id YGCIAnSZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:40 +0000 From: Vlastimil Babka Date: Fri, 31 May 2024 11:33:33 +0200 Subject: [PATCH RFC 2/4] error-injection: support static keys around injectable functions MIME-Version: 1.0 Message-Id: <20240531-fault-injection-statickeys-v1-2-a513fd0a9614@suse.cz> References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0156F4001D X-Stat-Signature: e1bj78r85kbaa34gttswstgr5sqymip3 X-Rspam-User: X-HE-Tag: 1717148021-625925 X-HE-Meta: U2FsdGVkX18iNVOe3eRvh712U4S/A0fwgxvWx/qe0x5KIQxAm2mNS5uA7q+QmYi2CUfrcSmVkUGrVX4FR7RktP3nkmlJ6lvl4zZdVuKPJ19O7TT6MUy48ImKNYWbRiPzGptKmm8tF05GYLWLQB52l9ATvsUL3fiS9TvI3kDvRRKyboKz426IC1ZI+PjnZfd2o+eMyiH1eHuBBsrJxxN3MBXMlBvdrPoMXXz1P0bUqMntDnSTjtPswPFZ45ylSN7gKlXLuAjpzP6agDxTbLjKQdvSShlMhE6HniLcSjYXyMUkR0wL86bYeRkdn+wCFnyeWub6Lowq/RIURCQ0CslpnqgIbork+7yiNQu3bp1aeNojPqQEo2S04ZyULaqC8hfyWSzEMFJSlZvD/AwOGWvOtqUte1wT7g7AMI+8feVQOBRxNas+2dRqhX5RafwwyvEE2DSisWZdu5UihKxh+UGe56OQqw2cbskUlrkoiegYCHLAnam70hZzpt6nBSESFnMjC/2GJY//XbnlJftPTc9tAzST0sIr7HE6euU1ReemkogKthYCkz9iadOC9YDILoYYwiasIB+R/4WtYm9JcalEOVmepBUjLsP8S1mndQdTeWiUE7ndyfFQCtYTKk8fIdImIREd4d+Jti481ALeMflgfqgoENyLA30EaYapXyh2qP4hr6voKxg/VazfW6TLL5l/tnBXELCBWXXiirHaipadpFZ3ME2N+xZLowiMlfR5n+jxLNOPyF1X2ZNfmPHJkgTcBTNTXrf8YbjaXbIvrbDnCnazPbZISlX9aWYg8hRjb/S6y8w3MthoyyCU5Hi4XaQEUyeeCDxRhxp6acUG0Rhag1BIosJRmuANSHjYGvn1JTZp6jJkZjWHN5hx5/DfiAyLgGQml90RtwNHdbOX9aElSKgNpAEvtI6w6fAVE9hLIk2A7mJyBY+pfAq2VsjYzH2G0FopshqFq3CqbwjI2BD cxHBn/M7 gkfiDSa9i1lIDsNQNuDHpBqnNvFhDTWoP687nUPkj+DBLZkXvAHEVXRlCF1u4in4zUqaF0ntOQ/NYIJopKfLXURTYM4ALZ7HUWWT2xwuhJ8tNDji7euRlDg8AcPpc9UMQO5EP+r7uVqLaMSxT14hPd7uHqKSBiX9gQp3HpDqkaSU0qfO8S0QsiaEE6zeM7NREi9J/26iW+SovPSNiJ8arZtx7QjXvZLiBqvEvpAbq7hS2y9NCBCsTwAm2Is4PjZfB7xdhZxKKa6xsRg8VdxB1WrbCtQta+89GSSaSlNXEl6tTXAo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Error injectable functions cannot be inlined and since some are called from hot paths, this incurrs overhead even if no error injection is enabled for them. To remove this overhead when disabled, allow the callsites of error injectable functions to put the calls behind a static key, which the framework can control when error injection is enabled or disabled for the function. Introduce a new ALLOW_ERROR_INJECTION_KEY() macro that adds a parameter with the static key's address, and store it in struct error_injection_entry. This new field has caused a mismatch when populating the injection list from the _error_injection_whitelist section with the current STRUCT_ALIGN(), so change the alignment to 8. During the population, copy the key's address also to struct ei_entry, and make it possible to retrieve it along with the error type by get_injectable_error_type(). Finally, make the processing of writes to the debugfs inject file enable the static key when the function is added to the injection list, and disable when removed. Signed-off-by: Vlastimil Babka --- include/asm-generic/error-injection.h | 13 ++++++++++++- include/asm-generic/vmlinux.lds.h | 2 +- include/linux/error-injection.h | 9 ++++++--- kernel/fail_function.c | 22 +++++++++++++++++++--- lib/error-inject.c | 6 +++++- 5 files changed, 43 insertions(+), 9 deletions(-) diff --git a/include/asm-generic/error-injection.h b/include/asm-generic/error-injection.h index b05253f68eaa..eed2731f3820 100644 --- a/include/asm-generic/error-injection.h +++ b/include/asm-generic/error-injection.h @@ -12,6 +12,7 @@ enum { struct error_injection_entry { unsigned long addr; + unsigned long static_key_addr; int etype; }; @@ -25,16 +26,26 @@ struct pt_regs; * 'Error Injectable Functions' section. */ #define ALLOW_ERROR_INJECTION(fname, _etype) \ -static struct error_injection_entry __used \ +static struct error_injection_entry __used __aligned(8) \ __section("_error_injection_whitelist") \ _eil_addr_##fname = { \ .addr = (unsigned long)fname, \ .etype = EI_ETYPE_##_etype, \ } +#define ALLOW_ERROR_INJECTION_KEY(fname, _etype, key) \ +static struct error_injection_entry __used __aligned(8) \ + __section("_error_injection_whitelist") \ + _eil_addr_##fname = { \ + .addr = (unsigned long)fname, \ + .static_key_addr = (unsigned long)key, \ + .etype = EI_ETYPE_##_etype, \ + } + void override_function_with_return(struct pt_regs *regs); #else #define ALLOW_ERROR_INJECTION(fname, _etype) +#define ALLOW_ERROR_INJECTION_KEY(fname, _etype, key) static inline void override_function_with_return(struct pt_regs *regs) { } #endif diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5703526d6ebf..1b15a0af2a00 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -248,7 +248,7 @@ #ifdef CONFIG_FUNCTION_ERROR_INJECTION #define ERROR_INJECT_WHITELIST() \ - STRUCT_ALIGN(); \ + . = ALIGN(8); \ BOUNDED_SECTION(_error_injection_whitelist) #else #define ERROR_INJECT_WHITELIST() diff --git a/include/linux/error-injection.h b/include/linux/error-injection.h index 20e738f4eae8..bec81b57a9d5 100644 --- a/include/linux/error-injection.h +++ b/include/linux/error-injection.h @@ -6,10 +6,12 @@ #include #include +struct static_key; + #ifdef CONFIG_FUNCTION_ERROR_INJECTION -extern bool within_error_injection_list(unsigned long addr); -extern int get_injectable_error_type(unsigned long addr); +bool within_error_injection_list(unsigned long addr); +int get_injectable_error_type(unsigned long addr, struct static_key **key_addr); #else /* !CONFIG_FUNCTION_ERROR_INJECTION */ @@ -18,7 +20,8 @@ static inline bool within_error_injection_list(unsigned long addr) return false; } -static inline int get_injectable_error_type(unsigned long addr) +static inline int get_injectable_error_type(unsigned long addr, + struct static_key **key_addr) { return -EOPNOTSUPP; } diff --git a/kernel/fail_function.c b/kernel/fail_function.c index d971a0189319..9240eb137e00 100644 --- a/kernel/fail_function.c +++ b/kernel/fail_function.c @@ -27,15 +27,16 @@ struct fei_attr { struct list_head list; struct kprobe kp; unsigned long retval; + struct static_key *key; }; static DEFINE_MUTEX(fei_lock); static LIST_HEAD(fei_attr_list); static DECLARE_FAULT_ATTR(fei_fault_attr); static struct dentry *fei_debugfs_dir; -static unsigned long adjust_error_retval(unsigned long addr, unsigned long retv) +static unsigned long __adjust_error_retval(int type, unsigned long retv) { - switch (get_injectable_error_type(addr)) { + switch (type) { case EI_ETYPE_NULL: return 0; case EI_ETYPE_ERRNO: @@ -53,9 +54,17 @@ static unsigned long adjust_error_retval(unsigned long addr, unsigned long retv) return retv; } +static unsigned long adjust_error_retval(unsigned long addr, unsigned long retv) +{ + int type = get_injectable_error_type(addr, NULL); + + return __adjust_error_retval(type, retv); +} + static struct fei_attr *fei_attr_new(const char *sym, unsigned long addr) { struct fei_attr *attr; + int type; attr = kzalloc(sizeof(*attr), GFP_KERNEL); if (attr) { @@ -66,7 +75,10 @@ static struct fei_attr *fei_attr_new(const char *sym, unsigned long addr) } attr->kp.pre_handler = fei_kprobe_handler; attr->kp.post_handler = fei_post_handler; - attr->retval = adjust_error_retval(addr, 0); + + type = get_injectable_error_type(addr, &attr->key); + attr->retval = __adjust_error_retval(type, 0); + INIT_LIST_HEAD(&attr->list); } return attr; @@ -218,6 +230,8 @@ static int fei_open(struct inode *inode, struct file *file) static void fei_attr_remove(struct fei_attr *attr) { + if (attr->key) + static_key_slow_dec(attr->key); fei_debugfs_remove_attr(attr); unregister_kprobe(&attr->kp); list_del(&attr->list); @@ -295,6 +309,8 @@ static ssize_t fei_write(struct file *file, const char __user *buffer, fei_attr_free(attr); goto out; } + if (attr->key) + static_key_slow_inc(attr->key); fei_debugfs_add_attr(attr); list_add_tail(&attr->list, &fei_attr_list); ret = count; diff --git a/lib/error-inject.c b/lib/error-inject.c index 887acd9a6ea6..e5f3b63f0dbb 100644 --- a/lib/error-inject.c +++ b/lib/error-inject.c @@ -17,6 +17,7 @@ struct ei_entry { struct list_head list; unsigned long start_addr; unsigned long end_addr; + struct static_key *key; int etype; void *priv; }; @@ -37,7 +38,7 @@ bool within_error_injection_list(unsigned long addr) return ret; } -int get_injectable_error_type(unsigned long addr) +int get_injectable_error_type(unsigned long addr, struct static_key **key_addr) { struct ei_entry *ent; int ei_type = -EINVAL; @@ -46,6 +47,8 @@ int get_injectable_error_type(unsigned long addr) list_for_each_entry(ent, &error_injection_list, list) { if (addr >= ent->start_addr && addr < ent->end_addr) { ei_type = ent->etype; + if (key_addr) + *key_addr = ent->key; break; } } @@ -86,6 +89,7 @@ static void populate_error_injection_list(struct error_injection_entry *start, ent->start_addr = entry; ent->end_addr = entry + size; ent->etype = iter->etype; + ent->key = (struct static_key *) iter->static_key_addr; ent->priv = priv; INIT_LIST_HEAD(&ent->list); list_add_tail(&ent->list, &error_injection_list); From patchwork Fri May 31 09:33:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1745BC25B75 for ; Fri, 31 May 2024 09:33:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 943D66B009A; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F11F6B009F; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 743936B009B; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5644D6B0095 for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0168BA1CE1 for ; Fri, 31 May 2024 09:33:43 +0000 (UTC) X-FDA: 82178178768.04.072C46A Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf30.hostedemail.com (Postfix) with ESMTP id CBBC280010 for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=5sHt+3lKlN8Bf+IcLW2oYc+uwJKSE7loId04scdWJMFQFyFQ+IYR8EwnV74r4OWdH/aqam /GTPzlknN6XwVIcH4fGpmLAS6leyjRma8kcJUHHU51J+f359FYwuZBuc94ZKzz0lJxEHnb 0MYsMnphlQ+tGUz4PPsCaBpWb2hh0oA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZP7fHUs/rbLHpmHtvQS8St+zXrEp6ICcSeYQ5zEHn4=; b=xB5RW1uNNC4RFk3y8eEvKNgQl4BhfzrKj0qx4+JX3M+zlQH4IS9yqJQcpgO2AKg6yLewEZ hYhU8SS1AwP/5T7tt6Y9mxuwefTSARC9PCsvdFZOhFeRgJ2KB+bIswJbXhS2XcDUgvgN2z qoGmEw6ppqH1qXLgmWpS9Y5ue5Gn73o= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 49B0E21B2B; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 2907813A91; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2I+7CXSZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:40 +0000 From: Vlastimil Babka Date: Fri, 31 May 2024 11:33:34 +0200 Subject: [PATCH RFC 3/4] mm, slab: add static key for should_failslab() MIME-Version: 1.0 Message-Id: <20240531-fault-injection-statickeys-v1-3-a513fd0a9614@suse.cz> References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CBBC280010 X-Stat-Signature: 8f6yddmgrmm76o59o9zc9n1ddfw7k64f X-HE-Tag: 1717148021-883203 X-HE-Meta: U2FsdGVkX1/X1wjibFM6CHwqxGbYpl/jpcUHCRZ+jTNXAhrcF5C/+UI5C660JRj8QLPIIdHb3f/ZTfnwvLK5tCsWnJ3cdvJRBIglmeJfwNCajXDcqNklquJR48DjV+bjo40j0PvWJwgQNw0S4lRsP0oqGPoZqp2tIcIShYx1b8kKHCP04MjiU+cyJBdWVS5iR78IzCAVDlW5o+Fzvi0HrhodjxdvYQ5AZWJ+GIvoLJZum2gzZvHr2iy9UvoULnEZFfMfpi4H14mg65D8k/ScdMeJKfTP7bbtmyeTdQLCKdCAo/5pLfgfN5XaTaA0tfP1ra2NYVyVx46xIv+HLiyN6UH4tnoXzangLITlKMzRkfQ1nVtTryCAnTDV4mTCIXk6gomoIcOmbHzKrm7so9g/4PoDMYfoHG68pFeuPBNVZhbr2FQMAfFdJBch4EQzs//j9spskJyDsPSAHAzEOS8/WNTErfuD/EZKrVFqYR72DJRIxjExrtkga3XJtHfTvL6sIwtYPmEuapk9Zg3uiQgTZa2X/zfRJ2yU8fr+sGLhZ3XrqIKWBY8yih055DEdiXZp2w3xTkbL9Xap6dzPphl/ww2NILZGqc6CuFIs0nRD8XGSyG3dxl2G+LAUiPhXIi/fCvT61evcZTiduMupTTBygPzvKDiFUkXmciPMrg4Jvxcx4a8FLjFXSC2PvojHD989ELVF0gtQ5L+WMH1cGHTB45JKQ+krRKroY13zc9Yp/ZiHsgJmg529usWb5wbIi8ImyLKTLd2RAsuhOM6nACJq2dHf9Z95r2HNNDfHFvbKT8ZK3UPB3ymZ9QuEAxb+ua/a+zTrtsPru4rWRTanhhhFXlIyxpvO0R5+p6zUMNx+Xavx4z7NNZg53vpcaTADSnkUQ14m94GPwpqOc9wDm6lLJ/vv5GF6z+ot/fzc/o5XBDCClayut/ft3tauq+wK9Guxf9q0+fo/jH9kjBRGkKT q309pR8d G0NxOGDPqQx3JqqTsDlHOlJXhTMZThtMWDQ/u3uck3c8MNwTvPQtZ0WnOmbI7KNc2lGuFvhlhStq+AvnVZ+1AHNbAljtV9qfWkE+witj4kpHs+6QIV3A1J0eWjrWcw+KC6CskqnWtLE3vmXn586gqtwhmVr2Pxvt94WbBTKf7NTiVql5mXNBDPJYTlbEIMNCp2y1WpbhTxEMWfP4S4FfHqf5eyCy9XZtguhJ/x9uKm7g32oo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since commit 4f6923fbb352 ("mm: make should_failslab always available for fault injection") should_failslab() is unconditionally a noinline function. This adds visible overhead to the slab allocation hotpath, even if the function is empty. With CONFIG_FAILSLAB=y there's additional overhead when the functionality is not enabled by a boot parameter or debugfs. The overhead can be eliminated with a static key around the callsite. Fault injection and error injection frameworks can now be told that the this function has a static key associated, and are able to enable and disable it accordingly. Signed-off-by: Vlastimil Babka Acked-by: Alexei Starovoitov Reviewed-by: Roman Gushchin --- mm/failslab.c | 2 +- mm/slab.h | 3 +++ mm/slub.c | 10 +++++++--- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/failslab.c b/mm/failslab.c index ffc420c0e767..878fd08e5dac 100644 --- a/mm/failslab.c +++ b/mm/failslab.c @@ -9,7 +9,7 @@ static struct { bool ignore_gfp_reclaim; bool cache_filter; } failslab = { - .attr = FAULT_ATTR_INITIALIZER, + .attr = FAULT_ATTR_INITIALIZER_KEY(&should_failslab_active.key), .ignore_gfp_reclaim = true, .cache_filter = false, }; diff --git a/mm/slab.h b/mm/slab.h index 5f8f47c5bee0..792e19cb37b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Internal slab definitions @@ -160,6 +161,8 @@ static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(freelist_aba_t) */ #define slab_page(s) folio_page(slab_folio(s), 0) +DECLARE_STATIC_KEY_FALSE(should_failslab_active); + /* * If network-based swap is enabled, sl*b must keep track of whether pages * were allocated from pfmemalloc reserves. diff --git a/mm/slub.c b/mm/slub.c index 0809760cf789..3bb579760a37 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3874,13 +3874,15 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, 0, sizeof(void *)); } +DEFINE_STATIC_KEY_FALSE(should_failslab_active); + noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) { if (__should_failslab(s, gfpflags)) return -ENOMEM; return 0; } -ALLOW_ERROR_INJECTION(should_failslab, ERRNO); +ALLOW_ERROR_INJECTION_KEY(should_failslab, ERRNO, &should_failslab_active); static __fastpath_inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) @@ -3889,8 +3891,10 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) might_alloc(flags); - if (unlikely(should_failslab(s, flags))) - return NULL; + if (static_branch_unlikely(&should_failslab_active)) { + if (should_failslab(s, flags)) + return NULL; + } return s; } From patchwork Fri May 31 09:33:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E25F8C27C44 for ; Fri, 31 May 2024 09:33:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2ACB26B0095; Fri, 31 May 2024 05:33:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 08EB66B00A0; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEDA16B0099; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8EF126B009C for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4B8B2805E3 for ; Fri, 31 May 2024 09:33:44 +0000 (UTC) X-FDA: 82178178768.27.17C6CD6 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf29.hostedemail.com (Postfix) with ESMTP id F1316120003 for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=RJ44r3mEV2SgE/KoVtmnKWZ2IGDv1l1PYxRL2TZKNuqnTs2c1ZIBmfEu7O3GZ3w9DjFH9H hlY6JjblYzaVweg7MWH/5kZJLtt7XdumuQyREYnalWwrLmCJz0Oq6F37vsO6qzj/ngN8/2 4UVnOihLJ9YHlo5UaOmJFw8IvX9gyX0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=cq3ZS3Bb; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=WQK9iHyh; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=nvv4gm6++P5wbF2mCqw2xnoLu6qjly35SHirIFmsuTRC0UaOnH8IUuY962O9549PsZEc5j o+fhB/je5d0tAOXDy7iLMSMvz1VmAWj+P1ftLjQrZ349EEZvd9u1DSm03+Uutg4pp3j5h9 55VpHXVrIRgNqtetr8yFcA/thhyifF4= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6322421B2C; Fri, 31 May 2024 09:33:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=cq3ZS3BbflQ7N2K6lBu8bapgCyqq54EbZLCUHI5AOJr2zGvzJwjgIv1pOlAsS/4VJ+HD// DqP8gM6quCsk/Rf4szRlG7kaJ5KOQrpnHm+OrwzFSPVsOugt54MfF1PraU/+2ZdMSbMJUb WfFBo7wBG5E4quordFfd4kyyCdsRO1E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=WQK9iHyhgXJjz5UKPeIRCrBHBTFp99Hwb64bZURii7O8xDLTgt/3LGZGt8RkE0fHxqXxKb zTsFKRohlCJOaEDw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=cq3ZS3BbflQ7N2K6lBu8bapgCyqq54EbZLCUHI5AOJr2zGvzJwjgIv1pOlAsS/4VJ+HD// DqP8gM6quCsk/Rf4szRlG7kaJ5KOQrpnHm+OrwzFSPVsOugt54MfF1PraU/+2ZdMSbMJUb WfFBo7wBG5E4quordFfd4kyyCdsRO1E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zr3/XI/R4rZyT+/e67qBFKZJGDvaAXhwDIbiD/WM1m0=; b=WQK9iHyhgXJjz5UKPeIRCrBHBTFp99Hwb64bZURii7O8xDLTgt/3LGZGt8RkE0fHxqXxKb zTsFKRohlCJOaEDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 45159132C2; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eAqVEHSZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:40 +0000 From: Vlastimil Babka Date: Fri, 31 May 2024 11:33:35 +0200 Subject: [PATCH RFC 4/4] mm, page_alloc: add static key for should_fail_alloc_page() MIME-Version: 1.0 Message-Id: <20240531-fault-injection-statickeys-v1-4-a513fd0a9614@suse.cz> References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Rspamd-Queue-Id: F1316120003 X-Stat-Signature: 5tfmk9yzcgc5wy9w3psxd6u9getdzz8i X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717148021-93072 X-HE-Meta: U2FsdGVkX18M8BhFO8MiWmHN/n6mPwLiGblOsy5ltKd5JVyMC8LluRcfmBhvMK/S/HdrQyfbAYH2tSQwMSjzoSCP+B6H3F2P81d/progZWIXGb4qSKqkSmElP2AZt9AgvpBQgL3pSuSiCm237+5YN+VhP2AnvteYSIN1Swg9M3IYfh+qo5j+1CKGR2rFkEZk+ah9rhaTIo8bWMik6WiqnTxiiB9sI4sogEot0aW+Tdv7L2qWwJZZyW7cqdSmssRUqZ9vyUHdNh/hJq2lyhO22LKRezmfFbMV7hwnFuYoHAYjz/Qr15kjaMVwaQh4b1Vh4vx5hgBtcgqHLCKukbMJKFyk/RKoK1YWD6NpWpyx1+ho16TS1t9LZ9Oz/ndWFDxSBtc7FwYU6hyDwgler2BlQkISQEZraoA75bMICpWPIK89SQdck14UmjwTxrg0v/+TOhNUpDAcXpyRQ03I8FEJaUQwKXoynfZeiU89vXgiwfdgGOc7sfbGJPreCdmsFrIBXq9F/c3yoVZNKb0Ga7lC+DiuMB0w94uc1oMcq0lCpZY4tXDf2+TSovE2hp8Lct2DPv0pYQilbKxyf5tQ1WJHJnigU3nzDpxRKx35P+lf7SpM6vjAUh9VVbxymeW6sXeNef7N5XqY5UWeK6/3XHg5ErEkQ/barXD7qSHAz6fGkV/DmVUelpRlrI8Vrudwid4fMXNDy7/5nJUhlJIx58zZ3g/rW0jTv1QN20gV1jsfMnht23CK2UKcnFVodPXyxgjWCp6XQC6L4q2hTzcEM/giuwVsEqhYcA806sTNjSfuio+ZignGkSpIAIUFQHrLgte+ub4YX691y6i4h/IdZaGPmMrpxHkBgxnOetRCUT/N0+NblSleYALYap0PToYbCDJqjOQ+f/z7E2yahf3hQlYsYKD80j74eksRuF4FPObbmKSTlV7gqDVvUjOHPBZU3Z6U9lUYth3I1gtoPSsLCSE jBzTy6YB /PbY8LuERryGtSISiBxSZv6I17TL2xtAjCtBzrapCNTIQmS4HRvskTr+GC1gzogERmre0ID8LCVgB+HamCNg+E/bcEY6RCxYks4iUeX4+blHCsPCB2sP9LkQxO027rSshIOcFAqTiVWRAEvk6bVTbZsIe2WKf3HqSvBwJGHDjElMr97G7nSDSkVNcjiPJ7JGcvHVaYQ9akpRrHck3BNsC49oGbYgYtKFkCKYY1GNxPgCayBpOzcHmd/sAlfBTCqQB/YQtkTqpo8fuaHdLgmaX+ibpYbnz2sEWYpzfDsTI109XT9B+qgMJmuXLnot95UrYJsHdlI0qEPp1als9U/2OSMcc2A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similarly to should_failslab(), remove the overhead of calling the noinline function should_fail_alloc_page() with a static key that guards the allocation hotpath callsite and is controlled by the fault and error injection frameworks. Signed-off-by: Vlastimil Babka Reviewed-by: Roman Gushchin --- mm/fail_page_alloc.c | 3 ++- mm/internal.h | 2 ++ mm/page_alloc.c | 11 ++++++++--- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/fail_page_alloc.c b/mm/fail_page_alloc.c index b1b09cce9394..0906b76d78e8 100644 --- a/mm/fail_page_alloc.c +++ b/mm/fail_page_alloc.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include "internal.h" static struct { struct fault_attr attr; @@ -9,7 +10,7 @@ static struct { bool ignore_gfp_reclaim; u32 min_order; } fail_page_alloc = { - .attr = FAULT_ATTR_INITIALIZER, + .attr = FAULT_ATTR_INITIALIZER_KEY(&should_fail_alloc_page_active.key), .ignore_gfp_reclaim = true, .ignore_gfp_highmem = true, .min_order = 1, diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..8539e39b02e6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -410,6 +410,8 @@ extern char * const zone_names[MAX_NR_ZONES]; /* perform sanity checks on struct pages being allocated or freed */ DECLARE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); +DECLARE_STATIC_KEY_FALSE(should_fail_alloc_page_active); + extern int min_free_kbytes; void setup_per_zone_wmarks(void); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2e22ce5675ca..e5dc3bafa549 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -274,6 +274,8 @@ int user_min_free_kbytes = -1; static int watermark_boost_factor __read_mostly = 15000; static int watermark_scale_factor = 10; +DEFINE_STATIC_KEY_FALSE(should_fail_alloc_page_active); + /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */ int movable_zone; EXPORT_SYMBOL(movable_zone); @@ -3012,7 +3014,7 @@ noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) { return __should_fail_alloc_page(gfp_mask, order); } -ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE); +ALLOW_ERROR_INJECTION_KEY(should_fail_alloc_page, TRUE, &should_fail_alloc_page_active); static inline long __zone_watermark_unusable_free(struct zone *z, unsigned int order, unsigned int alloc_flags) @@ -4430,8 +4432,11 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) - return false; + if (static_branch_unlikely(&should_fail_alloc_page_active)) { + if (should_fail_alloc_page(gfp_mask, order)) { + return false; + } + } *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags);