From patchwork Wed Jun 19 22:48:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704704 X-Patchwork-Delegate: mhiramat@kernel.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BE9C80C15; Wed, 19 Jun 2024 22:49:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837387; cv=none; b=arnBbHzdkOa6E3iTraKNCzH3zl5LctpUbHVSpv2gqfQAUg2loF8RXAUaz7QB4cO0LUPvTRt+1NiJUiZZPL+t7niNrXi8BwnGyRwWskQUF+pEeB1BUXIUSs54rkJAj8i3yja7WajnlzcqwzsEH05Ap0bCXce3mnEd/sSI1keTHoo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837387; c=relaxed/simple; bh=/cKM1m+IMVDzuJzFS+1fl48bPoWvR2tEgzVaVUXVT8A=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=MUzScgJFsBHOY4IOyTTKkV3zuNI1ZIIOY/8FyxDM9/ZuOZ+mcrHFKMsQ8y6Evp3bRmmAIqnYLX76euApKKrCLqtT9rMnfiVSTdNGRjTDLPkIaBlWlac+DCY3TkF5bseUJlXTCnAi33TSwHCN4KWlyADOqZO756rAvH3S5trC/u8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=smRbBT/c; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=g3E3XRzi; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=smRbBT/c; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=g3E3XRzi; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="smRbBT/c"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="g3E3XRzi"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="smRbBT/c"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="g3E3XRzi" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 25D261F7F5; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r7eLs3qWVHkCYzNUctfASvMgvFekhPcbd8a5b9Uty8M=; b=smRbBT/csZSGJtlsJ4JN3oklSIC7vbquyuWyYFQK4X7uOph5tB+4a752AntXvJy1S6SyqH IS3Q3mHFdaV3Nuk7YxxoaL7OyDll3aJ5H4tMiqF2zEByxywgSPn6DUWnsDAtqd2l0oVEwI Ih3Y7GYugjJN390WQpBeTcoBGFUj6Z4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r7eLs3qWVHkCYzNUctfASvMgvFekhPcbd8a5b9Uty8M=; b=g3E3XRziF6HgkAAPS88wKQ5lNo4texbHFNeFeEwV0v4mYHhP+i771BZm1quZ/kUsnkbfs9 f1WTtAqzsMR7PdDw== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r7eLs3qWVHkCYzNUctfASvMgvFekhPcbd8a5b9Uty8M=; b=smRbBT/csZSGJtlsJ4JN3oklSIC7vbquyuWyYFQK4X7uOph5tB+4a752AntXvJy1S6SyqH IS3Q3mHFdaV3Nuk7YxxoaL7OyDll3aJ5H4tMiqF2zEByxywgSPn6DUWnsDAtqd2l0oVEwI Ih3Y7GYugjJN390WQpBeTcoBGFUj6Z4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r7eLs3qWVHkCYzNUctfASvMgvFekhPcbd8a5b9Uty8M=; b=g3E3XRziF6HgkAAPS88wKQ5lNo4texbHFNeFeEwV0v4mYHhP+i771BZm1quZ/kUsnkbfs9 f1WTtAqzsMR7PdDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 0152E13ABD; Wed, 19 Jun 2024 22:49:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 0PuTO4Zgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:42 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:48:55 +0200 Subject: [PATCH v2 1/7] fault-inject: add support for static keys around fault injection sites Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-1-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4698; i=vbabka@suse.cz; h=from:subject:message-id; bh=/cKM1m+IMVDzuJzFS+1fl48bPoWvR2tEgzVaVUXVT8A=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2Byf1V3Bpp5LNGsY/c7Mzv3/GsPZ55FjBkLd LX7xvgBpSWJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNgcgAKCRC74LB10kWI mrooB/987gBj0pha0dvAhFXZiGUJdrVCDtQ6t2BV/SIlkveQM37wjVIcJKlT46POmu0EAVLbGts rfVWVX5e4aCI80nZ5KpkSs0COyqzR+4Eym09R0s0Mm29ctQ8BJYXISybtjvacF6lT2sDF1PvZG8 i3JzV58YERvX5BU+hxW2HjtgpEHgCzBBjFK/t/heKLFCeork00ZpNgQXER2fUDtpVCd4m62wTb9 aJr4p+K+XdUOvo164UnmBjTbqVN9BjUYA1laDaYyZ5+8942w7HBeyumYAqx3oJBarTi3ujoosjc 1Yyw2OhX6svXSoURISrllNMod+BTK/2jVgFrBLmcWyLPRwOW X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[20]; FREEMAIL_TO(0.00)[gmail.com,linux.com,google.com,kernel.org,iogearbox.net,linux.ibm.com,intel.com,davemloft.net,goodmis.org,arm.com]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[kernel.org,linux.dev,gmail.com,vger.kernel.org,kvack.org,suse.cz]; R_RATELIMIT(0.00)[to_ip_from(RL5nkphuxq5kxo98ppmuqoc8wo)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email] X-Spam-Flag: NO X-Spam-Score: -6.80 X-Spam-Level: Some fault injection sites are placed in hotpaths and incur overhead even if not enabled, due to one or more function calls leading up to should_fail_ex() that returns false due to attr->probability == 0. This overhead can be eliminated if the outermost call into the checks is guarded with a static key, so add support for that. The framework should be told that such static key exist for a fault_attr, by initializing fault_attr->active with the static key address. When it's not NULL, enable the static key from setup_fault_attr() when the fault probability is non-zero. Also wire up writing into debugfs "probability" file to enable or disable the static key when transitioning between zero and non-zero probability. For now, do not add configfs interface support as the immediate plan is to leverage this for should_failslab() and should_fail_alloc_page() after other necessary preparatory changes, and not for any of the configfs based fault injection users. Signed-off-by: Vlastimil Babka --- include/linux/fault-inject.h | 7 ++++++- lib/fault-inject.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/include/linux/fault-inject.h b/include/linux/fault-inject.h index 6d5edef09d45..cfe75cc1bac4 100644 --- a/include/linux/fault-inject.h +++ b/include/linux/fault-inject.h @@ -9,6 +9,7 @@ #include #include #include +#include /* * For explanation of the elements of this struct, see @@ -30,13 +31,14 @@ struct fault_attr { unsigned long count; struct ratelimit_state ratelimit_state; struct dentry *dname; + struct static_key *active; }; enum fault_flags { FAULT_NOWARN = 1 << 0, }; -#define FAULT_ATTR_INITIALIZER { \ +#define FAULT_ATTR_INITIALIZER_KEY(_key) { \ .interval = 1, \ .times = ATOMIC_INIT(1), \ .require_end = ULONG_MAX, \ @@ -44,8 +46,11 @@ enum fault_flags { .ratelimit_state = RATELIMIT_STATE_INIT_DISABLED, \ .verbose = 2, \ .dname = NULL, \ + .active = (_key), \ } +#define FAULT_ATTR_INITIALIZER FAULT_ATTR_INITIALIZER_KEY(NULL) + #define DECLARE_FAULT_ATTR(name) struct fault_attr name = FAULT_ATTR_INITIALIZER int setup_fault_attr(struct fault_attr *attr, char *str); bool should_fail_ex(struct fault_attr *attr, ssize_t size, int flags); diff --git a/lib/fault-inject.c b/lib/fault-inject.c index d608f9b48c10..de9552cb22d0 100644 --- a/lib/fault-inject.c +++ b/lib/fault-inject.c @@ -35,6 +35,9 @@ int setup_fault_attr(struct fault_attr *attr, char *str) atomic_set(&attr->times, times); atomic_set(&attr->space, space); + if (probability != 0 && attr->active) + static_key_slow_inc(attr->active); + return 1; } EXPORT_SYMBOL_GPL(setup_fault_attr); @@ -166,6 +169,12 @@ EXPORT_SYMBOL_GPL(should_fail); #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS +/* + * Protect updating probability from debugfs as that may trigger static key + * changes when changing between zero and non-zero. + */ +static DEFINE_MUTEX(probability_mutex); + static int debugfs_ul_set(void *data, u64 val) { *(unsigned long *)data = val; @@ -186,6 +195,38 @@ static void debugfs_create_ul(const char *name, umode_t mode, debugfs_create_file(name, mode, parent, value, &fops_ul); } +static int debugfs_prob_set(void *data, u64 val) +{ + struct fault_attr *attr = data; + + mutex_lock(&probability_mutex); + + if (attr->active) { + if (attr->probability != 0 && val == 0) { + static_key_slow_dec(attr->active); + } else if (attr->probability == 0 && val != 0) { + static_key_slow_inc(attr->active); + } + } + + attr->probability = val; + + mutex_unlock(&probability_mutex); + + return 0; +} + +static int debugfs_prob_get(void *data, u64 *val) +{ + struct fault_attr *attr = data; + + *val = attr->probability; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_prob, debugfs_prob_get, debugfs_prob_set, "%llu\n"); + #ifdef CONFIG_FAULT_INJECTION_STACKTRACE_FILTER static int debugfs_stacktrace_depth_set(void *data, u64 val) @@ -218,7 +259,7 @@ struct dentry *fault_create_debugfs_attr(const char *name, if (IS_ERR(dir)) return dir; - debugfs_create_ul("probability", mode, dir, &attr->probability); + debugfs_create_file("probability", mode, dir, attr, &fops_prob); debugfs_create_ul("interval", mode, dir, &attr->interval); debugfs_create_atomic_t("times", mode, dir, &attr->times); debugfs_create_atomic_t("space", mode, dir, &attr->space); From patchwork Wed Jun 19 22:48:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704701 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F26077F10; Wed, 19 Jun 2024 22:49:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837386; cv=none; b=bnQO/3hGfs1t7NJr4gGMVPm7vsm9uqaucs2DQxlYhI5YQjkAizsDiY5FazTSHJWomJwXSFZHSX6UtLf9bKdB9u/WE+sZbrr/6ftrNtqQBfP1pu1vN7dHstjTAUV2h3+iWymB9IfIhO3hlFqYseJw1at2t2/dkYrC03qO2XxwkdI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837386; c=relaxed/simple; bh=nCcr0F2SycRd4ZmkZlDmyM/fEKeGlbV/Lf6N6wPJuRc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fBB6QS6icyvN5LHJcPMurMQZqCwGFLZt7yrQi58ossh3gDo9SVPDQEkvus/3SYOAcqyxClrppVTMIlMtfd9iDOdMTgObpnmfiP2t676Ir1cbsiAiQizuD77jR5ucwC/DnOHvjPNPcqQEVbpOMFTF5Ne+cL5/Ds8XKwfynoVuZ4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 41DA621A2E; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Authentication-Results: smtp-out1.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1F2C313AC3; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id EHVGB4dgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:48:56 +0200 Subject: [PATCH v2 2/7] error-injection: support static keys around injectable functions Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-2-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6789; i=vbabka@suse.cz; h=from:subject:message-id; bh=nCcr0F2SycRd4ZmkZlDmyM/fEKeGlbV/Lf6N6wPJuRc=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2B1Frl4h+MJPB9HCCgG1OKOEt9S1LyDBVAEU 7M0ia0/FuiJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNgdQAKCRC74LB10kWI mp78B/0Vz6PDUOi8A2kc1lduJ93rou5JOVEFkLMCT4JoJKKbCqgoG658Fnnx3HlaQKP7S28r1fP 3vgXATaULFd5SCEjA0oxzDNE6YzK5EL7BA3K1xGx2ofu2hYba8jUh2vCZNZdX5+CZfVZ0ia4k1F HeqTxFmhEPAcJdAKjraSQVpcrsT7x5vV2wQojWCTDZ6JdWc8oL0+KJgVJG2ZQa3s8WdwYCP3nLu 7jplQLjhYrn1Z/XT2KfZoM2MkKZtaAbOHvn26QSrMM7DkuEfK4dHHHDCMXkU9NPJ7I3zN5sUNll X6uoyBbRQtQVe7e4xWwoXZNVn1VN3YvXt5fClIg6WZkAwvHt X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 41DA621A2E X-Rspamd-Action: no action X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org Error injectable functions cannot be inlined and since some are called from hot paths, this incurs overhead even if no error injection is enabled for them. To avoid this overhead when disabled, allow the callsites of error injectable functions to put the calls behind a static key, which the framework can control when error injection is enabled or disabled for the function. Introduce a new ALLOW_ERROR_INJECTION_KEY() macro that adds a parameter with the static key's address, and store it in struct error_injection_entry. This new field has caused a mismatch when populating the injection list from the _error_injection_whitelist section using the current STRUCT_ALIGN(), so change the alignment to 8. During the population, copy the key's address also to struct ei_entry, and make it possible to retrieve it by get_injection_key(). Finally, make the processing of writes to the debugfs inject file enable the static key when the function is added to the injection list, and disable when removed. Signed-off-by: Vlastimil Babka Reviewed-by: Steven Rostedt (Google) --- include/asm-generic/error-injection.h | 13 ++++++++++++- include/asm-generic/vmlinux.lds.h | 2 +- include/linux/error-injection.h | 12 ++++++++++-- kernel/fail_function.c | 10 ++++++++++ lib/error-inject.c | 19 +++++++++++++++++++ 5 files changed, 52 insertions(+), 4 deletions(-) diff --git a/include/asm-generic/error-injection.h b/include/asm-generic/error-injection.h index b05253f68eaa..eed2731f3820 100644 --- a/include/asm-generic/error-injection.h +++ b/include/asm-generic/error-injection.h @@ -12,6 +12,7 @@ enum { struct error_injection_entry { unsigned long addr; + unsigned long static_key_addr; int etype; }; @@ -25,16 +26,26 @@ struct pt_regs; * 'Error Injectable Functions' section. */ #define ALLOW_ERROR_INJECTION(fname, _etype) \ -static struct error_injection_entry __used \ +static struct error_injection_entry __used __aligned(8) \ __section("_error_injection_whitelist") \ _eil_addr_##fname = { \ .addr = (unsigned long)fname, \ .etype = EI_ETYPE_##_etype, \ } +#define ALLOW_ERROR_INJECTION_KEY(fname, _etype, key) \ +static struct error_injection_entry __used __aligned(8) \ + __section("_error_injection_whitelist") \ + _eil_addr_##fname = { \ + .addr = (unsigned long)fname, \ + .static_key_addr = (unsigned long)key, \ + .etype = EI_ETYPE_##_etype, \ + } + void override_function_with_return(struct pt_regs *regs); #else #define ALLOW_ERROR_INJECTION(fname, _etype) +#define ALLOW_ERROR_INJECTION_KEY(fname, _etype, key) static inline void override_function_with_return(struct pt_regs *regs) { } #endif diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5703526d6ebf..1b15a0af2a00 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -248,7 +248,7 @@ #ifdef CONFIG_FUNCTION_ERROR_INJECTION #define ERROR_INJECT_WHITELIST() \ - STRUCT_ALIGN(); \ + . = ALIGN(8); \ BOUNDED_SECTION(_error_injection_whitelist) #else #define ERROR_INJECT_WHITELIST() diff --git a/include/linux/error-injection.h b/include/linux/error-injection.h index 20e738f4eae8..48da027c0302 100644 --- a/include/linux/error-injection.h +++ b/include/linux/error-injection.h @@ -6,10 +6,13 @@ #include #include +struct static_key; + #ifdef CONFIG_FUNCTION_ERROR_INJECTION -extern bool within_error_injection_list(unsigned long addr); -extern int get_injectable_error_type(unsigned long addr); +bool within_error_injection_list(unsigned long addr); +int get_injectable_error_type(unsigned long addr); +struct static_key *get_injection_key(unsigned long addr); #else /* !CONFIG_FUNCTION_ERROR_INJECTION */ @@ -23,6 +26,11 @@ static inline int get_injectable_error_type(unsigned long addr) return -EOPNOTSUPP; } +static inline struct static_key *get_injection_key(unsigned long addr) +{ + return NULL; +} + #endif #endif /* _LINUX_ERROR_INJECTION_H */ diff --git a/kernel/fail_function.c b/kernel/fail_function.c index d971a0189319..d39a9606a448 100644 --- a/kernel/fail_function.c +++ b/kernel/fail_function.c @@ -27,6 +27,7 @@ struct fei_attr { struct list_head list; struct kprobe kp; unsigned long retval; + struct static_key *key; }; static DEFINE_MUTEX(fei_lock); static LIST_HEAD(fei_attr_list); @@ -67,6 +68,11 @@ static struct fei_attr *fei_attr_new(const char *sym, unsigned long addr) attr->kp.pre_handler = fei_kprobe_handler; attr->kp.post_handler = fei_post_handler; attr->retval = adjust_error_retval(addr, 0); + + attr->key = get_injection_key(addr); + if (IS_ERR(attr->key)) + attr->key = NULL; + INIT_LIST_HEAD(&attr->list); } return attr; @@ -218,6 +224,8 @@ static int fei_open(struct inode *inode, struct file *file) static void fei_attr_remove(struct fei_attr *attr) { + if (attr->key) + static_key_slow_dec(attr->key); fei_debugfs_remove_attr(attr); unregister_kprobe(&attr->kp); list_del(&attr->list); @@ -295,6 +303,8 @@ static ssize_t fei_write(struct file *file, const char __user *buffer, fei_attr_free(attr); goto out; } + if (attr->key) + static_key_slow_inc(attr->key); fei_debugfs_add_attr(attr); list_add_tail(&attr->list, &fei_attr_list); ret = count; diff --git a/lib/error-inject.c b/lib/error-inject.c index 887acd9a6ea6..982fbedd9ad5 100644 --- a/lib/error-inject.c +++ b/lib/error-inject.c @@ -17,6 +17,7 @@ struct ei_entry { struct list_head list; unsigned long start_addr; unsigned long end_addr; + struct static_key *key; int etype; void *priv; }; @@ -54,6 +55,23 @@ int get_injectable_error_type(unsigned long addr) return ei_type; } +struct static_key *get_injection_key(unsigned long addr) +{ + struct ei_entry *ent; + struct static_key *ei_key = ERR_PTR(-EINVAL); + + mutex_lock(&ei_mutex); + list_for_each_entry(ent, &error_injection_list, list) { + if (addr >= ent->start_addr && addr < ent->end_addr) { + ei_key = ent->key; + break; + } + } + mutex_unlock(&ei_mutex); + + return ei_key; +} + /* * Lookup and populate the error_injection_list. * @@ -86,6 +104,7 @@ static void populate_error_injection_list(struct error_injection_entry *start, ent->start_addr = entry; ent->end_addr = entry + size; ent->etype = iter->etype; + ent->key = (struct static_key *) iter->static_key_addr; ent->priv = priv; INIT_LIST_HEAD(&ent->list); list_add_tail(&ent->list, &error_injection_list); From patchwork Wed Jun 19 22:48:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704700 X-Patchwork-Delegate: mhiramat@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F4437F7C7; Wed, 19 Jun 2024 22:49:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837386; cv=none; b=igiavJmjvBAlIDCJ4bV1xv9JzLIRC0NjjVnwwpn6mBQf3tSVUTwgVLifI8WSGLULXpgPxxkn5UbmUDPMTdbClv+dofLb6hkh3/CVHn331TlNI4OrJ9AxhRWu9JQnNehWX/spqqjfBS4nx+jeE0V9Xt+BqeSFOsazLWLDB/PgtQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837386; c=relaxed/simple; bh=RHlcQH9C6ly4CbQFrHL5k0pueBGzmfFJHg6uosgSnqE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=N0pvB+lUsDQbQfv7zdCVF6OW2dgywUije4PuDqEHQPoSokWxOxfM5/TCxUdPg3ShXvyKBTDAf0UPtpSp3VUH8oUyWqoogz9kQDCZh/j/yhCsV1v5iHPLqDyv2sHmRJUM6578oqBjGgmcQohumF7M9Yu0Ly+OAgSw+ZPEq2m0WYs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=mtxCY5NO; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=2+cvj+i4; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=mtxCY5NO; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=2+cvj+i4; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="mtxCY5NO"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="2+cvj+i4"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="mtxCY5NO"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="2+cvj+i4" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5C51F21A5C; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GJmQfB+SFKwkGljPkMoSf+PzrKg2yTJDdkhioKlFbR0=; b=mtxCY5NOZUMLeNOQo2490WnZzrZz5lK6g9Qm7Ex4Shmq2UVpwBxwZP+nWMnZFY2Qk8nBs1 siLzlqttayrh4IykYENtS8rfrWC8ksDwhasAHMwn4I+ihPTkA8zNMcN6XlS3RF8s2wohXI U+CXOQeOPkjiu8wHU+JqsGy5rejV9nA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GJmQfB+SFKwkGljPkMoSf+PzrKg2yTJDdkhioKlFbR0=; b=2+cvj+i4DhZ6XKrKA7se2fWikyWEQBDFIpBjv+eA7nIqPwGQ03LS2LkiZhUbMyFqEGyfvQ qHnQkg3JJUtkfiDQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GJmQfB+SFKwkGljPkMoSf+PzrKg2yTJDdkhioKlFbR0=; b=mtxCY5NOZUMLeNOQo2490WnZzrZz5lK6g9Qm7Ex4Shmq2UVpwBxwZP+nWMnZFY2Qk8nBs1 siLzlqttayrh4IykYENtS8rfrWC8ksDwhasAHMwn4I+ihPTkA8zNMcN6XlS3RF8s2wohXI U+CXOQeOPkjiu8wHU+JqsGy5rejV9nA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GJmQfB+SFKwkGljPkMoSf+PzrKg2yTJDdkhioKlFbR0=; b=2+cvj+i4DhZ6XKrKA7se2fWikyWEQBDFIpBjv+eA7nIqPwGQ03LS2LkiZhUbMyFqEGyfvQ qHnQkg3JJUtkfiDQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 3D08413AAA; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id SCSeDodgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:48:57 +0200 Subject: [PATCH v2 3/7] bpf: support error injection static keys for perf_event attached progs Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-3-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4256; i=vbabka@suse.cz; h=from:subject:message-id; bh=RHlcQH9C6ly4CbQFrHL5k0pueBGzmfFJHg6uosgSnqE=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2B4jX+g3Xf4kDO7RN55ImULUpPLAS31+cJIL VWj6k4LZzmJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNgeAAKCRC74LB10kWI mk8pCACesvfkTPT71qvqKXfyLJtHc/HZ29/KezJP030k0GlFWTkG41XeSOf+Jc9jxoC1871A4SK EWFDuN7tbL8sEOdT7cBJU0MBEXQSDEYNDcoGyZxl98cLsAJ9XHVdkhNOzgJJgPRwcnPT10ejR6X Eb39gdyBoKUdHWD8IoRYiCpG6/Wx8PRQUjK59shva6E514qQRimzOvre6ZR5OQrcynjumq8WrVE n12/Bf0XbXqhrgaM1emVNUHExQn1U7iCYwHGEzlGPNvC8Mi+wQG9Xsq7xvjA5OSOUmmmDusj3eC 3soz/74FQINFipML/55CmhoPIFrOsjPyQbm5FZDYbWm25i7L X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[20]; FREEMAIL_TO(0.00)[gmail.com,linux.com,google.com,kernel.org,iogearbox.net,linux.ibm.com,intel.com,davemloft.net,goodmis.org,arm.com]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[kernel.org,linux.dev,gmail.com,vger.kernel.org,kvack.org,suse.cz]; R_RATELIMIT(0.00)[to_ip_from(RL5nkphuxq5kxo98ppmuqoc8wo)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email] X-Spam-Flag: NO X-Spam-Score: -6.80 X-Spam-Level: Functions marked for error injection can have an associated static key that guards the callsite(s) to avoid overhead of calling an empty function when no error injection is in progress. Outside of the error injection framework itself, bpf programs can be atteched to perf events and override results of error-injectable functions. To make sure these functions are actually called, attaching such bpf programs should control the static key accordingly. Therefore, add the static key's address to struct trace_kprobe and fill it in trace_kprobe_error_injectable(), using get_injection_key() instead of within_error_injection_list(). Introduce trace_kprobe_error_injection_control() to control the static key and call the control function when attaching or detaching programs with kprobe_override to perf events. Signed-off-by: Vlastimil Babka --- kernel/trace/bpf_trace.c | 6 ++++++ kernel/trace/trace_kprobe.c | 30 ++++++++++++++++++++++++++++-- kernel/trace/trace_probe.h | 5 +++++ 3 files changed, 39 insertions(+), 2 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index f5154c051d2c..944de1c41209 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2283,6 +2283,9 @@ int perf_event_attach_bpf_prog(struct perf_event *event, rcu_assign_pointer(event->tp_event->prog_array, new_array); bpf_prog_array_free_sleepable(old_array); + if (prog->kprobe_override) + trace_kprobe_error_injection_control(event->tp_event, true); + unlock: mutex_unlock(&bpf_event_mutex); return ret; @@ -2299,6 +2302,9 @@ void perf_event_detach_bpf_prog(struct perf_event *event) if (!event->prog) goto unlock; + if (event->prog->kprobe_override) + trace_kprobe_error_injection_control(event->tp_event, false); + old_array = bpf_event_rcu_dereference(event->tp_event->prog_array); ret = bpf_prog_array_copy(old_array, event->prog, NULL, 0, &new_array); if (ret == -ENOENT) diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c index 16383247bdbf..1c1ee95bd5de 100644 --- a/kernel/trace/trace_kprobe.c +++ b/kernel/trace/trace_kprobe.c @@ -61,6 +61,7 @@ struct trace_kprobe { unsigned long __percpu *nhit; const char *symbol; /* symbol name */ struct trace_probe tp; + struct static_key *ei_key; }; static bool is_trace_kprobe(struct dyn_event *ev) @@ -235,9 +236,34 @@ bool trace_kprobe_on_func_entry(struct trace_event_call *call) bool trace_kprobe_error_injectable(struct trace_event_call *call) { struct trace_kprobe *tk = trace_kprobe_primary_from_call(call); + struct static_key *ei_key; - return tk ? within_error_injection_list(trace_kprobe_address(tk)) : - false; + if (!tk) + return false; + + ei_key = get_injection_key(trace_kprobe_address(tk)); + if (IS_ERR(ei_key)) + return false; + + tk->ei_key = ei_key; + return true; +} + +void trace_kprobe_error_injection_control(struct trace_event_call *call, + bool enable) +{ + struct trace_kprobe *tk = trace_kprobe_primary_from_call(call); + + if (!tk) + return; + + if (!tk->ei_key) + return; + + if (enable) + static_key_slow_inc(tk->ei_key); + else + static_key_slow_dec(tk->ei_key); } static int register_kprobe_event(struct trace_kprobe *tk); diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h index 5803e6a41570..d9ddcabb9f97 100644 --- a/kernel/trace/trace_probe.h +++ b/kernel/trace/trace_probe.h @@ -212,6 +212,8 @@ DECLARE_BASIC_PRINT_TYPE_FUNC(symbol); #ifdef CONFIG_KPROBE_EVENTS bool trace_kprobe_on_func_entry(struct trace_event_call *call); bool trace_kprobe_error_injectable(struct trace_event_call *call); +void trace_kprobe_error_injection_control(struct trace_event_call *call, + bool enabled); #else static inline bool trace_kprobe_on_func_entry(struct trace_event_call *call) { @@ -222,6 +224,9 @@ static inline bool trace_kprobe_error_injectable(struct trace_event_call *call) { return false; } + +static inline void trace_kprobe_error_injection_control(struct trace_event_call *call, + bool enabled) { } #endif /* CONFIG_KPROBE_EVENTS */ struct probe_arg { From patchwork Wed Jun 19 22:48:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704705 X-Patchwork-Delegate: mhiramat@kernel.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A9958175E; Wed, 19 Jun 2024 22:49:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837388; cv=none; b=Ko7Mc7FqThmJoiuA6gjmS1WvWL4sWVshCIiV9JCngTIcH7zE3pp1AyQYEZL3OmvpD36vk9bH1oxyW2mMD1Nlk3OZvJAxSQOhP3sgQS0TSeZmZxnLrUNlsxMCZUIrTWpW74vbBWR900h+r7mxAR5eza9JBkIJM8LdxZDwbhdynfk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837388; c=relaxed/simple; bh=3qciDcxlabL8RPKApoikpRFG6GtFoTLNySS+QbqQz9Q=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=nrB8K+jY71UZV5aY6s6MAPBJe7E35lJ8+iddmjwplUJRJtqN/1bub97u4tYgQ9+B4Sar/moOcWBY27kYlGBZ5IGE+vvJ+FwC+gdARAYTaWVsXGjp2Or7YdLVXX+KsoMhaQfUNxO0b+BgdmQ2EQYHKDaRTabrW93HellzTt30doc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 78F4B21A5F; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Authentication-Results: smtp-out1.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 593BA13ABD; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id QO+DFYdgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:48:58 +0200 Subject: [PATCH v2 4/7] bpf: support error injection static keys for multi_link attached progs Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-4-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4495; i=vbabka@suse.cz; h=from:subject:message-id; bh=3qciDcxlabL8RPKApoikpRFG6GtFoTLNySS+QbqQz9Q=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2B7A+FYtqvio9GTWv3xme47MHxydtFu6JcXT U9aKA8HdJyJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNgewAKCRC74LB10kWI mtXfB/9Wf8ufmNZ+KgSqZhasEn0ZqY1pTjQK/1k4ql5arsN4WjQN5LONf7zFbsRueo1MccTXI8j /fH+X3PTcoSG6fRMjz3+vuHCOkrsWH3yUtKIWVCjSGbCnoTDSPalLr/doZP8HAWLrmhpFni/wZC 6VZHYbuRigc7AohRXcOu7R4Px5zhFhdbb+6z1EZkeKJzmzriFXhSIczEoPYa6g6RmKDL/HsZiEY Iwcv1O2H9gYU+430HbPN6am111EVHOeM6rY2MCwZaJjWJjd3at7bfhl/tIQWnAM0gal51sJCbR6 2uksKs0VJHgNrhMdsLGxDLtpwTXG5HrhbVle/JopZQNha/Sy X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 78F4B21A5F X-Rspamd-Action: no action X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org Functions marked for error injection can have an associated static key that guards the callsite(s) to avoid overhead of calling an empty function when no error injection is in progress. Outside of the error injection framework itself, bpf programs can be atteched to kprobes and override results of error-injectable functions. To make sure these functions are actually called, attaching such bpf programs should control the static key accordingly. Therefore, add an array of static keys to struct bpf_kprobe_multi_link and fill it in addrs_check_error_injection_list() for programs with kprobe_override enabled, using get_injection_key() instead of within_error_injection_list(). Introduce bpf_kprobe_ei_keys_control() to control the static keys and call the control function when doing multi_link_attach and release. Signed-off-by: Vlastimil Babka --- kernel/trace/bpf_trace.c | 59 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 6 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 944de1c41209..ef0fadb76bfa 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2613,6 +2613,7 @@ struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; unsigned long *addrs; + struct static_key **ei_keys; u64 *cookies; u32 cnt; u32 mods_cnt; @@ -2687,11 +2688,30 @@ static void free_user_syms(struct user_syms *us) kvfree(us->buf); } +static void bpf_kprobe_ei_keys_control(struct bpf_kprobe_multi_link *link, bool enable) +{ + u32 i; + + for (i = 0; i < link->cnt; i++) { + if (!link->ei_keys[i]) + break; + + if (enable) + static_key_slow_inc(link->ei_keys[i]); + else + static_key_slow_dec(link->ei_keys[i]); + } +} + static void bpf_kprobe_multi_link_release(struct bpf_link *link) { struct bpf_kprobe_multi_link *kmulti_link; kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link); + + if (kmulti_link->ei_keys) + bpf_kprobe_ei_keys_control(kmulti_link, false); + unregister_fprobe(&kmulti_link->fp); kprobe_multi_put_modules(kmulti_link->mods, kmulti_link->mods_cnt); } @@ -2703,6 +2723,7 @@ static void bpf_kprobe_multi_link_dealloc(struct bpf_link *link) kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link); kvfree(kmulti_link->addrs); kvfree(kmulti_link->cookies); + kvfree(kmulti_link->ei_keys); kfree(kmulti_link->mods); kfree(kmulti_link); } @@ -2985,13 +3006,19 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 return arr.mods_cnt; } -static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt) +static int addrs_check_error_injection_list(unsigned long *addrs, struct static_key **ei_keys, + u32 cnt) { - u32 i; + struct static_key *ei_key; + u32 i, j = 0; for (i = 0; i < cnt; i++) { - if (!within_error_injection_list(addrs[i])) + ei_key = get_injection_key(addrs[i]); + if (IS_ERR(ei_key)) return -EINVAL; + + if (ei_key) + ei_keys[j++] = ei_key; } return 0; } @@ -3000,6 +3027,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr { struct bpf_kprobe_multi_link *link = NULL; struct bpf_link_primer link_primer; + struct static_key **ei_keys = NULL; void __user *ucookies; unsigned long *addrs; u32 flags, cnt, size; @@ -3075,9 +3103,24 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr goto error; } - if (prog->kprobe_override && addrs_check_error_injection_list(addrs, cnt)) { - err = -EINVAL; - goto error; + if (prog->kprobe_override) { + ei_keys = kvcalloc(cnt, sizeof(*ei_keys), GFP_KERNEL); + if (!ei_keys) { + err = -ENOMEM; + goto error; + } + + if (addrs_check_error_injection_list(addrs, ei_keys, cnt)) { + err = -EINVAL; + goto error; + } + + if (ei_keys[0]) { + link->ei_keys = ei_keys; + } else { + kvfree(ei_keys); + ei_keys = NULL; + } } link = kzalloc(sizeof(*link), GFP_KERNEL); @@ -3132,10 +3175,14 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr return err; } + if (link->ei_keys) + bpf_kprobe_ei_keys_control(link, true); + return bpf_link_settle(&link_primer); error: kfree(link); + kvfree(ei_keys); kvfree(addrs); kvfree(cookies); return err; From patchwork Wed Jun 19 22:48:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704703 X-Patchwork-Delegate: mhiramat@kernel.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EE3082866; Wed, 19 Jun 2024 22:49:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837387; cv=none; b=ooCWMl1uMPbhEg+xCNtqBoyCULLM5NBSg4ficUlCp9Hqc4c43dB8CY2zIT6N9ccf+C5DaJ378PcsTTZKHi0HwIeK6qTVTr3xsabzUXTUgrWDM2ohNb4I2B0acANC9fHKUpqTT1MH5r4IAF31LMfKPG2Fy+e3xg+eBgv0PGeUL9Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837387; c=relaxed/simple; bh=+HcqYdgaWT9BrUacvRJSWp4CDbAKfRkRibADsrgT2bU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Xr2du9jHi1mCaLJ7lThD6wAHe6VX+q0VapdsLXZ6f3R/40o4DAhvHZRMuDK/JIVjFtc/q+5ZA0wUj85tfabZP4VYzk9gRG2e4thzuvFbtPXMptyU6ALHs1ijZvQBCZ+A7YPQHvMaGasl4SN4jj9sMGCSWax1I3nkkYe9rab+X2E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=LNWxnnz1; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=roVh2sJJ; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=LNWxnnz1; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=roVh2sJJ; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="LNWxnnz1"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="roVh2sJJ"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="LNWxnnz1"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="roVh2sJJ" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 994E01F7F7; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X7+XAotVIvWS2vTh75oVQOPFwsOci6tWQZhDoSRQGx4=; b=LNWxnnz1OKuGoWGrMXrwTd2enyzJN73ypy1Cuy1qZtBotKAAqp2aq65A1gK7qZ8U54E2Lz FILu5L+orwbhDShXDMWDHPD659eTlwmmC78MWsxkCr/n6HetuhQ8GeFlOmDbT43Vpg8zeO i6yA9AvNuoYhNrFm1SRXWVMJ9lI5Fo0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X7+XAotVIvWS2vTh75oVQOPFwsOci6tWQZhDoSRQGx4=; b=roVh2sJJLOmM0oiL35sPyLjBSwOuoRsf/nGyJvEjjBe2kt/FPev0yfzWJK/1XF1/iK2GOP 56CsF5e9QNpjsaBg== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X7+XAotVIvWS2vTh75oVQOPFwsOci6tWQZhDoSRQGx4=; b=LNWxnnz1OKuGoWGrMXrwTd2enyzJN73ypy1Cuy1qZtBotKAAqp2aq65A1gK7qZ8U54E2Lz FILu5L+orwbhDShXDMWDHPD659eTlwmmC78MWsxkCr/n6HetuhQ8GeFlOmDbT43Vpg8zeO i6yA9AvNuoYhNrFm1SRXWVMJ9lI5Fo0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X7+XAotVIvWS2vTh75oVQOPFwsOci6tWQZhDoSRQGx4=; b=roVh2sJJLOmM0oiL35sPyLjBSwOuoRsf/nGyJvEjjBe2kt/FPev0yfzWJK/1XF1/iK2GOP 56CsF5e9QNpjsaBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 76B2913AC3; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id gBasHIdgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:48:59 +0200 Subject: [PATCH v2 5/7] bpf: do not create bpf_non_sleepable_error_inject list when unnecessary Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-5-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1591; i=vbabka@suse.cz; h=from:subject:message-id; bh=+HcqYdgaWT9BrUacvRJSWp4CDbAKfRkRibADsrgT2bU=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2B9IzDEAVKG5FXS38RcYZUyeL9j5X2CdCydU Xm/auj9IMKJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNgfQAKCRC74LB10kWI mieMB/0Z2oYYCWNAmVEuHX7NW7yYno6Akg2QEWBe10L/+QsV4Wfjt61ibXRZyOYv5dL16q+TyVM Gofehb9TgIuIG9fzbiv/IkDt0D9UULaagQwhJUpdnLAOYxw7FRr1nYbjmJt/cNOUtR8cMo6x1PG 33Gevml4l4phxiCJEnvCSO9Xum+ZBP/F3GOvVXDBDIF/vXVBhmF0iRyvefoi9o66f+guBziQc0j DDNd0wet/qYXslsYfyo9XtYY9+dP/5/d9pwYbcsvP8BN9uQkE3IeYQHe/Thldx8B/855/bItiKc AkAI9cbY8cY4/mKYag0vVzO6xNXmffLfuOg01lsVKyX58qWE X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Spamd-Result: default: False [-3.85 / 50.00]; REPLY(-4.00)[]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; BAYES_HAM(-0.05)[59.54%]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[20]; FREEMAIL_TO(0.00)[gmail.com,linux.com,google.com,kernel.org,iogearbox.net,linux.ibm.com,intel.com,davemloft.net,goodmis.org,arm.com]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[kernel.org,linux.dev,gmail.com,vger.kernel.org,kvack.org,suse.cz]; R_RATELIMIT(0.00)[to_ip_from(RL5nkphuxq5kxo98ppmuqoc8wo)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email] X-Spam-Flag: NO X-Spam-Score: -3.85 X-Spam-Level: When CONFIG_FUNCTION_ERROR_INJECTION is disabled, within_error_injection_list() will return false for any address and the result of check_non_sleepable_error_inject() denylist is thus redundant. The bpf_non_sleepable_error_inject list thus does not need to be constructed at all, so #ifdef it out. This will allow to inline functions on the list when CONFIG_FUNCTION_ERROR_INJECTION is disabled as there will be no BTF_ID() reference for them. Signed-off-by: Vlastimil Babka --- kernel/bpf/verifier.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 77da1f438bec..5cd93de37d68 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -21044,6 +21044,8 @@ static int check_attach_modify_return(unsigned long addr, const char *func_name) return -EINVAL; } +#ifdef CONFIG_FUNCTION_ERROR_INJECTION + /* list of non-sleepable functions that are otherwise on * ALLOW_ERROR_INJECTION list */ @@ -21061,6 +21063,19 @@ static int check_non_sleepable_error_inject(u32 btf_id) return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id); } +#else /* CONFIG_FUNCTION_ERROR_INJECTION */ + +/* + * Pretend the denylist is empty, within_error_injection_list() will return + * false anyway. + */ +static int check_non_sleepable_error_inject(u32 btf_id) +{ + return 0; +} + +#endif + int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, From patchwork Wed Jun 19 22:49:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704706 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6C2682869; Wed, 19 Jun 2024 22:49:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837388; cv=none; b=sQKXDHwXPenm44P4dDnFh2cEdv4KKeqCqpb/nJMU0Emry70Ca/JQE57wTHuzRTc9j67T5ClQwCZRgA+odVC4UN55R6L12O3DmQBXcqSgr4AFz7AlC0Ufhdymmy8SNY+v/AmYfc/oXP7RAEhBZyBg2I3sYf9oan92W6YVxna0Gm0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837388; c=relaxed/simple; bh=ldvIP2V3S2F517OMbSSDg7KW3uQS+0XOj9fkVOyN4sA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=l3cRLwL36IVZK3GrsZdYnydwi/CiAdfsmOKxtQs+t1DUsFYRcHaH8Q/jWPn9N8YiTXPbVW1bUomHGqRTf++j/pYJlGlcjs0XSFw6Br6SETcoBCNZrnzAUvLQUhMQeLYCDZ/XiHC6LTiMhCRBTpalSN326G8pWZmZDAo8gWuSvBM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=OctDuuKS; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=j+2cW9Bf; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=njbQ6Amv; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=W6ttRn13; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="OctDuuKS"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="j+2cW9Bf"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="njbQ6Amv"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="W6ttRn13" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B6C7121A61; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837385; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IKuT+N7+Lb5VqApn1kbaOItiOKwDSbQEIH+cYViP3f4=; b=OctDuuKSC4UF+wjKtoVyH0QvGLpF8XAqLyIbYfZfqUqDqbAwL2bcO3E1Pn+AnZ3eyibQfJ 6tYt02lob+xX6VBjZqZpNYNoAFLn6kK0OzL6sjX7j4tN4Djr5FmBiXSud2Nwi+ztUQq/Zn 90QlsQZrnKA+lNEaCE4uWXQHIrkgSYU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837385; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IKuT+N7+Lb5VqApn1kbaOItiOKwDSbQEIH+cYViP3f4=; b=j+2cW9Bfl+vq0dFzwR4Z8VJ/wS4ujClcEsbZnU5mUWOOkqMaN8pzA61v7a4cLcVPI6cou5 oo7W4FlMxtS0HUCw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IKuT+N7+Lb5VqApn1kbaOItiOKwDSbQEIH+cYViP3f4=; b=njbQ6AmvOqf/I3slNL642nz7dBav5v8cX1pr5wFiS4LsJeEcIXUKP4JeIGDMoYpMT1ZHst BoNAr/7ULUHsi0/LTIHKAQvyJs2XwZsVIxHGH1g8+c1W4/i2GGHXiON2JJcEy7S/UNuL8L xJNvP++tshk5rMe3z2U9GcJdCDP19Jw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IKuT+N7+Lb5VqApn1kbaOItiOKwDSbQEIH+cYViP3f4=; b=W6ttRn13nlD24fhIhJolQ8tSxHCj36d2kgYmU/rAvTHIozAnRoU5Z5qBqx+Xe4FxOmedZz 1dd53nZh0GPG6TCQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 972F913AAA; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eNSiJIdgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:49:00 +0200 Subject: [PATCH v2 6/7] mm, slab: add static key for should_failslab() Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-6-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5370; i=vbabka@suse.cz; h=from:subject:message-id; bh=ldvIP2V3S2F517OMbSSDg7KW3uQS+0XOj9fkVOyN4sA=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2CAyioCEBExL2f8WncWFmFMWNZmaTFD1Cae+ fsvxwAVOTSJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNggAAKCRC74LB10kWI moEGB/9t+BV2+IAq6SNJND2pIHnMkyYyiUiRMgkXxQEQLNZbKEmMr1y1PoT7g7IbtFutozCfC6P bWrARbt2jfDNJOj/yOgVfB0zY6h9vQlmoqe8jjatje4Lt3s25DwGW6BgQl8PwkCPUNdQLfBevS/ AYIXYRcHbUOYW/qvyFYFXpUaCDhSmv0MLe1twno9XP42wpIZW1tw1gBuz+4UY284502W7uAXa2g cySWbAiKUS9owip7X8JnR7HvnnWXngktOPjsSzDdpuHPNzMa+FjgMBXyHemCi0nrD5+CCZyLUBo WPwX6uGMzVlf+cD+P/EYDIixpxBcEuaOvo6PBJfQfQxFxkWP X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Spam-Score: -6.80 X-Spam-Level: X-Spam-Flag: NO X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[20]; FREEMAIL_TO(0.00)[gmail.com,linux.com,google.com,kernel.org,iogearbox.net,linux.ibm.com,intel.com,davemloft.net,goodmis.org,arm.com]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_CC(0.00)[kernel.org,linux.dev,gmail.com,vger.kernel.org,kvack.org,suse.cz]; R_RATELIMIT(0.00)[to_ip_from(RL5nkphuxq5kxo98ppmuqoc8wo)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_COUNT_TWO(0.00)[2]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email,linux.dev:email] Since commit 4f6923fbb352 ("mm: make should_failslab always available for fault injection") should_failslab() is unconditionally a noinline function. This adds visible overhead to the slab allocation hotpath, even if the function is empty. With CONFIG_FAILSLAB=y there's additional overhead, even when the functionality is not activated by a boot parameter or via debugfs. The overhead can be eliminated with a static key around the callsite. Fault injection and error injection frameworks including bpf can now be told that this function has a static key associated, and are able to enable and disable it accordingly. Additionally, compile out all relevant code if neither CONFIG_FAILSLAB nor CONFIG_FUNCTION_ERROR_INJECTION is enabled. When only the latter is not enabled, make should_failslab() static inline instead of noinline. To demonstrate the reduced overhead of calling an empty should_failslab() function, a kernel build with CONFIG_FUNCTION_ERROR_INJECTION enabled but CONFIG_FAILSLAB disabled, and CPU mitigations enabled, was used in a qemu-kvm (virtme-ng) on AMD Ryzen 7 2700 machine, and execution of a program trying to open() a non-existent file was measured 3 times: for (int i = 0; i < 10000000; i++) { open("non_existent", O_RDONLY); } After this patch, the measured real time was 4.3% smaller. Using perf profiling it was verified that should_failslab was gone from the profile. With CONFIG_FAILSLAB also enabled, the patched kernel performace was unaffected, as expected, while unpatched kernel's performance was worse, resulting in the relative speedup being 10.5%. This means it no longer needs to be an option suitable only for debug kernel builds. Acked-by: Alexei Starovoitov Reviewed-by: Roman Gushchin Signed-off-by: Vlastimil Babka --- include/linux/fault-inject.h | 4 +++- mm/failslab.c | 2 +- mm/slab.h | 3 +++ mm/slub.c | 30 +++++++++++++++++++++++++++--- 4 files changed, 34 insertions(+), 5 deletions(-) diff --git a/include/linux/fault-inject.h b/include/linux/fault-inject.h index cfe75cc1bac4..0d0fa94dc1c8 100644 --- a/include/linux/fault-inject.h +++ b/include/linux/fault-inject.h @@ -107,9 +107,11 @@ static inline bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) } #endif /* CONFIG_FAIL_PAGE_ALLOC */ +#ifdef CONFIG_FUNCTION_ERROR_INJECTION int should_failslab(struct kmem_cache *s, gfp_t gfpflags); +#endif #ifdef CONFIG_FAILSLAB -extern bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags); +bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags); #else static inline bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags) { diff --git a/mm/failslab.c b/mm/failslab.c index ffc420c0e767..878fd08e5dac 100644 --- a/mm/failslab.c +++ b/mm/failslab.c @@ -9,7 +9,7 @@ static struct { bool ignore_gfp_reclaim; bool cache_filter; } failslab = { - .attr = FAULT_ATTR_INITIALIZER, + .attr = FAULT_ATTR_INITIALIZER_KEY(&should_failslab_active.key), .ignore_gfp_reclaim = true, .cache_filter = false, }; diff --git a/mm/slab.h b/mm/slab.h index 5f8f47c5bee0..792e19cb37b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Internal slab definitions @@ -160,6 +161,8 @@ static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(freelist_aba_t) */ #define slab_page(s) folio_page(slab_folio(s), 0) +DECLARE_STATIC_KEY_FALSE(should_failslab_active); + /* * If network-based swap is enabled, sl*b must keep track of whether pages * were allocated from pfmemalloc reserves. diff --git a/mm/slub.c b/mm/slub.c index 0809760cf789..11980aa94631 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3874,13 +3874,37 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, 0, sizeof(void *)); } -noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) +#if defined(CONFIG_FUNCTION_ERROR_INJECTION) || defined(CONFIG_FAILSLAB) +DEFINE_STATIC_KEY_FALSE(should_failslab_active); + +#ifdef CONFIG_FUNCTION_ERROR_INJECTION +noinline +#else +static inline +#endif +int should_failslab(struct kmem_cache *s, gfp_t gfpflags) { if (__should_failslab(s, gfpflags)) return -ENOMEM; return 0; } -ALLOW_ERROR_INJECTION(should_failslab, ERRNO); +ALLOW_ERROR_INJECTION_KEY(should_failslab, ERRNO, &should_failslab_active); + +static __always_inline int should_failslab_wrapped(struct kmem_cache *s, + gfp_t gfp) +{ + if (static_branch_unlikely(&should_failslab_active)) + return should_failslab(s, gfp); + else + return 0; +} +#else +static __always_inline int should_failslab_wrapped(struct kmem_cache *s, + gfp_t gfp) +{ + return false; +} +#endif static __fastpath_inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) @@ -3889,7 +3913,7 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) might_alloc(flags); - if (unlikely(should_failslab(s, flags))) + if (should_failslab_wrapped(s, flags)) return NULL; return s; From patchwork Wed Jun 19 22:49:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704707 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1BD88286D; Wed, 19 Jun 2024 22:49:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837388; cv=none; b=j4It1QX0ql8bzH3VdP2tqtTD1dNPOt8xp7OklR8fK4jtUftSysLkGMLIeMQl2AH9CYxjUYkDWFUeBTK54mbq3oc0PTONpeqXbu0QPdu6aQ9RQ9DCZ8dLWW6nhuNhCgvI99VFwbzfu8WnbUw5OTMo6Z6R/7PjFQlGqlOtUHx4MRg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718837388; c=relaxed/simple; bh=lrkvm9m+5V6rPH6PUfaXjT75wU3B7FL1fWu0EmMg4jI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=UB0ehRKwEkG7Q0IofgjgtfhbwqBIOG0hm+TWygaA5xRSBLtoUFrhkt1WdAwOps/1cffDjgnQHkxfE0eZsZHI4oT1HO/b9SPB1EUG7lihKrnTd0K297QU8VUlskx8F/hhqJd5b7oVpglc0RDiMQln0bu/qe88/4oP0S1aj/LpUks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=oBxQ6Ia7; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=hkjarJfU; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=TynXPAks; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=L985X4si; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="oBxQ6Ia7"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="hkjarJfU"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="TynXPAks"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="L985X4si" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id D50FF21A6D; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837385; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=En1fgwGBp3akxqcTlyA7z2SXZocSGyKqoIqkTPWBt38=; b=oBxQ6Ia700L8P7MHC2ZSj7lYOClW/b+nn74PtnoV77STyNfZuJ3s/3FxFvQ5ON/6/R+4CE h1v2V61d1EUmbIDPC9LkVI94UA1fx7FZyzcOsTllmIb3LpdGX+e/CqM1Qq4PPQSvsp62af TuKzhlqNZ2MZIuhtI7+mCA6VvjGjO2o= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837385; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=En1fgwGBp3akxqcTlyA7z2SXZocSGyKqoIqkTPWBt38=; b=hkjarJfUCCuhuk1r1+VTczH1jn0SrePROWGO6/JuDQUTIuFa3Kwopbx0FvR7OLsgc+7hff PXUElSpJDIwp0hBA== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=En1fgwGBp3akxqcTlyA7z2SXZocSGyKqoIqkTPWBt38=; b=TynXPAkshMLovoWJEyMGcwthd+0VNECCmYJf+upAlVaI4TBVa+V6WzmLxCDjoVKizHIo8z beBAnPxk8Tok4UFc8dN+SfSIuFPzGJKktIyQ5FT2rmT7UliqCKiboQSnua+P+muJ5d4aOd fgXUBfIUGrD6WUh4tQSVTqKwK0xvg9Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1718837383; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=En1fgwGBp3akxqcTlyA7z2SXZocSGyKqoIqkTPWBt38=; b=L985X4siXKopQT7WhbmtjCdC4M0g8YD/5UflTdx8IHiUaLXCG9QyIeO73rbief4APg1G1f rLISpErk9/ETi2Cg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B4FCC13ABD; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id +GboK4dgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:49:01 +0200 Subject: [PATCH v2 7/7] mm, page_alloc: add static key for should_fail_alloc_page() Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-7-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4477; i=vbabka@suse.cz; h=from:subject:message-id; bh=lrkvm9m+5V6rPH6PUfaXjT75wU3B7FL1fWu0EmMg4jI=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2CDq4C5vfd3OGsJTJXwWPnIR++a/CyqxA3pF yu5aRhqYhCJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNggwAKCRC74LB10kWI mtY9B/9F48jU4mg4DvmDG0e36QW+ooxwNCrSVM1MXSjv46DlcqK9YtApZrvOymhCnd03BmprQnE cTZiAw/ZIIuNLlW8D09hcNpB/oGulWuhpMehEB0iTP3UfItTOMOU1N30LODDF/abWn0v2qzp6LD ZtThwT/w0IM8z8/K4/oMDfoZinMXGXnFPV+b4fBbs/27R3btbUs9N3VZFkDLvC7eXmK3vAl0Bzw updjoZ7nNW0dqQnrOjDOl1T/DzDGQYdzGRoD3k7b3RnW8x25qFRY/ZejhS3tKEntPDl8Cg3C89i t0Jo4FimVrnSavpJWzGth87QCgsmpfVPo2Almyo5yxTbQ8Ee X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Spamd-Result: default: False [-6.80 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWELVE(0.00)[20]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com,linux.com,google.com,kernel.org,iogearbox.net,linux.ibm.com,intel.com,davemloft.net,goodmis.org,arm.com]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_CC(0.00)[kernel.org,linux.dev,gmail.com,vger.kernel.org,kvack.org,suse.cz]; R_RATELIMIT(0.00)[to_ip_from(RL5nkphuxq5kxo98ppmuqoc8wo)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; DBL_BLOCKED_OPENRESOLVER(0.00)[linux.dev:email,suse.cz:email] X-Spam-Flag: NO X-Spam-Score: -6.80 X-Spam-Level: Similarly to should_failslab(), remove the overhead of calling the noinline function should_fail_alloc_page() with a static key that guards the callsite in the page allocator hotpath, and is controlled by the fault and error injection frameworks and bpf. Additionally, compile out all relevant code if neither CONFIG_FAIL_ALLOC_PAGE nor CONFIG_FUNCTION_ERROR_INJECTION is enabled. When only the latter is not enabled, make should_fail_alloc_page() static inline instead of noinline. No measurement was done other than verifying the should_fail_alloc_page is gone from the perf profile. A measurement with the analogical change for should_failslab() suggests that for a page allocator intensive workload there might be noticeable improvement. It also makes CONFIG_FAIL_ALLOC_PAGE an option suitable not only for debug kernels. Reviewed-by: Roman Gushchin Signed-off-by: Vlastimil Babka --- include/linux/fault-inject.h | 3 ++- mm/fail_page_alloc.c | 3 ++- mm/internal.h | 2 ++ mm/page_alloc.c | 30 +++++++++++++++++++++++++++--- 4 files changed, 33 insertions(+), 5 deletions(-) diff --git a/include/linux/fault-inject.h b/include/linux/fault-inject.h index 0d0fa94dc1c8..1a782042ae80 100644 --- a/include/linux/fault-inject.h +++ b/include/linux/fault-inject.h @@ -96,8 +96,9 @@ static inline void fault_config_init(struct fault_config *config, struct kmem_cache; +#ifdef CONFIG_FUNCTION_ERROR_INJECTION bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order); - +#endif #ifdef CONFIG_FAIL_PAGE_ALLOC bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order); #else diff --git a/mm/fail_page_alloc.c b/mm/fail_page_alloc.c index b1b09cce9394..0906b76d78e8 100644 --- a/mm/fail_page_alloc.c +++ b/mm/fail_page_alloc.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include "internal.h" static struct { struct fault_attr attr; @@ -9,7 +10,7 @@ static struct { bool ignore_gfp_reclaim; u32 min_order; } fail_page_alloc = { - .attr = FAULT_ATTR_INITIALIZER, + .attr = FAULT_ATTR_INITIALIZER_KEY(&should_fail_alloc_page_active.key), .ignore_gfp_reclaim = true, .ignore_gfp_highmem = true, .min_order = 1, diff --git a/mm/internal.h b/mm/internal.h index b2c75b12014e..8539e39b02e6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -410,6 +410,8 @@ extern char * const zone_names[MAX_NR_ZONES]; /* perform sanity checks on struct pages being allocated or freed */ DECLARE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); +DECLARE_STATIC_KEY_FALSE(should_fail_alloc_page_active); + extern int min_free_kbytes; void setup_per_zone_wmarks(void); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2e22ce5675ca..b6e246acb4aa 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3008,11 +3008,35 @@ struct page *rmqueue(struct zone *preferred_zone, return page; } -noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) +#if defined(CONFIG_FUNCTION_ERROR_INJECTION) || defined(CONFIG_FAIL_PAGE_ALLOC) +DEFINE_STATIC_KEY_FALSE(should_fail_alloc_page_active); + +#ifdef CONFIG_FUNCTION_ERROR_INJECTION +noinline +#else +static inline +#endif +bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) { return __should_fail_alloc_page(gfp_mask, order); } -ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE); +ALLOW_ERROR_INJECTION_KEY(should_fail_alloc_page, TRUE, &should_fail_alloc_page_active); + +static __always_inline bool +should_fail_alloc_page_wrapped(gfp_t gfp_mask, unsigned int order) +{ + if (static_branch_unlikely(&should_fail_alloc_page_active)) + return should_fail_alloc_page(gfp_mask, order); + + return false; +} +#else +static __always_inline bool +should_fail_alloc_page_wrapped(gfp_t gfp_mask, unsigned int order) +{ + return false; +} +#endif static inline long __zone_watermark_unusable_free(struct zone *z, unsigned int order, unsigned int alloc_flags) @@ -4430,7 +4454,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, might_alloc(gfp_mask); - if (should_fail_alloc_page(gfp_mask, order)) + if (should_fail_alloc_page_wrapped(gfp_mask, order)) return false; *alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, *alloc_flags);