From patchwork Fri May 31 09:33:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49B02C27C4F for ; Fri, 31 May 2024 09:33:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFF556B009F; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C55A26B009B; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93FDD6B0098; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 64E1F6B0098 for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DBA231C06CC for ; Fri, 31 May 2024 09:33:43 +0000 (UTC) X-FDA: 82178178726.12.87D96A4 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf13.hostedemail.com (Postfix) with ESMTP id B44D920015 for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2C4S76QU; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=j0ilQ40F; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=b77OsGwl; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=y5h1qKUz; spf=pass (imf13.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=BpUFTa1dewe3YkhU5VOZFY9dIPMuL7ZfkALMLa25ypg=; b=igkjElRMvCJnKGYycqRmoK3vxBi9RW82UWK6tgs8rtUFNf+H4MB7YsNfd3iARlmrNgDxQy lsGgCOt3LD2ShQ8Kl0tFWSoUKJVFt3UVdQaH5I8Y58CYurMaETT3CL2K0UjWmgvEtU3UPm pe5c7d7i8XhzrBs9IDzXbVVx4d5O3Lg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2C4S76QU; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=j0ilQ40F; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=b77OsGwl; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=y5h1qKUz; spf=pass (imf13.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=L1hfxmM5ls3c0+14C1nLBjNA09gRYLYeQ6SLt0XO088ioVD02T5F1dGI6ZLl8z3nF74chj 101z1BlN1zkDFqlhZQ3BCLqh6fy2yH6s2UEoheJ1PsH5094GfNwq1UCicLIbozEiGGDYb7 q+GGEIWm9cXjLJv8U+7NPfyqY9VHyNU= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E94B41F81C; Fri, 31 May 2024 09:33:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BpUFTa1dewe3YkhU5VOZFY9dIPMuL7ZfkALMLa25ypg=; b=2C4S76QUkLnMyJhMS3n1JXtMaEMxmhlKGrNi5oOY9Iei0ux4xJEpSwUDIBu4OWB4EVi8Fc KT4wFnFgNNGHhRkVI+mo8lSTtBBXrQQ5ya8oU1/lgkFSu6bE5lVs0svSF0VItvQnw8OhJ6 FQR1uLE7AdOyJiAJH/KBUabWBYoZrW4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148020; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BpUFTa1dewe3YkhU5VOZFY9dIPMuL7ZfkALMLa25ypg=; b=j0ilQ40FCdAHLD0Ckw+QFwmO1U9OInYDiCMybIXTpUjXlKuawOD4PdXzA9978BHaNCS+6O Tmok1FhxgWXD66Cg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1717148019; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BpUFTa1dewe3YkhU5VOZFY9dIPMuL7ZfkALMLa25ypg=; b=b77OsGwlvfpGYJeQdsEuU0i1rdrN3nA8p45wXqKUe9MhscqB0SpPIAUeNmEFQeELMLviGt UgWZ3c/l0CYrdOSI0u1L8F3W6khOwDcb+WVj29z2oNgjFhPK7DmklGqKU7m/PPHCQx1ro5 mgJoeHolbmO+hbwTrBgidiFssRwaZuU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1717148019; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BpUFTa1dewe3YkhU5VOZFY9dIPMuL7ZfkALMLa25ypg=; b=y5h1qKUzsPEuaYo9RplJOsl+dgFCZHqjV+0toux2jIT8eqsdle6VNfYgPwwtuI/Xcf98Wz q5/UAZ0tq0QLP5AA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id C3088132C2; Fri, 31 May 2024 09:33:39 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id FZAJL3OZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:39 +0000 From: Vlastimil Babka Subject: [PATCH RFC 0/4] static key support for error injection functions Date: Fri, 31 May 2024 11:33:31 +0200 Message-Id: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> MIME-Version: 1.0 X-B4-Tracking: v=1; b=H4sIAGuZWWYC/6tWKk4tykwtVrJSqFYqSi3LLM7MzwNyDHUUlJIzE vPSU3UzU4B8JSMDIxMDU2MD3bTE0pwS3cy8rNTkEqBa3eKSxJLM5OzUymJdM7MkcyMjo1RLwyR zJaABBUWpaZkVYMOjlYLcnJVia2sB0kvgPXEAAAA= To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Stat-Signature: guoa3u7hcr1asniny7nahk3migeh64qc X-Rspamd-Queue-Id: B44D920015 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1717148021-585813 X-HE-Meta: U2FsdGVkX19xJogFYXMAF8zZ+Li10lN4xk49ahU0J0V01FwxjK7XER8WbwCcSR/DBUdZZzm/rUFfjk13oaU818jYw9uxdXVLlZSKXb/C8GS2g28spGhBWqxhKPDisiU8UPcFkOj3F8KsxFGVbGB++i8k3iWO4l9prREmwJTQDt+sA9BtOvyfUHh2ZKLMlj9/WxaKppO0KINWI1rExDRcTBxbQ8B29VV/2IZBPeiTvty4fjfNK2vPiz4hlaa8wobTDRZY6BWXs14XKqsMLMQMxAf4mlYhTAeq4FbQ/L8Fh5tfoErSHkQt9TzpQxae/dRvdnFNhNi3W/oDc5gCI1JLF8dCoeIdGJ3ZXqYLq/38yExqgx4E89AvLoi0ZmYObhV6ZpOK0TNscdsnLbCOPEimd9tSvgDbn710Vs530AEH0J857OPjnRd0d/bKlnPhbD77WwdFnMYlA6P1VO6pEa4aeTt4X53aJ/okxqxChluNgauX7oJuaED50z9oCNjvkbWUQQ3qf3VNEafY4ceuGQ1ynhJvoNotRkY1xKnkyX5PHVyy398bEc1c4RdcglzKn6dWHCUzlvclk22bBiksQSO9f6HPIMsNl9bpXiUW65kd7bVRqhAdglg/mtblDfDP2uoUVW1j1oWgtjSV410j2hha1YwHC/OgIjwPbKk+YbO3E/u3Xyn3rK4ve1Dx6JBvR7qOvu5yaGI6tJF0WWDpIWo3Xy0nqIccCk4U28gA2oHwA42kNKjLPz/6H/Tul4hhRJ0OhtxYK0Vl/qlEVaGo+RgLN1OnzGZ0Tw3eF1BxOBUWjf5yIn4QsxGPKxDrif+6zmoFLpnCTY4gt7oXyrnUNeDH2URE++5bT11hbiqIwf1373rHJrocf2yBMXlx7ASEfOjnOK2XG9VmFtyiYLOk3030epX3UUR2ICEftX5Ej3yeEREDhEZ0Bh0b7q92gM/UH24HOLo1cvy87s1yRvqY4eW /1efkw9j yKiuBbEifpJTWQLAE3CzU9Rnv0+j8ILJA2ZOAAW/mYQjXL1JhEiB/8V48idtqQ5aSoYM6fNY+Q1pFPgM2B8Lz0xtLci/aXwjl5aByMOqOY4EBvqx9E7ENeEEibSoo9Qukybc62PBvZnx78d2NTGdXuyCVh6Wb+qOEM+DaKA1VEXr6Mv9vxW+E4cTxukv6NgLriBnLbwR5lDtwzfwtJeatUj7KeeJJJvq1o9l1LsiGsVDbHjn2+i5C+drOHq47WfuPxg+94SSSbJDnmHqAR/yCjCAe8fAh25HrqET2EeqMk2QCPoh6GyTUfqmIHjppnnc0pd5pNSh/9+X9ByNVpYqQ2bgqvB+Adax7aaMzDEIgz/85nTRNUZBFeAeMTwCfZr6OGUgQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Incomplete, help needed from ftrace/kprobe and bpf folks. As previously mentioned by myself [1] and others [2] the functions designed for error injection can bring visible overhead in fastpaths such as slab or page allocation, because even if nothing hooks into them at a given moment, they are noninline function calls regardless of CONFIG_ options since commits 4f6923fbb352 ("mm: make should_failslab always available for fault injection") and af3b854492f3 ("mm/page_alloc.c: allow error injection"). Live patching their callsites has been also suggested in both [1] and [2] threads, and this is an attempt to do that with static keys that guard the call sites. When disabled, the error injection functions still exist and are noinline, but are not being called. Any of the existing mechanisms that can inject errors should make sure to enable the respective static key. I have added that support to some of them but need help with the others. - the legacy fault injection, i.e. CONFIG_FAILSLAB and CONFIG_FAIL_PAGE_ALLOC is handled in Patch 1, and can be passed the address of the static key if it exists. The key will be activated if the fault injection probability becomes non-zero, and deactivated in the opposite transition. This also removes the overhead of the evaluation (on top of the noninline function call) when these mechanisms are configured in the kernel but unused at the moment. - the generic error injection using kretprobes with override_function_with_return is handled in patch 2. The ALLOW_ERROR_INJECTION() annotation is extended so that static key address can be passed, and the framework controls it when error injection is enabled or disabled in debugfs for the function. There are two more users I know of but am not familiar enough to fix up myself. I hope people that are more familiar can help me here. - ftrace seems to be using override_function_with_return from #define ftrace_override_function_with_return but I found no place where the latter is used. I assume it might be hidden behind more macro magic? But the point is if ftrace can be instructed to act like an error injection, it would also have to use some form of metadata (from patch 2 presumably?) to get to the static key and control it. If ftrace can only observe the function being called, maybe it wouldn't be wrong to just observe nothing if the static key isn't enabled because nobody is doing the fault injection? - bpftrace, as can be seen from the example in commit 4f6923fbb352 description. I suppose bpf is already aware what functions the currently loaded bpf programs hook into, so that it could look up the static key and control it. Maybe using again the metadata from patch 2, or extending its own, as I've noticed there's e.g. BTF_ID(func, should_failslab) Now I realize maybe handling this at the k(ret)probe level would be sufficient for all cases except the legacy fault injection from Patch 1? Also wanted to note that by AFAIU by using the static_key_slow_dec/inc API (as done in patches 1/2) should allow all mechanisms to coexist naturally without fighting each other on the static key state, and also handle the reference count for e.g. active probes or bpf programs if there's no similar internal mechanism. Patches 3 and 4 implement the static keys for the two mm fault injection sites in slab and page allocators. For a quick demonstration I've run a VM and the simple test from [1] that stresses the slab allocator and got this time before the series: real 0m8.349s user 0m0.694s sys 0m7.648s with perf showing 0.61% nonexistent [kernel.kallsyms] [k] should_failslab.constprop.0 0.00% nonexistent [kernel.kallsyms] [k] should_fail_alloc_page ▒ And after the series real 0m7.924s user 0m0.727s sys 0m7.191s and the functions gone from perf report. There might be other such fault injection callsites in hotpaths of other subsystems but I didn't search for them at this point. [1] https://lore.kernel.org/all/6d5bb852-8703-4abf-a52b-90816bccbd7f@suse.cz/ [2] https://lore.kernel.org/all/3j5d3p22ssv7xoaghzraa7crcfih3h2qqjlhmjppbp6f42pg2t@kg7qoicog5ye/ Signed-off-by: Vlastimil Babka --- Vlastimil Babka (4): fault-inject: add support for static keys around fault injection sites error-injection: support static keys around injectable functions mm, slab: add static key for should_failslab() mm, page_alloc: add static key for should_fail_alloc_page() include/asm-generic/error-injection.h | 13 ++++++++++- include/asm-generic/vmlinux.lds.h | 2 +- include/linux/error-injection.h | 9 +++++--- include/linux/fault-inject.h | 7 +++++- kernel/fail_function.c | 22 +++++++++++++++--- lib/error-inject.c | 6 ++++- lib/fault-inject.c | 43 ++++++++++++++++++++++++++++++++++- mm/fail_page_alloc.c | 3 ++- mm/failslab.c | 2 +- mm/internal.h | 2 ++ mm/page_alloc.c | 11 ++++++--- mm/slab.h | 3 +++ mm/slub.c | 10 +++++--- 13 files changed, 114 insertions(+), 19 deletions(-) --- base-commit: 1613e604df0cd359cf2a7fbd9be7a0bcfacfabd0 change-id: 20240530-fault-injection-statickeys-66b7222e91b7 Best regards,