From patchwork Mon Aug 5 12:39:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 13753577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F019DC3DA7F for ; Mon, 5 Aug 2024 12:43:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54E126B0089; Mon, 5 Aug 2024 08:43:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FD536B0092; Mon, 5 Aug 2024 08:43:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39DB06B0093; Mon, 5 Aug 2024 08:43:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 192296B0089 for ; Mon, 5 Aug 2024 08:43:33 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C04E9140261 for ; Mon, 5 Aug 2024 12:43:32 +0000 (UTC) X-FDA: 82418157864.30.E5DA887 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 0F2C118000F for ; Mon, 5 Aug 2024 12:43:30 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=K3ymFT2J; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 38ciwZgUKCIIkr1kxmuumrk.iusrot03-ssq1giq.uxm@flex--elver.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38ciwZgUKCIIkr1kxmuumrk.iusrot03-ssq1giq.uxm@flex--elver.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722861742; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=LrC8PMqmgoYq1tMdewJcFduXfqGeN4f8xWTu+1cWz7M=; b=QfvWaYB8MQj2+qM1s7bpfsHZ3UlDNi4eZyrJbzhVe6J9M1BPJ3iNCs0L70lnUgWSv3cKJg s5a9lKU+9SkmaAHG6qwZ/mRJI7PvWTFrVrvhCfxDtM8N1FeNfeoYGwq9fF3L4S/Q8+n5ux KLLymuBPR1+u7WYzWx4qBn9xK9NdW9w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722861742; a=rsa-sha256; cv=none; b=LZ4WsLpsUFA0t7BWLqaQwwfARAgi1qucg4qT9sxcYKhM4j1jZ+diq13ifoNL1vnCRXRCiz 4Gz/iL0KR7QMxSaaCJ0xhMl1uJ5mjAbLV/aHwfuai92dXUaP5M9K07ZrsBbCkDrW2BS+T2 UwyfCvYuFxX616ozDcOEiuZBx8E+KTE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=K3ymFT2J; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 38ciwZgUKCIIkr1kxmuumrk.iusrot03-ssq1giq.uxm@flex--elver.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38ciwZgUKCIIkr1kxmuumrk.iusrot03-ssq1giq.uxm@flex--elver.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e087ed145caso15576423276.3 for ; Mon, 05 Aug 2024 05:43:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722861810; x=1723466610; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=LrC8PMqmgoYq1tMdewJcFduXfqGeN4f8xWTu+1cWz7M=; b=K3ymFT2JtvBoYOAtWNZlqENxk5HYWNGcKC09I4+i3DPr69lSeiPMWsB9zo7zf/8gCA g2nctDSXFJnGC6SDDo24wXmDCehVW8vkhtKzfL8l35wrMN1rYdqun4Q2aWREPxuqLzjH qNzFu+2cbvdd8w09qk6siYQDyllu3DwivbSSYqI8wAbUTTDhbJ1r1ngstU4Pxo4X2IcT DF8uVpA1HNAeLOpnULk57M/9kldEty+Swhe9K08QvR9BQU1PZls5TuVFZ7BADBsDQLMs bzsrfEC8shD7gNtsBEV9j2gg6hWG3TOdXbJkVYF6PkhlsVyKmKv0EU9AbaWjDMV8kYhp NWYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722861810; x=1723466610; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=LrC8PMqmgoYq1tMdewJcFduXfqGeN4f8xWTu+1cWz7M=; b=RqeIBz+tlAqGDrG0z7zkeTh5pFVg7ahUfYN7rYVhQB6/MqT4AW10pfo4y2SwFbn5Cw 7YsccFMwxP9iGJxchQUFCHmD1sHuD7HJGXqzdcxLoQ8NnIJaRqZU3lzVPhqCZYDSZW/I P8BoRsBHVMEIJHcwHV4RLsvwfR3XB84JH/S+vSyPALnRChHLwrUsE/oQ0j15t0ABlvq9 AoJESvzEUpSZnWEL15XO6m7RrbUAA64KdeUehqnUFC6Fn+1dM9v6sxAul2GTpub0OqJx LyNjc7XywOjlMHkMXOclGhAWxMZvczekyFTqe0RI9kVzSvetqTNQTikDafIcLINKBiUj J6cQ== X-Forwarded-Encrypted: i=1; AJvYcCWKClP6uEAyIn7+HdMzPJO9/kIk6jdHOQPhzZxmfDYIGVhWABF1CPvj2rx3Jxv1CXodoZFBG48Hgl2jBAhupy9PW1s= X-Gm-Message-State: AOJu0Yybxkpwa9MDVy6uei0Y4Lg6V78ioftr5u0XLd2+K40SLF26VdUy cOkVt82VyHxULsAslDUuBWDe5vSg1QasyamZeLTNb8L5bPrTtjwM2fZbJz+gcCnc3LGRsJj5cw= = X-Google-Smtp-Source: AGHT+IGqkRac4PWjShJ8ZuJcg3lK53X4rsTvmZrworTa4PKf1hQ7xcsnsuhYAGll2uBuhCm9g3UE71/Pzw== X-Received: from elver.muc.corp.google.com ([2a00:79e0:9c:201:575f:bb7d:b516:3c23]) (user=elver job=sendgmr) by 2002:a05:6902:100c:b0:dfa:8ed1:8f1b with SMTP id 3f1490d57ef6-e0bde22affamr334185276.1.1722861809962; Mon, 05 Aug 2024 05:43:29 -0700 (PDT) Date: Mon, 5 Aug 2024 14:39:39 +0200 Mime-Version: 1.0 X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240805124203.2692278-1-elver@google.com> Subject: [PATCH] kfence: introduce burst mode From: Marco Elver To: elver@google.com, Andrew Morton Cc: Andrey Konovalov , Alexander Potapenko , Dmitry Vyukov , Jann Horn , kasan-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0F2C118000F X-Stat-Signature: azwzekcacj644m8a9gu7m1zs96ah5g4u X-Rspam-User: X-HE-Tag: 1722861810-603902 X-HE-Meta: U2FsdGVkX19tbyutzmsmqyzXc0yTfMIgn+QKPLQtHTwP9l1fOdaMLDllVn902cqlF+WBiFQyi2XswzFW4jvbsMYLgDUrXd024hjzU/bLWvLum2S3Fjpc3zr4msx2TO5zY1RStzxuXlO5v/g0v8aC/549U/HrSvelrKA2TG1CtWtRRhgXIej/2qbOLhfCRdDx3vl/f4dcDL5gzl77Vq8eK+m510ilSxPdKglPdjw4yvN/6L8MzUnJjHxbTiPHcg56/mnYg41bMPKgF1PDXLVYfDKEVmXeaZMuARBBwE09IHiA7plpHG/xxRTqT6IXNcjaie6brpReZn2Vxd/OF6F/dnt5pfjd6GkCRwjo7Li0Tt3v82TjLGrkAz0MWlnog2KqBQ8SL/N0n+4XFZigR7NeQaeAgKgIO2jcbPdfXUXNn675liQFARb8m/x+11NomEe77PHgc46v8RqLTiTAbZioKDHvf0dgBZjCUJObsk4gRVOgR5kv6lPcFjiGtmYjiiiU+XT6niTdhlppXCYySXJQN7zlSGn1IbifqAtwwvLf07jwyX0tldXVLrPrPedT0ZIkjb3hVo20DpXc82WK3BCc8F3d25cm1D+N/xRtcfhLl+Leao9T70FKBWYNtqC3Aw1Bx99z496oB77pzABT30GiB/b/mKveRs57eLD3EStSFTwGFadvSWSSmvQrMgXyQLPSk+3d2ITPXoWOGM+x85g1Dtgwf44PTjlaxMmHeb69DUlQtVkYDChkVnBPZfHBPEmwYTQRU5srf3bt0hmArm0vIz4tunIPsSdqGwHQ7BqKju30DftoJakTWC0QLmp8HTFM5FFpzUQXlpGKP+lcGO6oi73LqHBE0n4KlWIL+L8SHflb7deukfT8d+aS4g4ak+BSPpPyztILS0Vx/B2Ld/1jC87NEGnw8miY8E/wJuIaLaf52onqUL3aOVJNlh3LtMV8w512BBpm8KUKpnGTLPE 56JDoQKe gVdeiVi8lxmtoTbu2VuiD1gWm0vSgLYkWS32YJOWg0YxWrXpiYNgQII2K0FQZkhxPdHFIkiKtBYJN9f/GNBvB+B4x8zHVG8NmsQFQnZPV39PSUYib+vmha2SXzUsHqIaB7+s7Nl6kidAEb2iz7Y25UrwT9VYVLXxmrso6GRo8I4wNQYKwu7eOE5X7Nq5x623Jm91jBszO5UD1aF6Q0jVRBOUxRhwl6x9JDwBHQp82C2SxWlffxcDnJTJakO8wlcbyGOWWR4tiqKbzxFOnnUuq8d58O86CesvpwSETeCBEWDYrg0XFJ5ClX6aNn6hQ4DPLthBdeOTkIRIRLVwTWdXD8Gs+RDYXBwYT9LES3r5k0btgtrdWNUaDO2u1mA2wadbsTLYf1oZ1DV5B5T4xcxXZgVQnNDAXR2PC1VbLcivzDTxSvOdza0sbpOzJ/ccCeIYtl95NurBpirAE4bi5frueWjtRPTNqZ2u898IO48Doy3CFJN4TwfmZNnrHxjfgCvk/Oddx+Q3eilwTpua8JCjPIl0ANUoXoouedpmUs/LiKVjM+WoHU+DjsrHhdQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce burst mode, which can be configured with kfence.burst=$count, where the burst count denotes the additional successive slab allocations to be allocated through KFENCE for each sample interval. The idea is that this can give developers an additional knob to make KFENCE more aggressive when debugging specific issues of systems where either rebooting or recompiling the kernel with KASAN is not possible. Experiment: To assess the effectiveness of the new option, we randomly picked a recent out-of-bounds [1] and use-after-free bug [2], each with a reproducer provided by syzbot, that initially detected these bugs with KASAN. We then tried to reproduce the bugs with KFENCE below. [1] Fixed by: 7c55b78818cf ("jfs: xattr: fix buffer overflow for invalid xattr") https://syzkaller.appspot.com/bug?id=9d1b59d4718239da6f6069d3891863c25f9f24a2 [2] Fixed by: f8ad00f3fb2a ("l2tp: fix possible UAF when cleaning up tunnels") https://syzkaller.appspot.com/bug?id=4f34adc84f4a3b080187c390eeef60611fd450e1 The following KFENCE configs were compared. A pool size of 1023 objects was used for all configurations. Baseline kfence.sample_interval=100 kfence.skip_covered_thresh=75 kfence.burst=0 Aggressive kfence.sample_interval=1 kfence.skip_covered_thresh=10 kfence.burst=0 AggressiveBurst kfence.sample_interval=1 kfence.skip_covered_thresh=10 kfence.burst=1000 Each reproducer was run 10 times (after a fresh reboot), with the following detection counts for each KFENCE config: | Detection Count out of 10 | | OOB [1] | UAF [2] | ------------------+-------------+-------------+ Default | 0/10 | 0/10 | Aggressive | 0/10 | 0/10 | AggressiveBurst | 8/10 | 8/10 | With the Default and even the Aggressive configs the results are unsurprising, given KFENCE has not been designed for deterministic bug detection of small test cases. However, when enabling burst mode with relatively large burst count, KFENCE can start to detect heap memory-safety bugs even in simpler test cases with high probability (in the above cases with ~80% probability). Signed-off-by: Marco Elver --- Documentation/dev-tools/kfence.rst | 7 +++++++ include/linux/kfence.h | 2 +- mm/kfence/core.c | 14 ++++++++++---- 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/Documentation/dev-tools/kfence.rst b/Documentation/dev-tools/kfence.rst index 936f6aaa75c8..541899353865 100644 --- a/Documentation/dev-tools/kfence.rst +++ b/Documentation/dev-tools/kfence.rst @@ -53,6 +53,13 @@ configurable via the Kconfig option ``CONFIG_KFENCE_DEFERRABLE``. The KUnit test suite is very likely to fail when using a deferrable timer since it currently causes very unpredictable sample intervals. +By default KFENCE will only sample 1 heap allocation within each sample +interval. *Burst mode* allows to sample successive heap allocations, where the +kernel boot parameter ``kfence.burst`` can be set to a non-zero value which +denotes the *additional* successive allocations within a sample interval; +setting ``kfence.burst=N`` means that ``1 + N`` successive allocations are +attempted through KFENCE for each sample interval. + The KFENCE memory pool is of fixed size, and if the pool is exhausted, no further KFENCE allocations occur. With ``CONFIG_KFENCE_NUM_OBJECTS`` (default 255), the number of available guarded objects can be controlled. Each object diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 88100cc9caba..0ad1ddbb8b99 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -124,7 +124,7 @@ static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp if (!static_branch_likely(&kfence_allocation_key)) return NULL; #endif - if (likely(atomic_read(&kfence_allocation_gate))) + if (likely(atomic_read(&kfence_allocation_gate) > 0)) return NULL; return __kfence_alloc(s, size, flags); } diff --git a/mm/kfence/core.c b/mm/kfence/core.c index c5cb54fc696d..c3ef7eb8d4dc 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -99,6 +99,10 @@ module_param_cb(sample_interval, &sample_interval_param_ops, &kfence_sample_inte static unsigned long kfence_skip_covered_thresh __read_mostly = 75; module_param_named(skip_covered_thresh, kfence_skip_covered_thresh, ulong, 0644); +/* Allocation burst count: number of excess KFENCE allocations per sample. */ +static unsigned int kfence_burst __read_mostly; +module_param_named(burst, kfence_burst, uint, 0644); + /* If true, use a deferrable timer. */ static bool kfence_deferrable __read_mostly = IS_ENABLED(CONFIG_KFENCE_DEFERRABLE); module_param_named(deferrable, kfence_deferrable, bool, 0444); @@ -827,12 +831,12 @@ static void toggle_allocation_gate(struct work_struct *work) if (!READ_ONCE(kfence_enabled)) return; - atomic_set(&kfence_allocation_gate, 0); + atomic_set(&kfence_allocation_gate, -kfence_burst); #ifdef CONFIG_KFENCE_STATIC_KEYS /* Enable static key, and await allocation to happen. */ static_branch_enable(&kfence_allocation_key); - wait_event_idle(allocation_wait, atomic_read(&kfence_allocation_gate)); + wait_event_idle(allocation_wait, atomic_read(&kfence_allocation_gate) > 0); /* Disable static key and reset timer. */ static_branch_disable(&kfence_allocation_key); @@ -1052,6 +1056,7 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) unsigned long stack_entries[KFENCE_STACK_DEPTH]; size_t num_stack_entries; u32 alloc_stack_hash; + int allocation_gate; /* * Perform size check before switching kfence_allocation_gate, so that @@ -1080,14 +1085,15 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) if (s->flags & SLAB_SKIP_KFENCE) return NULL; - if (atomic_inc_return(&kfence_allocation_gate) > 1) + allocation_gate = atomic_inc_return(&kfence_allocation_gate); + if (allocation_gate > 1) return NULL; #ifdef CONFIG_KFENCE_STATIC_KEYS /* * waitqueue_active() is fully ordered after the update of * kfence_allocation_gate per atomic_inc_return(). */ - if (waitqueue_active(&allocation_wait)) { + if (allocation_gate == 1 && waitqueue_active(&allocation_wait)) { /* * Calling wake_up() here may deadlock when allocations happen * from within timer code. Use an irq_work to defer it.