From patchwork Wed Apr 21 10:51:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 12215969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57C8AC433ED for ; Wed, 21 Apr 2021 10:52:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D25E961447 for ; Wed, 21 Apr 2021 10:52:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D25E961447 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 57E756B0070; Wed, 21 Apr 2021 06:52:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 555346B0071; Wed, 21 Apr 2021 06:52:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CEFD6B0072; Wed, 21 Apr 2021 06:52:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 245206B0070 for ; Wed, 21 Apr 2021 06:52:04 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D713C181AEF2A for ; Wed, 21 Apr 2021 10:52:03 +0000 (UTC) X-FDA: 78056059326.27.91D9760 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 8CC3980192D8 for ; Wed, 21 Apr 2021 10:52:02 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id k5-20020a2524050000b02904e716d0d7b1so15203528ybk.0 for ; Wed, 21 Apr 2021 03:52:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OxgBhA98k3tGthKxuSN/4brjg9G3Dacp3flVNK8rvYo=; b=kZCrucRDppiGDBPjAXIIXIo0MxE4/IyqxJJhXwKZnRZitxeGk0n/gU50Zw9DaWrpD+ dIpvk0jbhDyEE//vafgmDxLrgr2YyYm66YH+SiaJKfmQozMx5XcaeO5hf4eWzECEKjp0 DelqluZ8nf+mHKY0XVzdVwGk8ty9BSH78j0/6K/wFDufY5AfFSbPeh8A4xraPkTh886n 9Z4kUhLBAsqyKAVGsv02WcAeZV3sEBGaDgqRuEJcNLLQHQPf3JIiEw3lCSUgcKZvdDXG xeUD6XPpxzPQXIngH07/btqdDuRqSxVDtooRU6+D+Q1pVBCH9B7GNwaUSnOrJcrjCtc4 Zjng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=OxgBhA98k3tGthKxuSN/4brjg9G3Dacp3flVNK8rvYo=; b=oN4p76FYRKWRlScPK/hi29fwTr+7gp2IYaS19YU8lehophHFBweT+FF3dS2GTr7Vgl PNoQFmK5yoaV0r/GeCCVQrUHq+NBzUSsGsJxGXKNqv1DXbYqQY2/1zrO6SrIf57gc/tn HJoBnlgHF3ZDK+ezRIPYHAw2sngmj00N0n36sM7LJm0gJRXhueCYCzTQ19TQHveMTeKV VYvVOoDIEYVqnd8LY0fcSEYjbiROlJXd2dKEoHwxlcOhVGFXoe8MnWNRE2w0Ui+mddF7 mBI9i/x/p6QVLd6kBwWvapAAa5skgNXpnYWbKQ4DcgqVvA4ndB+337qECcHmQxniQ+AC FOpw== X-Gm-Message-State: AOAM531N1u0cBOpeC4CXypj8zbwoeVTdx74zbTOpYJkhQCft1+04+beY CVuRc3hBQ2/MMBLxY7tR0nZB4fr9gA== X-Google-Smtp-Source: ABdhPJxnxD3CB/9+5BCPqRdLBDuWnvcCjjRpgFvxXOW0RsfmOBJPVhGilgvJZ/ty/H3/94vwogLNxmc1JQ== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:c552:ee7c:6a14:80cc]) (user=elver job=sendgmr) by 2002:a25:8b86:: with SMTP id j6mr29548944ybl.470.1619002322474; Wed, 21 Apr 2021 03:52:02 -0700 (PDT) Date: Wed, 21 Apr 2021 12:51:30 +0200 In-Reply-To: <20210421105132.3965998-1-elver@google.com> Message-Id: <20210421105132.3965998-2-elver@google.com> Mime-Version: 1.0 References: <20210421105132.3965998-1-elver@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v2 1/3] kfence: await for allocation using wait_event From: Marco Elver To: elver@google.com, akpm@linux-foundation.org Cc: glider@google.com, dvyukov@google.com, jannh@google.com, mark.rutland@arm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com, hdanton@sina.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8CC3980192D8 X-Stat-Signature: 418ag1uiixh77ijy8n4kbf7witb5f6qj Received-SPF: none (flex--elver.bounces.google.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from="<30gOAYAUKCOMJQaJWLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--elver.bounces.google.com>"; helo=mail-yb1-f202.google.com; client-ip=209.85.219.202 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619002322-690153 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On mostly-idle systems, we have observed that toggle_allocation_gate() is a cause of frequent wake-ups, preventing an otherwise idle CPU to go into a lower power state. A late change in KFENCE's development, due to a potential deadlock [1], required changing the scheduling-friendly wait_event_timeout() and wake_up() to an open-coded wait-loop using schedule_timeout(). [1] https://lkml.kernel.org/r/000000000000c0645805b7f982e4@google.com To avoid unnecessary wake-ups, switch to using wait_event_timeout(). Unfortunately, we still cannot use a version with direct wake_up() in __kfence_alloc() due to the same potential for deadlock as in [1]. Instead, add a level of indirection via an irq_work that is scheduled if we determine that the kfence_timer requires a wake_up(). Fixes: 0ce20dd84089 ("mm: add Kernel Electric-Fence infrastructure") Signed-off-by: Marco Elver --- v2: * Replace kfence_timer_waiting with simpler waitqueue_active() check. --- lib/Kconfig.kfence | 1 + mm/kfence/core.c | 45 +++++++++++++++++++++++++++++---------------- 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/lib/Kconfig.kfence b/lib/Kconfig.kfence index 78f50ccb3b45..e641add33947 100644 --- a/lib/Kconfig.kfence +++ b/lib/Kconfig.kfence @@ -7,6 +7,7 @@ menuconfig KFENCE bool "KFENCE: low-overhead sampling-based memory safety error detector" depends on HAVE_ARCH_KFENCE && (SLAB || SLUB) select STACKTRACE + select IRQ_WORK help KFENCE is a low-overhead sampling-based detector of heap out-of-bounds access, use-after-free, and invalid-free errors. KFENCE is designed diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 768dbd58170d..235d726f88bc 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -587,6 +588,17 @@ late_initcall(kfence_debugfs_init); /* === Allocation Gate Timer ================================================ */ +#ifdef CONFIG_KFENCE_STATIC_KEYS +/* Wait queue to wake up allocation-gate timer task. */ +static DECLARE_WAIT_QUEUE_HEAD(allocation_wait); + +static void wake_up_kfence_timer(struct irq_work *work) +{ + wake_up(&allocation_wait); +} +static DEFINE_IRQ_WORK(wake_up_kfence_timer_work, wake_up_kfence_timer); +#endif + /* * Set up delayed work, which will enable and disable the static key. We need to * use a work queue (rather than a simple timer), since enabling and disabling a @@ -604,25 +616,13 @@ static void toggle_allocation_gate(struct work_struct *work) if (!READ_ONCE(kfence_enabled)) return; - /* Enable static key, and await allocation to happen. */ atomic_set(&kfence_allocation_gate, 0); #ifdef CONFIG_KFENCE_STATIC_KEYS + /* Enable static key, and await allocation to happen. */ static_branch_enable(&kfence_allocation_key); - /* - * Await an allocation. Timeout after 1 second, in case the kernel stops - * doing allocations, to avoid stalling this worker task for too long. - */ - { - unsigned long end_wait = jiffies + HZ; - - do { - set_current_state(TASK_UNINTERRUPTIBLE); - if (atomic_read(&kfence_allocation_gate) != 0) - break; - schedule_timeout(1); - } while (time_before(jiffies, end_wait)); - __set_current_state(TASK_RUNNING); - } + + wait_event_timeout(allocation_wait, atomic_read(&kfence_allocation_gate), HZ); + /* Disable static key and reset timer. */ static_branch_disable(&kfence_allocation_key); #endif @@ -729,6 +729,19 @@ void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) */ if (atomic_read(&kfence_allocation_gate) || atomic_inc_return(&kfence_allocation_gate) > 1) return NULL; +#ifdef CONFIG_KFENCE_STATIC_KEYS + /* + * waitqueue_active() is fully ordered after the update of + * kfence_allocation_gate per atomic_inc_return(). + */ + if (waitqueue_active(&allocation_wait)) { + /* + * Calling wake_up() here may deadlock when allocations happen + * from within timer code. Use an irq_work to defer it. + */ + irq_work_queue(&wake_up_kfence_timer_work); + } +#endif if (!READ_ONCE(kfence_enabled)) return NULL;