From patchwork Tue Aug 29 17:11:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78A9DC83F14 for ; Tue, 29 Aug 2023 17:11:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E15368E0029; Tue, 29 Aug 2023 13:11:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC4218E002F; Tue, 29 Aug 2023 13:11:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C95608E0029; Tue, 29 Aug 2023 13:11:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B986A8E0029 for ; Tue, 29 Aug 2023 13:11:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 599D28024E for ; Tue, 29 Aug 2023 17:11:39 +0000 (UTC) X-FDA: 81177783918.11.E84B519 Received: from out-244.mta1.migadu.com (out-244.mta1.migadu.com [95.215.58.244]) by imf23.hostedemail.com (Postfix) with ESMTP id 7D8D714001B for ; Tue, 29 Aug 2023 17:11:37 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=DeG8gVsP; spf=pass (imf23.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.244 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329097; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Gxqstvn+9KrGAurRzEA5+xZd4FpeRy2hmiRTg5iKOfw=; b=MpK2AihyULpZ0WSv5Ae+vSYjGGWHFv4T8M8Kl2s7O1QH+10WGf2ItxHCuNx8glcn9B+4s9 00IQ+am/qL8mxkPrJOYyvlFnMPzYg16cACiq9WN2ctH2SEHzWnu1q7mOK8q3V9Ew08WUoL qIhIt936E35izVLJOSZnCyDGXJrVqaw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=DeG8gVsP; spf=pass (imf23.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.244 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329097; a=rsa-sha256; cv=none; b=VrCoTZqIkIqszzPBmf0QnetPoi9l2XkwsO6gSS4mOFLX4r7Bva0BrZoerO5zRTte2WrWm+ Lp7fdiyCBc9KD8Pkw8mcawQ9ZYUadUCGcgle6166WMXzAqYC3DAgWM5MadjFfV93oHDMju R5IgnChNPnxMJzjTlj7AyS9EaYOO+3I= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329095; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gxqstvn+9KrGAurRzEA5+xZd4FpeRy2hmiRTg5iKOfw=; b=DeG8gVsPVA6eHp6GYAtPbcN1W4Kc4hW54VJN2cPa1oAVV0Vtv+XvIoxvZ1he5+lPDyKPzU jVtmhpgvmpCpTClSH2nhy/NQ4GtPpio2EnuYBWWAcI0xhaei4Zp0zBBpeO5Q7FjGGTEUej XwtRIWiCy9Atd2ddYEKCng61HMvXEZ8= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 01/15] stackdepot: check disabled flag when fetching Date: Tue, 29 Aug 2023 19:11:11 +0200 Message-Id: <43b26d397d4f0d76246f95a74a8a38cfd7297bbc.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 7D8D714001B X-Rspam-User: X-Stat-Signature: 7cqepu4w4bj83qe3kcie93y9smubopqc X-Rspamd-Server: rspam01 X-HE-Tag: 1693329097-414465 X-HE-Meta: U2FsdGVkX191QRbY/g8v0nf+yQ0hqBSEyCFfb8zmx0Pvvzo784lH++XTBXTvwUNFAdZyhKf7EurS0Eprt3LE2RFlD2cFgGpfdh/mkBGiXkmjHcabCwJicdNnSeJ4PMbbvOdV7wKto747NoHv+8CkIvZgAyutiP6qVMmQSmoEYS64N6OsLa56Q2jqolLedrMHygg3NMSEONXS6rM+Uq4DDKwElP3IZu00nKqUfsTO+BJeBCp6iKStPNCqTkL13BBORFisMd4uL2Zt0v/aALqkTz8JUm8GKWUbPIZSvh9hb/CXWgxDOpv1gpc1kOjNJbJD1fDE6u89ilIHa7+K7U5G5f/R/YZMU2IhigkxpkmigxiiPzUaqQVrOQdN8d4vPhw7+qwqJmhbh/pYJLe2xSXQOamIMXhwazAkvhRPNuRYz0D5JbP8cSv6m4V/1O1rHMJDLtUL39dLwMlsIkHJ8EVwdxCP9BVkrMe9wKV7g2Na64T5h8LDjG2LQsMbF3Y1n3NOMD4/gE6PYsTFFWVQTfz4HP/YdLIfCutAC6XWbfPkvYmoHJNn3t0AXNGYUE++cNfauqzjNvQ4qWBvE9ErQUMMuU1npisJfX1VaXPDqepLfg+s2AfEpt6TfQE8IPhXYIhnn3j7Y/i4csAwRUfcBedZQubgCL/6kJs3baO6a6s7kvg0/OEn7LX/T2eX13dpuCv7poyLT23VArUoUa20a/cCq2s9JhNaSzzSQoM1gVs9Rl+UjZ/Lhm7nt+BYD1qlw6pzI8rVRUViYS/yJmwSWtthnfWeRbocge7oq3vISekyqTXUzH8D3KlwhHvGPOM5/zHxbXBOtuc9IzPY5hRFvAXj8aE6zLhvuETd/D53oenkUNeLQyn/0ndL/nlQ8kM9GFR+/pV3Ta2o+IJc4ZR4Vohd2UN+pJKg5aNVno8uMCA2SBFe4EfliZKS8I3Jtkb0ouphSBRCPtOCGXt6pfeUXXO 3bwk7VUg pkR0jlf0oYTeRo0UBW258yOwlxXcR7sUR7ffLxTxJdNEMHtaZSKu/kfPYNiMDPe7dqe9xsM0c1M3FsiPQ3Tuvqjzd0V2/vC+nwtpZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Do not try fetching a stack trace from the stack depot if the stack_depot_disabled flag is enabled. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 2f5aa851834e..3a945c7206f3 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -477,7 +477,7 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, */ kmsan_unpoison_memory(entries, sizeof(*entries)); - if (!handle) + if (!handle || stack_depot_disabled) return 0; if (parts.pool_index > pool_index_cached) { From patchwork Tue Aug 29 17:11:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A969CC83F12 for ; Tue, 29 Aug 2023 17:11:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2067E8E002E; Tue, 29 Aug 2023 13:11:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B6E48E002F; Tue, 29 Aug 2023 13:11:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E37918E002E; Tue, 29 Aug 2023 13:11:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C4A4C8E002E for ; Tue, 29 Aug 2023 13:11:39 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9585F801E1 for ; Tue, 29 Aug 2023 17:11:39 +0000 (UTC) X-FDA: 81177783918.06.489F274 Received: from out-246.mta1.migadu.com (out-246.mta1.migadu.com [95.215.58.246]) by imf21.hostedemail.com (Postfix) with ESMTP id DA8031C0027 for ; Tue, 29 Aug 2023 17:11:37 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=FMOUlvJx; spf=pass (imf21.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.246 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329098; a=rsa-sha256; cv=none; b=DX8+n9o+Q7jlYAYQvX6G08w5p5qW3TJ1Fhd9UDMt0XKzOjNfU95xkPb0dAH38LJc6g3XsX gUX+NwqFAF3sRu+mp9yohexvW0IcUbRVg3CZTeTrS82AgqWyVN/Os9LYVEG+PfODLCz6FW RJM2FI1SLh731GaJyOW/R0+xjj52QTo= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=FMOUlvJx; spf=pass (imf21.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.246 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Opt9t2Sh3BPQLxjav3raWw4j3L9ASrpPJpFprr4WIwg=; b=JnbOx9J9wyObtCblZLKuuXpr3dF937ve0syZSFq57kkdmLv9KZC5br+TDSrBms3CdmNOYp PkcTZok8PRk35GKX9r55mq7JxpF2j8VjjajUVcKPgmTuR6I/YRVU5olLtCL/H6PZ+VGEUO weiUke9hVheVWROQ7wQL64DMWxyz1V4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329096; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Opt9t2Sh3BPQLxjav3raWw4j3L9ASrpPJpFprr4WIwg=; b=FMOUlvJxgMB5WpKRQ7t3KX2aGQnwsNKzQ2ccI31MOspuXDov1WZjg8f3zBxS9aiip16KLg QsPLydt7gf0VxZWrz0MHedBxpNNjUHO1AeoE3D3p5RwVNU7SOLfTAhOveTf+35NDZNtujl ResgGbrFAGlj6khMSxwabRc53OkvSPc= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 02/15] stackdepot: simplify __stack_depot_save Date: Tue, 29 Aug 2023 19:11:12 +0200 Message-Id: <20dbc3376fccf2e7824482f56a75d6670bccd8ff.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DA8031C0027 X-Stat-Signature: x4f969xzz1puoyme5us4zsaaozrwxuw6 X-Rspam-User: X-HE-Tag: 1693329097-494968 X-HE-Meta: U2FsdGVkX1+npRKnIHwx1v9QC+zrDcveDgBrnlQ3E9IjbWRpQpdefwLqNczVFyvNGDoHEMvBo1FuSg3nUM4DbpEQPTauoEmcChQN33VzXQHrkfbHk1yG9D/9xV96SVv1yA88bvaDEyM5IakccIjIZi6WH9GU7POg8pXvndthoycegb+qHWc8abK5ZwUVBHbf1/JqBZCs0ILO5fXhc06YPiySR33tAW1CDN97wqAGRmeRYV70+gDHygrwNzPruIfsxJTYO+YD1cM8bMMMxQs/TbF7qaI++H8aWKo6dszAa9oeYVS1Y9lL7QwOnoMY0BqLfrMxyzws8Q2KIO/JtQsEmBJ1ImsCwEf9N9Y6bBBBDFAy3PE10g+wyyQM5hMloBReakWmql3EAyGjqT1RPmaPESxSiFLDeAOtawF9KV7RZ5dKDUBeRk01+/DHtjIlg3ELh0ayw1j3En6xBTfn9VOVc/fo+AQsdThgxGIz6r1TR75rcs9V//Li4Bdh0PRQuSZ++qLdkDJqFUbpjyIAc4ltS2rZDbiJwMLKXvkUj6TB/eNwRVC80c1ZN/RToa00cBsMy4v1n0ESYc+4ucSFZUKzWDgFNoy7Y26p1M74NHRvcXv2NcPH7GhVOi1dW55WWkGOFn8eCZzp/lwCkLAflqj2zVWCGwUiCLCl9golyvD8BZ9MGvx0wO5tZkJ8EQdZfjXPwXuAFYFiHL6i2ecoVlmtmHJ/wpb4RGDQ3Uf0alIv+PR2iSdmsJh3U7VwLHKJvnKZvOBBCTHu/n1gh10tWswbjh63QJHyzFf6vv5lUqXFj+hhoQAh8p9AFfsoVrpjTVxQiSCxnPe5DI8Vv+fIZh7Ojr6TYicK+JOF0hHQtWxqqz/RXo3qBEWvrN9Ip3Xvpangh3D792h02t+cQ8Zp5WDnV2vRHr7PQghBvw7OpF8fV9Nb5u1R658wSgcWE894/FNhR6kHv/Y7sAOH0CmA6X3 oMb0EtQd Yd1B40IA9WhUDcjujKYQrRS7rm0D4rO5pYOOEWdwxRrj7ZiIrzfbPNU5oBGJY3+d7xa4PJMw4zAUvGxlSIUFBJA50f8Nt3omhNav5EyMxR+EV4/O2SHKHL5RsFvn1nxZXeBKJBUB20gfPj04yvqZ6r1g1Cg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov The retval local variable in __stack_depot_save has the union type handle_parts, but the function never uses anything but the union's handle field. Define retval simply as depot_stack_handle_t to simplify the code. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 3a945c7206f3..0772125efe8a 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -360,7 +360,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, gfp_t alloc_flags, bool can_alloc) { struct stack_record *found = NULL, **bucket; - union handle_parts retval = { .handle = 0 }; + depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; unsigned long flags; @@ -377,7 +377,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, nr_entries = filter_irq_stacks(entries, nr_entries); if (unlikely(nr_entries == 0) || stack_depot_disabled) - goto fast_exit; + return 0; hash = hash_stack(entries, nr_entries); bucket = &stack_table[hash & stack_hash_mask]; @@ -443,9 +443,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); } if (found) - retval.handle = found->handle.handle; -fast_exit: - return retval.handle; + handle = found->handle.handle; + return handle; } EXPORT_SYMBOL_GPL(__stack_depot_save); From patchwork Tue Aug 29 17:11:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42BBCC83F14 for ; Tue, 29 Aug 2023 17:11:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C01D8E0030; Tue, 29 Aug 2023 13:11:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 86DF58E002F; Tue, 29 Aug 2023 13:11:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7350F8E0030; Tue, 29 Aug 2023 13:11:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5ED8B8E002F for ; Tue, 29 Aug 2023 13:11:40 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 392BA4024F for ; Tue, 29 Aug 2023 17:11:40 +0000 (UTC) X-FDA: 81177783960.03.64DEB31 Received: from out-244.mta1.migadu.com (out-244.mta1.migadu.com [95.215.58.244]) by imf29.hostedemail.com (Postfix) with ESMTP id 8801412002A for ; Tue, 29 Aug 2023 17:11:38 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=FdmqrQoR; spf=pass (imf29.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.244 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J9x0A1WNFQDshCpK/emmSalBuss8B7zIgs4IPlFr17A=; b=kG6H9MJwzu8rzLrY2PGe8x3r0nVTDoFq8rIk+2cGQNIXid4Qr5KeC/fQu+T6TfuiyOAbkq toIhcDZUYE5IWBllZzesN5n7xCV4PcETxjUP8ZscsBaYo6YLX18nHXsKd0+Z8Qx2kWNvJe nr3fGOrOUNhpLRmFAWvTqCaPlUhaJ7A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329098; a=rsa-sha256; cv=none; b=O7wFfzBPggljMwGFJpbSEyVQgrFqEHtoSoCz0P+/aFdxNarEtIuz6Y4rolYJrbXq/9M2js zaUm6dcGfSYcyj20dw6d5ycgLMEABaE7vowbA39oGqK1wsTkj+1UPKrNu6ajUk15uYOUY1 kcaH6t8evUkYFp2X+8UvnaZl/6CQEto= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=FdmqrQoR; spf=pass (imf29.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.244 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329097; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J9x0A1WNFQDshCpK/emmSalBuss8B7zIgs4IPlFr17A=; b=FdmqrQoRG911hkZBrs9+qOuzijl9EyrM9zkCfxSRiSKdthhj9DynENOHTygZicJGL6ttQd b8bicv/bN9d7rHnZGH7XvWbnvE0ExxkcPNoHVf4CMtYnM6Xf5U4ohouTeDLn5GYyzrjaNE Ef5USp/vQmHcZo4oq0IjadJbI/g7Dw4= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 03/15] stackdepot: drop valid bit from handles Date: Tue, 29 Aug 2023 19:11:13 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8801412002A X-Rspam-User: X-Stat-Signature: wxaaafe7rt8pa8m9pwynbhebn7npsjed X-Rspamd-Server: rspam03 X-HE-Tag: 1693329098-173220 X-HE-Meta: U2FsdGVkX18ap0cVTt6OaSEpjmebK5n5K7ZvM/y7WmuXFNG5Z4ezTZttwETlHTgbPePxDnOwpGrRzkZ5PbdB95Y4HTikmfeUgpU7nUTBToJjpy5wd2TcmDZv5e2Izy95HEXZF/kHBLzgR1+n2wbdThsi6L+DiFYa9PBt/MP6Qiysg1wVccgYcVzg2vqlW2mui4ZX7WcHnrOii/JtMCj7KYK92dC8H+sFxA+na2e48pz+4nNs73h4eAkkGJo1eJLm5VDe15wAZPYb5DIbIge/RqG6sFikgc/vWrpbLJNQo6LpPfPSQLkxVp+3wSAznP9UmiglzV54yrsfRWjITIwlB8sbv7vOA8/Hu5CmjQbpRMucVDTnsMBUvDbkzIMu6VXY+FKQS18M3QWdxo8kcHkipG4U2ex1omC28iEgHbdU1dzrHCvzk4Gsd8PXwMaE4w9ukWYKjz8BRQbm48dnyPJh3aHel2DFmPJc5PKiwAuPtDNrLz9yP1TrJ3WnhSmRhyaH9aL5QQUiA3IPKK0UMghL9ws9oq2rfGN9fNHw3WW6UYMd9l4ZnlyWo55h0BnGoyIN0ToKyJzYUJJZ+aA2XI6j6zktozMlxJkgRxE+1DGviGHxsmJuwjRJ0VVkhifkeZOPeQYdMXlgRAmIAY4OAD7nvmUyzIKqBwKIUNGnW1UWrCHPc9EnN1Bv6+uPuVendwYQ4k/s0+b4fb4NzqR+/lp6aH8urkh5b875Gz6cxmBobVDL7IisA1u8qWWKBW7eKYTK3tItPxg6Wt4nFT6umv2Pldkgv+p2bkYC5el+jJ4TB5UMwBWhPZ66+Fu9bOnxWxELX01MIYtn/rMaQsbrc9tzjYL01rUqvJgMMJabpAVyw4Wu1rNuZvAD7Odk732f5YV8Y3Szij+3DNuan3MHwej7R/cZxO8LN1e+uN3UALIbZ5RGE/aFa52yx4f5sGQiECsOesrq3EET0gGlKBMWHlq QiTud8Nu WiBW86dFLf4itiVr7IAx0fmY2kkicK36wsQeL7bcWpO8hs+aQyjZ53LItRVYvccag2Qdx8d99dAapPR3uXVEywdLFeQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Stack depot doesn't use the valid bit in handles in any way, so drop it. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 0772125efe8a..482eac40791e 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -32,13 +32,12 @@ #define DEPOT_HANDLE_BITS (sizeof(depot_stack_handle_t) * 8) -#define DEPOT_VALID_BITS 1 #define DEPOT_POOL_ORDER 2 /* Pool size order, 4 pages */ #define DEPOT_POOL_SIZE (1LL << (PAGE_SHIFT + DEPOT_POOL_ORDER)) #define DEPOT_STACK_ALIGN 4 #define DEPOT_OFFSET_BITS (DEPOT_POOL_ORDER + PAGE_SHIFT - DEPOT_STACK_ALIGN) -#define DEPOT_POOL_INDEX_BITS (DEPOT_HANDLE_BITS - DEPOT_VALID_BITS - \ - DEPOT_OFFSET_BITS - STACK_DEPOT_EXTRA_BITS) +#define DEPOT_POOL_INDEX_BITS (DEPOT_HANDLE_BITS - DEPOT_OFFSET_BITS - \ + STACK_DEPOT_EXTRA_BITS) #define DEPOT_POOLS_CAP 8192 #define DEPOT_MAX_POOLS \ (((1LL << (DEPOT_POOL_INDEX_BITS)) < DEPOT_POOLS_CAP) ? \ @@ -50,7 +49,6 @@ union handle_parts { struct { u32 pool_index : DEPOT_POOL_INDEX_BITS; u32 offset : DEPOT_OFFSET_BITS; - u32 valid : DEPOT_VALID_BITS; u32 extra : STACK_DEPOT_EXTRA_BITS; }; }; @@ -303,7 +301,6 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) stack->size = size; stack->handle.pool_index = pool_index; stack->handle.offset = pool_offset >> DEPOT_STACK_ALIGN; - stack->handle.valid = 1; stack->handle.extra = 0; memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); pool_offset += required_size; From patchwork Tue Aug 29 17:11:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A548FC6FA8F for ; Tue, 29 Aug 2023 17:11:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 676408E0032; Tue, 29 Aug 2023 13:11:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6241B8E002F; Tue, 29 Aug 2023 13:11:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C5B48E0032; Tue, 29 Aug 2023 13:11:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 35F338E002F for ; Tue, 29 Aug 2023 13:11:42 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 06A5B80247 for ; Tue, 29 Aug 2023 17:11:42 +0000 (UTC) X-FDA: 81177784044.08.6CBD8E4 Received: from out-249.mta1.migadu.com (out-249.mta1.migadu.com [95.215.58.249]) by imf25.hostedemail.com (Postfix) with ESMTP id 5DD0CA001E for ; Tue, 29 Aug 2023 17:11:39 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=LIEnc9qf; spf=pass (imf25.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.249 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329099; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iO6rOKx0E5Y80aTOb2gdl835Yi8CPfP2ICeqYsm38a8=; b=YFsDq7onN0nbXyWn8GLsY7e/Xdpml/n2dZ2maT6dkeVthHo49VGi4WS7TGP2hKVZBn38ro 1M8IP2VIfdNPgc5u5qzJ4f1nimM6ykTdbrD4V4sKd98Q5eQPcDqtIf3wjRgkVG2Hwywz9y uSl1pNbn+aXC52isTVAmeJgRgKzpauE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329099; a=rsa-sha256; cv=none; b=k0GA9WnzNM9QXdmn9zc5xsYrvRZA7n7urgtSQG+YDuzesbTSg9D/Rhlo2lvJDUU0m2V5ON h/ONbwUUi5tw3o5ZH03fSXhMzEWvesRgIWN82RnjZsqaIivM8isn3Dyyr29RDj2RA68rES 4srNl3KbQAr7JSK/iNeUkYyfNpSKykk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=LIEnc9qf; spf=pass (imf25.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.249 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329097; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iO6rOKx0E5Y80aTOb2gdl835Yi8CPfP2ICeqYsm38a8=; b=LIEnc9qf0bdzjRuVfvd2JdO2h3KjGuPt1jpYgEgLcKhd+uMTNri0079VJZ7BuNqzfKhWpH nChiTXEn/GCsQoyhhHm4/OnyjGw6tvxpMZ/P/+tL4MFSJeV+Aw2nZXmyYGKoQ8vfzv/ONv 7MlOomIeIzvZ5a8LNRxAPzvNoJy8ViI= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 04/15] stackdepot: add depot_fetch_stack helper Date: Tue, 29 Aug 2023 19:11:14 +0200 Message-Id: <757ff72866010146fafda3049cb3749611cd7dd3.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 5DD0CA001E X-Rspam-User: X-Stat-Signature: 3auhtgpf189nuza69gwx3tsz3b1y6z7a X-Rspamd-Server: rspam03 X-HE-Tag: 1693329099-572030 X-HE-Meta: U2FsdGVkX19nl9+otGqTnH6tQgUR79L5TzI6u6X5XeJrgo+pPpW0UHzvfMy3MGRLhpXFEHgZ9WUKHkMNGInG8rNjWITbezrRUYTDc56OoNgMjzw6TDa0eMNw7wkK7OFVRsEu59LaQUjgs2r2fdxSMmB6ZpQQFZdy7mAi3z5Q88PARMp0Tkl1L9HO55ENmJZYh/h1mVaF8CSQqygMxtmUgonRWQhbxMp+r0/avO2pRWFWedXnknn12iWAK28kO8GG5jwCJPLm9HW4QSs78P5nCKocz9T6M+4YLwuZoCiYgNQqAxnhzMcPbgKwAcZfOmqHh0DfoZaVmNw5NobifIDzeS80eR2zgg7akLXV27sCLsqO0Owc+/NMzG8bqNoKHtRCLSOQrhTxgx4MouJaNgiPx/SHME29oeaeQagGqED72P7zsQUsUN47fuSMedKLSkSQoJWox6AJ30op9lrN2eF2lPw5zphvojjZobMtnIewyFC9tpiM1k3d7WxDkEzTcJqBSZnvvzmklA29XY/QOHrukybisiqPjYnt26RcPJq7QZatMSz2THH8UVvSDSLnpUC7ZZcc3+9z3AsTD/ZturmyqOmiqWQh3atXqNmHHnokF+PYreQOCTv5Oa3BVz1uYSURh7Q+W9qMqzxQ/QFay8bs67WexJPsiBrjLLemg2gNpynYyVapLfaHPfH1/aTuK67XvAp+4PXuRTBStTiUC13mkUjSIWFY7Fve038fEpuU5125GiEpeZPc/VuW6YP4Mq/d0CcrbIhC4lhIZGKGKBgQ0xVCi2QP0G4DQ3bWBZ8NbaI08Map9KceJUKkODaPqVs/gd5FdPLoak7W0zJFP3A0KQDzRQlvRJbcgiwhiL941bir957iZ9ktqIrv3Pag2g4IQprSSC0GlaeB7vYRVZ2h2E5V0PjaWskVN1AtduwMXMchgz/ciefBvJsyiY7g9yrC6YKgIMpq4LFsljymm/t r5NbZDzh InmqNwgEXs4X50PuuFBuoZ/3YA4gxkZLc14rZ63JxMhkvvbxTim1Qoe82vpr9rHMSfS+tFar1nOwkICF19eJT/pSFa3LrynQlg6Liz4g2rW5UIWKFnduLWMJdTqVtjEpBDXK+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Add a helper depot_fetch_stack function that fetches the pointer to a stack record. With this change, all static depot_* functions now operate on stack pools and the exported stack_depot_* functions operate on the hash table. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 45 ++++++++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 482eac40791e..2128108f2acb 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -304,6 +304,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) stack->handle.extra = 0; memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); pool_offset += required_size; + /* * Let KMSAN know the stored stack record is initialized. This shall * prevent false positive reports if instrumented code accesses it. @@ -313,6 +314,32 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) return stack; } +static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) +{ + union handle_parts parts = { .handle = handle }; + /* + * READ_ONCE pairs with potential concurrent write in + * depot_alloc_stack. + */ + int pool_index_cached = READ_ONCE(pool_index); + void *pool; + size_t offset = parts.offset << DEPOT_STACK_ALIGN; + struct stack_record *stack; + + if (parts.pool_index > pool_index_cached) { + WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", + parts.pool_index, pool_index_cached, handle); + return NULL; + } + + pool = stack_pools[parts.pool_index]; + if (!pool) + return NULL; + + stack = pool + offset; + return stack; +} + /* Calculates the hash for a stack. */ static inline u32 hash_stack(unsigned long *entries, unsigned int size) { @@ -456,14 +483,6 @@ EXPORT_SYMBOL_GPL(stack_depot_save); unsigned int stack_depot_fetch(depot_stack_handle_t handle, unsigned long **entries) { - union handle_parts parts = { .handle = handle }; - /* - * READ_ONCE pairs with potential concurrent write in - * depot_alloc_stack. - */ - int pool_index_cached = READ_ONCE(pool_index); - void *pool; - size_t offset = parts.offset << DEPOT_STACK_ALIGN; struct stack_record *stack; *entries = NULL; @@ -476,15 +495,7 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, if (!handle || stack_depot_disabled) return 0; - if (parts.pool_index > pool_index_cached) { - WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", - parts.pool_index, pool_index_cached, handle); - return 0; - } - pool = stack_pools[parts.pool_index]; - if (!pool) - return 0; - stack = pool + offset; + stack = depot_fetch_stack(handle); *entries = stack->entries; return stack->size; From patchwork Tue Aug 29 17:11:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 195D3C83F12 for ; Tue, 29 Aug 2023 17:11:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC7CD8E0031; Tue, 29 Aug 2023 13:11:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D768E8E002F; Tue, 29 Aug 2023 13:11:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3E6D8E0031; Tue, 29 Aug 2023 13:11:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A9F868E002F for ; Tue, 29 Aug 2023 13:11:41 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7D2FB801E1 for ; Tue, 29 Aug 2023 17:11:41 +0000 (UTC) X-FDA: 81177784002.16.DFE7FBC Received: from out-253.mta1.migadu.com (out-253.mta1.migadu.com [95.215.58.253]) by imf27.hostedemail.com (Postfix) with ESMTP id CA2F440020 for ; Tue, 29 Aug 2023 17:11:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=vWRwTzQB; spf=pass (imf27.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.253 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329100; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=etRrGOIqY063bnkRK0UomaeBxyJpZf9uP5poWvb3gqc=; b=uchqxte+6iEk0GoLxehLcqTyeN/Gs5Lmj3STQGaRwkdqKWI/fWNWsmTt4iUAkWm7D+VeX9 A3dstMZ/6amKVozQy1L9xzvrlSSShI4l7UyDeo7sNdOpjAJVKQ3phb1XY8uKgiVt7Pzkkk hjJ98xK8DePZj+TbeWDM76hNzeo5zXY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=vWRwTzQB; spf=pass (imf27.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.253 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329100; a=rsa-sha256; cv=none; b=C3o1+6mWEHH6b24xI0QoaLUDag3b7T0jGxFyLnt85qFJ+i6cjJ9XUascEaahpomnZ1twzh /MU89IYmCvb5b/1QNqV6RypPsqzcpUzTSS5Toor9GXPljuUdRWgsndMdTIZSjaeMB9jcDV BqFm0uh157nD0gZk/bQ2qqOxdrrJtp4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=etRrGOIqY063bnkRK0UomaeBxyJpZf9uP5poWvb3gqc=; b=vWRwTzQB5vMYolMaxpPDvyneGGFmDZgws8+tFLq59cWPXaVn3GV0VaYcUF5RzitZxoe+T3 ZMExK1/RpceRrBL2M3n8y25ttAi3ULTCCP+y0Sbt1jt2hAnxV/2IzWEEpOegOi8gP9Lc6d hE8F8O5yYsTdFlsVFcDlLuY0x9gpyyU= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 05/15] stackdepot: use fixed-sized slots for stack records Date: Tue, 29 Aug 2023 19:11:15 +0200 Message-Id: <89c2f64120a7dd6b2255a9a281603359a50cf6f7.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: CA2F440020 X-Rspam-User: X-Stat-Signature: 349qhifmubgpbpkanhwzezrngab45p6u X-Rspamd-Server: rspam01 X-HE-Tag: 1693329099-294711 X-HE-Meta: U2FsdGVkX1/P/JbTwauP3NvmmZNj1OWaD+AiZ0ObXMbUD0cGMTcYkLWiCi5gwpborZ5ifUo3KkmtaiFsA70OKEGhlL9Cq7wSDkXcBUJR1wCARjE9+pa2asagVSYtkpGNAzDFuAREWkr+G4bYuAkMDzhqjlv6cHkwpOknSFlO0lWFN8DnpjJ6okbisGja+iEDrmwaDE4o6i8OAo1/zEcw+tA3QdgnSP6EtzeM4uCQVsCS0YjYjut0Su1mDeCNaoWKtwo0EeabfV0hXKn+LvsL6r1SysVtvWWwTOJfO6AEVxZZrJ7uRyTo/wCBM1/gdcSXA/wg2aXebkKvQJ9TKexfDDlP1igd0z95sj5qS++7N3py0yYNnAPF3M1Gu4pZtUOdUdgDmPL3oiyuePWviSgtlzdHV2PF7fUJ48Paah2nuphHen7uRw8DJn3A5u8oG7oBATEUJAZJYTX0+5Y3e89+cwMd4EJ2Po2vOLdZaSsjlxIRGJ+mN70UJ1EcmHF7nAsmY7jzmZT5gpNNCClIr5zIbx5bC7xrfRL4TFtLGXC/47vCjwWDyyTnkYZfMd8/YQe1mZG3f3aPkuAbsNOE6rx3lN484kGKvdeG3QRt6bX+f9MPVuDzczXtvKZ7Z7hZ0zeu24auhe3GiHREQnQgFDK/GigkrJcx9xgICbJ37/ZVNr30sxQpjUtaJThd/spLpeZ2mU4QXhxzmUvmW4J9eFc0uc4kwIW3289SKugRcW5LVUp5G16CYbHJ0C4UyKaGf1Vp9vtqkJDFODK7Vw11ZF940nMbvhnZ2ECFR7QAIruZAJei7j/0xNh1l+HYLzi96rK7W2LvXuQp7++INmeNUxPtB+7eaj0PAhwh6CiB68aGkklvZyqWhgy8KI3imNUJlBguvoOJobedJeGhjeYXG1E7qhdvPZih8bgzc4rkrNrvzygrr+7VhK91hNILsQGVC9mLylFbz2DEkHA8/yxT0Is XVDTQtcF KQeEc1vWeNeZpvOlkMmTOrfjJqwel0iW9uQmNkCaU4JHUUishd1I405oXsWLmzq4+YMVT62LQQ4M8j9MRw3mGSkPvVbpCTBqWow1v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Instead of storing stack records in stack depot pools one right after another, use 32-frame-sized slots. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 2128108f2acb..93191ee70fc3 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -42,6 +42,7 @@ #define DEPOT_MAX_POOLS \ (((1LL << (DEPOT_POOL_INDEX_BITS)) < DEPOT_POOLS_CAP) ? \ (1LL << (DEPOT_POOL_INDEX_BITS)) : DEPOT_POOLS_CAP) +#define DEPOT_STACK_MAX_FRAMES 32 /* Compact structure that stores a reference to a stack. */ union handle_parts { @@ -58,9 +59,12 @@ struct stack_record { u32 hash; /* Hash in the hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; - unsigned long entries[]; /* Variable-sized array of frames */ + unsigned long entries[DEPOT_STACK_MAX_FRAMES]; /* Frames */ }; +#define DEPOT_STACK_RECORD_SIZE \ + ALIGN(sizeof(struct stack_record), 1 << DEPOT_STACK_ALIGN) + static bool stack_depot_disabled; static bool __stack_depot_early_init_requested __initdata = IS_ENABLED(CONFIG_STACKDEPOT_ALWAYS_INIT); static bool __stack_depot_early_init_passed __initdata; @@ -258,9 +262,7 @@ static struct stack_record * depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) { struct stack_record *stack; - size_t required_size = struct_size(stack, entries, size); - - required_size = ALIGN(required_size, 1 << DEPOT_STACK_ALIGN); + size_t required_size = DEPOT_STACK_RECORD_SIZE; /* Check if there is not enough space in the current pool. */ if (unlikely(pool_offset + required_size > DEPOT_POOL_SIZE)) { @@ -295,6 +297,10 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) if (stack_pools[pool_index] == NULL) return NULL; + /* Limit number of saved frames to DEPOT_STACK_MAX_FRAMES. */ + if (size > DEPOT_STACK_MAX_FRAMES) + size = DEPOT_STACK_MAX_FRAMES; + /* Save the stack trace. */ stack = stack_pools[pool_index] + pool_offset; stack->hash = hash; From patchwork Tue Aug 29 17:11:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96AC2C83F21 for ; Tue, 29 Aug 2023 17:12:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A5618E0033; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2531B8E002F; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F51A8E0033; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ED19E8E002F for ; Tue, 29 Aug 2023 13:12:42 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BA90EA0235 for ; Tue, 29 Aug 2023 17:12:42 +0000 (UTC) X-FDA: 81177786564.22.66B0F10 Received: from out-249.mta1.migadu.com (out-249.mta1.migadu.com [95.215.58.249]) by imf25.hostedemail.com (Postfix) with ESMTP id CE28EA0013 for ; Tue, 29 Aug 2023 17:12:40 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=xUAvBItl; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.249 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+57664S6bw4mOO/qPj8h7l/H8qh2Ip2xAdoPp4fmyag=; b=IvsHwssZQ3vLWaD7S7JbOJ+MgGJh0dE46WplhuyqF0UU04ymTcyEF1iu+ZGN2SVasMZ2Iz CaK812gOnCHykh7AgHVA+iJKkcdvQnIDq3kTwcNHes5sM7/Z3zbyy1YyKVR+QBxY/1CbH+ y1y9vC+rHbcNFiuH9gBJpbibchvzsuE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=xUAvBItl; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.249 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329161; a=rsa-sha256; cv=none; b=jtzNntCqgCnmzt7N8NIFitBVEHbum+QNulI2qnH/REwioJLSyETisaFaZYoDNSHhwGBg6j iVwWKL16wltHbxOcJM1x8ryg7dbVXkTg6hVRAPi7oqz79aVQY6tIPTHuuq4RkaGJY79ll0 01bNUg+cumUYzKyTTVHFSwXdtfyTMM8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329159; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+57664S6bw4mOO/qPj8h7l/H8qh2Ip2xAdoPp4fmyag=; b=xUAvBItl08LNbG+js05Euj/uDwk7mnKgQkEClKLARFeIBi8pvWrOlUBmEdpKyMjEBpe08t MZxZZK8u8HPHUPrktgetKtdD93OVx+rnbApYTVTui9WIUJLSE/Xb3wolF7HpROmuiVuy/N gklA8fWRmILe9ypho6CUw2WGHHwPn0E= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 06/15] stackdepot: fix and clean-up atomic annotations Date: Tue, 29 Aug 2023 19:11:16 +0200 Message-Id: <8ad8f778b43dab49e4e6214b8d90bed31b75436f.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: CE28EA0013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: krj8t9rs83bw7e33d9yiy6a4najxt9zd X-HE-Tag: 1693329160-672446 X-HE-Meta: U2FsdGVkX19xvSdge75skjUX/VJhBUuVuXj6pDg05VsmuYukDgirR+bkDZ1SCp9MvONhqSrLmsWsk4CjBOt9EiFuQHEMCHDwbAj+6Gn+wJCQQgT3T+q4NfknwEg5Fxs5o9uIDBGbNaJr1RP+OLpo/fEvTPVHv0j23vqoO8f/TKw3QcIM3YUUEctBr1zZprwNX6lsOk0RGWU0ayejpjM7TVMd6/nHFt04FD2b4LjuKxkNqNMZ+5BkWzMHGCSt/F3hO3tmyfQ9aTEtHkGbWDoCXhGaS8NzdB7n9A6CHscaKsbcmehtk5bxZNLp/BHmo/KuioSj8gYeC9ARpcQg+6KS1FbwZiLEHzMSMOBBtMpwaQ3mCScsHKrz7mWlWQug5S9OjiF/ezOClRw5Dt7KQYhxvH9gzTSqwsbYVLy8Gu9P7dDuuCbGOYrWLgFSOskfTo05G1+ieGpOaNtoGofSwmg6zCGEq45qZM9lnsIlw3vMe3QxzrtnVCSUMitsx1xXgb0CKcLrgOlKU5KmTF2YE7UfQekndd+YYxbX4j9NpL5UA52gr4YP+JSXn8NFsuMGLX4JIrwlr34PG47ku/odrqJwLmYKjAF4/ImiZ56qPT9hBjeFw+mK8sGwqqr+KRgOdECHI5iv33UtfukwvQPU2PoFfBKqG+6AWrKp8wOIc+4SDvJKQLuQ2FFEKwESs3d9Cy6KhzdqdXXB8n/C/dtBn+DlJWGSuMge1owDOe6S0h7u1wmf0dYAUSo8bq5OMr9GWLC6YW6kFv0hM6X/r+hUC8eK63QHx0cwQz/SE0Ho6sOkjX/HpxOi/RYD2rm4WeGpvvXvE+lhir4j2O9DDeANNSZg/Tm57HgQuRO0i5Ss9pwZysxtxEktCF+tl+FQpGcTsyFukqVvnONjmTG6seQGcRJfThc1H3zEyeltijSQIDFvZ5rZQKtxZLHjAEeu8lsI5XSQcryEAODqEUG9olI2Rl8 P+/UlzrB 3HdphzJ9lJ6strll6o3L5q7ATSLe9T9QIBjno9jxkswOF2mqhCrtHdbYoozZMVh4NUbaCtMHNbXaFqC/GXSCsLAVL4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Simplify comments accompanying the use of atomic accesses in the stack depot code. Also turn smp_load_acquire from next_pool_required in depot_init_pool into READ_ONCE, as both depot_init_pool and the all smp_store_release's to this variable are executed under the stack depot lock. Signed-off-by: Andrey Konovalov --- This patch is not strictly required, as the atomic accesses are fully removed in one of the latter patches. However, I decided to keep the patch just in case we end up needing these atomics in the following iterations of this series. --- lib/stackdepot.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 93191ee70fc3..9ae71e1ef1a7 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -226,10 +226,10 @@ static void depot_init_pool(void **prealloc) /* * If the next pool is already initialized or the maximum number of * pools is reached, do not use the preallocated memory. - * smp_load_acquire() here pairs with smp_store_release() below and - * in depot_alloc_stack(). + * READ_ONCE is only used to mark the variable as atomic, + * there are no concurrent writes. */ - if (!smp_load_acquire(&next_pool_required)) + if (!READ_ONCE(next_pool_required)) return; /* Check if the current pool is not yet allocated. */ @@ -250,8 +250,8 @@ static void depot_init_pool(void **prealloc) * At this point, either the next pool is initialized or the * maximum number of pools is reached. In either case, take * note that initializing another pool is not required. - * This smp_store_release pairs with smp_load_acquire() above - * and in stack_depot_save(). + * smp_store_release pairs with smp_load_acquire in + * stack_depot_save. */ smp_store_release(&next_pool_required, 0); } @@ -275,15 +275,15 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) /* * Move on to the next pool. * WRITE_ONCE pairs with potential concurrent read in - * stack_depot_fetch(). + * stack_depot_fetch. */ WRITE_ONCE(pool_index, pool_index + 1); pool_offset = 0; /* * If the maximum number of pools is not reached, take note * that the next pool needs to initialized. - * smp_store_release() here pairs with smp_load_acquire() in - * stack_depot_save() and depot_init_pool(). + * smp_store_release pairs with smp_load_acquire in + * stack_depot_save. */ if (pool_index + 1 < DEPOT_MAX_POOLS) smp_store_release(&next_pool_required, 1); @@ -414,8 +414,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, /* * Fast path: look the stack trace up without locking. - * The smp_load_acquire() here pairs with smp_store_release() to - * |bucket| below. + * smp_load_acquire pairs with smp_store_release to |bucket| below. */ found = find_stack(smp_load_acquire(bucket), entries, nr_entries, hash); if (found) @@ -425,8 +424,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * Check if another stack pool needs to be initialized. If so, allocate * the memory now - we won't be able to do that under the lock. * - * The smp_load_acquire() here pairs with smp_store_release() to - * |next_pool_inited| in depot_alloc_stack() and depot_init_pool(). + * smp_load_acquire pairs with smp_store_release + * in depot_alloc_stack and depot_init_pool. */ if (unlikely(can_alloc && smp_load_acquire(&next_pool_required))) { /* @@ -452,8 +451,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, if (new) { new->next = *bucket; /* - * This smp_store_release() pairs with - * smp_load_acquire() from |bucket| above. + * smp_store_release pairs with smp_load_acquire + * from |bucket| above. */ smp_store_release(bucket, new); found = new; From patchwork Tue Aug 29 17:11:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69DBCC83F14 for ; Tue, 29 Aug 2023 17:12:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A94018E0034; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1BB68E002F; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86E978E0034; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 71AA08E002F for ; Tue, 29 Aug 2023 13:12:43 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 53501C0228 for ; Tue, 29 Aug 2023 17:12:43 +0000 (UTC) X-FDA: 81177786606.14.EA855DB Received: from out-252.mta1.migadu.com (out-252.mta1.migadu.com [95.215.58.252]) by imf22.hostedemail.com (Postfix) with ESMTP id 7D5CBC0019 for ; Tue, 29 Aug 2023 17:12:41 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=rTza+MTD; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf22.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.252 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TBLA9eVhfpPnUCaW6dvoHE+HvIWz4FYhXI+BxxI+EVo=; b=iIn0fCBjLpYDYyb7eFPukpnRjeyktlq694fs2ysdAUdnHSetFLCha+GKpqQ3GZ322JpIf2 hCFo5J8fy804Po84Aj+KH1GYmgt5VuFeU1HMiapdvPXoFQwaRcsnu2sVtaunO3S6GmFQft /KcTywos67CQegshIxgK7kD3qzO0mMw= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=rTza+MTD; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf22.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.252 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329161; a=rsa-sha256; cv=none; b=6CE7LdzLs4M7TylcDAewQEkFcBhLqQk1bx3umkCPqV2h+9yrySh+eSV63HUqzbZtuBeIT0 /iQYRhuwBUKj4wEISBhWAuhmRJwUR5tOK5gPrZW/SiGc7Jk+2qYC0PUXCJrLt6yl1ORJ0f 1cQJULZNHhG6zPEtnZr0PSWDWNu1jJ8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TBLA9eVhfpPnUCaW6dvoHE+HvIWz4FYhXI+BxxI+EVo=; b=rTza+MTDtXsApdY3UtmvCOQ7kVEgxoogWmU+0/N+/sEo8Jl8pyma+tVj6hbC217kRw1J6E W2W+iu1c4gZpxYAIOpkN1sNFMFS3da9ylVLiov/nqPc0JE4H1dH8MM7fvRWFW9ucQ0eS+L WhjGfi5dKdpIeZyWgv2oUZtl5JJ4KEI= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 07/15] stackdepot: rework helpers for depot_alloc_stack Date: Tue, 29 Aug 2023 19:11:17 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7D5CBC0019 X-Stat-Signature: kidkkyotbcyf139jpwsostzfck7nxuzy X-Rspam-User: X-HE-Tag: 1693329161-559602 X-HE-Meta: U2FsdGVkX1+M0dp7YKnE8TQLyyoCWBGrg8LTlGnNHMmLrMedFQIME7AzeRWGUG17ZAOLaIawXj4viyGrow2q7AXbQSqk4h9+Bcm2sEfYzM6EfYqbDCtCfGXV0vBEVR9Lb27bjdiOy6KP7ttSPtfi1hJYwjKdSRXSVnOIMAw1yuOzFRL4ZaxTO+l4ogTC2ajt7UcQ3GJdCE23Knd0Xa4QRQ1nwPlErLEn7ts6hlqfQf5Im9WAgCsAfXtEMy7s7y40R/yJK5HMb+KnLauB4LB5JD5dIT+1eQpbMM4dQrwVX6r2RQ+O0jzg7MlDhq8PqqaT7yVIy/mxXwAUBbDjU1LB7XrFV6waCqbAGg0TW48TenF47c/mgDS6QcEddROE3aX/U1S2+bdXPDTUAAZ4R0LSqro8f1UgrxLdrb3QIucogEyXYlHxgn40d4GB9lkuy5fyJrB1jnOahd3KfvKzawhe2Rt9o6jyAy0K15itvLwoNdTH1T44VX8N0C9cf4u4iCiFTzm1bseEz1QacOBirikgldpVd/7sCZDrGw1Eeo3Bey/YBiF8dYaUahWTREOrnUGlQr6ahI0R+bq9JWGA1iXiXdtNIbBOCveFTj7mjdD2dubbyTjfa3V8EcbvKjW0HVieGskzErQEnppCcaBJJPYzP/0LDcSnvUhzqwk3qbH5NZv2rvuvRYPcNl+1LQO/wcssQeUfdEDKEVLZtlM2p1Jrd5kAOmqm2ys2FsNPC/+Uf4OPq5/ckPDM/kUbhU8T9HBG+Sx2e1w4UT+YR8rAQp8bM4yEuopPImsHFCq3I2LlghGb8LNgKmEHAZIR52UB8tdIP5Gsrvsfabji28tApdzEmMy5bGl1KphYSIZQS8ILtgCjJkl2uESJwKA8Op+V6yXODrO7yTyDjy+igFoXuMogcwsw3pdvg8PuZj3Oy9PNGyDZxIM4S0Ur6ew9LwYS21hqdJ/dRiH952UVT812Hq0 tPuhGpXL CsBhgH15gxHtdlLtlNjcYiYdzOMcjKo89I/j/N1FGB99TCqTtrJooa3BOjOFu2noKZRwGst4DA+Wt6lHqZRpbT+UTj3UjUUPE6hu9VssvWX4GN18aHcKnvgGXKYPctDfX1puSI33QyecYoI5NM9kSR65Qog== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Split code in depot_alloc_stack and depot_init_pool into 3 functions: 1. depot_keep_next_pool that keeps preallocated memory for the next pool if required. 2. depot_update_pools that moves on to the next pool if there's no space left in the current pool, uses preallocated memory for the new current pool if required, and calls depot_keep_next_pool otherwise. 3. depot_alloc_stack that calls depot_update_pools and then allocates a stack record as before. This makes it somewhat easier to follow the logic of depot_alloc_stack and also serves as a preparation for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 85 +++++++++++++++++++++++++++--------------------- 1 file changed, 48 insertions(+), 37 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 9ae71e1ef1a7..869d520bc690 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -220,11 +220,11 @@ int stack_depot_init(void) } EXPORT_SYMBOL_GPL(stack_depot_init); -/* Uses preallocated memory to initialize a new stack depot pool. */ -static void depot_init_pool(void **prealloc) +/* Keeps the preallocated memory to be used for the next stack depot pool. */ +static void depot_keep_next_pool(void **prealloc) { /* - * If the next pool is already initialized or the maximum number of + * If the next pool is already saved or the maximum number of * pools is reached, do not use the preallocated memory. * READ_ONCE is only used to mark the variable as atomic, * there are no concurrent writes. @@ -232,44 +232,33 @@ static void depot_init_pool(void **prealloc) if (!READ_ONCE(next_pool_required)) return; - /* Check if the current pool is not yet allocated. */ - if (stack_pools[pool_index] == NULL) { - /* Use the preallocated memory for the current pool. */ - stack_pools[pool_index] = *prealloc; + /* + * Use the preallocated memory for the next pool + * as long as we do not exceed the maximum number of pools. + */ + if (pool_index + 1 < DEPOT_MAX_POOLS) { + stack_pools[pool_index + 1] = *prealloc; *prealloc = NULL; - } else { - /* - * Otherwise, use the preallocated memory for the next pool - * as long as we do not exceed the maximum number of pools. - */ - if (pool_index + 1 < DEPOT_MAX_POOLS) { - stack_pools[pool_index + 1] = *prealloc; - *prealloc = NULL; - } - /* - * At this point, either the next pool is initialized or the - * maximum number of pools is reached. In either case, take - * note that initializing another pool is not required. - * smp_store_release pairs with smp_load_acquire in - * stack_depot_save. - */ - smp_store_release(&next_pool_required, 0); } + + /* + * At this point, either the next pool is kept or the maximum + * number of pools is reached. In either case, take note that + * keeping another pool is not required. + * smp_store_release pairs with smp_load_acquire in stack_depot_save. + */ + smp_store_release(&next_pool_required, 0); } -/* Allocates a new stack in a stack depot pool. */ -static struct stack_record * -depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) +/* Updates refences to the current and the next stack depot pools. */ +static bool depot_update_pools(size_t required_size, void **prealloc) { - struct stack_record *stack; - size_t required_size = DEPOT_STACK_RECORD_SIZE; - /* Check if there is not enough space in the current pool. */ if (unlikely(pool_offset + required_size > DEPOT_POOL_SIZE)) { /* Bail out if we reached the pool limit. */ if (unlikely(pool_index + 1 >= DEPOT_MAX_POOLS)) { WARN_ONCE(1, "Stack depot reached limit capacity"); - return NULL; + return false; } /* @@ -279,9 +268,10 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) */ WRITE_ONCE(pool_index, pool_index + 1); pool_offset = 0; + /* * If the maximum number of pools is not reached, take note - * that the next pool needs to initialized. + * that the next pool needs to be initialized. * smp_store_release pairs with smp_load_acquire in * stack_depot_save. */ @@ -289,9 +279,30 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) smp_store_release(&next_pool_required, 1); } - /* Assign the preallocated memory to a pool if required. */ + /* Check if the current pool is not yet allocated. */ + if (*prealloc && stack_pools[pool_index] == NULL) { + /* Use the preallocated memory for the current pool. */ + stack_pools[pool_index] = *prealloc; + *prealloc = NULL; + return true; + } + + /* Otherwise, try using the preallocated memory for the next pool. */ if (*prealloc) - depot_init_pool(prealloc); + depot_keep_next_pool(prealloc); + return true; +} + +/* Allocates a new stack in a stack depot pool. */ +static struct stack_record * +depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) +{ + struct stack_record *stack; + size_t required_size = DEPOT_STACK_RECORD_SIZE; + + /* Update current and next pools if required and possible. */ + if (!depot_update_pools(required_size, prealloc)) + return NULL; /* Check if we have a pool to save the stack trace. */ if (stack_pools[pool_index] == NULL) @@ -325,7 +336,7 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) union handle_parts parts = { .handle = handle }; /* * READ_ONCE pairs with potential concurrent write in - * depot_alloc_stack. + * depot_update_pools. */ int pool_index_cached = READ_ONCE(pool_index); void *pool; @@ -425,7 +436,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * the memory now - we won't be able to do that under the lock. * * smp_load_acquire pairs with smp_store_release - * in depot_alloc_stack and depot_init_pool. + * in depot_update_pools and depot_keep_next_pool. */ if (unlikely(can_alloc && smp_load_acquire(&next_pool_required))) { /* @@ -462,7 +473,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * Stack depot already contains this stack trace, but let's * keep the preallocated memory for the future. */ - depot_init_pool(&prealloc); + depot_keep_next_pool(&prealloc); } raw_spin_unlock_irqrestore(&pool_lock, flags); From patchwork Tue Aug 29 17:11:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EB4FC83F18 for ; Tue, 29 Aug 2023 17:12:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47B948E0035; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 401E18E002F; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27C338E0035; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 107758E002F for ; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DA1901C941E for ; Tue, 29 Aug 2023 17:12:43 +0000 (UTC) X-FDA: 81177786606.05.8A8EA68 Received: from out-248.mta1.migadu.com (out-248.mta1.migadu.com [95.215.58.248]) by imf19.hostedemail.com (Postfix) with ESMTP id 05D441A0014 for ; Tue, 29 Aug 2023 17:12:41 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=dEoDb6yz; spf=pass (imf19.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.248 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z7PnYrb1S+8x+6HixwHlp8BbvsuCogeXMw7DcmkZ+jU=; b=qmx9mO41F97l8dMU8xZmdWuqLOKXED12yaOOmhrk6l1yGlpwbpa4liGyzMQw0EKCU1g67e WBe3fjOU3jtMq4cuUJvXH5FHQC3bi7qqjkOC0htTyb0JTLpV1+M2Gh06efitLvYWe4Ak66 ly0bo0NpNMWD5e5YmjOe2z3VbDH7VB8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329162; a=rsa-sha256; cv=none; b=OdTGgMgnytjC4GalCE/k/yVv1m/jh4SnRVm5qlYra/BrhfGvrReX4/3Ypb2pVq7cuu64vX 6m7ysOnCZanBo5PT47GnzI3DDydkNnzOkMLinV7XdGVDsn4Akfz88eBRozmxXSDA2YVDXW 15bXvmaXPVuU09aECoYgL/xrZh+WchY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=dEoDb6yz; spf=pass (imf19.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.248 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z7PnYrb1S+8x+6HixwHlp8BbvsuCogeXMw7DcmkZ+jU=; b=dEoDb6yzJM9iX6McraduAfQqnJZCVuZ61muNam0W1Q8u/Mj2MWXslfimq4jnFNa8v8+Yj+ /Lw1sjBeAZ25rXrIel+1C6rNGA5rk9DVJDQRgkWLe5Nd7RsIlPk+jzgO9LM5Ew++A1tMf5 NgbpsmDhYFFdEXDI5r4VFPiqImx4blM= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 08/15] stackdepot: rename next_pool_required to new_pool_required Date: Tue, 29 Aug 2023 19:11:18 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: pwrqqhbpdajioqgqhu3p87qw1zag4wsp X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 05D441A0014 X-Rspam-User: X-HE-Tag: 1693329161-946980 X-HE-Meta: U2FsdGVkX1+PRemfxtn4j7PVX9WNEQp/0bZDpSEYcPcaKAIosPDjjh0VxD/Uw21PvKWC0v8oTW/08R/tCq3TAszA+ODdLlkOBJBbYgIUeqCzvy3o1IJYYgJYS5Qo6PJDoRm3pXo0pdguWKo5PiHPXHQMZgA0SpYzNmC5u01TzYoNyp257RMBnFlzW46IKbSHhd78OctwaU04KJWUUWnTw4jYdwy0LZaDF6HecniR3uS+Zs//8KvciKNycHj7kyAXbwwoD46xTF1JajlajgwPCPLziwZ7iXKVACbCP614i1ImprOzwARtFzJ6TBeXZgML/SDHmAetSRy2j53RKUmiBzkBL/F1siO7GBTq5lLvQF1wyg40/SSbrAGLmjtyim0rsywQHKE/enOQsD4iDlxU4+mS3hIszBrX35uoiONPT9sTVk8zzNG3yaekANE8Fnxqr5Ol6O0+cBnfELJ1/gHLBvfovY/s3VeqZLA6am+Iqzo9RPge25O90FScXaQQD5sZIKhcy8f9WQIZFs0eBE2LYcdPDg24wW6rwM50Tr53IM5+hu2THk5jZfGc4+zckHKOcnXO87s5Ow0jw3aUcWInWFMwwaSS03ooQzKb10Q9GZCXCyk7xHLbZmtXexO3Nk32C7bw5xvLOcG7xhX0vDCxgPebJwvK1fFludDFXjMqtbErNqsE/68AgeyPvSSCbA4BVUWhC0IFWOZQtenbGYYkdc5NQFmY3y+N7DitDWNMvWx6X0ozQxV4NEXXEIA/AGTndoRMMXbyKxODGw+oejB21wap02/bCZNgWx7h4yHITHkzsBJ1BlC8uFuqYG5zynGgkqVDOvoMjbpo6ANwPpf1JxKXK7qrlDsC8u/8ScBCMvdHXMarWQVmmjFKwxYKT2jjA6rl91xFs6QN+F1k2GlnnAF6BzbGBi8PhmZcaX1B4Dn2ujHzs5WVmVNkICx7j/PitebHEY2kO6D/0il7klF NFj+V7tS 1BjX39IKCYp/9tiDFkayXLCF4tcX5d10/nFPp9n3RCpFRmYEGQoFPvMDBuHHxlt97jaqEo+vhLk/2Ewc8/kO8+CLuBEdWoQhjOBmUavV5UJwaFpPVt6fMzAhU6ecIGIsO7vzeD+G6gozq/Mk6eY1A0oEnmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Rename next_pool_required to new_pool_required. This a purely code readability change: the following patch will change stack depot to store the pointer to the new pool in a separate variable, and "new" seems like a more logical name. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 47 +++++++++++++++++++++++------------------------ 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 869d520bc690..11934ea3b1c2 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -94,12 +94,11 @@ static size_t pool_offset; static DEFINE_RAW_SPINLOCK(pool_lock); /* * Stack depot tries to keep an extra pool allocated even before it runs out - * of space in the currently used pool. - * This flag marks that this next extra pool needs to be allocated and - * initialized. It has the value 0 when either the next pool is not yet - * initialized or the limit on the number of pools is reached. + * of space in the currently used pool. This flag marks whether this extra pool + * needs to be allocated. It has the value 0 when either an extra pool is not + * yet allocated or if the limit on the number of pools is reached. */ -static int next_pool_required = 1; +static int new_pool_required = 1; static int __init disable_stack_depot(char *str) { @@ -220,20 +219,20 @@ int stack_depot_init(void) } EXPORT_SYMBOL_GPL(stack_depot_init); -/* Keeps the preallocated memory to be used for the next stack depot pool. */ -static void depot_keep_next_pool(void **prealloc) +/* Keeps the preallocated memory to be used for a new stack depot pool. */ +static void depot_keep_new_pool(void **prealloc) { /* - * If the next pool is already saved or the maximum number of + * If a new pool is already saved or the maximum number of * pools is reached, do not use the preallocated memory. * READ_ONCE is only used to mark the variable as atomic, * there are no concurrent writes. */ - if (!READ_ONCE(next_pool_required)) + if (!READ_ONCE(new_pool_required)) return; /* - * Use the preallocated memory for the next pool + * Use the preallocated memory for the new pool * as long as we do not exceed the maximum number of pools. */ if (pool_index + 1 < DEPOT_MAX_POOLS) { @@ -242,12 +241,12 @@ static void depot_keep_next_pool(void **prealloc) } /* - * At this point, either the next pool is kept or the maximum + * At this point, either a new pool is kept or the maximum * number of pools is reached. In either case, take note that * keeping another pool is not required. * smp_store_release pairs with smp_load_acquire in stack_depot_save. */ - smp_store_release(&next_pool_required, 0); + smp_store_release(&new_pool_required, 0); } /* Updates refences to the current and the next stack depot pools. */ @@ -262,7 +261,7 @@ static bool depot_update_pools(size_t required_size, void **prealloc) } /* - * Move on to the next pool. + * Move on to the new pool. * WRITE_ONCE pairs with potential concurrent read in * stack_depot_fetch. */ @@ -271,12 +270,12 @@ static bool depot_update_pools(size_t required_size, void **prealloc) /* * If the maximum number of pools is not reached, take note - * that the next pool needs to be initialized. + * that yet another new pool needs to be allocated. * smp_store_release pairs with smp_load_acquire in * stack_depot_save. */ if (pool_index + 1 < DEPOT_MAX_POOLS) - smp_store_release(&next_pool_required, 1); + smp_store_release(&new_pool_required, 1); } /* Check if the current pool is not yet allocated. */ @@ -287,9 +286,9 @@ static bool depot_update_pools(size_t required_size, void **prealloc) return true; } - /* Otherwise, try using the preallocated memory for the next pool. */ + /* Otherwise, try using the preallocated memory for a new pool. */ if (*prealloc) - depot_keep_next_pool(prealloc); + depot_keep_new_pool(prealloc); return true; } @@ -300,7 +299,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) struct stack_record *stack; size_t required_size = DEPOT_STACK_RECORD_SIZE; - /* Update current and next pools if required and possible. */ + /* Update current and new pools if required and possible. */ if (!depot_update_pools(required_size, prealloc)) return NULL; @@ -432,13 +431,13 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, goto exit; /* - * Check if another stack pool needs to be initialized. If so, allocate - * the memory now - we won't be able to do that under the lock. + * Check if another stack pool needs to be allocated. If so, allocate + * the memory now: we won't be able to do that under the lock. * * smp_load_acquire pairs with smp_store_release - * in depot_update_pools and depot_keep_next_pool. + * in depot_update_pools and depot_keep_new_pool. */ - if (unlikely(can_alloc && smp_load_acquire(&next_pool_required))) { + if (unlikely(can_alloc && smp_load_acquire(&new_pool_required))) { /* * Zero out zone modifiers, as we don't have specific zone * requirements. Keep the flags related to allocation in atomic @@ -471,9 +470,9 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, } else if (prealloc) { /* * Stack depot already contains this stack trace, but let's - * keep the preallocated memory for the future. + * keep the preallocated memory for future. */ - depot_keep_next_pool(&prealloc); + depot_keep_new_pool(&prealloc); } raw_spin_unlock_irqrestore(&pool_lock, flags); From patchwork Tue Aug 29 17:11:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D65C83F19 for ; Tue, 29 Aug 2023 17:12:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0A3E8E002F; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F6368E0036; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D3778E002F; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5DB538E0036 for ; Tue, 29 Aug 2023 13:12:44 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3427214026B for ; Tue, 29 Aug 2023 17:12:44 +0000 (UTC) X-FDA: 81177786648.04.2ED848B Received: from out-244.mta1.migadu.com (out-244.mta1.migadu.com [95.215.58.244]) by imf14.hostedemail.com (Postfix) with ESMTP id 7F8B1100036 for ; Tue, 29 Aug 2023 17:12:42 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tY6t+d+v; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf14.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.244 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MjxW8cEdQiVOI1Uv8f/KLmoitLM15XxpvrzjQCh/NXc=; b=o5P+hyKarVaGd4gCRHyRRCukG6kxfkHvEcy6vkrmPV2fT0ODUQFV1CJmuwdRU3yGFvq/bR 13cGqfjF6+oNv/Ej84dlWX/qsJEYa5vVTSswDR1ZanJz8KWLTDaTlOhK806GRRNztW+XEU aKLAvnAqzfOLbyL30zCaG3eJkrGFy4o= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tY6t+d+v; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf14.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.244 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329162; a=rsa-sha256; cv=none; b=Yu1jt+1M0qZXVyAfbV8HgWtaeSAIJu42TgT2QDgGSpG1lldxHLBRNrbSidBIw/lJPDVaJf QoU5GQmTx+dA3kfxxAQzgxOw6HPj5qoMA/ELnXNHBlQVGIhimd6x8jhVFmYNzcf8zhleFs 4Ah3aORMeFc1oXP43DiS64BG4htrp2U= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MjxW8cEdQiVOI1Uv8f/KLmoitLM15XxpvrzjQCh/NXc=; b=tY6t+d+vBtGNax4kT5p4Y7yBgtMWSnu/QUCRoJmsSpGYGi+JfvAHktxIuYT65MqPmBkvxp 8RKOWRZXFpn759z55PlIk/UrGpleF+qwN8UTuRs6xh3eea3xHUbqZVYdzJbVmPrc/bwhgG rJ4DhJcJpXlAotO4gn30Fc2AmkPTb4c= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 09/15] stackdepot: store next pool pointer in new_pool Date: Tue, 29 Aug 2023 19:11:19 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 7F8B1100036 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: kkmcudjyk855t5reh161pycz9stefzd4 X-HE-Tag: 1693329162-816897 X-HE-Meta: U2FsdGVkX19dLXZIJ7o4D7vSMBd4ERiItpWZEeRPx+nyFYr5sJqEBHiTWw/orGxixVboEq7JRt1s8s2kkj3lMORr8wAOd3ArumCYvG8KpYa8eGHNIRa8/c9gc1UewcAMOOEr0U0EYh2qCyAaA7LW6CWisRUDI0Z00fbRQb8uJIZqG9RcWb8CWfLG4yabKgnijohs2U7x/gH7kqCow8219QMNSBH/cqeDJ49K07fhvB+szuEMksSpb4Qy69cXwvyZLZOyLK2PLNfybVVtE9Umzlz/e6AlWFMRlES5wSGck1bil/0cR9W/Zx2pmNqTJlFuWobhJ1ypu+3ilz2cY/EeFJmcxPe3s9oVZkDqGPPGFL6xMKr5Vu+o/ly3HvkRk4VjND/7P3oyFcxulJ0EL2nR4cI020QX2z0473nukUg3tCTwxvdv8bJQDzgHpGKvMgZfQpANL9haIfvPsMXYp0s9ysGpzDqt30f0uzHdjjVOSeQ5+P7ebYPVGd32snY7lbGrOPjdsbm1SoUviXCfRSY1DCfzP9LDF/gT0uj2Y5qq3EqaqTT5RIGlnI1uAh6HvBdzIu44PyDZA0ym/h9UOW60kZ5w23L5KEe5GZklJ3ujF53pYyRom7Llec6Stk6fcGkvMUaDvzEJn712Topo/vENKXJ93mciqjAYO8Xkl+qCRiC5pS+SgdQgR5tOFoO4YxQJvkqaAnASXqkKbyfzMvOlozWbyYJCtLEKn0HB3ZBgeqx7T1x6735mkGjv8uD7kTtf/z77eP5t9FBBSioQT+IAqD+iAobbeyDlnD0DF6VmfIfd7yyXdjiVEu/qYFlsMTZbirBq+Cv31kspYm3r6u+XPrA+2KnfhuetYtCUTWeLXSx3dM2rmvN4kpZe7COOZunSS8Og47qTXxjJrfzymstxx9IdXDZ7Uh6dhxfjyHeO/mFu16/IL5J+HZUFiHwyWeTaMUWtB6sLYuMDBqfzDtg PXhXHu75 mrtXe1lqt6pf7sFevrLDWDgdxeYG18j+Gy3zzptehUJ4G68UB6tk/4eQotPgwsEOmKVXVFtHFYrvb4dr+GbPyxiPVupmWQ0+yzabr2+mr/eUlL2RB3++GVA1GL2d6JkT3d0LMyyj18KzXYGgfdVDfl5j1iQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Instead of using the last pointer in stack_pools for storing the pointer to a new pool (which does not yet store any stack records), use a new new_pool variable. This a purely code readability change: it seems more logical to store the pointer to a pool with a special meaning in a dedicated variable. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 11934ea3b1c2..5982ea79939d 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -86,6 +86,8 @@ static unsigned int stack_hash_mask; /* Array of memory regions that store stack traces. */ static void *stack_pools[DEPOT_MAX_POOLS]; +/* Newly allocated pool that is not yet added to stack_pools. */ +static void *new_pool; /* Currently used pool in stack_pools. */ static int pool_index; /* Offset to the unused space in the currently used pool. */ @@ -236,7 +238,7 @@ static void depot_keep_new_pool(void **prealloc) * as long as we do not exceed the maximum number of pools. */ if (pool_index + 1 < DEPOT_MAX_POOLS) { - stack_pools[pool_index + 1] = *prealloc; + new_pool = *prealloc; *prealloc = NULL; } @@ -266,6 +268,8 @@ static bool depot_update_pools(size_t required_size, void **prealloc) * stack_depot_fetch. */ WRITE_ONCE(pool_index, pool_index + 1); + stack_pools[pool_index] = new_pool; + new_pool = NULL; pool_offset = 0; /* From patchwork Tue Aug 29 17:11:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4017AC6FA8F for ; Tue, 29 Aug 2023 17:12:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FBCD8E0037; Tue, 29 Aug 2023 13:12:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 982B18E0036; Tue, 29 Aug 2023 13:12:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 760248E0037; Tue, 29 Aug 2023 13:12:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 52BC38E0036 for ; Tue, 29 Aug 2023 13:12:45 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 15477B245D for ; Tue, 29 Aug 2023 17:12:45 +0000 (UTC) X-FDA: 81177786690.25.E183873 Received: from out-249.mta1.migadu.com (out-249.mta1.migadu.com [95.215.58.249]) by imf07.hostedemail.com (Postfix) with ESMTP id 4EA0C40024 for ; Tue, 29 Aug 2023 17:12:43 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=aVdkqRkg; spf=pass (imf07.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.249 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329163; a=rsa-sha256; cv=none; b=Zr5pCUgYdFUYsbfBN99FDM3SJLzw4i+TdBKPeYM5MFLJ6m2xOfcB8u+KMV3a8VLQm+4VMB vJtusSOL6N0d44MuSXJBI4lm6D3Owf5OjRIIT+Y/MAFqDly1u0CLLWZGvTnaRYOzz2COQe Ep34/0IHZRcXAOorHNa73AHdyVK3rsk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=aVdkqRkg; spf=pass (imf07.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.249 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329163; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=78m2o0At4EqoxYvPO8GgUXqZmV/z9BTR2roI196o/ig=; b=BaJxm5FItQUAFOArT+EHMi5yQDIBF3bW/lx3wK+R8pQNYMd3df4g5+lXmGR+pS4zWbHWbK HxKDfjUAzCOi8+N++gt3j3LGKlJdWQ8hDr1JiwRlra2sT9vxnvU6wFs22lGFtr0V9ybtIG ABI7SS4iOopzOePPqpeqG6kHHCB0huE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=78m2o0At4EqoxYvPO8GgUXqZmV/z9BTR2roI196o/ig=; b=aVdkqRkgoIFSKNeh42TLnbqZmPMoSkLTIn6T1JEhVD3Qqk6MuAku7bMKEsC8wSyrSX5enC FPGjSthHCdAdFwFBUcruYbc6A6xXWGBHMtC2szN93Qqo9L4dPl6NYeHve9SBanQLnxvT1U U8zvI/sIeAjU3YhEOjSFZSV+LFNUxSw= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 10/15] stackdepot: store free stack records in a freelist Date: Tue, 29 Aug 2023 19:11:20 +0200 Message-Id: <0853a38f849f75a428a76fe9bcd093c0502d26f4.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4EA0C40024 X-Stat-Signature: d6kortwwuytr7q67y5k4eqgq794ytnuf X-Rspam-User: X-HE-Tag: 1693329163-371807 X-HE-Meta: U2FsdGVkX19ZUpHGEBTHikOiHYfXRZtnFjBpT96eM8bCqg2mfax8gks+zVQIlsV1NWxmaLepbCYTJNsMnCHmJkvJ38FroIlBubEaSNSyufl8Ux+yjXIqNRa7F67L14DBIuqiZT2nDBhhU+SXjxGsNGCav1KxWO+zpOjTmeex+DEDh5QI87HsBl20Zycw/gFuo5vRviekkrrATgcVtTFy1QMvb7+tyXc4MDXfIzUHI6C3N46hEWcdRS8ZGWB4gzQH1DSdLADrnMBBcTwOJTBhEau8RA4X7b60lNLscdJThdKwomCB6NOJc8nLEvaK+MsbcQsEeG2jAwqT6IZ1XxUY3VqKtHZQEgbgu9hnSzckpaGJN8NINItovoXwBo4PU4tToj8pOSkLBGPgfVme6Ghnm1CLexwICSwBvrZJfbMdTrQqSE7mZjGliHryiLvNYbgwm0cgHD2LUZpm3nqUkGQjN///rQVuPUR4wxqkcuG9n1mDlt81Uw0hBVPZWztf0DenYA2UvHtxQm5Hos7dGSKK6tryZC+IUDN33tOgNaLcGsmqt7xK97bYj0R+RFtF0a9tk0cEU5Ebum7D54nj2oSaBIufn7+JOX6PKl/cVKWnE13Do352++Qqsa2BYzMjAt2xdZ3WAxBE3aYhyeq6EMu5jVBPe59vtwuV14T5DbjbYPChKU86erSX2m05eJAAUwMjtW1wHeHrswxA2LbbJOEvz1sRCORHM62YnfN6ihIRHem6dFzGcSE57LdcEFK20py1W90qyivhfU5xBK7ZSBm6svlDQZjyZUXw33nTxDGTTy8IAznlgKnxK5d1YRNzAt+77VxE77cR549uJDst8aAGLSLUB5v28M9Fb/HjEs6ou1sRPXpNDyGSL273iMoSEcGZaAynGhtscsj2P9woQd9lBhC+waLYga4r0YDw+IwJTOYTdfoKyxFit41HlwIpw1mm73w49vtVk0985VwF9C+ YunrL4Wa 8I8M/zfX6BP9vxrDs2N6dWyfoJrAjFeRSLkYdDyVZ/wqXHtJv0kPCef+sU8y7eTrNtB5enx/D/Nx/lEAF3U+/q7/y/n40A1eCwjua X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Instead of using the global pool_offset variable to find a free slot when storing a new stack record, mainlain a freelist of free slots within the allocated stack pools. A global next_stack variable is used as the head of the freelist, and the next field in the stack_record struct is reused as freelist link (when the record is not in the freelist, this field is used as a link in the hash table). This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 130 +++++++++++++++++++++++++++++------------------ 1 file changed, 81 insertions(+), 49 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 5982ea79939d..9011f4adcf20 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -55,8 +55,8 @@ union handle_parts { }; struct stack_record { - struct stack_record *next; /* Link in the hash table */ - u32 hash; /* Hash in the hash table */ + struct stack_record *next; /* Link in hash table or freelist */ + u32 hash; /* Hash in hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; unsigned long entries[DEPOT_STACK_MAX_FRAMES]; /* Frames */ @@ -88,10 +88,10 @@ static unsigned int stack_hash_mask; static void *stack_pools[DEPOT_MAX_POOLS]; /* Newly allocated pool that is not yet added to stack_pools. */ static void *new_pool; -/* Currently used pool in stack_pools. */ -static int pool_index; -/* Offset to the unused space in the currently used pool. */ -static size_t pool_offset; +/* Number of pools in stack_pools. */ +static int pools_num; +/* Next stack in the freelist of stack records within stack_pools. */ +static struct stack_record *next_stack; /* Lock that protects the variables above. */ static DEFINE_RAW_SPINLOCK(pool_lock); /* @@ -221,6 +221,41 @@ int stack_depot_init(void) } EXPORT_SYMBOL_GPL(stack_depot_init); +/* Initializes a stack depol pool. */ +static void depot_init_pool(void *pool) +{ + const int records_in_pool = DEPOT_POOL_SIZE / DEPOT_STACK_RECORD_SIZE; + int i, offset; + + /* Initialize handles and link stack records to each other. */ + for (i = 0, offset = 0; offset < DEPOT_POOL_SIZE; + i++, offset += DEPOT_STACK_RECORD_SIZE) { + struct stack_record *stack = pool + offset; + + stack->handle.pool_index = pools_num; + stack->handle.offset = offset >> DEPOT_STACK_ALIGN; + stack->handle.extra = 0; + + if (i < records_in_pool - 1) + stack->next = (void *)stack + DEPOT_STACK_RECORD_SIZE; + else + stack->next = NULL; + } + + /* Link stack records into the freelist. */ + WARN_ON(next_stack); + next_stack = pool; + + /* Save reference to the pool to be used by depot_fetch_stack. */ + stack_pools[pools_num] = pool; + + /* + * WRITE_ONCE pairs with potential concurrent read in + * depot_fetch_stack. + */ + WRITE_ONCE(pools_num, pools_num + 1); +} + /* Keeps the preallocated memory to be used for a new stack depot pool. */ static void depot_keep_new_pool(void **prealloc) { @@ -237,7 +272,7 @@ static void depot_keep_new_pool(void **prealloc) * Use the preallocated memory for the new pool * as long as we do not exceed the maximum number of pools. */ - if (pool_index + 1 < DEPOT_MAX_POOLS) { + if (pools_num < DEPOT_MAX_POOLS) { new_pool = *prealloc; *prealloc = NULL; } @@ -252,45 +287,42 @@ static void depot_keep_new_pool(void **prealloc) } /* Updates refences to the current and the next stack depot pools. */ -static bool depot_update_pools(size_t required_size, void **prealloc) +static bool depot_update_pools(void **prealloc) { - /* Check if there is not enough space in the current pool. */ - if (unlikely(pool_offset + required_size > DEPOT_POOL_SIZE)) { - /* Bail out if we reached the pool limit. */ - if (unlikely(pool_index + 1 >= DEPOT_MAX_POOLS)) { - WARN_ONCE(1, "Stack depot reached limit capacity"); - return false; - } + /* Check if we still have objects in the freelist. */ + if (next_stack) + goto out_keep_prealloc; - /* - * Move on to the new pool. - * WRITE_ONCE pairs with potential concurrent read in - * stack_depot_fetch. - */ - WRITE_ONCE(pool_index, pool_index + 1); - stack_pools[pool_index] = new_pool; + /* Check if we have a new pool saved and use it. */ + if (new_pool) { + depot_init_pool(new_pool); new_pool = NULL; - pool_offset = 0; - /* - * If the maximum number of pools is not reached, take note - * that yet another new pool needs to be allocated. - * smp_store_release pairs with smp_load_acquire in - * stack_depot_save. - */ - if (pool_index + 1 < DEPOT_MAX_POOLS) + /* Take note that we might need a new new_pool. */ + if (pools_num < DEPOT_MAX_POOLS) smp_store_release(&new_pool_required, 1); + + /* Try keeping the preallocated memory for new_pool. */ + goto out_keep_prealloc; + } + + /* Bail out if we reached the pool limit. */ + if (unlikely(pools_num >= DEPOT_MAX_POOLS)) { + WARN_ONCE(1, "Stack depot reached limit capacity"); + return false; } - /* Check if the current pool is not yet allocated. */ - if (*prealloc && stack_pools[pool_index] == NULL) { - /* Use the preallocated memory for the current pool. */ - stack_pools[pool_index] = *prealloc; + /* Check if we have preallocated memory and use it. */ + if (*prealloc) { + depot_init_pool(*prealloc); *prealloc = NULL; return true; } - /* Otherwise, try using the preallocated memory for a new pool. */ + return false; + +out_keep_prealloc: + /* Keep the preallocated memory for a new pool if required. */ if (*prealloc) depot_keep_new_pool(prealloc); return true; @@ -301,35 +333,35 @@ static struct stack_record * depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) { struct stack_record *stack; - size_t required_size = DEPOT_STACK_RECORD_SIZE; /* Update current and new pools if required and possible. */ - if (!depot_update_pools(required_size, prealloc)) + if (!depot_update_pools(prealloc)) return NULL; - /* Check if we have a pool to save the stack trace. */ - if (stack_pools[pool_index] == NULL) + /* Check if we have a stack record to save the stack trace. */ + stack = next_stack; + if (!stack) return NULL; + /* Advance the freelist. */ + next_stack = stack->next; + /* Limit number of saved frames to DEPOT_STACK_MAX_FRAMES. */ if (size > DEPOT_STACK_MAX_FRAMES) size = DEPOT_STACK_MAX_FRAMES; /* Save the stack trace. */ - stack = stack_pools[pool_index] + pool_offset; + stack->next = NULL; stack->hash = hash; stack->size = size; - stack->handle.pool_index = pool_index; - stack->handle.offset = pool_offset >> DEPOT_STACK_ALIGN; - stack->handle.extra = 0; + /* stack->handle is already filled in by depot_init_pool. */ memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); - pool_offset += required_size; /* * Let KMSAN know the stored stack record is initialized. This shall * prevent false positive reports if instrumented code accesses it. */ - kmsan_unpoison_memory(stack, required_size); + kmsan_unpoison_memory(stack, DEPOT_STACK_RECORD_SIZE); return stack; } @@ -339,16 +371,16 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) union handle_parts parts = { .handle = handle }; /* * READ_ONCE pairs with potential concurrent write in - * depot_update_pools. + * depot_init_pool. */ - int pool_index_cached = READ_ONCE(pool_index); + int pools_num_cached = READ_ONCE(pools_num); void *pool; size_t offset = parts.offset << DEPOT_STACK_ALIGN; struct stack_record *stack; - if (parts.pool_index > pool_index_cached) { + if (parts.pool_index > pools_num_cached) { WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", - parts.pool_index, pool_index_cached, handle); + parts.pool_index, pools_num_cached, handle); return NULL; } From patchwork Tue Aug 29 17:11:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8434CC83F12 for ; Tue, 29 Aug 2023 17:12:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 399E08E0038; Tue, 29 Aug 2023 13:12:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 319378E0036; Tue, 29 Aug 2023 13:12:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D1D58E0038; Tue, 29 Aug 2023 13:12:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DFF7C8E0036 for ; Tue, 29 Aug 2023 13:12:45 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BE4EB1A01A7 for ; Tue, 29 Aug 2023 17:12:45 +0000 (UTC) X-FDA: 81177786690.15.34D383A Received: from out-247.mta1.migadu.com (out-247.mta1.migadu.com [95.215.58.247]) by imf26.hostedemail.com (Postfix) with ESMTP id 05DBD140007 for ; Tue, 29 Aug 2023 17:12:43 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=W3s6IJQi; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf26.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.247 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329164; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VaH0nAjv38F7wcUNXUY6R+pPhb94jTIKfgVe1zCE0Y8=; b=BZHivEdIXBwOAriCKhEe1HKSA0KWEU4Hz52nNmLeChwZwTTpm1b1O5MzaRMTn7rp84YYAo 7F3+Y0waufgH/TwLyL00mtCcevgsNlRCUKgav6EQNK5+Gxz77yDjFQFU5ZSxsL4lbz3pVu DbEKHNKuZ1JjkGqTt4PnciIz/sbWTSk= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=W3s6IJQi; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf26.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.247 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329164; a=rsa-sha256; cv=none; b=DG8OBAjXFiH1ol5LSkJ3An5aGWuxaeGFv3C1yqCn6W6iJzHj3MtJPThwBuZ3OR7IPF3Xdn RnNE/E3mQ5dIxbBXNQyZxBrB/ht9jcOxxQh8aTF6N1/i5oQ2CDw5PbQK0Mik07PNhSwjjU LklK7CPJ56ODda2CKQ7FM6Tfwjd+SWo= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VaH0nAjv38F7wcUNXUY6R+pPhb94jTIKfgVe1zCE0Y8=; b=W3s6IJQiOTNsLFWQX22lPhTUoCW/6QB06+fYLBi4rvyMjQlY8pZ19z1qHG5iTI4lSOXxc8 JzhL/UMWrVr1qyOR/ywQgLHEg1XcWVwHXHXc51/PGHp/Ttkp9aeef95fXkmWHkZlAw9eZI YwpKJwq1b7mI2IEYsDbLMikoFrHR7d4= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 11/15] stackdepot: use read/write lock Date: Tue, 29 Aug 2023 19:11:21 +0200 Message-Id: <6db160185d3bd9b3312da4ccc073adcdac58709e.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 05DBD140007 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: cthm8a8zsf7ey869bw6m6wnha9ebe8xc X-HE-Tag: 1693329163-776610 X-HE-Meta: U2FsdGVkX1/+DLFhARLLTevJC8CJfwLcJ4lTS6KHJIDslySZy4UXB5Do+bOFTb5V8YBdpljBxgO/GR+WW3od5AYDxLyjH7QaS1YvjH4Ueinmj0Uw2ISioExvVRKlsY0DhYnY7v6qZZON09QnU4osSjRLbpfjVgwpvq7bAz5fRqOD/JmuB3vyFr/VFAZ3A6HZljxyCDTMI/djxYHIjhEhYfMwvLoUaPu9tJpIwCoDH9EbXCw65aelBlNJvrjnuIGugj1zmk2uvZma+5jftWnoC75eiDVpuTrEVbKYi/JfTUb5DsUCSRfwUcuF9S9FVfJ1US2huoWUXwr/krh6EIcxLk/tX44k11OZnFRi+LxtG6qyck069evZWkKPvuc0po/tQ90nA36RJ06rk2qlFhprnWB5ohUptZ4G0c2Wh+BwMkDgy+dBmgGhxVCgCx3NTKWnKTkzT5oGd2fYe/t92tuiCaKPPOz+ZQDWo3/ITq87MKGcmwouIkC7V7zCYiH1ADeOtmDsAT3SKYOyrIt6zVgCG5FyfvWZrCh/wiQKQEL8Z449tCsPTkJFfh6nSUniLuLr31VZQa2nBC56rlPQuVWGEcCh4oO0/9R6PyNw6mgeay978INHP9hjWdJi1PMwEG0E3rzPptTDYobcg68VFXlg/9y4UuuPpmEdHLurSvis/XifkxukrEvTMVGmlNGU3DskobLzdiaba3Nfdloho6XlfNcjwTm8UYkHHH7BVYqw6JgfSB+U/Q9Ob+Fp3kGS3nkeqqK+d5kVFkykyjaQnKNqr3b7CyTTjJqUdADk4pbz8MN1v4012diAJqeNgbXz03P2DUAw5GlQJUJB0XyAzuIEJXib71AKPw12hhKAB+bcBAOyqPMnhJ/XpWjJm/0oab9hFuVMMptM8BxIjAS4w+nvbc0VU/M/jrHZuuxCXngU+DJDgoJOop8I/LsD66H1q2wG47vYcHYFsSTNJMy6qSo qOmmxyJj UPQPGsuctNdEK1d+wy94r7UGuyeWSKOeWXxxy/3hgNapnHqtnvgQJtmwF3jRk7K/ykRxMcQ9fkIfTwWvgJhfV0EOkWkP1fcQOsnZo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Currently, stack depot uses the following locking scheme: 1. Lock-free accesses when looking up a stack record, which allows to have multiple users to look up records in parallel; 2. Spinlock for protecting the stack depot pools and the hash table when adding a new record. For implementing the eviction of stack traces from stack depot, the lock-free approach is not going to work anymore, as we will need to be able to also remove records from the hash table. Convert the spinlock into a read/write lock, and drop the atomic accesses, as they are no longer required. Looking up stack traces is now protected by the read lock and adding new records - by the write lock. One of the following patches will add a new function for evicting stack records, which will be protected by the write lock as well. With this change, multiple users can still look up records in parallel. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 76 ++++++++++++++++++++++-------------------------- 1 file changed, 35 insertions(+), 41 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 9011f4adcf20..5ad454367379 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -92,15 +93,15 @@ static void *new_pool; static int pools_num; /* Next stack in the freelist of stack records within stack_pools. */ static struct stack_record *next_stack; -/* Lock that protects the variables above. */ -static DEFINE_RAW_SPINLOCK(pool_lock); /* * Stack depot tries to keep an extra pool allocated even before it runs out * of space in the currently used pool. This flag marks whether this extra pool * needs to be allocated. It has the value 0 when either an extra pool is not * yet allocated or if the limit on the number of pools is reached. */ -static int new_pool_required = 1; +static bool new_pool_required = true; +/* Lock that protects the variables above. */ +static DEFINE_RWLOCK(pool_rwlock); static int __init disable_stack_depot(char *str) { @@ -248,12 +249,7 @@ static void depot_init_pool(void *pool) /* Save reference to the pool to be used by depot_fetch_stack. */ stack_pools[pools_num] = pool; - - /* - * WRITE_ONCE pairs with potential concurrent read in - * depot_fetch_stack. - */ - WRITE_ONCE(pools_num, pools_num + 1); + pools_num++; } /* Keeps the preallocated memory to be used for a new stack depot pool. */ @@ -262,10 +258,8 @@ static void depot_keep_new_pool(void **prealloc) /* * If a new pool is already saved or the maximum number of * pools is reached, do not use the preallocated memory. - * READ_ONCE is only used to mark the variable as atomic, - * there are no concurrent writes. */ - if (!READ_ONCE(new_pool_required)) + if (!new_pool_required) return; /* @@ -281,9 +275,8 @@ static void depot_keep_new_pool(void **prealloc) * At this point, either a new pool is kept or the maximum * number of pools is reached. In either case, take note that * keeping another pool is not required. - * smp_store_release pairs with smp_load_acquire in stack_depot_save. */ - smp_store_release(&new_pool_required, 0); + new_pool_required = false; } /* Updates refences to the current and the next stack depot pools. */ @@ -300,7 +293,7 @@ static bool depot_update_pools(void **prealloc) /* Take note that we might need a new new_pool. */ if (pools_num < DEPOT_MAX_POOLS) - smp_store_release(&new_pool_required, 1); + new_pool_required = true; /* Try keeping the preallocated memory for new_pool. */ goto out_keep_prealloc; @@ -369,18 +362,13 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) { union handle_parts parts = { .handle = handle }; - /* - * READ_ONCE pairs with potential concurrent write in - * depot_init_pool. - */ - int pools_num_cached = READ_ONCE(pools_num); void *pool; size_t offset = parts.offset << DEPOT_STACK_ALIGN; struct stack_record *stack; - if (parts.pool_index > pools_num_cached) { + if (parts.pool_index > pools_num) { WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", - parts.pool_index, pools_num_cached, handle); + parts.pool_index, pools_num, handle); return NULL; } @@ -439,6 +427,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; + bool need_alloc = false; unsigned long flags; u32 hash; @@ -458,22 +447,26 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, hash = hash_stack(entries, nr_entries); bucket = &stack_table[hash & stack_hash_mask]; - /* - * Fast path: look the stack trace up without locking. - * smp_load_acquire pairs with smp_store_release to |bucket| below. - */ - found = find_stack(smp_load_acquire(bucket), entries, nr_entries, hash); - if (found) + read_lock_irqsave(&pool_rwlock, flags); + + /* Fast path: look the stack trace up without full locking. */ + found = find_stack(*bucket, entries, nr_entries, hash); + if (found) { + read_unlock_irqrestore(&pool_rwlock, flags); goto exit; + } + + /* Take note if another stack pool needs to be allocated. */ + if (new_pool_required) + need_alloc = true; + + read_unlock_irqrestore(&pool_rwlock, flags); /* - * Check if another stack pool needs to be allocated. If so, allocate - * the memory now: we won't be able to do that under the lock. - * - * smp_load_acquire pairs with smp_store_release - * in depot_update_pools and depot_keep_new_pool. + * Allocate memory for a new pool if required now: + * we won't be able to do that under the lock. */ - if (unlikely(can_alloc && smp_load_acquire(&new_pool_required))) { + if (unlikely(can_alloc && need_alloc)) { /* * Zero out zone modifiers, as we don't have specific zone * requirements. Keep the flags related to allocation in atomic @@ -487,7 +480,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, prealloc = page_address(page); } - raw_spin_lock_irqsave(&pool_lock, flags); + write_lock_irqsave(&pool_rwlock, flags); found = find_stack(*bucket, entries, nr_entries, hash); if (!found) { @@ -496,11 +489,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, if (new) { new->next = *bucket; - /* - * smp_store_release pairs with smp_load_acquire - * from |bucket| above. - */ - smp_store_release(bucket, new); + *bucket = new; found = new; } } else if (prealloc) { @@ -511,7 +500,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, depot_keep_new_pool(&prealloc); } - raw_spin_unlock_irqrestore(&pool_lock, flags); + write_unlock_irqrestore(&pool_rwlock, flags); exit: if (prealloc) { /* Stack depot didn't use this memory, free it. */ @@ -535,6 +524,7 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, unsigned long **entries) { struct stack_record *stack; + unsigned long flags; *entries = NULL; /* @@ -546,8 +536,12 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, if (!handle || stack_depot_disabled) return 0; + read_lock_irqsave(&pool_rwlock, flags); + stack = depot_fetch_stack(handle); + read_unlock_irqrestore(&pool_rwlock, flags); + *entries = stack->entries; return stack->size; } From patchwork Tue Aug 29 17:11:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369246 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F032C83F1D for ; Tue, 29 Aug 2023 17:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED9118E0039; Tue, 29 Aug 2023 13:13:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E61D08E0036; Tue, 29 Aug 2023 13:13:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D03588E0039; Tue, 29 Aug 2023 13:13:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BBE5E8E0036 for ; Tue, 29 Aug 2023 13:13:46 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 99ABD160199 for ; Tue, 29 Aug 2023 17:13:46 +0000 (UTC) X-FDA: 81177789252.26.11F1176 Received: from out-248.mta1.migadu.com (out-248.mta1.migadu.com [95.215.58.248]) by imf13.hostedemail.com (Postfix) with ESMTP id DE80F20008 for ; Tue, 29 Aug 2023 17:13:44 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=h6llS9CH; spf=pass (imf13.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.248 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SXcX5gR3dSvkxNFtne9AIJ3MreGstDXokl9zt3D1S94=; b=GKO/u6kxhtIWVnZZmCJL9fCdvSeg/8dC3SE/v352yxJ0pavNqk9zjpMz3iegs3JnDzz6Fy ETGKs/Q0JbkRK/2a0sgFyhZGHza+bm+dl/T50DQn3v+20DWFVSHqj79XeLI66okYYuNgzY FjB4aB/xj0S+HPPxl5BTBN4NK0gsqd0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329225; a=rsa-sha256; cv=none; b=jPxWSAEJkspct1lYXuHERG+dHg65LOJ9UVpiMt8esFfTbMySJP151hZXR7OjnDSLgFqc1D Zgd67ue9feo8JQEY9gqovghnAc6zQMYtQjXGNYpNuKsqcJxW/LBBHKaSSpncu1YV1p2/Ob qLFMj58hyxcgIstP6Cm7lO6sTOE2F14= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=h6llS9CH; spf=pass (imf13.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.248 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SXcX5gR3dSvkxNFtne9AIJ3MreGstDXokl9zt3D1S94=; b=h6llS9CHe5RoqXvZREMQb8iyAp/Kb6KqjJZ0snmlmlR4rFM6dED6F0JcGGQU+5vqbXo8HX elNoG7XJ6jnDUln2rKKJ4l8c8KxNHcrBz76e+Dr/1x/lSC9m6R4ZN8FJDGgM6G/w4Breap 7U2QWpyTEl7oegpOYYamAOWgDo9QJW0= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 12/15] stackdepot: add refcount for records Date: Tue, 29 Aug 2023 19:11:22 +0200 Message-Id: <306aeddcd3c01f432d308043c382669e5f63b395.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: DE80F20008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 6sffeoauhssd1m987t355ww1xottusrz X-HE-Tag: 1693329224-832451 X-HE-Meta: U2FsdGVkX1+KD4tCEIlJmJXskov9BMgH5SblgAZfc4iPmNFBQERXRH+WAwLG14EtxnynTQFvD/zvIgyTDszb0/ZICTZWMTM4eNTiERseOFw3/9z7zmTVjdoFBpKYMWpdA1exqhScnFWJcXg4MrRhipQfVV59RcKa7sQsiIaW/1SI5n67AIBK6lcQlU16RwsiRGZ2Jz9wBWGPAOnhGxRWjPUtW5PBCSqEzQcwsuD6H/P6VzGr9dX+7y1dYLjLnKzYwi46gmwf+g8EepHi5TsiTprI2LjSYrrRIaYXqCS0GqbuEzB0tH6uRCAv+Yzpaun6Ldw8am/JNX7Pp/bzaVKmd4NrpkPBfFTGOS6L9gPPfzaMD9TPtQsNXWh+15hO6kFj15UULTv8cPq7YQkvPfjFQxUZinNOEAJSAvvCnhBxVY28xR1wn+jWfyIRR/Wksgv9AfwKFBuu0q4to+NngJV0Hqcg78Xn/kbO/x8ztY2JNiZCMkXX41b3F+eBgZH0nBlPSxdHv0gaYSaCUdzMR4dhsGMw+W5cqr5UIZ5FYoqg+lspJeXAsyvnSiB27yBqH9f2OYv2SRycsz7sk5QDJNFZrtmcXqKkt3e1whQkzcE8xyXKnHbl10RfwSB3ur4pya1dLZqRXe2l8UnKlrn1A/n8jHKylnoX+i2W12Is1P2s6gob8dCMhbklNY9S+sD4DrfPTcNeBfHRiDsg8srEijTpaCgpZYISjaUBHZFaQkCzflFgD3bwr4RsNCR8hRigGz0Y16/h8q9jZSAEfa7SjSku0kdeRh1IpFhj5/x42Fq4GTT6t+ndl2VrWvuGN9MvA+cEcqbxpf8XLqbsm4r+i7R0/uOkhzW5HPIv8Y7M59NlBf86aHayWM6TG0OCiv4SnZuTvD2hIJn7a++5VRNh4XQA6Hxw8fB07izsWbnzmr1hDRC6OoVeD+5tcha/QMtSZpc67CEECxxWfjirHgHx6f4 anrFVllB 7nx8kaWT6oMoXQkC2JVsREB46bREXZW6QyR/pQ8qraSF/w/JDyTUYmUWwAlab2wm0mFdzKJmVzRQ1We6dGZSXPc5G2ii/kCiLOzRWyVjJZL0xD7v8vgjsBAyf9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Add a reference counter for how many times a stack records has been added to stack depot. Do no yet decrement the refcount, this is implemented in one of the following patches. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 5ad454367379..a84c0debbb9e 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -60,6 +61,7 @@ struct stack_record { u32 hash; /* Hash in hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; + refcount_t count; unsigned long entries[DEPOT_STACK_MAX_FRAMES]; /* Frames */ }; @@ -348,6 +350,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) stack->hash = hash; stack->size = size; /* stack->handle is already filled in by depot_init_pool. */ + refcount_set(&stack->count, 1); memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); /* @@ -452,6 +455,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, /* Fast path: look the stack trace up without full locking. */ found = find_stack(*bucket, entries, nr_entries, hash); if (found) { + refcount_inc(&found->count); read_unlock_irqrestore(&pool_rwlock, flags); goto exit; } From patchwork Tue Aug 29 17:11:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BEBFC83F12 for ; Tue, 29 Aug 2023 17:13:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA10B8E003A; Tue, 29 Aug 2023 13:13:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2B0A8E0036; Tue, 29 Aug 2023 13:13:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A455B8E003A; Tue, 29 Aug 2023 13:13:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 86D258E0036 for ; Tue, 29 Aug 2023 13:13:47 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 64FCE40241 for ; Tue, 29 Aug 2023 17:13:47 +0000 (UTC) X-FDA: 81177789294.16.2929464 Received: from out-248.mta1.migadu.com (out-248.mta1.migadu.com [95.215.58.248]) by imf28.hostedemail.com (Postfix) with ESMTP id 8B8C7C0019 for ; Tue, 29 Aug 2023 17:13:45 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=PT1yVeNi; spf=pass (imf28.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.248 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6r+jJInS3pVfd7hB9KGynpwXDuP9tHCSgnjNZ7TlokM=; b=eB6kv/u5rODTWPhyevob++wTXWPJINrrUNwmPclgn/Gu6a9uJbB6nkS+h0W6UfuM+4iGCc cnBqoLo3L76HmMR2rOYs8lRrg7E1/HsnhRwh4E+R//GQCJwvVnCWDYY4jIA/MbwpV3I0Zq bJMA1Wj8KC1hag+iqsLOC+Gg66i/OeQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329225; a=rsa-sha256; cv=none; b=RHdiWkpwmAf/YOeQI8Ui4gLepBQShYZ2PyyYoIG7VGnFKjdyQ/Wh3t4Lvh51H6A89+NHIg HwLvB5wDsnKdHt5i5XYutVlJnkNX1KS6Cja31GCNbDidtOfMHhMkXq8i4ZMBKkrbL+NBXr rkQ9W7JCFCDiGVU4uV/nb3uIeBp95Lg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=PT1yVeNi; spf=pass (imf28.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.248 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6r+jJInS3pVfd7hB9KGynpwXDuP9tHCSgnjNZ7TlokM=; b=PT1yVeNi/WFW0iBcrm2Hn4oAVO5Mp4hu85tRrRP1HYBbHSPjNipz67GtVxzpFdOYzHKM/M EOna2VPwPAI1BS2TmFOrLFsWZmPHYOllcK3aTIE2DJg0Lqwv7JOqQ+rn0o60sbtux81qug hC+Sq5lFDkk29LN4JU6YC08g6eDKk+8= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 13/15] stackdepot: add backwards links to hash table buckets Date: Tue, 29 Aug 2023 19:11:23 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8B8C7C0019 X-Rspam-User: X-Stat-Signature: 3ksgobeqr48846xudx5qnbyy88fjcqeh X-Rspamd-Server: rspam03 X-HE-Tag: 1693329225-788023 X-HE-Meta: U2FsdGVkX1+GlVlmbeO1TPiVRdddH7eDeOz8ttl5FL0TVvn+n9gRmhsdYjZfjKa9v44ml1FLCQDlHex5E2ul92Gra5oRSTSW23HNGeQcWAu3Joo4jr/LqIMNAK+Bd8nRaSQ6wp0W2OQBCJ4KReZgxoASDjw13pW2uISuVHnpDw6miVz5rd16Ea6CNw8vquNJBPKEgbOiuYcP8uEn7xi7QyhW3g7Wzye5uNkfEzTW87Ngm1EXkBVL/GBmzL7EE0HuItTjQDymaNGuv5ip+0JuMZEkXDAJCetOnp2yD0YNXWPM+IiMHP7keISwG1q7okYzVKpgWk/OdX1ipzPkQJOf20jNG+MwDHKAqF4K1GK/TrW/hYIi6iCPvRMXVvOd9LV0Z+j8mDIEPUaB0A9SL1cPmCATDBj2mxPLO9wx3VBVATJIjUiZm7P/F8zxTqpYXeAsVWRyaZh2+00sYProRqdoZAHUZGJrWP7uyiJKjWaQAcjZJO67Xi62u6qG3NsHOge/YMIpxzAcHtPL/TmPiynqEL7eNmn9PXsHR0UOkT8/nd9wpP/u/QchtgAB2CPmMqSQNyibyteEFnDGFKe72AXYG5uEu++b1USNDTYvG6X82Jh/VuKbuVIv/050BG+OewyMNWBpR1rC+MGcv7CzLA0sUfPdnbbyTb7/HGH/eMB5ASlhFEWnsf8GcDpdsDqCKA2lfIabSUb93YDutfTfSzvVCgjwl9kMLKxHVAJ8nsB2ApAVXiNTqbfXw8C++OM98ns3ATIRfHgb/USwXjwJZHMqTdrRSfCPpEcfCQjTpjQ9fpzyUHC5y69tfnhljKarLy42Wg3rbaN8VXpwGwLrThW1BEiOND0OdeW/BOIpPdIY9LcICJVCe6+Zz8XW54qgY4jkSQ/JD6WK6wGUd0fE4fk7+cA6ZZrMOgjn1WpZwNomgHcOjCKtqczZLqdCbtbZEW4zw3S/584Hk9+sF0VnL4g uR6cQqmP vsj7J8T9VemgiNEYmMCmrX1kXZDOGuK1TUu26VogZoEFQxUWf6RlNNDvpJmwSv/ZXwk8AuCO1U52Ujc9KfYn/QLoe3DARcU7H0MSq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Maintain links in the stack records to previous entries within the hash table buckets. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index a84c0debbb9e..641db97d8c7c 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -58,6 +58,7 @@ union handle_parts { struct stack_record { struct stack_record *next; /* Link in hash table or freelist */ + struct stack_record *prev; /* Link in hash table */ u32 hash; /* Hash in hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; @@ -493,6 +494,9 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, if (new) { new->next = *bucket; + new->prev = NULL; + if (*bucket) + (*bucket)->prev = new; *bucket = new; found = new; } From patchwork Tue Aug 29 17:11:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EF5CC83F1E for ; Tue, 29 Aug 2023 17:13:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A16E8E0036; Tue, 29 Aug 2023 13:13:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 429F18E003B; Tue, 29 Aug 2023 13:13:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A4878E0036; Tue, 29 Aug 2023 13:13:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EFA498E003B for ; Tue, 29 Aug 2023 13:13:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CD965140260 for ; Tue, 29 Aug 2023 17:13:47 +0000 (UTC) X-FDA: 81177789294.09.945D546 Received: from out-252.mta1.migadu.com (out-252.mta1.migadu.com [95.215.58.252]) by imf14.hostedemail.com (Postfix) with ESMTP id 22B6F100039 for ; Tue, 29 Aug 2023 17:13:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=T1iG7qQW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf14.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.252 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c+OqyPrsOPGHg5XrLPNZBNE7kZ+J8BEkGBD6zb/JuYQ=; b=TZ5YbBhYXVZyq9KoRd5bNLQg5FzQhGgCMmp5RddvokH0I5L5TXJ71oiTPv9qhdAWTR3FeG ZyLUtgbwSXoI7HTeU2ypskm5erAut8/Zkt3lO73yMe9EQtj4p2rgfjtPdP7WTKSH9KWyyX 49cY0Rkfv6sK1j8DQqqZXnMSNTrYSpM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=T1iG7qQW; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf14.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.252 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329226; a=rsa-sha256; cv=none; b=jcxcX9909/AjIxgv0sK6gD58gPFt6wmJJwqCjK1rSUd9Il34B+w/cxZVd2Xjm7Fatqd6Ge lSlV5/XauWigF8NEFayLI4xU68qjqfIEQlxPEzwUfb/vwyO09lm3J9nRY2TFTozl1XBkZ/ iAPNNve3tq3//NQ1ozheF9+uFAX1ejE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c+OqyPrsOPGHg5XrLPNZBNE7kZ+J8BEkGBD6zb/JuYQ=; b=T1iG7qQWFqXknvW3oxV4ppqhlGJ2B4GzTHDs5oEMMGMfeCMjQjH9IJfBalbYfv0NVvUDzy EwkpyMnG6Nxt+AMl/9tnMXZ4f/nw+qGm5hpPJdpaKfCfaWqaErDfKrIusJZnN6iT+BpvCJ ObS/4jb3NCEUDuj8JPJ9Tc91Spi4dxo= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 14/15] stackdepot: allow users to evict stack traces Date: Tue, 29 Aug 2023 19:11:24 +0200 Message-Id: <99cd7ac4a312e86c768b933332364272b9e3fb40.1693328501.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 22B6F100039 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: hsqwoum64ou764gjki6fac56yf91u8qj X-HE-Tag: 1693329225-558056 X-HE-Meta: U2FsdGVkX18Xh6d6N6GAcV0Sq+bCyRbWFucXxNzLj/UJzNMrZiQA3A6uswm8DHDXiBjiCSTNXPRnR4Mz8CCqRN1YEDAk+UMsXIrVou7N0ssIJUGaqovz6Hv1cJHL6V0o7sAkE3gMN65E1NdekncnyAuuJoawCjmW1e2Jl6HBpPDIenGzqjgWViSSKVE8bz+6Tj6ulEQVdqhvckl1GiJ5sPoVfwhyQM02bV8bmieBnznWyWcU4ukbaOPXp1sUBhqQN1V1GIQIvK+PGFfqhvgtAA9FWxPip6OiFYvy58Lbwp+wTojrlSSV1h2NE2lkSSnAnzareuIGH6MSBuJT24q8feikHlBvy0WDimhCEADEZAia9UwgpimkM0UAkpsNNvo7V7T6e7icmNrhQC6/FzE8OnNff7mjHZ/V4rxKd78ZZCgnAMJ8SkZ5iYVPLs7nw+UXc2scWjN8CRoRUZbG/QxmT7Czrc8qqhVhsqZoL3mjy+cAwSgLay6KxuUX3wBzL51lQjAd0rajgRdfm+Jk8zcKvNtZba3UEfbZM1VXbQJwFAjBXW5mmVNLzh2viPZoDRrLoURR7GvFUWMuuan6xv1+jbX5QXM2Jc6RoS21ZKNNcvW9Xl5IunhGoay2avVHiNLzG13Su6p4XAUMbv92bC4gEWrcnxEzlcL8Qj4duHQGNteL74aGVvGKv6D3HEOqOVCu5xlJAxAmbtUOadANF2NkFptzp29lQggnLGxdRbWM9LWjDpPCVDDOaBJsxO2SHrO+S4WcHMTv8oFMLCswU+YQtp18NUidxV5iC/bjpszNLV8xRvMj1NUkKmV54IvXUnJxEV7XhOz9k6SIJAPWavhpk3QHAUgaQibRLVMtaLygYGJLpFwl4RPJ3wPWQCtMNSXDZuqsIhfz3kg1DdRYr+AQJrakuvIt+geuI2xe65gq2AqPzprPV5kVpGRF/oIzOocChN9i7qI5DJUOiLhuxCi EeyWY0F+ VJpRF5to4bvk19zGsHOgdHrC3qjLuRbdfTjikPPnyDjYxRRg0/di5Ku9hksSFmAad+GqsyVJzjvFcnFBxC1XAvxCyGkb8ST+I2sR7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Add stack_depot_evict, a function that decrements a reference counter on a stack record and removes it from the stack depot once the counter reaches 0. Internally, when removing a stack record, the function unlinks it from the hash table bucket and returns to the freelist. With this change, the users of stack depot can call stack_depot_evict when keeping a stack trace in the stack depot is not needed anymore. This allows avoiding polluting the stack depot with irrelevant stack traces and thus have more space to store the relevant ones before the stack depot reaches its capacity. Signed-off-by: Andrey Konovalov --- include/linux/stackdepot.h | 11 ++++++++++ lib/stackdepot.c | 43 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index e58306783d8e..b14da6797714 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -121,6 +121,17 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries, unsigned int stack_depot_fetch(depot_stack_handle_t handle, unsigned long **entries); +/** + * stack_depot_evict - Drop a reference to a stack trace from stack depot + * + * @handle: Stack depot handle returned from stack_depot_save() + * + * The stack trace gets fully removed from stack depot once all references + * to it has been dropped (once the number of stack_depot_evict calls matches + * the number of stack_depot_save calls for this stack trace). + */ +void stack_depot_evict(depot_stack_handle_t handle); + /** * stack_depot_print - Print a stack trace from stack depot * diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 641db97d8c7c..cf28720b842d 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -384,6 +384,13 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) return stack; } +/* Frees stack into the freelist. */ +static void depot_free_stack(struct stack_record *stack) +{ + stack->next = next_stack; + next_stack = stack; +} + /* Calculates the hash for a stack. */ static inline u32 hash_stack(unsigned long *entries, unsigned int size) { @@ -555,6 +562,42 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, } EXPORT_SYMBOL_GPL(stack_depot_fetch); +void stack_depot_evict(depot_stack_handle_t handle) +{ + struct stack_record *stack, **bucket; + unsigned long flags; + + if (!handle || stack_depot_disabled) + return; + + write_lock_irqsave(&pool_rwlock, flags); + + stack = depot_fetch_stack(handle); + if (WARN_ON(!stack)) + goto out; + + if (refcount_dec_and_test(&stack->count)) { + /* Drop stack from the hash table. */ + if (stack->next) + stack->next->prev = stack->prev; + if (stack->prev) + stack->prev->next = stack->next; + else { + bucket = &stack_table[stack->hash & stack_hash_mask]; + *bucket = stack->next; + } + stack->next = NULL; + stack->prev = NULL; + + /* Free stack. */ + depot_free_stack(stack); + } + +out: + write_unlock_irqrestore(&pool_rwlock, flags); +} +EXPORT_SYMBOL_GPL(stack_depot_evict); + void stack_depot_print(depot_stack_handle_t stack) { unsigned long *entries; From patchwork Tue Aug 29 17:11:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 13369249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDDD9C83F14 for ; Tue, 29 Aug 2023 17:13:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11E438E003C; Tue, 29 Aug 2023 13:13:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 07FD38E003B; Tue, 29 Aug 2023 13:13:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DECA88E003C; Tue, 29 Aug 2023 13:13:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C81688E003B for ; Tue, 29 Aug 2023 13:13:48 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8EB828018C for ; Tue, 29 Aug 2023 17:13:48 +0000 (UTC) X-FDA: 81177789336.04.D7E457D Received: from out-245.mta1.migadu.com (out-245.mta1.migadu.com [95.215.58.245]) by imf09.hostedemail.com (Postfix) with ESMTP id C5264140012 for ; Tue, 29 Aug 2023 17:13:46 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=GlaqijiJ; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf09.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.245 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693329227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nl/mi/sJhm8d2IDMyyx9SK432XHk6oEEQ1PTooRDAe8=; b=0kjPXHRGBUl3pnJDftxzJVRTFXpTmnk/ERo0gFhzsA0WhQRkX9avs2t3QgfaZEQ6qCi6QI BwVMtQWHV+XFMT/xnDqGbftY6SPlhniZpsffWJ0BdmikD1VspLLGDTsz7t0IbB73rKhJC+ HXAH6rcRAOMmjh36aStyzFI86U4kjOI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=GlaqijiJ; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf09.hostedemail.com: domain of andrey.konovalov@linux.dev designates 95.215.58.245 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693329227; a=rsa-sha256; cv=none; b=4zsFHPR8BauZzRT3G1NJoKlZ1OYCCCyok5lVlEqdMHp7A5pzrEkUC6kPqDlsWbqDmvIthS Ic/xBEweLdFZX3+/H+5K0oNvM2NL2RUBP1KdVM5V1qvzvHClJIOkuzDyNa6JJB/euKOfiX fWYfDbTa13IE8dcnneCbTDxi8vDz2zc= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1693329225; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nl/mi/sJhm8d2IDMyyx9SK432XHk6oEEQ1PTooRDAe8=; b=GlaqijiJCNuxG7rw/crBXwYIu+bsbRgtfNjI15SsNv8t1sbXPGyc+8Qf9Y8QvqLJiZuTi7 ZNtWdGIvQFFzknC6Cszo4DXXBT6fdVhnEoWLc28FUoQtxU/76o7hIYC2gdCczCKWh6/THq P/TGvd/MEaLMkz+3R7oGxuH+BvdITeQ= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 15/15] kasan: use stack_depot_evict for tag-based modes Date: Tue, 29 Aug 2023 19:11:25 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: C5264140012 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: r54gf56kx6uu8ujn1w4o9dikbm8jcpf9 X-HE-Tag: 1693329226-355505 X-HE-Meta: U2FsdGVkX19nxiT/bJ5OhUGa06YEjjlBskMJpw1Ijp/Xlka3rq/TIAwtgl9u81c3eygOWE0WKN+R2vqxriAuvhCgm2oL5UolFRzAz8pklntRFpH87CH9xnOkztO9jYDYPEE1Ispx3vhhsj9Js/iPe3FCDiw+omUTdCWexZ4VlYdFpip132ar4Nx6n6/WwokaA/AlzsiFjQly2p9VXW/4dHz6ZXHJcLjiC8xffxvxqvXJC0bD1pohPe+2lDY8o3OuJDyfO6sfjUA1PphtTzkpPS161+8jK43hBdH22jZcjl4Jchu+zlKeccSEU++L7g+/aUeTpl1DOyR/TuCPjD2rbdDRmxXK+/oA66dBtX3CZw7jXOjt2AP1PQN0NDa77gildcrfOFTRM9T5E8cMLo4+zu9SiygddaJjttMg4J61taYTmihj2cpUp4hi4jhLyj0bIj/uKzO7AwDDj2gM0tb6h8c8lIMyDE10Z9Dmbs7syWvKZ/JQ6S7lnnsVVnvNEprjB0pnT180waA8DfM5d77YvGFQjI1mq+R3DeHKsRvQr3lRCAzDgu8/ckx9365VPj2eUcNmaQGGPSSEaPtsEyr51dRb8RGhCK9PboX1FzvXyxSAB5VtAD+W37plUCMLasNRDhXW+j/RNEWZZk6/se6vZaoyK8q9/Ovz0xjtHqKCV9QrzCpRByV8Of2UBnY/Dfxdt3OtFfoa+HZjBzB0B0JpMwUXBaMX6HDmr9uthOA5gB2qkM6TtN+02P2qiErd45663MV5D9dN1mJpSEMkaNSY1Q+JT7YTcMISCpVUImXJHF7FcUoiqm8MbVRyqJ4iV7/K2JDDT99dLr/q965mpUBtOOQYXkbWkeTCQLsfXxN3i9zRNVoCq72SYbbk8ypQxsGjs3QvYluqcMyvnsuDSZWa0sY9b5K50h8kLgYSShJGckByHdkGyLw/plY40+cJg6zrHZfssyCQUPR//Ghm2yD e9sMPY+F xhBkpkAasemI0lPOZaUeNSYuaOSVpvs51jADza4ogaNiDDg0OWSA1zp1Z7D+kMKePB9/y+6AvCBzrir9lxQNKI/hUdSnOG6kFZrIs+1JhfXd4Ure5YvjN4PAFWweNujtqJZ/5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Evict stack traces from the stack depot for the tag-based KASAN modes once they are evicted from the stack ring. Signed-off-by: Andrey Konovalov --- mm/kasan/tags.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index 7dcfe341d48e..fa6b0f77a7dd 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -96,7 +96,7 @@ static void save_stack_info(struct kmem_cache *cache, void *object, gfp_t gfp_flags, bool is_free) { unsigned long flags; - depot_stack_handle_t stack; + depot_stack_handle_t stack, old_stack; u64 pos; struct kasan_stack_ring_entry *entry; void *old_ptr; @@ -120,6 +120,8 @@ static void save_stack_info(struct kmem_cache *cache, void *object, if (!try_cmpxchg(&entry->ptr, &old_ptr, STACK_RING_BUSY_PTR)) goto next; /* Busy slot. */ + old_stack = READ_ONCE(entry->stack); + WRITE_ONCE(entry->size, cache->object_size); WRITE_ONCE(entry->pid, current->pid); WRITE_ONCE(entry->stack, stack); @@ -131,6 +133,9 @@ static void save_stack_info(struct kmem_cache *cache, void *object, smp_store_release(&entry->ptr, (s64)object); read_unlock_irqrestore(&stack_ring.lock, flags); + + if (old_stack) + stack_depot_evict(old_stack); } void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)