From patchwork Mon Sep 2 04:41:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13786701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61731CA0ED3 for ; Mon, 2 Sep 2024 04:41:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C75018D006E; Mon, 2 Sep 2024 00:41:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C26A48D0065; Mon, 2 Sep 2024 00:41:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A28958D006E; Mon, 2 Sep 2024 00:41:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7DDCC8D0065 for ; Mon, 2 Sep 2024 00:41:37 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 27BEBC175B for ; Mon, 2 Sep 2024 04:41:37 +0000 (UTC) X-FDA: 82518549834.04.631634F Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 62C374000C for ; Mon, 2 Sep 2024 04:41:35 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LO2cvda6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3_kHVZgYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_kHVZgYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725252019; a=rsa-sha256; cv=none; b=ttq5qMzknG/FeMAw17gwQqan9I2xr3FXVT8/nN3Cn0ft/GYxaZDDULY2LAXan/oXq+o4NI iNzgFe7R7FEfXgNA1uJfre8ALObpZMmJ1AV1Bw6r93aj3yNHrgoOP1S6gCcdnzU4yQftX1 kpVhENYNyUH8JShLnowt5J6eGMjfRQM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LO2cvda6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3_kHVZgYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_kHVZgYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725252019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kLpU39ux3x2TTO/xeONsOF6qZRhVUZNcpjltFBicrmM=; b=mO0lTsJfDytIO8Ljg7NnQthyUtWiaGD090ey4jKjH7klb9jYhRUHoPJl0l0gF4JFhOphws aoIA4eC6wkOfBVaJwTEJZhBEafYxxAPZnvwGktd3IJGl05rjILii7lgCXjtYzSt+xNW/Nh G3+urVr4mRHiHzF7JE3+kYu1SayOvvY= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0353b731b8so6521035276.2 for ; Sun, 01 Sep 2024 21:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725252094; x=1725856894; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kLpU39ux3x2TTO/xeONsOF6qZRhVUZNcpjltFBicrmM=; b=LO2cvda6wAecAHAMNk0YzCiHvG7uRE9CwlzZo/c5boL1WQgpNwCJ3noebNFtFIT4XN dj3dpcK1xHPLScsFPXKcYN6RX98h5YZuIOemHWYr7BOBDsuBv2zA0oSBzN/qJKEpLb1J oD0z4Cb6enk4z3yVGMjET9NGQIv7ASdyIalqtDa/6fu6CdAZTdYfxbWAmDq0SzuHW9FQ q7g5PzNfDFoejm7L0oPlohWcmljEiuenxG8nhYTCaz+kJX7e/dsTXyf+osyQHrH27XDc EvfUNlYeMvlYevbRd/ZsDLXypY3MXp7+qtBCA2i41J0NqaeANCDSmBl/53gqPGVQGILU ifnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725252094; x=1725856894; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kLpU39ux3x2TTO/xeONsOF6qZRhVUZNcpjltFBicrmM=; b=R62swJIzplzenHgfFYqxs/+fXDBaUMEUsFd7KVQ5acskp/rV1GDCmrtgNjsMBuk8tE KbIIaA1W12+kwU3iHecAR81v2Gm7h/cWRIuKIhSKRmTJB8pCPyX4qHYiyPxz2SIXF5hS 1iG2Xe1iitmyRGf4jNQizfFYaETa2klHBY26Rp9mEqwtBPQr+jQZnkmxGE7YDsT9tzY1 Fm9B8txuO/vgp+bKSNiCeDOytXYZoCNKu2BsqNEMWPsFmfSorpeelFb2XbyutMpAKcFp u1eupzdyAKvfSDsoN9/0xF8pTNe5hsYsccoWuPp8W1NtSL8Aq6kkSUeTut2iCDDCJxaW pfdw== X-Forwarded-Encrypted: i=1; AJvYcCWtxvMCHQUjx0jcVucQkh9NO3WhQHNS5eDbamL/TQqEJESH5rd233Vove0sO5SOSCuNvo7RYnxAaw==@kvack.org X-Gm-Message-State: AOJu0YzyVTgXN0ZHhKjrlxbrdCdlheCsFxQfjg7BifIyMHjJfujWDXm/ GsbWi4l+M86b2nFMxL67IxIHHYnXdevtAVC4fTvY/HZbqSbE8hhGWWnl2lMsSe1wutyAyPkzCHb XtA== X-Google-Smtp-Source: AGHT+IE+2G0xxDfv2kbdFSiqwhGjdz1DeNRKuCcVpqvSyerK6sADotCLCA7OoS3t5ommCEFjIJVLeySgq3I= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:7343:ecd2:aed:5b8f]) (user=surenb job=sendgmr) by 2002:a25:83c4:0:b0:e03:3683:e67f with SMTP id 3f1490d57ef6-e1a79fed1aamr15821276.5.1725252094100; Sun, 01 Sep 2024 21:41:34 -0700 (PDT) Date: Sun, 1 Sep 2024 21:41:23 -0700 In-Reply-To: <20240902044128.664075-1-surenb@google.com> Mime-Version: 1.0 References: <20240902044128.664075-1-surenb@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240902044128.664075-2-surenb@google.com> Subject: [PATCH v2 1/6] maple_tree: add mas_for_each_rev() helper From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 62C374000C X-Stat-Signature: onq9sga3cffpkh5z8z8d874fcuqnypzg X-Rspam-User: X-HE-Tag: 1725252095-276518 X-HE-Meta: U2FsdGVkX18JWeODbnWb/sgzlbNSFwCksN2fCkaoxYN+0zPkt6flKBcW4tmM24Q/1wRWEviHT/MC+1OZVn0AVM04SsWm2uFAa3RVMz2VI72DA4kgfEfdOV+8R2lIJ9ZjaR4Vb04ZN59WZNCqRXKSmkfSdxp57h4hOSquCVpxKbwtxWiY8yHZjPRN+c8SCmna1v6QL9N/t6YYTij0CMGobCXE8j8O8uNQONnWoG9c92mFZulldRuaTjW6ZHtKxEX1c5UIMqPBuBMf1UXDOzv04qqnyabG2WxliZAkO+YnLgJL47x7mLzAOe3xI/WgPhjG26p2yQbai2x1hlg/H/4MvCFwXgb24jVB30165m6CCdYPcxY0YHUJUMSKxSXfgJucGbzNrZZYx9C9JxXNG4M8hdDCipFtgm2mbtiFPfRtY8vQ7XhyhPXQn+2vj2Yz/mVyASD9LhsQwjKP/j3Tu3Sf0ERTsuUWhfH76hgmTUPWiSlq5RHFpXeLyfTXiKMVIavRK74mTMTtBqh28oeV+5gAuCu8QDbUB7CgBAVbYdn32mTxQBxo4apnybjwL5yMKVAcJm80jLxZpr4pI2yevSl2YK1k099lsV67F1RwmcU1T8io1fCCNHNXvLBU5HhxFOBqZ6CCwV5iSKkLEF3J6P4wCT3PJGneZLGjcGe2gUzgS8mSqtu08rwAKChzN9F4o994NUaHMvwJyl7Z+lAPfOVCY5heo6mFojTvUma6hgPWgCweD9JLl6eJrxkgu/s3WQGaPse9ZUKnCJtzWD0LzDmmHIwVY7/3gztXhQ2zAoEaurHqBM+0v4y64TiArAVa82PInbDj6oVxTYDRFW9IlCHaNw8mDOA/2GYX0aaaRJO/K8nJmgf00Uz4T+rl0KroIplwvEKTRtKSUSlDcexmeygphNB+TWvTISkZ8vSd/1k6iGkITKI1FV8KH98wza5Mnll1PO3ja+c8RNGJTxJvmaC WtHwyMAF teKeE8DKaPfSfKU/aS7p+boafZ0Bm0JUOu5Ij8tEyESKRbAPo+Hv6cr1oTR0JC3QRDBn+LwrcGuGiSXdTCcOzgwB0xliCnWCc8NhIRG64/+ZMNXqG57CDmPV0+5+67IXjvdlQdupUx3V4UVJugSe/od4gqy92RmBG/uklWrF1c+YdAmFBVDmjAcRuycx/JxJaLvkFm215RcMbnfjWyWleY6NkhNN+AEl5GVuoFPSJC36UCw3g0fv0HuNwOviXq8LDjfC/mwthS46PuSrL10XTCd+MBPj4f0iLtmrnYo3nRP66VDTZUzq3dxeF191/DduKoyQA4XTI5N+9TAWfk2MauOacHAxFIe/SWdrZph3tFO7KGXCnagpbTsaf3ZMt2IkorUFB2Teh7ICAqGhPHKDgnGcQsV5P1HeFkF4iXzQ4wSJ3BsbhJZdVQ3epzzPHOAOfVB02dDl89TDw8KuLI0rAdbUkw4/aZnxAM+eC07jOWpY59cr63b/kxLuEe5YISQfCeELAl30Qj2YAUcbh3wAH+yBIySxvGWWGythijzQf7TGDs/aK1xlfzVtowxHZYfPdiEAI4euwyTj+nMlfSm/Cqm9oDg/eDqO8bGpY8XLvc4Z4Cgk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add mas_for_each_rev() function to iterate maple tree nodes in reverse order. Suggested-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/maple_tree.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 8e1504a81cd2..45e633806da2 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -592,6 +592,20 @@ static __always_inline void mas_reset(struct ma_state *mas) #define mas_for_each(__mas, __entry, __max) \ while (((__entry) = mas_find((__mas), (__max))) != NULL) +/** + * mas_for_each_rev() - Iterate over a range of the maple tree in reverse order. + * @__mas: Maple Tree operation state (maple_state) + * @__entry: Entry retrieved from the tree + * @__min: minimum index to retrieve from the tree + * + * When returned, mas->index and mas->last will hold the entire range for the + * entry. + * + * Note: may return the zero entry. + */ +#define mas_for_each_rev(__mas, __entry, __min) \ + while (((__entry) = mas_find_rev((__mas), (__min))) != NULL) + #ifdef CONFIG_DEBUG_MAPLE_TREE enum mt_dump_format { mt_dump_dec, From patchwork Mon Sep 2 04:41:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13786702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4133CD3427 for ; Mon, 2 Sep 2024 04:41:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23F3F8D006F; Mon, 2 Sep 2024 00:41:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 178618D0065; Mon, 2 Sep 2024 00:41:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6FB18D006F; Mon, 2 Sep 2024 00:41:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BC1BB8D0065 for ; Mon, 2 Sep 2024 00:41:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 65255AB0B2 for ; Mon, 2 Sep 2024 04:41:39 +0000 (UTC) X-FDA: 82518549918.11.056E239 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 93073140004 for ; Mon, 2 Sep 2024 04:41:37 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ldj2mbjp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3AELVZgYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3AELVZgYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725251993; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dp4HLFsSdaCRLqkUULTSVafVtTTe/LDKPcvkJXQF4Ug=; b=npcDQrBSsOEuY6XHmvDHit8TY0GiBJFkFXvt3UDAtDBu0ek6Xsegd4wUip0RPV0A5AoFUp zCepK/rUUqbZRWoR9KHpkhzIlkTgWqswmRwPM4GC/QYsvTLaCdSfSxZsLu1oqxc0Q7Isb4 iwOcHSyoIUY+xpIIGsNTlIlBlk1i69I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725251993; a=rsa-sha256; cv=none; b=YkeLiXGzBbhziY7mT5N9JFzTEefQpIakQWqeqUogztmHJRu7WDXhxk1myneqidFfAd0yq3 MmNFBR4qyQ8EcP2cd9NxRa3JJklg/ckEmDaNlmCaL+KAHTe/RpkYydwZQA27ZGYr7wjYcG MznrJ3NDS6iCliegB2POOKmdNGxZTq0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ldj2mbjp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3AELVZgYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3AELVZgYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6b41e02c293so79581327b3.0 for ; Sun, 01 Sep 2024 21:41:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725252096; x=1725856896; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dp4HLFsSdaCRLqkUULTSVafVtTTe/LDKPcvkJXQF4Ug=; b=ldj2mbjp2yuaFncqYHV3VYLW6cchWLu1RbjPfgb0CtQTspJa0aol7//zwLfYidL9TJ dv4CM7NFRMwO6JIuOaEmwb8x6b5qA/7S/gJ/4UowZm2cmCxgkdfWAycv2CLLreURepYQ 0nzTniZ9uL8y+Uq22UE303qlHeUxhuer2HyfzbIs712jIHF65rhTKaeSqIEZDtdFTXnt l5jWHudjmOiFLNR5hxmee2NxGvdnZi6tSb3BntU3BN0aFmnSohXf12UvJ/GRYyBTsB8y xBmVZ5iPDzxKnVBnbK7hq33MvK8xqgKNeX4r/OxkEAtPytxklV2hU9WbcghU8DSuYZKf azfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725252096; x=1725856896; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dp4HLFsSdaCRLqkUULTSVafVtTTe/LDKPcvkJXQF4Ug=; b=p8q7ZlYW6p40+hPDE2//sgHYz3GK1On9phacU7D94t9FnN/vmecmr8XhYQDv4Ia6FV 6QiPwzTz2wBqh7RQWlfMELmHKHzlHsWFwmJQiPR1o5UL+PWs9hDvur7ZbtCqpguzpvXe 59Lzqua+O+Mv3feHtiQGkf3oROe/aZp7wRehC2gki8ME/loagVRhtiKRVH4JgHvIBHnX DIbje2WFLmqKv6mrBARy7FEr48oFsEkJAMsq9zEAZW6FBRBCSsSvlDtrdzpsZ0YoVZ1q OnESsGI9EFLTlaovwWGkpm1YMbwvyOK/0F4CJCWYar1lxcK6s03S1KwX/QTwyrRMikzT DAng== X-Forwarded-Encrypted: i=1; AJvYcCXjsVIAZBndmvvgrX63RCU1ZcipHFAbgDx6FkLESwnYZN8tl1XFDAphraBJLItI5JfSfNUWPySzQw==@kvack.org X-Gm-Message-State: AOJu0YwdLtnAS8mX/7gT5ZD7ofM2gF1z9TC0VOjvuL6wbKtV4MMuIzJC BGuTPu/GbG5c1SQQusLh9nLA9wHIEHwmQtnd/l3FYza2bar4X1kweFQOkJZyqRsN7CkQZ+E7Epv HDg== X-Google-Smtp-Source: AGHT+IEB96AfQMtGaWG34HjG3YUsNj+jjgM1XA7e+3hjtS0U7EuT1s7DnlwBcK3PHwYEM5jDMTEBB/kupyA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:7343:ecd2:aed:5b8f]) (user=surenb job=sendgmr) by 2002:a05:690c:3386:b0:68d:52a1:bed with SMTP id 00721157ae682-6d40d88f6eemr5995947b3.1.1725252096434; Sun, 01 Sep 2024 21:41:36 -0700 (PDT) Date: Sun, 1 Sep 2024 21:41:24 -0700 In-Reply-To: <20240902044128.664075-1-surenb@google.com> Mime-Version: 1.0 References: <20240902044128.664075-1-surenb@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240902044128.664075-3-surenb@google.com> Subject: [PATCH v2 2/6] alloc_tag: load module tags into separate continuous memory From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 93073140004 X-Stat-Signature: 7xd7kt33zp83anafbazx5xuf9w111abi X-Rspam-User: X-HE-Tag: 1725252097-47551 X-HE-Meta: U2FsdGVkX1+zR57M962ouiQ5mk/EedwCQrjtZeq3O4IjEsbp9g0dZsSq5UwyNfTSrG42Aj2Dgc3PqqLzJYPJM5VIElGMU6ymC9OSRLEDJxd9kqOymOI6JfaadiyD+C/xLQjDPvQgh0GucSEl2g7mcKhUhnBN/LcpcinJKOPCg0JRq+1fkaIR8mQ9XP7It1LcsuZnvXimczwJlIRfh7eV8iG53CLFbGU6qe39HGnIBofjOfiMXXEG8aYxp8Ufso4xrFhoVPrf4+pvKWgHZOKzLuHLff8g5w5IHH7lOg4BuxzfxqsQjkPtvNPVeY7xI2HJI+afvrcbecw6zl9qTmKjPaLlDIJehC4FeQ6RAFoLLiJ11KeTnznsqlaFTB4mhlZkxFV6P6cfnxNNMBAfhRRni2Neyn5FJTfpGgUrNZlQEffXIP3A3DzfPzHLCSnNTbOarTjSUS5ByEfXdiy/nN5fMtmlaJ8pETGjKz0czCIoPsXRef/5ticy9arQZ5OpzYYbt4z62e/8BZCz1eKcyv6ijh5HU6lXnrdostBMFvAvfGHJeEUs2NeN8ypkwVBybmVnwtFuaNnrbziDZkB8pciHKD5CYVjfIXohxNC1n3y2xr1llGIyZ1v+Z99vRy2qDB1JF/D230fbpMlquOJjXGYX1jRnIkS1EA4FxS/esETrnzGkClt45Ey/DfagsaXra3jXl7oh0puqwlIH8INdGJl5DE/EccPWocmkw5SvMXq1jJkscEFj33GKVMksgzMBS1VmPLdkik1hP90nMt2QpTH+6lsrBQt3DLx9n2HF5/gkKJ7rgocyNEEKWHnyRk3VvrmBpXb3wbPmwjq0dDjdgj/MfxWKkp3kzTHZziRXsWQwWgYXFmTCtDiu7hgkSpuyJmlUPXy93naigAiluHnzM+9B5ozcj/KDlMvok0NE1AwyPwlR6JJmiXMG8o6Tmjqkj5SzvtCdFUZFxj6rFOIU1Wa 0K6C+iOx 6zBxgco4JMNlClouOe0YlOnoy0dpQgcwrvB2xvWcXwCkyh2UyVwuwQ+SiwN9+tlyJPl60VnxzHFkkyVxXTjwaCR62MLOLtCWk6JgtgJFR6sxt3Bzi1CmN9DaZkrvrkvt5vNGbq7W3YGXW7WTMGcbfGbDKv4jTbCTtms3DCqwZQEPtJpMQFC8PZS3XQqWsq6vuFTtEACkeO+aM4ngK4vFpw56peABaqzGeXaaa8XviOsJyKPszQEc6cIL5hiTnEYRuUBa71y/V8SJAgVB0bC6JxZ+sXkkqlRnI6pxeBkBIHelG6220MFMT4cNgRsyTEKLBwnMNnVTWEPjvtab62Z89POWNMT5M3VdWjUQUqS1VD9cUWBn3d2f1iTxLp1GNGnfvUbM63D0ZXquYufy4sMII3tcKvdzP2+dqXE7skFbsSXVhYQ+3xOlQlXOu66uGAppBhCntowBLk4T2AeNyfGfw4Ze9+xfgkVmgoLgsYwT/d3h58lGx2ZMy5lw5BzB1D/ZlBWzenVogIH1Iv5ngilcCw7H/bh08rNUPaPVSUUgtnaGzHtlu2KE0a7ZIl8+W9txwllYc8DonLsEezNcQ5EB/2jSweQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a module gets unloaded there is a possibility that some of the allocations it made are still used and therefore the allocation tags corresponding to these allocations are still referenced. As such, the memory for these tags can't be freed. This is currently handled as an abnormal situation and module's data section is not being unloaded. To handle this situation without keeping module's data in memory, allow codetags with longer lifespan than the module to be loaded into their own separate memory. The in-use memory areas and gaps after module unloading in this separate memory are tracked using maple trees. Allocation tags arrange their separate memory so that it is virtually contiguous and that will allow simple allocation tag indexing later on in this patchset. The size of this virtually contiguous memory is set to store up to 100000 allocation tags and max_module_alloc_tags kernel parameter is introduced to change this size. Signed-off-by: Suren Baghdasaryan --- .../admin-guide/kernel-parameters.txt | 4 + include/asm-generic/codetag.lds.h | 19 ++ include/linux/alloc_tag.h | 13 +- include/linux/codetag.h | 37 ++- kernel/module/main.c | 67 +++-- lib/alloc_tag.c | 265 ++++++++++++++++-- lib/codetag.c | 100 ++++++- scripts/module.lds.S | 5 +- 8 files changed, 451 insertions(+), 59 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 8dd0aefea01e..991b1c9ecf0e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3205,6 +3205,10 @@ devices can be requested on-demand with the /dev/loop-control interface. + + max_module_alloc_tags= [KNL] Max supported number of allocation tags + in modules. + mce [X86-32] Machine Check Exception mce=option [X86-64] See Documentation/arch/x86/x86_64/boot-options.rst diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h index 64f536b80380..372c320c5043 100644 --- a/include/asm-generic/codetag.lds.h +++ b/include/asm-generic/codetag.lds.h @@ -11,4 +11,23 @@ #define CODETAG_SECTIONS() \ SECTION_WITH_BOUNDARIES(alloc_tags) +/* + * Module codetags which aren't used after module unload, therefore have the + * same lifespan as the module and can be safely unloaded with the module. + */ +#define MOD_CODETAG_SECTIONS() + +#define MOD_SEPARATE_CODETAG_SECTION(_name) \ + .codetag.##_name : { \ + SECTION_WITH_BOUNDARIES(_name) \ + } + +/* + * For codetags which might be used after module unload, therefore might stay + * longer in memory. Each such codetag type has its own section so that we can + * unload them individually once unused. + */ +#define MOD_SEPARATE_CODETAG_SECTIONS() \ + MOD_SEPARATE_CODETAG_SECTION(alloc_tags) + #endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 8c61ccd161ba..99cbc7f086ad 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +struct alloc_tag_module_section { + unsigned long start_addr; + unsigned long end_addr; + /* used size */ + unsigned long size; +}; + #ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG #define CODETAG_EMPTY ((void *)1) @@ -54,6 +61,8 @@ static inline void set_codetag_empty(union codetag_ref *ref) {} #ifdef CONFIG_MEM_ALLOC_PROFILING +#define ALLOC_TAG_SECTION_NAME "alloc_tags" + struct codetag_bytes { struct codetag *ct; s64 bytes; @@ -76,7 +85,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { \ + __section(ALLOC_TAG_SECTION_NAME) = { \ .ct = CODE_TAG_INIT, \ .counters = &_shared_alloc_tag }; @@ -85,7 +94,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { \ + __section(ALLOC_TAG_SECTION_NAME) = { \ .ct = CODE_TAG_INIT, \ .counters = &_alloc_tag_cntr }; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c2a579ccd455..fb4e7adfa746 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,8 +35,15 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - bool (*module_unload)(struct codetag_type *cttype, + void (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); +#ifdef CONFIG_MODULES + void (*module_replaced)(struct module *mod, struct module *new_mod); + bool (*needs_section_mem)(struct module *mod, unsigned long size); + void *(*alloc_section_mem)(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align); + void (*free_section_mem)(struct module *mod, bool unused); +#endif }; struct codetag_iterator { @@ -71,11 +78,31 @@ struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) + +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size); +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align); +void codetag_free_module_sections(struct module *mod); +void codetag_module_replaced(struct module *mod, struct module *new_mod); void codetag_load_module(struct module *mod); -bool codetag_unload_module(struct module *mod); -#else +void codetag_unload_module(struct module *mod); + +#else /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ + +static inline bool +codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) { return false; } +static inline void * +codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) { return NULL; } +static inline void codetag_free_module_sections(struct module *mod) {} +static inline void codetag_module_replaced(struct module *mod, struct module *new_mod) {} static inline void codetag_load_module(struct module *mod) {} -static inline bool codetag_unload_module(struct module *mod) { return true; } -#endif +static inline void codetag_unload_module(struct module *mod) {} + +#endif /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 71396e297499..d195d788835c 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1225,18 +1225,12 @@ static int module_memory_alloc(struct module *mod, enum mod_mem_type type) return 0; } -static void module_memory_free(struct module *mod, enum mod_mem_type type, - bool unload_codetags) +static void module_memory_free(struct module *mod, enum mod_mem_type type) { - void *ptr = mod->mem[type].base; - - if (!unload_codetags && mod_mem_type_is_core_data(type)) - return; - - execmem_free(ptr); + execmem_free(mod->mem[type].base); } -static void free_mod_mem(struct module *mod, bool unload_codetags) +static void free_mod_mem(struct module *mod) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1247,25 +1241,20 @@ static void free_mod_mem(struct module *mod, bool unload_codetags) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod, type, unload_codetags); + module_memory_free(mod, type); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod, MOD_DATA, unload_codetags); + module_memory_free(mod, MOD_DATA); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { - bool unload_codetags; - trace_module_free(mod); - unload_codetags = codetag_unload_module(mod); - if (!unload_codetags) - pr_warn("%s: memory allocation(s) from the module still alive, cannot unload cleanly\n", - mod->name); + codetag_unload_module(mod); mod_sysfs_teardown(mod); @@ -1308,7 +1297,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod, unload_codetags); + free_mod_mem(mod); } void *__symbol_get(const char *symbol) @@ -1573,6 +1562,14 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i if (WARN_ON_ONCE(type == MOD_INVALID)) continue; + /* + * Do not allocate codetag memory as we load it into + * preallocated contiguous memory. s->sh_entsize will + * not be used for this section, so leave it as is. + */ + if (codetag_needs_module_section(mod, sname, s->sh_size)) + continue; + s->sh_entsize = module_get_offset_and_type(mod, type, s, i); pr_debug("\t%s\n", sname); } @@ -2247,6 +2244,7 @@ static int move_module(struct module *mod, struct load_info *info) int i; enum mod_mem_type t = 0; int ret = -ENOMEM; + bool codetag_section_found = false; for_each_mod_mem_type(type) { if (!mod->mem[type].size) { @@ -2257,7 +2255,7 @@ static int move_module(struct module *mod, struct load_info *info) ret = module_memory_alloc(mod, type); if (ret) { t = type; - goto out_enomem; + goto out_err; } } @@ -2267,11 +2265,27 @@ static int move_module(struct module *mod, struct load_info *info) void *dest; Elf_Shdr *shdr = &info->sechdrs[i]; enum mod_mem_type type = shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; + const char *sname; if (!(shdr->sh_flags & SHF_ALLOC)) continue; - dest = mod->mem[type].base + (shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK); + sname = info->secstrings + shdr->sh_name; + /* + * Load codetag sections separately as they might still be used + * after module unload. + */ + if (codetag_needs_module_section(mod, sname, shdr->sh_size)) { + dest = codetag_alloc_module_section(mod, sname, shdr->sh_size, + arch_mod_section_prepend(mod, i), shdr->sh_addralign); + if (IS_ERR(dest)) { + ret = PTR_ERR(dest); + goto out_err; + } + codetag_section_found = true; + } else { + dest = mod->mem[type].base + (shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK); + } if (shdr->sh_type != SHT_NOBITS) { /* @@ -2283,7 +2297,7 @@ static int move_module(struct module *mod, struct load_info *info) if (i == info->index.mod && (WARN_ON_ONCE(shdr->sh_size != sizeof(struct module)))) { ret = -ENOEXEC; - goto out_enomem; + goto out_err; } memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size); } @@ -2299,9 +2313,12 @@ static int move_module(struct module *mod, struct load_info *info) } return 0; -out_enomem: +out_err: for (t--; t >= 0; t--) - module_memory_free(mod, t, true); + module_memory_free(mod, t); + if (codetag_section_found) + codetag_free_module_sections(mod); + return ret; } @@ -2422,6 +2439,8 @@ static struct module *layout_and_allocate(struct load_info *info, int flags) /* Module has been copied to its final place now: return it. */ mod = (void *)info->sechdrs[info->index.mod].sh_addr; kmemleak_load_module(mod, info); + codetag_module_replaced(info->mod, mod); + return mod; } @@ -2431,7 +2450,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod, true); + free_mod_mem(mod); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..19ef02a18611 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include @@ -149,30 +150,244 @@ static void __init procfs_init(void) proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); } -static bool alloc_tag_module_unload(struct codetag_type *cttype, - struct codetag_module *cmod) +#ifdef CONFIG_MODULES + +static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +/* A dummy object used to indicate an unloaded module */ +static struct module unloaded_mod; +/* A dummy object used to indicate a module prepended area */ +static struct module prepend_mod; + +static struct alloc_tag_module_section module_tags; + +static bool needs_section_mem(struct module *mod, unsigned long size) { - struct codetag_iterator iter = codetag_get_ct_iter(cttype); - struct alloc_tag_counters counter; - bool module_unused = true; + return size >= sizeof(struct alloc_tag); +} + +/* Called under RCU read lock */ +static void clean_unused_module_areas_locked(void) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_for_each(&mas, val, module_tags.size) { + if (val == &unloaded_mod) { + struct alloc_tag *tag; + struct alloc_tag *last; + bool unused = true; + + tag = (struct alloc_tag *)(module_tags.start_addr + mas.index); + last = (struct alloc_tag *)(module_tags.start_addr + mas.last); + while (tag <= last) { + struct alloc_tag_counters counter; + + counter = alloc_tag_read(tag); + if (counter.bytes) { + unused = false; + break; + } + tag++; + } + if (unused) + mas_erase(&mas); + } + } +} + +static void *reserve_module_tags(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align) +{ + unsigned long section_size = module_tags.end_addr - module_tags.start_addr; + MA_STATE(mas, &mod_area_mt, 0, section_size - 1); + bool cleanup_done = false; + unsigned long offset; + void *ret; + + /* If no tags return NULL */ + if (size < sizeof(struct alloc_tag)) + return NULL; + + /* + * align is always power of 2, so we can use IS_ALIGNED and ALIGN. + * align 0 or 1 means no alignment, to simplify set to 1. + */ + if (!align) + align = 1; + + mas_lock(&mas); +repeat: + /* Try finding exact size and hope the start is aligned */ + if (mas_empty_area(&mas, 0, section_size - 1, prepend + size)) + goto cleanup; + + if (IS_ALIGNED(mas.index + prepend, align)) + goto found; + + /* Try finding larger area to align later */ + mas_reset(&mas); + if (!mas_empty_area(&mas, 0, section_size - 1, + size + prepend + align - 1)) + goto found; + +cleanup: + /* No free area, try cleanup stale data and repeat the search once */ + if (!cleanup_done) { + clean_unused_module_areas_locked(); + cleanup_done = true; + mas_reset(&mas); + goto repeat; + } else { + ret = ERR_PTR(-ENOMEM); + goto out; + } + +found: + offset = mas.index; + offset += prepend; + offset = ALIGN(offset, align); + + if (offset != mas.index) { + unsigned long pad_start = mas.index; + + mas.last = offset - 1; + mas_store(&mas, &prepend_mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + goto out; + } + mas.index = offset; + mas.last = offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + mas.index = pad_start; + mas_erase(&mas); + goto out; + } + + } else { + mas.last = offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + goto out; + } + } + + if (module_tags.size < offset + size) + module_tags.size = offset + size; + + ret = (struct alloc_tag *)(module_tags.start_addr + offset); +out: + mas_unlock(&mas); + + return ret; +} + +static void release_module_tags(struct module *mod, bool unused) +{ + MA_STATE(mas, &mod_area_mt, module_tags.size, module_tags.size); + struct alloc_tag *last; struct alloc_tag *tag; - struct codetag *ct; + struct module *val; - for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { - if (iter.cmod != cmod) - continue; + if (unused) + return; + + mas_lock(&mas); + mas_for_each_rev(&mas, val, 0) + if (val == mod) + break; + + if (!val) /* module not found */ + goto out; + + tag = (struct alloc_tag *)(module_tags.start_addr + mas.index); + last = (struct alloc_tag *)(module_tags.start_addr + mas.last); + + unused = true; + while (tag <= last) { + struct alloc_tag_counters counter; - tag = ct_to_alloc_tag(ct); counter = alloc_tag_read(tag); + if (counter.bytes) { + struct codetag *ct = &tag->ct; + + pr_info("%s:%u module %s func:%s has %llu allocated at module unload\n", + ct->filename, ct->lineno, ct->modname, + ct->function, counter.bytes); + unused = false; + break; + } + tag++; + } + + mas_store(&mas, unused ? NULL : &unloaded_mod); + val = mas_prev_range(&mas, 0); + if (val == &prepend_mod) + mas_store(&mas, NULL); +out: + mas_unlock(&mas); +} + +static void replace_module(struct module *mod, struct module *new_mod) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; - if (WARN(counter.bytes, - "%s:%u module %s func:%s has %llu allocated at module unload", - ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) - module_unused = false; + mas_lock(&mas); + mas_for_each(&mas, val, module_tags.size) { + if (val != mod) + continue; + + mas_store_gfp(&mas, new_mod, GFP_KERNEL); + break; } + mas_unlock(&mas); +} + +static unsigned long max_module_alloc_tags __initdata = 100000; - return module_unused; +static int __init set_max_module_alloc_tags(char *arg) +{ + if (!arg) + return -EINVAL; + + return kstrtoul(arg, 0, &max_module_alloc_tags); } +early_param("max_module_alloc_tags", set_max_module_alloc_tags); + +static int __init alloc_mod_tags_mem(void) +{ + unsigned long module_tags_mem_sz; + + module_tags_mem_sz = max_module_alloc_tags * sizeof(struct alloc_tag); + pr_info("%lu bytes reserved for module allocation tags\n", module_tags_mem_sz); + + /* Allocate space to copy allocation tags */ + module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, + module_tags_mem_sz); + if (!module_tags.start_addr) + return -ENOMEM; + + module_tags.end_addr = module_tags.start_addr + module_tags_mem_sz; + + return 0; +} + +static void __init free_mod_tags_mem(void) +{ + execmem_free((void *)module_tags.start_addr); + module_tags.start_addr = 0; +} + +#else /* CONFIG_MODULES */ + +static inline int alloc_mod_tags_mem(void) { return 0; } +static inline void free_mod_tags_mem(void) {} + +#endif /* CONFIG_MODULES */ #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support __meminitdata = true; @@ -255,14 +470,26 @@ static inline void sysctl_init(void) {} static int __init alloc_tag_init(void) { const struct codetag_type_desc desc = { - .section = "alloc_tags", - .tag_size = sizeof(struct alloc_tag), - .module_unload = alloc_tag_module_unload, + .section = ALLOC_TAG_SECTION_NAME, + .tag_size = sizeof(struct alloc_tag), +#ifdef CONFIG_MODULES + .needs_section_mem = needs_section_mem, + .alloc_section_mem = reserve_module_tags, + .free_section_mem = release_module_tags, + .module_replaced = replace_module, +#endif }; + int res; + + res = alloc_mod_tags_mem(); + if (res) + return res; alloc_tag_cttype = codetag_register_type(&desc); - if (IS_ERR(alloc_tag_cttype)) + if (IS_ERR(alloc_tag_cttype)) { + free_mod_tags_mem(); return PTR_ERR(alloc_tag_cttype); + } sysctl_init(); procfs_init(); diff --git a/lib/codetag.c b/lib/codetag.c index afa8a2d4f317..60463ef4bb85 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -207,6 +207,94 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) } #ifdef CONFIG_MODULES +#define CODETAG_SECTION_PREFIX ".codetag." + +/* Some codetag types need a separate module section */ +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) +{ + const char *type_name; + struct codetag_type *cttype; + bool ret = false; + + if (strncmp(name, CODETAG_SECTION_PREFIX, strlen(CODETAG_SECTION_PREFIX))) + return false; + + type_name = name + strlen(CODETAG_SECTION_PREFIX); + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) == 0) { + if (!cttype->desc.needs_section_mem) + break; + + down_write(&cttype->mod_lock); + ret = cttype->desc.needs_section_mem(mod, size); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) +{ + const char *type_name = name + strlen(CODETAG_SECTION_PREFIX); + struct codetag_type *cttype; + void *ret = NULL; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) == 0) { + if (WARN_ON(!cttype->desc.alloc_section_mem)) + break; + + down_write(&cttype->mod_lock); + ret = cttype->desc.alloc_section_mem(mod, size, prepend, align); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void codetag_free_module_sections(struct module *mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.free_section_mem) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.free_section_mem(mod, true); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + +void codetag_module_replaced(struct module *mod, struct module *new_mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.module_replaced) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.module_replaced(mod, new_mod); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + void codetag_load_module(struct module *mod) { struct codetag_type *cttype; @@ -220,13 +308,12 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -bool codetag_unload_module(struct module *mod) +void codetag_unload_module(struct module *mod) { struct codetag_type *cttype; - bool unload_ok = true; if (!mod) - return true; + return; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -243,18 +330,17 @@ bool codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - if (!cttype->desc.module_unload(cttype, cmod)) - unload_ok = false; + cttype->desc.module_unload(cttype, cmod); cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); } up_write(&cttype->mod_lock); + if (found && cttype->desc.free_section_mem) + cttype->desc.free_section_mem(mod, false); } mutex_unlock(&codetag_lock); - - return unload_ok; } #endif /* CONFIG_MODULES */ diff --git a/scripts/module.lds.S b/scripts/module.lds.S index 3f43edef813c..711c6e029936 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -50,7 +50,7 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } .rodata : { @@ -59,9 +59,10 @@ SECTIONS { } #else .data : { - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } #endif + MOD_SEPARATE_CODETAG_SECTIONS() } /* bring in arch-specific sections */ From patchwork Mon Sep 2 04:41:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13786703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE7A8C54FC6 for ; Mon, 2 Sep 2024 04:41:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B0A28D0070; Mon, 2 Sep 2024 00:41:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23D2F8D0065; Mon, 2 Sep 2024 00:41:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 067028D0070; Mon, 2 Sep 2024 00:41:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DA8178D0065 for ; Mon, 2 Sep 2024 00:41:41 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 59121141661 for ; Mon, 2 Sep 2024 04:41:41 +0000 (UTC) X-FDA: 82518550002.25.1CE71FF Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 8A85640014 for ; Mon, 2 Sep 2024 04:41:39 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MLPguSra; spf=pass (imf11.hostedemail.com: domain of 3AkLVZgYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3AkLVZgYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725252076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GtHW3QbMIbL8VkRhZ9UoruDWIR+5Ejb5CjRZSzRSfG0=; b=xABWTpMET1qJTzonvpfus8PrJsX4LbHxWIgoqMbWcvoTmCcFVYnQAfSez6RGeVzv82DLzd 71z5kcmXCAnBu9cVyKwi2kpNGcIk7VufRBzW++Vv5aJy++ytK/rTuZysxnHK7WiYNBSRHM yT+Zi4Mb29VKsqrP+h2Qa91qt355oW4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MLPguSra; spf=pass (imf11.hostedemail.com: domain of 3AkLVZgYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3AkLVZgYKCBkHJG3C05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725252076; a=rsa-sha256; cv=none; b=6CZxEb9wiQuv5sKQLS0ZF58idtYa3VrXcW8AlXD67yqt0ZhQD/L24GGuXeAga3RiVMqqc8 roaH52ioOifV/b4xA4Ofubj6xojQHKKjMO7oT5e8y8Nn75FyBvhnop+0M0paFSs0YQmGXt q6a+s+wqGt30VhgJyspedHDwrgQ+0n4= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1a7aedfff1so5015706276.3 for ; Sun, 01 Sep 2024 21:41:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725252099; x=1725856899; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GtHW3QbMIbL8VkRhZ9UoruDWIR+5Ejb5CjRZSzRSfG0=; b=MLPguSrarxhYBXDPZXbCtavzdrTCz5Uf3AawaveyfWxmXDdpZ7kBh4QH+alLIZISHJ rIE/yyK3ShqIdw/ASVU7y/ispUO1PRihdbd04FRHm6thuW7zR8VJiN6g9MXS2brcEfsz uKKPaT0O3z4Xu9xYzuBmZM2cDqfRUk/2gO4KShQMvk+kODWCUMCe2bdSf9D2wEAojtUT lJTvIMg5LLvWqVObzV3HtB5OLPJRMoUTgIVw/3ARO1q2+hOMIsQTqkLMrkV9DTWOAPq1 Z2AnrvHQPU+Q51/p2hicaMRImI5LbErug5amFA3G8N7y/wZfr67fWdLteK4kV0a/ZHYm ZysA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725252099; x=1725856899; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GtHW3QbMIbL8VkRhZ9UoruDWIR+5Ejb5CjRZSzRSfG0=; b=YHpZyOcUIMLG0bFsxIpRpXK1PW8mKRDa2kfCzO7ubQda0qVaMzt1kaQC7jH0NY9M37 9wbWdqJAsau1v8N0b/tnGxi8zJ4xFccYLMq3nbFt0E8ZrJry3inZzB+pWDCa3XtFXdlZ QN9ErOta4OUuFvzbXHfS0LgH3XV/fwrBYf6QtdpXaF75wNEJjLNp8oBpQSUyB7mEs26h Uz6uU27fqIB+FF3u0B55L7zC8A6nmn7QPw3GE+MyQElBL8EwOIQkdhMA4rwosk+4i2qR 7L6AeXCNUvkazepVxwJZjQbwxOgGuRLL+jb1qnP8a2r+NvwccSMqRXcaUsScbvt+lIkh 3khA== X-Forwarded-Encrypted: i=1; AJvYcCUYwlnhBrxb2GZvvx7BA7nB7DrFPf89cSxB4QBddSfBdZiDyTsAqyNdaySH8LlRMlkmvD0W3m4yWg==@kvack.org X-Gm-Message-State: AOJu0YyguWxZTnoCjD0fATFpzxVsynAcyXoGlRT8ipXkFPPH3EfZOt46 pDRzrvPpGR6e/GZKzTphf+zGbBOkg/vriS+q17g0+4oQcGrncb7wQFE1zkBamPzqq5drnIrg650 FzA== X-Google-Smtp-Source: AGHT+IGHUG44Vos/pxlFWbZNjhZtZ0OaAmv4KSGmEpEw821HAQXG3t5pT+l+pURZYG9PiSEp/j/9qgeoB6s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:7343:ecd2:aed:5b8f]) (user=surenb job=sendgmr) by 2002:a25:7e46:0:b0:e0b:ab63:b9c8 with SMTP id 3f1490d57ef6-e1a7a3dbab3mr14893276.11.1725252098507; Sun, 01 Sep 2024 21:41:38 -0700 (PDT) Date: Sun, 1 Sep 2024 21:41:25 -0700 In-Reply-To: <20240902044128.664075-1-surenb@google.com> Mime-Version: 1.0 References: <20240902044128.664075-1-surenb@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240902044128.664075-4-surenb@google.com> Subject: [PATCH v2 3/6] alloc_tag: eliminate alloc_tag_ref_set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Stat-Signature: pw8jip4diactz1y1wghgknksit8swfis X-Rspamd-Queue-Id: 8A85640014 X-Rspamd-Server: rspam11 X-HE-Tag: 1725252099-519610 X-HE-Meta: U2FsdGVkX19NtDoYpnF22CB5VnimINf2UfRSm00CUrDwsmHssYLvXWUj9zLCycvIHknomND18mTQXrE7xB7VeVZrTSbiWWnpR0Zwe6pCocaW+8mJO7ggoWT73ZlpGko1y14qHi6jfOcgFfQ4Z2y4tizaH+izOefTzc9OrlvR8W5ffo10rRYAf7NY5ORefpvuBDRzhQsQM6IOFtzBEecnR6j8H7NXwCWSBaZTXtKmTEQfgNnlP4aC2RdOE0rlDLq4pAH0bpQJILBPCE3fknts5gIs2urb9SC5bhBBaIRV05nndO4pJTT2u5b+xC/+8LeHr1Q3skmuBPbduJJA9bp5yAts7/icrOZnOKnlYjN9uNaBh5p0z+SfeDyAcdglpmIebhRx3JkEOTj3/FmZUjTIdaLEoUdBBTsLZWGAS2hO0btx+fGtqayo4gghR/V4sv46d0Q+TwSR0QXTWP8knPWKjW4cMOST0DhFZ3+wooVSBES+CEacrBTMpv9JYFgekGqQweiLO7NyHXX+3dx78wB4Ek10gjb/9dWqH6rQ6didYhwgUPODk7niHPQfbOUnwM6JhhyMLCGifjV0WeIpmWLe6BERn6PujRUpiaB/bGMol5YvgPV2h7JYAAJQFDbqoX0v4sW4bqpiyJH1y9eoNLN+t4P/Xh8Zz9DcVazp9fDI4nEGkGrD7aGvmm07bLrX832USjUcqBgIckALyL7SGBGc+iZJnATn/O17Ni+R0ZRSqgS54Ij/tbeBpS+U0NiQ8uRsxpwQrP6s0/Ciq1cKvUsQevPuLhDqmd+ehfxKyNM7AMk5PdBGLScG9o//d/8BKonaIZItzFX5N7OHQbXSTnnNw0ukgixtjI8DC9A/PX/cbqWDcR3gRbH2zp2dkCwG4RnddAKoVEHJB+pCrQJRFsLaMml78QMV0yp1CHpq56y28XtY09aIqlsKx/gDu20M5V60cz7iHp1TOg0KSCqhAHG J/gYT6v5 omngpqW0n6HJPOZwt2hwv6l0evmLw8itgp2LlTbeFg7WqnOjEC0itlj3nlQHU7nqgLI9sVVmm+SFUCJO1xtebqTQqOun9Rv+LWBGzQU8cEj06VrxHkXEcH401PsxjjjNOqDWXk71VLFvQuFYdx69yanxgt95ycWNUl2pWehb0asVzlL48qeVKm6STBcLmv9uwowEOVj13oKkQptgFn2FxU6/9RZa/seFPLswdESuW7pbxKvj6RlomDwJXa9OasUCkRZ6E/bkgbdUhzreaQZWa9jyGvyzyDTUcEGmzlVRJ9T5pdEr4kTuSRCNl2F3DztBut9QryVzXzfqoS+Zu9y1yNXsryoi5iBrq7/54P0GWt4sfVAcDoc1a+IAOT4W5VWegFZkWwlXuRWCAlfxwd+/iKb0vbKZFSOajb4oQXXYYSPEP/Y19AqhrroHAt14sXrrr7l2i0Or+kjkcaiX0+cSLme70URpOuvo/fMeqnA7jRiFubG+qmnYYfpxlYHNzBOWXAgMa6L00EZ0SsxOv/MkNdQcN7bIw964uoqnWCE4wBA1uQZ0XWVWPonOnZnvPUVEqYXYrj9dskEj183ObseZ+YLG7eg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000052, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To simplify further refactoring, open-code the only two callers of alloc_tag_ref_set(). Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 25 ++----------------------- include/linux/pgalloc_tag.h | 12 +++++++++++- 2 files changed, 13 insertions(+), 24 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 99cbc7f086ad..21e3098220e3 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -143,36 +143,15 @@ static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag static inline void alloc_tag_sub_check(union codetag_ref *ref) {} #endif -/* Caller should verify both ref and tag to be valid */ -static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) -{ - ref->ct = &tag->ct; - /* - * We need in increment the call counter every time we have a new - * allocation or when we split a large allocation into smaller ones. - * Each new reference for every sub-allocation needs to increment call - * counter because when we free each part the counter will be decremented. - */ - this_cpu_inc(tag->counters->calls); -} - -static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) -{ - alloc_tag_add_check(ref, tag); - if (!ref || !tag) - return; - - __alloc_tag_ref_set(ref, tag); -} - static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) { alloc_tag_add_check(ref, tag); if (!ref || !tag) return; - __alloc_tag_ref_set(ref, tag); + ref->ct = &tag->ct; this_cpu_add(tag->counters->bytes, bytes); + this_cpu_inc(tag->counters->calls); } static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 207f0c83c8e9..244a328dff62 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -103,7 +103,17 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) page_ext = page_ext_next(page_ext); for (i = 1; i < nr; i++) { /* Set new reference to point to the original tag */ - alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), tag); + ref = codetag_ref_from_page_ext(page_ext); + alloc_tag_add_check(ref, tag); + if (ref) { + ref->ct = &tag->ct; + /* + * We need in increment the call counter every time we split a + * large allocation into smaller ones because when we free each + * part the counter will be decremented. + */ + this_cpu_inc(tag->counters->calls); + } page_ext = page_ext_next(page_ext); } out: From patchwork Mon Sep 2 04:41:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13786704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A10FBC54FC6 for ; Mon, 2 Sep 2024 04:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 091326B00B8; Mon, 2 Sep 2024 00:41:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F34918D0065; Mon, 2 Sep 2024 00:41:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D37A38D0071; Mon, 2 Sep 2024 00:41:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A94538D0065 for ; Mon, 2 Sep 2024 00:41:43 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4E184AB108 for ; Mon, 2 Sep 2024 04:41:43 +0000 (UTC) X-FDA: 82518550086.12.D48E600 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 8A4E1C000B for ; Mon, 2 Sep 2024 04:41:41 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="mIYC77/c"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3BELVZgYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BELVZgYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725252025; a=rsa-sha256; cv=none; b=i0zQcpUV//1OVb0YizDoo16GZCHeOscARzXsVGWc7k9MEa3hWt/yGxB3q/FL+QmJ0HxBkJ Hxccm/pA3Knp+hP1hSNCpiBL3Jj782GX+Lg+TKuB9vcKU6QtxLVkHe4BU4ujdGPPMaZP/w HsQdPxf3zcfdI6rIi1Nan9GXOVTzrAI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="mIYC77/c"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3BELVZgYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BELVZgYKCBsJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725252025; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nlt0d2kKMzLV8FpvC+sskMXVjz4TJlE3ZyIfZQBAJUA=; b=nViQcTg+AAORNBuTUYhI5CwHnNrabC3+p/JRyH8hJwVd7yQMJJ4G8dYL/ChS7C5CN+LwpW /Ie+1/zuTa6q7Pa19adhghE+T0JTNZ/pJ+KJudp16QK1E/fyXnomnCNT3dwAeIetd94qED Nrr308yu8zmooom1lHgPR//o+oKsIMY= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6d4b832fb63so51626317b3.2 for ; Sun, 01 Sep 2024 21:41:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725252100; x=1725856900; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Nlt0d2kKMzLV8FpvC+sskMXVjz4TJlE3ZyIfZQBAJUA=; b=mIYC77/czyLMBgjCINDn0eCP4r7gTuzumyhuMEDZnTHYEHkCtRkShvOSTXsGpTYTj7 5jQ3ETtKbFrPzFIT+8L7tuapkzlOICMSBHNuHbu3v6KuvQnWsnmrCX5RHzCrHffsrg6i PDF718uOm+LNT6STucoiT6c2vxZVM1JEBLORgbJCbrCbGBF4Se7FgkDtYTBNb5qsqqSG ufGl3xzdbqEQDXqY2sKzJZ8pHkTpr6GOU9xkZNbEp+2Mv3ay27/p6lOt8w9Lf73O6lDX HSoHrvOZrqwJGg/SOLtuuHyuKyPCQUzCe1lj3Es7gP5O3MN2EddKJ2SGM7dYaUvQNQMD OYSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725252100; x=1725856900; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Nlt0d2kKMzLV8FpvC+sskMXVjz4TJlE3ZyIfZQBAJUA=; b=lAHcvai2BOOJzreSE1Nr1VBUhsBb/ahM9Zy0RZMhjJTvMjY9x88TOild2uC66J81e+ CeBbeSk02Mm+X3XaJ3ssTO5B1bHuJ0K0d4kdaBPiAM+DM23elxsVghMBqprMztpKWkDK TG4W5GCfnM6HfGvpM+eMm+C03JQ/+2ycSRtf/RyuY4/mXiF1MMQK2r9dgP/qOHeWv3Iu r37HJbOOuDS1OeLJC5c3zbL6je5gfmgS7IsOh8y/KQ0crrAXgBwvbI7nDtuJ0UygejPE WvkreysEwJtsJz2E+Qoa8dQ/UhkcyB7vhRYEBNRNclS1lDuEHYap5co8gyoE1rs5rjob co5g== X-Forwarded-Encrypted: i=1; AJvYcCVuohOFFNGjJ5G3TsAuPoxqQy8byTYTlvM/7KKlw6wGvbD56CvjyMF5oLyuO0Sn8oDpz0bJsZ+ZEw==@kvack.org X-Gm-Message-State: AOJu0Yy5B45MlgvmjumUdU4i7soEhn1sSouxzNv2YaJhPHPzgvJCgyy2 rPXoxN7osuS3cHwXpf4U6q0CjH3m9pUBbBVlRrlDU1o4KOL+4XmGkQLVnafHp6R/oWsUW+BmRuX cIA== X-Google-Smtp-Source: AGHT+IHSPd8nZJZH/UC+NYWl2T7llxAohdiabcKHZvTC/kXLRpqPrIn5YCFkqt976gAwR4y0XtmrRfdj/lI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:7343:ecd2:aed:5b8f]) (user=surenb job=sendgmr) by 2002:a05:690c:6309:b0:691:41f5:7f42 with SMTP id 00721157ae682-6d40f340f7dmr6255217b3.4.1725252100605; Sun, 01 Sep 2024 21:41:40 -0700 (PDT) Date: Sun, 1 Sep 2024 21:41:26 -0700 In-Reply-To: <20240902044128.664075-1-surenb@google.com> Mime-Version: 1.0 References: <20240902044128.664075-1-surenb@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240902044128.664075-5-surenb@google.com> Subject: [PATCH v2 4/6] alloc_tag: introduce pgalloc_tag_ref to abstract page tag references From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8A4E1C000B X-Stat-Signature: 8fh9tz8wy9xy7dqring5x3xmoy6b534a X-Rspam-User: X-HE-Tag: 1725252101-985999 X-HE-Meta: U2FsdGVkX1+Z17NM+Bdv2mk3gUUu9j7F2UievRpM8Pv10TVyuhipK2otF7Wf0RmNaVXjl1R5oMopweZ0XCMwssS01U27ksWrAq9l+B9YQlslsuWNF2EheGmlLVb4LoeWls8T8jIwuVY1lfM6VHoQJy3GZv2YunW9HMlrgpvWpeFHucmGzddnkhQc2wPeQEZHiyCXF06CzeMPJAzZsFeQP2P2ijaO+yIHYiv+mbpdml94p2XQdaqA/77tbeVK6Fk7hGA1KzgnuecrjDVl4kAWs5D+9N2O1fmzl3uSvoSk38SUbw95ZdRnEtXrNyNgFR+RKcmPb1xrh5HZqcW6oKDF1eyXDQ0QHV+CBCQw4hcCuuoR7l03T/42gENGwTpLWAR2vdkQnTPXLLpWGbU3wnYpCvpG5v1utKpESWsBU74RU7LRrZKq+PwLzPq0qy3Shz8TCKE4Q6/lR1jhxsahBPCQwhIwaQg7U9X9LEDbQUkl9ayeSPQVsttCiOYIsXfOcNDamPg9pBnfowMd3w30YH7cFWUY0nKSfG5XWnJHQQGp1QbyKcdYMq+tat2u9+lxf2sQKtCoPvimyzlY5UxLXlLHrpJrOwM4pnn2cYkaqt6T1jHD/ADSKltwubDj6EAydiNBuEc8bj6a9xOhA2Xfdf/hhD1jrpqbJQaG2XyorCurKA0zD8p8xwCb4OgaVu6Gu3y5QmozYj3ICAG2RYr+pHRBiOJQ7VsTurER4aC4VcFejkf26WPkwToWeGhRUKwQQ2Z6D9CtGgfrPkc9DOUjns/UNIZ7QnyjGySWXCMhWbFzOlFLJ7GHdKuzPF+B3kxbuwrQXL8W2+xdK627j3uXake61f9b6gPRGgF61At9kHg0xs25/01wn/KB84jQl8WfS/EkdaKpYn5dg8yyIthRqHUoO9NXw1elQ8NM4H9lN1FaznVL/seE0DTeB3pog5vUoAy5tz9hm30fZRMko2MU8hU GhGcCOp4 w2viH4jxUeu9QzZrh/XOHtuoS69Pd/ysRnS6nFIEHlhXyOOMFKVKKQo9H8yQRnDehCreecbZ+mvsSZrAy2MZrbb5Jk9PVhWl5waOLj1pyt/wPplxHy7l6M9WbJHiimJBQK0IVa38sdrhBhegotUzl//GDb465pXuSEDbmo4HN/r+ePLG7L09QL4WwprOMzuDJFRSjo7+48p9+urBOPQ9bvu2kRIAuChLlCNh1bU8zS9W51GUmfV3srN0zd0hlye1veqOIe8I6FGaenQTLxrUa1gW5iD4VQ+A+x63+rqYxoB8qOpGNAS7e8bm8FDKcCcIAc7zBGBLykM+ODo2U/XQ8s61tsYNNEkPQqrEU/L5pkRLUjg7QlLthMzq8NPgH25q00wup37Q0dSUPI3FeUribCAoW1t6NGs5F4+fkx42HpozeUKZKHjmPWoLGc0LzzZLUcIXsDQS3ALdBd4tskX7C1JfBjwMmn6W1EfkUwjxYiJANkEIyEZyTTTAghXzV47SRIcJbAN0Fx32OIxYrh1fuFy+7PRLdVqL3hsmHKXNtbZ+NenYRtZHpW6JPgA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000045, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To simplify later changes to page tag references, introduce new pgalloc_tag_ref and pgtag_ref_handle types. This allows easy replacement of page_ext as a storage of page allocation tags Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 144 +++++++++++++++++++++++------------- lib/alloc_tag.c | 3 +- 2 files changed, 95 insertions(+), 52 deletions(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 244a328dff62..c76b629d0206 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -9,48 +9,76 @@ #ifdef CONFIG_MEM_ALLOC_PROFILING +typedef union codetag_ref pgalloc_tag_ref; + +static inline void read_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + ref->ct = pgref->ct; +} + +static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + pgref->ct = ref->ct; +} #include extern struct page_ext_operations page_alloc_tagging_ops; -static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +static inline pgalloc_tag_ref *pgref_from_page_ext(struct page_ext *page_ext) { - return (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); + return (pgalloc_tag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); } -static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +static inline struct page_ext *page_ext_from_pgref(pgalloc_tag_ref *pgref) { - return (void *)ref - page_alloc_tagging_ops.offset; + return (void *)pgref - page_alloc_tagging_ops.offset; } +typedef pgalloc_tag_ref *pgtag_ref_handle; + /* Should be called only if mem_alloc_profiling_enabled() */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) +static inline pgtag_ref_handle get_page_tag_ref(struct page *page, union codetag_ref *ref) { if (page) { struct page_ext *page_ext = page_ext_get(page); - if (page_ext) - return codetag_ref_from_page_ext(page_ext); + if (page_ext) { + pgalloc_tag_ref *pgref = pgref_from_page_ext(page_ext); + + read_pgref(pgref, ref); + return pgref; + } } return NULL; } -static inline void put_page_tag_ref(union codetag_ref *ref) +static inline void put_page_tag_ref(pgtag_ref_handle pgref) { - if (WARN_ON(!ref)) + if (WARN_ON(!pgref)) return; - page_ext_put(page_ext_from_codetag_ref(ref)); + page_ext_put(page_ext_from_pgref(pgref)); +} + +static inline void update_page_tag_ref(pgtag_ref_handle pgref, union codetag_ref *ref) +{ + if (WARN_ON(!pgref || !ref)) + return; + + write_pgref(pgref, ref); } static inline void clear_page_tag_ref(struct page *page) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); - - if (ref) { - set_codetag_empty(ref); - put_page_tag_ref(ref); + pgtag_ref_handle handle; + union codetag_ref ref; + + handle = get_page_tag_ref(page, &ref); + if (handle) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -59,11 +87,14 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); - - if (ref) { - alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE * nr); - put_page_tag_ref(ref); + pgtag_ref_handle handle; + union codetag_ref ref; + + handle = get_page_tag_ref(page, &ref); + if (handle) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -71,53 +102,58 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); - - if (ref) { - alloc_tag_sub(ref, PAGE_SIZE * nr); - put_page_tag_ref(ref); + pgtag_ref_handle handle; + union codetag_ref ref; + + handle = get_page_tag_ref(page, &ref); + if (handle) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } static inline void pgalloc_tag_split(struct page *page, unsigned int nr) { - int i; - struct page_ext *first_page_ext; - struct page_ext *page_ext; - union codetag_ref *ref; + pgtag_ref_handle first_pgref; + union codetag_ref first_ref; struct alloc_tag *tag; + int i; if (!mem_alloc_profiling_enabled()) return; - first_page_ext = page_ext = page_ext_get(page); - if (unlikely(!page_ext)) + first_pgref = get_page_tag_ref(page, &first_ref); + if (unlikely(!first_pgref)) return; - ref = codetag_ref_from_page_ext(page_ext); - if (!ref->ct) + if (!first_ref.ct) goto out; - tag = ct_to_alloc_tag(ref->ct); - page_ext = page_ext_next(page_ext); + tag = ct_to_alloc_tag(first_ref.ct); for (i = 1; i < nr; i++) { - /* Set new reference to point to the original tag */ - ref = codetag_ref_from_page_ext(page_ext); - alloc_tag_add_check(ref, tag); - if (ref) { - ref->ct = &tag->ct; + pgtag_ref_handle handle; + union codetag_ref ref; + + page++; + handle = get_page_tag_ref(page, &ref); + if (handle) { + /* Set new reference to point to the original tag */ + alloc_tag_add_check(&ref, tag); + ref.ct = &tag->ct; /* * We need in increment the call counter every time we split a * large allocation into smaller ones because when we free each * part the counter will be decremented. */ this_cpu_inc(tag->counters->calls); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } - page_ext = page_ext_next(page_ext); } out: - page_ext_put(first_page_ext); + put_page_tag_ref(first_pgref); } static inline struct alloc_tag *pgalloc_tag_get(struct page *page) @@ -125,13 +161,15 @@ static inline struct alloc_tag *pgalloc_tag_get(struct page *page) struct alloc_tag *tag = NULL; if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); - - alloc_tag_sub_check(ref); - if (ref) { - if (ref->ct) - tag = ct_to_alloc_tag(ref->ct); - put_page_tag_ref(ref); + pgtag_ref_handle handle; + union codetag_ref ref; + + handle = get_page_tag_ref(page, &ref); + if (handle) { + alloc_tag_sub_check(&ref); + if (ref.ct) + tag = ct_to_alloc_tag(ref.ct); + put_page_tag_ref(handle); } } @@ -146,8 +184,12 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } -static inline void put_page_tag_ref(union codetag_ref *ref) {} +typedef void *pgtag_ref_handle; + +static inline pgtag_ref_handle +get_page_tag_ref(struct page *page, union codetag_ref *ref) { return NULL; } +static inline void put_page_tag_ref(pgtag_ref_handle handle) {} +static inline void update_page_tag_ref(pgtag_ref_handle handle, union codetag_ref *ref) {} static inline void clear_page_tag_ref(struct page *page) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 19ef02a18611..53bd3236d30b 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -436,7 +437,7 @@ static __init void init_page_alloc_tagging(void) } struct page_ext_operations page_alloc_tagging_ops = { - .size = sizeof(union codetag_ref), + .size = sizeof(pgalloc_tag_ref), .need = need_page_alloc_tagging, .init = init_page_alloc_tagging, }; From patchwork Mon Sep 2 04:41:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13786705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4BB8CD3427 for ; Mon, 2 Sep 2024 04:41:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFD678D0071; Mon, 2 Sep 2024 00:41:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D84ED8D0065; Mon, 2 Sep 2024 00:41:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3A138D0071; Mon, 2 Sep 2024 00:41:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 951138D0065 for ; Mon, 2 Sep 2024 00:41:45 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4A3AB121A2B for ; Mon, 2 Sep 2024 04:41:45 +0000 (UTC) X-FDA: 82518550170.01.40BFA82 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id 859E51C000F for ; Mon, 2 Sep 2024 04:41:43 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XgAMbi2x; spf=pass (imf18.hostedemail.com: domain of 3BkLVZgYKCB0LNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BkLVZgYKCB0LNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725252010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jIAzqL0tmqWuUadipZSgPIzfdbVk1YeOcL9yjZr6O/Q=; b=j8KOnao+8pWZOyUkHcSdaG/3gIjZDYGdv6DGtOavj0rPd6kQFKVq/AsMsHj6/CS0MVsU09 aQnm/kKNfG0lwcJF9bRSpB2znzkzzOLpYybEoMVSSW0Gr5W1DXQP5qoV6B8CIJxawJhY77 SNBMVsxyYT9+8iYijyjH5RztzVNXrbg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725252010; a=rsa-sha256; cv=none; b=HynPlZLpscyg70q01nkxpb1tvHkzWT6oHHulqpR4piyQl7k5ozZjmlTVnQi3YzQDTWlX4D gapuy620AlbKu/CiLenReoAbgJWp0ajUJe9UiilTehgHkBOfIPs+Wnp0whhnr2Ra2wyGib qtL/3ZqQk4ovIwc9Mh5oEtF9EZ+pqCI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XgAMbi2x; spf=pass (imf18.hostedemail.com: domain of 3BkLVZgYKCB0LNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BkLVZgYKCB0LNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1ce191f74fso639349276.2 for ; Sun, 01 Sep 2024 21:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725252102; x=1725856902; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jIAzqL0tmqWuUadipZSgPIzfdbVk1YeOcL9yjZr6O/Q=; b=XgAMbi2xt/IsxE7wIWaYaZepwMqLLbad+UQAIl8ymmOCzIeo3Oa8+K9ZiMETylLfyy hTq1gBw1S6D3zVnfu3kaJG3/nmKxAJ58g7fv3Ta5TNhJenVs0R0+rs1DVD6z2yIB9M9k SbGp+BJ1reFclNeyHnWGpnELrxVgRMJry2BQmiiYXNmiMivkaL7kypJugJ/BdhyTA/Xo crl0K5n8DkKPdPEAjabwAMbOJTSWGSg6nBhWtCSL708kR4xlTRR7dMoGecY0+cqCqLdO lpeaVjviV89S5fdJpL0NXRCQUC6MFy+usH+3sVDiP9vVS4TUyhULI80MFC8ILOLUJU6B cIsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725252102; x=1725856902; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jIAzqL0tmqWuUadipZSgPIzfdbVk1YeOcL9yjZr6O/Q=; b=dz56Ol/d+G/uCpT9JjEB5lNKRXXY80L3KhPGWxhpAl+vJc55DI9tKSCk0PLODZtBEJ 6y+zDCPxip3ab4IIekxCbsoe2D2DElhMWlZgn6cHWd8nX0qH4Sggs5zuOw+lnWcrAoj6 ZS9hFg4fgpYkAOTYwHSCrzVyGleLGQeWo8O6+2/O0CXWi2Hs3oO+aBY8Etbl7pX9nN8c 2p6q4KgLtW+M3pwNclCkHRki9F73xvxbN+DkogiMZM2DaBC5bgusFySxV0ORLAkQZYH1 gH2im2dXfsZ/M0EL6S3CI8f65z+DTNyIt971BIEhHelCuaMy1jwFoLH2cKZJhjUtdUzX wfHg== X-Forwarded-Encrypted: i=1; AJvYcCUI1+1+TvSRSWhS7RqTEHVIysq1e6IFLorp36Ovywl/pSEFyR5WzyI3I3M7yPVd/ynOjwFrpuD+MQ==@kvack.org X-Gm-Message-State: AOJu0YwwzbHcwCSonVPemOj2T95POyEcZdz4V/WUbUVwsjUE//rJybVW 90QVLfx36Z8Z4oq8blbYT+135Zv+fmWcOBDGA/l+QWF8OysNsNsahaeeTG7JauGHO+wMb2hsckh 4QQ== X-Google-Smtp-Source: AGHT+IGweUdtvzfkyW3WmJuGFzyyd2e8B2d2obHsNq5jvXmRb4BdxuAkJtcuoa6tBEzRO4VbtFxeXiTxIrs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:7343:ecd2:aed:5b8f]) (user=surenb job=sendgmr) by 2002:a5b:d06:0:b0:e11:6a73:b0d with SMTP id 3f1490d57ef6-e1a7a171852mr16567276.6.1725252102521; Sun, 01 Sep 2024 21:41:42 -0700 (PDT) Date: Sun, 1 Sep 2024 21:41:27 -0700 In-Reply-To: <20240902044128.664075-1-surenb@google.com> Mime-Version: 1.0 References: <20240902044128.664075-1-surenb@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240902044128.664075-6-surenb@google.com> Subject: [PATCH v2 5/6] alloc_tag: make page allocation tag reference size configurable From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: f1451fb9ugptgkqodkz6kdeyz5i9nrky X-Rspamd-Queue-Id: 859E51C000F X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1725252103-621083 X-HE-Meta: U2FsdGVkX19EbgTJ894YqLoF3cIBS+NKmY6hDjsDKuHYhSLMX0cG8pCGiE3tmlLXTA/4iy0LpsmHc1AblsqSTBgG73SvX63GTr8veG5N3AMnWl48rAdUO5xuwS7PMw4h5J/ar0MT1KaQKS6MXg+YGcxiHRzORvTl0oHa9vir/mlPzjJjFPapGzdpdaPDhYhWGF678fP/QpkH/CgQJx5iwnicylbptc/uLMT97kliDR9/53sXc9qzqQYiQwQqRVqcGa+kaIElftlf0WeBIXarmdKA2Gh+26vUiwghYfdNixDPubA7rNf64rzG6vpDuxikIY424DxrIUsxNzqeBjppPN2BL67AdZrkWic0cidwLCuzE33bJG/qC2QFG2p84c4CXHfUV0GhXRgJ0Yfcsdtgm5Vxbkyl+pEW9ebUJSWzfvwsE6X4N0ljt+Zrd9G1NWSTEeYsoO5vypQzkZ2AIt3uffRIjaKh8bgEFuw2rCL7blwVy5YYX3KHyK1x/K8WTZeV8slzNBhqfeAbgke7isSaemzGWH6Blr5eU3IxpDid90FHR8yhkmGpujMPZHJ+tdJuhajZ2tWpFiP9imNtbFTD0HT9XTPKt47SBf84u1yWUC6jWBYoahMrxq+ZY+WAbiVueODmsTX8QhYo5smv892SSL8N+hngxLj2BujcCzkbR6gFPa3bxM7kpra3CaT/OBKLKbtT+L0XNCOsjBGzDknJ9w3wDSnhgDFLaKpzK7qoG2/jy7qU48vq8EIvr1stWLLYtzhT0CHAprAd1A53cKgPBtZaNeucGub0auC0YxqudpfEVM9tdeU+x1TZACZ5oQ6jeLpIYJxvpoJJGUJvhVHMhkHXvyrKCDkMiGSb7GQCTGRb2N+ns8Ym4AB7FzRRtrKdD7ZRgInFtNQPTUAoLE6sBJXUg6uVDCBeE+dk/B30ukgJ/gL5LIJNalOl1Aj07wKOhwsqBmCOcHYkJrRK6Jy UYdiGTXN faXHhBcXRPI1qM56CYgHeGkHaL348nCSlZWTNK2YKLrc/uJZDE3eY+pSRTzZH63pDmelqC+SxnalLicVAILLwFFFjZ/g4LWiveJtEekcXulE2wU5LywLuULoIC4CeNZkrGgUQGLoVA7GLWSS/ZhW7FbHPm6/pziGBjNECuCPRU6n/PkwYTEJg7o+Oznpqc+7w5SeIgjx219oMIV5xt4d8JRGWxR7XurEglb/ivV+YbzgsEnGYg5zdtgGt6ci3Ghyb3I47Cd51QAwQf7oLdWIcPa3RMz6Vc7oUlHyBuAdXIgyes1kCFoe5qXMSnfp3/lO+uoy+nJ5uc2d8Sjc649+6DDN7oeo1zi+2RNheEKvW24+oJ5J79tZRpdZBfBrlEJ0FBtFCaPLerLpnjkwQE8FyKYc8Sdwxva5wtM5EppMaEnmNt0tqYQEJATXZUQ3rs9XqV/+YeeLaKbVam9GLSf+isvoUvwT342yRbuPq7J+t9goyk3nzmJ1B3TLwH91X83vnnUgP3L1x2YEhMLCBLE7EBqUWmWv5f4Pgx7xfZwiVLgFopG5zGf7+6QHHUaTLgUD0v7KOZcG19+SncFyvFDRquWfB5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce CONFIG_PGALLOC_TAG_REF_BITS to control the size of the page allocation tag references. When the size is configured to be less than a direct pointer, the tags are searched using an index stored as the tag reference. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 10 +++- include/linux/codetag.h | 3 ++ include/linux/pgalloc_tag.h | 99 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 11 +++++ lib/alloc_tag.c | 51 ++++++++++++++++++- lib/codetag.c | 4 +- mm/mm_init.c | 1 + 7 files changed, 175 insertions(+), 4 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 21e3098220e3..b5cf24517333 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,8 +30,16 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +struct alloc_tag_kernel_section { + struct alloc_tag *first_tag; + unsigned long count; +}; + struct alloc_tag_module_section { - unsigned long start_addr; + union { + unsigned long start_addr; + struct alloc_tag *first_tag; + }; unsigned long end_addr; /* used size */ unsigned long size; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index fb4e7adfa746..401fc297eeda 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -13,6 +13,9 @@ struct codetag_module; struct seq_buf; struct module; +#define CODETAG_SECTION_START_PREFIX "__start_" +#define CODETAG_SECTION_STOP_PREFIX "__stop_" + /* * An instance of this structure is created in a special ELF section at every * code location being tagged. At runtime, the special section is treated as diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index c76b629d0206..a7f8f00c118f 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -9,7 +9,18 @@ #ifdef CONFIG_MEM_ALLOC_PROFILING +#if !defined(CONFIG_PGALLOC_TAG_REF_BITS) || CONFIG_PGALLOC_TAG_REF_BITS > 32 +#define PGALLOC_TAG_DIRECT_REF typedef union codetag_ref pgalloc_tag_ref; +#else /* !defined(CONFIG_PGALLOC_TAG_REF_BITS) || CONFIG_PGALLOC_TAG_REF_BITS > 32 */ +#if CONFIG_PGALLOC_TAG_REF_BITS > 16 +typedef u32 pgalloc_tag_ref; +#else +typedef u16 pgalloc_tag_ref; +#endif +#endif /* !defined(CONFIG_PGALLOC_TAG_REF_BITS) || CONFIG_PGALLOC_TAG_REF_BITS > 32 */ + +#ifdef PGALLOC_TAG_DIRECT_REF static inline void read_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) { @@ -20,6 +31,93 @@ static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) { pgref->ct = ref->ct; } + +static inline void alloc_tag_sec_init(void) {} + +#else /* PGALLOC_TAG_DIRECT_REF */ + +extern struct alloc_tag_kernel_section kernel_tags; + +#define CODETAG_ID_NULL 0 +#define CODETAG_ID_EMPTY 1 +#define CODETAG_ID_FIRST 2 + +#ifdef CONFIG_MODULES + +extern struct alloc_tag_module_section module_tags; + +static inline struct codetag *get_module_ct(pgalloc_tag_ref pgref) +{ + return &module_tags.first_tag[pgref - kernel_tags.count].ct; +} + +static inline pgalloc_tag_ref get_module_pgref(struct alloc_tag *tag) +{ + return CODETAG_ID_FIRST + kernel_tags.count + (tag - module_tags.first_tag); +} + +#else /* CONFIG_MODULES */ + +static inline struct codetag *get_module_ct(pgalloc_tag_ref pgref) +{ + pr_warn("invalid page tag reference %lu\n", (unsigned long)pgref); + return NULL; +} + +static inline pgalloc_tag_ref get_module_pgref(struct alloc_tag *tag) +{ + pr_warn("invalid page tag 0x%lx\n", (unsigned long)tag); + return CODETAG_ID_NULL; +} + +#endif /* CONFIG_MODULES */ + +static inline void read_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + pgalloc_tag_ref pgref_val = *pgref; + + switch (pgref_val) { + case (CODETAG_ID_NULL): + ref->ct = NULL; + break; + case (CODETAG_ID_EMPTY): + set_codetag_empty(ref); + break; + default: + pgref_val -= CODETAG_ID_FIRST; + ref->ct = pgref_val < kernel_tags.count ? + &kernel_tags.first_tag[pgref_val].ct : + get_module_ct(pgref_val); + break; + } +} + +static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) +{ + struct alloc_tag *tag; + + if (!ref->ct) { + *pgref = CODETAG_ID_NULL; + return; + } + + if (is_codetag_empty(ref)) { + *pgref = CODETAG_ID_EMPTY; + return; + } + + tag = ct_to_alloc_tag(ref->ct); + if (tag >= kernel_tags.first_tag && tag < kernel_tags.first_tag + kernel_tags.count) { + *pgref = CODETAG_ID_FIRST + (tag - kernel_tags.first_tag); + return; + } + + *pgref = get_module_pgref(tag); +} + +void __init alloc_tag_sec_init(void); + +#endif /* PGALLOC_TAG_DIRECT_REF */ #include extern struct page_ext_operations page_alloc_tagging_ops; @@ -197,6 +295,7 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} +static inline void alloc_tag_sec_init(void) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index a30c03a66172..253f9c2028da 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1000,6 +1000,17 @@ config MEM_ALLOC_PROFILING_DEBUG Adds warnings with helpful error messages for memory allocation profiling. +config PGALLOC_TAG_REF_BITS + int "Number of bits for page allocation tag reference (10-64)" + range 10 64 + default "64" + depends on MEM_ALLOC_PROFILING + help + Number of bits used to encode a page allocation tag reference. + + Smaller number results in less memory overhead but limits the number of + allocations which can be tagged (including allocations from modules). + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 53bd3236d30b..73791aa55ab6 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -151,6 +152,26 @@ static void __init procfs_init(void) proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); } +#ifndef PGALLOC_TAG_DIRECT_REF + +#define SECTION_START(NAME) (CODETAG_SECTION_START_PREFIX NAME) +#define SECTION_STOP(NAME) (CODETAG_SECTION_STOP_PREFIX NAME) + +struct alloc_tag_kernel_section kernel_tags = { NULL, 0 }; + +void __init alloc_tag_sec_init(void) +{ + struct alloc_tag *last_codetag; + + kernel_tags.first_tag = (struct alloc_tag *)kallsyms_lookup_name( + SECTION_START(ALLOC_TAG_SECTION_NAME)); + last_codetag = (struct alloc_tag *)kallsyms_lookup_name( + SECTION_STOP(ALLOC_TAG_SECTION_NAME)); + kernel_tags.count = last_codetag - kernel_tags.first_tag; +} + +#endif /* PGALLOC_TAG_DIRECT_REF */ + #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); @@ -159,7 +180,16 @@ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ static struct module prepend_mod; -static struct alloc_tag_module_section module_tags; +struct alloc_tag_module_section module_tags; + +#ifndef PGALLOC_TAG_DIRECT_REF +static inline unsigned long alloc_tag_align(unsigned long val) +{ + if (val % sizeof(struct alloc_tag) == 0) + return val; + return ((val / sizeof(struct alloc_tag)) + 1) * sizeof(struct alloc_tag); +} +#endif /* PGALLOC_TAG_DIRECT_REF */ static bool needs_section_mem(struct module *mod, unsigned long size) { @@ -216,6 +246,21 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, if (!align) align = 1; +#ifndef PGALLOC_TAG_DIRECT_REF + /* + * If alloc_tag size is not a multiple of required alignment tag + * indexing does not work. + */ + if (!IS_ALIGNED(sizeof(struct alloc_tag), align)) { + pr_err("%s: alignment %lu is incompatible with allocation tag indexing (CONFIG_PGALLOC_TAG_REF_BITS)", + mod->name, align); + return ERR_PTR(-EINVAL); + } + + /* Ensure prepend consumes multiple of alloc_tag-sized blocks */ + if (prepend) + prepend = alloc_tag_align(prepend); +#endif /* PGALLOC_TAG_DIRECT_REF */ mas_lock(&mas); repeat: /* Try finding exact size and hope the start is aligned */ @@ -373,6 +418,10 @@ static int __init alloc_mod_tags_mem(void) return -ENOMEM; module_tags.end_addr = module_tags.start_addr + module_tags_mem_sz; +#ifndef PGALLOC_TAG_DIRECT_REF + /* Ensure the base is alloc_tag aligned */ + module_tags.start_addr = alloc_tag_align(module_tags.start_addr); +#endif return 0; } diff --git a/lib/codetag.c b/lib/codetag.c index 60463ef4bb85..cb3aa1631417 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -149,8 +149,8 @@ static struct codetag_range get_section_range(struct module *mod, const char *section) { return (struct codetag_range) { - get_symbol(mod, "__start_", section), - get_symbol(mod, "__stop_", section), + get_symbol(mod, CODETAG_SECTION_START_PREFIX, section), + get_symbol(mod, CODETAG_SECTION_STOP_PREFIX, section), }; } diff --git a/mm/mm_init.c b/mm/mm_init.c index 4ba5607aaf19..231a95782455 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2650,6 +2650,7 @@ void __init mm_core_init(void) report_meminit(); kmsan_init_shadow(); stack_depot_early_init(); + alloc_tag_sec_init(); mem_init(); kmem_cache_init(); /* From patchwork Mon Sep 2 04:41:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13786706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5846DC54FC6 for ; Mon, 2 Sep 2024 04:41:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9C8F8D0072; Mon, 2 Sep 2024 00:41:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C5D058D0065; Mon, 2 Sep 2024 00:41:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA04C8D0072; Mon, 2 Sep 2024 00:41:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8435E8D0065 for ; Mon, 2 Sep 2024 00:41:47 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4105341A28 for ; Mon, 2 Sep 2024 04:41:47 +0000 (UTC) X-FDA: 82518550254.13.230F5AB Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 6D37E140007 for ; Mon, 2 Sep 2024 04:41:45 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=N6qazXVZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3CELVZgYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3CELVZgYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725252001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o+zkIQ79nR91+bZ9fiMOs0/SZD3fit87wxI6zeuv5qM=; b=QQgNuqGYS7ud3zXXEq0z0ZfSE7eEXYTtpDCPC1sKZSMiZVUtnhTT0/aiFozmaTrqq7Y7+W G09D75VXdZpjUHm0++EO/ttJ4sXPsTBDihd0iwSJLdco12fqKnJYz4T4Ef03DY9f0dtDYf lL7TKKgWD0EVYTnlfggxXVSIFRlsv9s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725252001; a=rsa-sha256; cv=none; b=JuL1s1CvChHLUA6jyDLdHRkOPsOcymZ1F79FkEiwHUr7j0RqBVCGcJFIbVx9v360Q7uRqB xb0emaPGMpexrr1PTPRUuNSBL2Is5q7kvKwB7XTazJlfFVjw0PaWtD2SXRzg85nS/HiEVB qQ5W0t3SEJIxxii/A9PqNUJExjBmSmk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=N6qazXVZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3CELVZgYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3CELVZgYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e1ce98dcafaso70265276.0 for ; Sun, 01 Sep 2024 21:41:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725252104; x=1725856904; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o+zkIQ79nR91+bZ9fiMOs0/SZD3fit87wxI6zeuv5qM=; b=N6qazXVZtYrPlQrJ/4juZx/lqIiMRbEiqq84g+V3/6xuYEfqMtgDBsfGCKeqcv4X8i zaaomHyq7H5KyMhFu60bsuDO/VfRgwgFVoON8nSGM8gbYzrq7MOOylBh9Iq7dq7wm1iL pxH/JRlzAqXOUfwjrKcoUapB5BujkRDNmAVQDySit4VQzwGzvZZhMjpF7xn4xwsNbEPl fiioh2AP3eOPMd8XWHOm2zbOYfD5cPvHD9pvA7W/26jOXZrUXxLh+EVC0lrwPRLQkrzP y0dzY8E3RFR+G+/a+p7J4iDoze3WPbuTQnRoat28CWINnWDOv0+8F2itoH1GG77jJwrt peLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725252104; x=1725856904; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o+zkIQ79nR91+bZ9fiMOs0/SZD3fit87wxI6zeuv5qM=; b=L0SnOMvU54xsA4aZtt8hE5N7KTLgvQzz5x+5QIm2fkx03QEj3phvkukhPclngxzsDj BtMD+0tIevtk6aNKHEOTh1BRgzM99ITSIXLQENPiEELz2Awp0IP5q7Rj2KZup/gKa6To MCPiSq0qR6DNcyHfu9PEeCe47aftwl/45IRYKQaG6eCwfYAVWx4vVouwJl1bPg4V3TcA OfjZk2PTwk3s13DUAYzBNMh7p71gJUBJLTl0R5j+xXuPt1cHkMkeaaS74q/m+GyPZvlC /n0Pn/F7snDh1iCeF4epoEqBL2Vbz13y+iE3P2ySfLD62E15R3WH8qa1AHV/B3xp6x+H JIRg== X-Forwarded-Encrypted: i=1; AJvYcCUKzFyVE4O8d2goMrKWaouujn/H8B2+90K4OXUhLl4ZX+ZNNVUCydhGO8IajJhOShqDtr22M59xTg==@kvack.org X-Gm-Message-State: AOJu0YwCSghMeMknuyuThAGSt5syxY3Q71B49iHRTvcg/pc9/f2+0Loo OY41BdxHR+T6BD2J62M2K/4jTYTWPkWAw5KNem3WqHIMseUKqCxqFjYNsrhhAl0plUAEOYyqosf QFQ== X-Google-Smtp-Source: AGHT+IGESmFEaOyUz7H46Wj9ZUskazUKRvv1KzwKzYF26vqNkzVhO5DbeT/NbEzV1QldF78OxMgid1k7qes= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:7343:ecd2:aed:5b8f]) (user=surenb job=sendgmr) by 2002:a05:6902:1b0a:b0:e0b:a2a7:df77 with SMTP id 3f1490d57ef6-e1a79faf966mr117475276.2.1725252104383; Sun, 01 Sep 2024 21:41:44 -0700 (PDT) Date: Sun, 1 Sep 2024 21:41:28 -0700 In-Reply-To: <20240902044128.664075-1-surenb@google.com> Mime-Version: 1.0 References: <20240902044128.664075-1-surenb@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240902044128.664075-7-surenb@google.com> Subject: [PATCH v2 6/6] alloc_tag: config to store page allocation tag refs in page flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6D37E140007 X-Stat-Signature: h1xzbnptmocm5tigzrujth8c7jfonxwo X-Rspam-User: X-HE-Tag: 1725252105-56374 X-HE-Meta: U2FsdGVkX189QFLseU3H6sCcfrYYRXlkNHpjr5rlAU9uipx3iDw85EstR1S88NdgZhFg0jvKq0g+mvcnPsKPQCLWxtcD35anZ9nCgoheU6H8E1PTbwVLLKJjYJ+yHqzQ8nxPdBNTXlvj47mNIayU/P4i7tsYNghMSPuhHDE2k2oXLJf/MNSgc/tNaD1V7tAt8jX0sbbmlTT1Yar82NLszv1qgOEUDCnGKDRjOn9WKce+lrrjiyumKJlZl0ClV4vBeVeKEAj3qoVgEzB+oSmNRdkevfFBX9qSa61kl96+Nl7l5vsN/UXlpfHtKHRYfXJTp0e06qHdyyjou4xYXD3J1Cib/M0j3nzxPFxCf3B+ODpo+YdgzmXZjRwt86o/xu3Jgj4NDA8nUQeADpxsZt6WgCTMEz5Vzp75PS8rL8t3CLqAETWVfOkkh9w9/UmMFtfLupNB2gL393MUePQAAQ07ER0sARJQwxc00s5N0O5OC3WSZ+QtSg1SrykyH2azuXbaT3Ytphr3BY4IHUmUR9dPt0fIkU/kg33zqi0jsbvFvANva/8PTNyUyNRMo0Jep2sUD7aoSE5p47hzv+NRfaCnjvvbvauePf0KLV+P47h5hxGCpuAXiTT4PDQseq8cdRZAn0Vw6th1HJnS/P/nCA5jr4RYhGHIXyxGTaXa9MgvPT+9FL2Cdw3A07MytMJBGtnMLh7Fcm5u5WEmWofdpWQT/Up3OBM8PPUMoRk5s47EryqLycOKb0J3kS7Rcs3cGVr6eQZ8cYGAtLcI2v694SAXZzoPeqM8KFEyZ5QNVf/sdU81FHEAQ9l3M6CkFFqfwpvnFm1GNV/Lfcmq9RoWRusUyUqIkM2cJbG9mqBj8lrAx+I37+Jd4CZfpbtkh2ruXOHaFYBJl1pJyN+t+XxdXYiERCMZNUmdkgylgv1mOfW2erNk4322oJ+IWviXf3hUw1x9l/JVpKI2bKL5G2+0/Gj h//2tWnc lwqBkuDz4nXZd0eN2K1y4NxMJdNKhGtGAFlSTsUeztiCfCctSOXj4QqbxWA4Fdzi/GZbMQo3blxL6u69TYZHHfxHZeDxRl+u0bRG6+KYagJ0l/hfZ42JL674Tm58tqUmotx3Cx76/pIvmgKMYHfiwQYGNv94scZjFxISmFefUNGUkG7PpAYc78aCOF+H42tXqFgwcqnUf3nBVhTZSBHV1okMDJ2Knsh33tlIE0/sw6hVQtX/IB5vK3ZLy1r7a68bHex7Kxg/WY3yFbZon7ISBoHzs/ZG7pBajzC5T/wPi3nTrND0xGARuaJATiJOGJmZOk9lDDzvBwNA8I1qRc6eYdSP1jhg7uLDf/z2fLnZtiZTS8VUn94oGyVQTZOa7aIejN11ZzvrsGK0wgL1qrAIInQTw7fT/zG14vuHA0Vu8L0e3ntd30sS/ZXKeQ3xMq/Tfx0qrQVEMvVIQMREBDJaxjiPZODRUgWfjmTLzGB8GvpKY2H9Z33aL2ZMexGDtZIKEaHY1yB5w7MuEvtZ8xez70brGy8dpOpgyZPDFaPacUEGZny1DZ7p7zbnlf10ovQijM7FgRmaIeRIav95+oS5rmyPvhw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add CONFIG_PGALLOC_TAG_USE_PAGEFLAGS to store allocation tag references directly in the page flags. This removes dependency on page_ext and results in better performance for page allocations as well as reduced page_ext memory overhead. CONFIG_PGALLOC_TAG_REF_BITS controls the number of bits required to be available in the page flags to store the references. If the number of page flag bits is insufficient, the build will fail and either CONFIG_PGALLOC_TAG_REF_BITS would have to be lowered or CONFIG_PGALLOC_TAG_USE_PAGEFLAGS should be disabled. Signed-off-by: Suren Baghdasaryan --- include/linux/mmzone.h | 3 ++ include/linux/page-flags-layout.h | 10 +++++-- include/linux/pgalloc_tag.h | 48 +++++++++++++++++++++++++++++++ lib/Kconfig.debug | 27 +++++++++++++++-- lib/alloc_tag.c | 4 +++ mm/page_ext.c | 2 +- 6 files changed, 89 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 17506e4a2835..0dd2b42f7cb6 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1085,6 +1085,7 @@ static inline bool zone_is_empty(struct zone *zone) #define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) #define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) #define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH) +#define ALLOC_TAG_REF_PGOFF (LRU_REFS_PGOFF - ALLOC_TAG_REF_WIDTH) /* * Define the bit shifts to access each section. For non-existent @@ -1096,6 +1097,7 @@ static inline bool zone_is_empty(struct zone *zone) #define ZONES_PGSHIFT (ZONES_PGOFF * (ZONES_WIDTH != 0)) #define LAST_CPUPID_PGSHIFT (LAST_CPUPID_PGOFF * (LAST_CPUPID_WIDTH != 0)) #define KASAN_TAG_PGSHIFT (KASAN_TAG_PGOFF * (KASAN_TAG_WIDTH != 0)) +#define ALLOC_TAG_REF_PGSHIFT (ALLOC_TAG_REF_PGOFF * (ALLOC_TAG_REF_WIDTH != 0)) /* NODE:ZONE or SECTION:ZONE is used to ID a zone for the buddy allocator */ #ifdef NODE_NOT_IN_PAGE_FLAGS @@ -1116,6 +1118,7 @@ static inline bool zone_is_empty(struct zone *zone) #define LAST_CPUPID_MASK ((1UL << LAST_CPUPID_SHIFT) - 1) #define KASAN_TAG_MASK ((1UL << KASAN_TAG_WIDTH) - 1) #define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1) +#define ALLOC_TAG_REF_MASK ((1UL << ALLOC_TAG_REF_WIDTH) - 1) static inline enum zone_type page_zonenum(const struct page *page) { diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h index 7d79818dc065..21bba7c8c965 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -5,6 +5,12 @@ #include #include +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS +#define ALLOC_TAG_REF_WIDTH CONFIG_PGALLOC_TAG_REF_BITS +#else +#define ALLOC_TAG_REF_WIDTH 0 +#endif + /* * When a memory allocation must conform to specific limitations (such * as being suitable for DMA) the caller will pass in hints to the @@ -91,7 +97,7 @@ #endif #if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \ - KASAN_TAG_WIDTH + LAST_CPUPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS + KASAN_TAG_WIDTH + ALLOC_TAG_REF_WIDTH + LAST_CPUPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT #else #define LAST_CPUPID_WIDTH 0 @@ -102,7 +108,7 @@ #endif #if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \ - KASAN_TAG_WIDTH + LAST_CPUPID_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS + KASAN_TAG_WIDTH + ALLOC_TAG_REF_WIDTH + LAST_CPUPID_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS #error "Not enough bits in page flags" #endif diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index a7f8f00c118f..dcb6706dee15 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -118,6 +118,52 @@ static inline void write_pgref(pgalloc_tag_ref *pgref, union codetag_ref *ref) void __init alloc_tag_sec_init(void); #endif /* PGALLOC_TAG_DIRECT_REF */ + +#ifdef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + +typedef struct page *pgtag_ref_handle; + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline pgtag_ref_handle get_page_tag_ref(struct page *page, + union codetag_ref *ref) +{ + if (page) { + pgalloc_tag_ref pgref; + + pgref = (page->flags >> ALLOC_TAG_REF_PGSHIFT) & ALLOC_TAG_REF_MASK; + read_pgref(&pgref, ref); + return page; + } + + return NULL; +} + +static inline void put_page_tag_ref(pgtag_ref_handle page) +{ + WARN_ON(!page); +} + +static inline void update_page_tag_ref(pgtag_ref_handle page, union codetag_ref *ref) +{ + unsigned long old_flags, flags, val; + pgalloc_tag_ref pgref; + + if (WARN_ON(!page || !ref)) + return; + + write_pgref(&pgref, ref); + val = (unsigned long)pgref; + val = (val & ALLOC_TAG_REF_MASK) << ALLOC_TAG_REF_PGSHIFT; + do { + old_flags = READ_ONCE(page->flags); + flags = old_flags; + flags &= ~(ALLOC_TAG_REF_MASK << ALLOC_TAG_REF_PGSHIFT); + flags |= val; + } while (unlikely(!try_cmpxchg(&page->flags, &old_flags, flags))); +} + +#else /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + #include extern struct page_ext_operations page_alloc_tagging_ops; @@ -166,6 +212,8 @@ static inline void update_page_tag_ref(pgtag_ref_handle pgref, union codetag_ref write_pgref(pgref, ref); } +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + static inline void clear_page_tag_ref(struct page *page) { if (mem_alloc_profiling_enabled()) { diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 253f9c2028da..9fc8c1981f27 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -979,7 +979,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING - select PAGE_EXTENSION + select PAGE_EXTENSION if !PGALLOC_TAG_USE_PAGEFLAGS select SLAB_OBJ_EXT help Track allocation source code and record total allocation size @@ -1000,10 +1000,26 @@ config MEM_ALLOC_PROFILING_DEBUG Adds warnings with helpful error messages for memory allocation profiling. +config PGALLOC_TAG_USE_PAGEFLAGS + bool "Use pageflags to encode page allocation tag reference" + default n + depends on MEM_ALLOC_PROFILING + help + When set, page allocation tag references are encoded inside page + flags, otherwise they are encoded in page extensions. + + Setting this flag reduces memory and performance overhead of memory + allocation profiling but also limits how many allocations can be + tagged. The number of bits is set by PGALLOC_TAG_USE_PAGEFLAGS and + they must fit in the page flags field. + + Say N if unsure. + config PGALLOC_TAG_REF_BITS int "Number of bits for page allocation tag reference (10-64)" range 10 64 - default "64" + default "16" if PGALLOC_TAG_USE_PAGEFLAGS + default "64" if !PGALLOC_TAG_USE_PAGEFLAGS depends on MEM_ALLOC_PROFILING help Number of bits used to encode a page allocation tag reference. @@ -1011,6 +1027,13 @@ config PGALLOC_TAG_REF_BITS Smaller number results in less memory overhead but limits the number of allocations which can be tagged (including allocations from modules). + If PGALLOC_TAG_USE_PAGEFLAGS is set, the number of requested bits should + fit inside the page flags. + + If PGALLOC_TAG_USE_PAGEFLAGS is not set, the number of bits used to store + a reference is rounded up to the closest basic type. If set higher than 32, + a direct pointer to the allocation tag is stored for performance reasons. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 73791aa55ab6..34820650186d 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -476,6 +476,8 @@ static int __init setup_early_mem_profiling(char *str) } early_param("sysctl.vm.mem_profiling", setup_early_mem_profiling); +#ifndef CONFIG_PGALLOC_TAG_USE_PAGEFLAGS + static __init bool need_page_alloc_tagging(void) { return mem_profiling_support; @@ -492,6 +494,8 @@ struct page_ext_operations page_alloc_tagging_ops = { }; EXPORT_SYMBOL(page_alloc_tagging_ops); +#endif /* CONFIG_PGALLOC_TAG_USE_PAGEFLAGS */ + #ifdef CONFIG_SYSCTL static struct ctl_table memory_allocation_profiling_sysctls[] = { { diff --git a/mm/page_ext.c b/mm/page_ext.c index 641d93f6af4c..5f993c271ee7 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -83,7 +83,7 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif -#ifdef CONFIG_MEM_ALLOC_PROFILING +#if defined(CONFIG_MEM_ALLOC_PROFILING) && !defined(CONFIG_PGALLOC_TAG_USE_PAGEFLAGS) &page_alloc_tagging_ops, #endif #ifdef CONFIG_PAGE_TABLE_CHECK