From patchwork Wed Oct 23 17:07:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED309CFA454 for ; Wed, 23 Oct 2024 17:08:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D5126B0096; Wed, 23 Oct 2024 13:08:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75DB96B0098; Wed, 23 Oct 2024 13:08:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B0536B009B; Wed, 23 Oct 2024 13:08:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3A8FF6B0096 for ; Wed, 23 Oct 2024 13:08:07 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D915EAC5FA for ; Wed, 23 Oct 2024 17:07:31 +0000 (UTC) X-FDA: 82705499352.15.DB4CD9A Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf05.hostedemail.com (Postfix) with ESMTP id B25A2100027 for ; Wed, 23 Oct 2024 17:07:31 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=J4GooH+c; spf=pass (imf05.hostedemail.com: domain of 3cy0ZZwYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3cy0ZZwYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Cy4VkAGTnueEjEvw1TSjR77Zk0lX1faSNj+Oljrwa2Y=; b=1GUE62OQmjC+sXGoMcKq+VW0zx98uQLq4bKL+k/ugCnsP4wFac39GRLdodScaas1xYE8iL IjpOZR9MDUI1E7PeyVcs70hE5094eBDwd4TgGz1xepo9eSA/lOlh+mM/prkHF/WQyCGBDT E8aEc+JwTQ+ivl8Dv2jNWeMk96cHacI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703132; a=rsa-sha256; cv=none; b=aJA6/23rkF6iJdN8ztWxsxGIqi5oF/1YFv7H3q+z4bEb01/TjwNQFRmt0tinKX57J/Girs 5I18lfJMh9dVwb3fpt087inurk/WOptAH/EVaJYouH8rxgnz30gbSbPsvkhYXQYdCila0P NTO1GOICk8Xw3JxIqVQOB/vXRDOSE+I= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=J4GooH+c; spf=pass (imf05.hostedemail.com: domain of 3cy0ZZwYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3cy0ZZwYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e2954ada861so17274276.3 for ; Wed, 23 Oct 2024 10:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703284; x=1730308084; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy4VkAGTnueEjEvw1TSjR77Zk0lX1faSNj+Oljrwa2Y=; b=J4GooH+c+yDEKfWScuX06MUnjdtMF9da+l9PUrh228OL2OGK0YlODh7Hr51i60aVgc ygUlOj2L8Q+uE/PHC/HE62LQiGomaLDqdzcA7e1iBoWupE6JQipE7nNBJXQE+hnGUVHz NTQvYb3+CZVvaAokHUx+AVdUfZ1icUhLhwVM5ugvVjK8wcA8x3e17N/plAA5bAfeuVQu tVc9Q2f9R8IRC89NjaAN5D/uXL16O0YTB5vLWSQe2iZqJ9ST3kseT52FIpIOvRKI3s1l SW6aus4WcC0O1/VjGtcANZOeKoBVWMpcm+HnHIBC5PKufcWtK1JrhloXr6/PgB1jfEvT poDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703284; x=1730308084; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy4VkAGTnueEjEvw1TSjR77Zk0lX1faSNj+Oljrwa2Y=; b=mSSaxmkL70PAAVk/vAqpbBdUUeX9P8Bh8V4OP9CoANlzYHUwjK3jzjeLhHrC50Z2h7 Bt9t8GYG9Oln9CV6O7tsCkYBeohtgusJK9S1Vix5jrOjh8dz/Dw07PSAlAUcNlivUdVu +GYmQHYrgYLKN1jhktG77jVg5vsWtthVXGds4ue+i4Eby2E/XJcYDjbeRuqDNWdfIkPA BH8PM69wcEMOYrIutiFInF5AGaBxYcKNThWA69GsJWwoe9pjipi/Bs7uutoN9wDC7SA+ MI4Al1Scw87wcreYyMnVKBVSSI9oZVTVB0pJwB4Jxz6ske5Kcp+ARqsVdyFu5euUqZ8A 5tCg== X-Forwarded-Encrypted: i=1; AJvYcCVEnu3Fo2/BTStkf0FYzKpx7YT9YHW9dT+oT4NQB0kc6tFeMui8M7gwF1r7IRbBNHPfgwxuvmk5gw==@kvack.org X-Gm-Message-State: AOJu0YxF7/pem1XnAlsApatBEERn5uWB+IknuIKVn71zN/o+1g2RJZ0q jzSQJ28B7lELfMcK+13YH2C1RFd6zEnq9PQ1A+7bM7Qn6qTZiM8QX/6Sp0nhUP9yoazvh7+gksy VqQ== X-Google-Smtp-Source: AGHT+IG5P3EOBq94xAhs93OihXu6fn1+vh8Hj0b9Xh0nro0ohDjLm9Tm7P85Bm8yXiSBBXuIOUCegUy3OSE= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:7241:0:b0:e2e:2b0f:19fc with SMTP id 3f1490d57ef6-e2e3a631bf2mr3476276.4.1729703283882; Wed, 23 Oct 2024 10:08:03 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:54 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-2-surenb@google.com> Subject: [PATCH v4 1/6] maple_tree: add mas_for_each_rev() helper From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B25A2100027 X-Stat-Signature: kp8wnjk3cxiot3jywykjtsirac4kxytz X-HE-Tag: 1729703251-525926 X-HE-Meta: U2FsdGVkX1/JlAGg5Vm80/L4HISZrxogqYVBEiZAZt6O+cK6hLZDPvy/u0Os+kzp/IzsA5e+jZ3iQg4sF31VWDtDp7Oa42p3OQTWdoZVS77KhijOlbrXpKWDo44oRcrkB0hDMo154sWk4c0yE0gZ9MuMaDX0OUQo5AwTtt7cwg8HoddYY3ot/VoB6dDVvwUBnei3yjtBLMIcSIulZTnzZNISY0e96TE2kX8mSaM2Jh1+15ZdjQclz2ZAh9To6d6qAEYI7Lnbt/Sihft4kbb78ZIg8CWt7g3wlUaZiaFDMe/zTxllyxNU7IbjL6pHiMLdM+XOPCzZ22G/Z9ZHPgrrKA43acxhi996Huku1FS47rfLPCu52BBPZZSgzhB2N8P7SplKuBjruKe3wsBP0D1VdsGi0TB9BOiaJ1recIykYX58rdO2U83SAMrFhfTfK9rBDIa98zMuBtIihMP4IJinn3aJi8hQNTURPtSccmZWdOgifzM3WavVpOcQTg5cGPPPGIws7gzC/CJsfTJI+z3Idv6b02wbtYYCLydyaL2uU5q6HhUGyN9s1OsSHBKG17OVO5cqOY6M+knCh8HHlf+hUt2i7h0QmckJe2iztjNACqNzA0RF1ng5daB71zBc2/uN0+lWhDr5Bts2zvi6tR9IfM6ik2QdX0HdbMi1ReAciUu/vNhPo4JCyhbhkNwpvGDpaE3+4L6hlJGquWzbs7pAeponjXN7QgmRFhu1ChF6xYQqTUfvr2h7lpMLYs3dHnDtUpyZmFHP6H7G0+C0UU1tt9KzP0L4sUzjP56oqc3/d7RGywzjEk9Iwjcvltt9ihi5QGvu1QvwLnrcbJKtjKJqnRPbqe+aVHLkhrnA0N0mij0WXTxyAXjMJHoFYr2zYT8UZqfu/ylj9j/Wa5bMGskXnA7V4LWxxuwzv360rz9W3CT8zEJ9324dU5MWYZIlxxgG4TWl/J5BbP9pySkYhn+ vo9fHUVK xtCOhpUL1J5MHWO0ZeINIIV843K1YQ/S4/E3scUAeuGxQVy/+bAxdqRgAtCo/CzO8xfmiW4P4jGzB5llPXUTBVOSYVipWHFnNoeoXyj4J/jrPyRB9lDz2ZVKGNpcKimE9pHHVizeVNCSFGe5bmjxX3bX6lPNEnfZcgwiscyewnY1erQCsNIZe+IVoLk3xC0qtWejswEFWfPOgT63/haol9Gv5dGHZwzhe436Wv7xpZYIM1nSMEMOGmIQNZD6HPT3QlxfQrxJ1Gg+t/Ho06yEkgKZDxyjl8tAuLIoUQKGdPLCGgCjvjwAT/x1Dy60AYQUmMU0JLenxpJqB5bXz7jrXGUK8HtjQ6oLcz+/7Es502UJUP9M+s0q9RQrdZ0Tw0r5UG0PP4l0m60PfVJsGZDQeO8GVFStto6yk3fD4Rx2Cg+cfLpjnFJZHk6LJef++/9tWSlvsWgaOtMHYTaEfmyGhLy3u0geErKpg+Lh0iDecLa3ltDydqGfcTLaLP7X/zCWCr0EGqv/urG0KsrZ3p6mFtc3YoheXEPGWlbr7G9oV82U6kUB0Zf+48xRONew44ZNSIC0xpMbi1iUocLBGJUbB642JfEzLgvlFPAVFvp4g7w0klHE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add mas_for_each_rev() function to iterate maple tree nodes in reverse order. Suggested-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett Reviewed-by: Pasha Tatashin --- include/linux/maple_tree.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 61c236850ca8..cbbcd18d4186 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -592,6 +592,20 @@ static __always_inline void mas_reset(struct ma_state *mas) #define mas_for_each(__mas, __entry, __max) \ while (((__entry) = mas_find((__mas), (__max))) != NULL) +/** + * mas_for_each_rev() - Iterate over a range of the maple tree in reverse order. + * @__mas: Maple Tree operation state (maple_state) + * @__entry: Entry retrieved from the tree + * @__min: minimum index to retrieve from the tree + * + * When returned, mas->index and mas->last will hold the entire range for the + * entry. + * + * Note: may return the zero entry. + */ +#define mas_for_each_rev(__mas, __entry, __min) \ + while (((__entry) = mas_find_rev((__mas), (__min))) != NULL) + #ifdef CONFIG_DEBUG_MAPLE_TREE enum mt_dump_format { mt_dump_dec, From patchwork Wed Oct 23 17:07:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D91ACFA453 for ; Wed, 23 Oct 2024 17:08:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 897A56B009B; Wed, 23 Oct 2024 13:08:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 823D06B009C; Wed, 23 Oct 2024 13:08:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64E1B6B009D; Wed, 23 Oct 2024 13:08:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 408DF6B009B for ; Wed, 23 Oct 2024 13:08:09 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1A732C0DCC for ; Wed, 23 Oct 2024 17:07:50 +0000 (UTC) X-FDA: 82705499352.10.E720F28 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id C9C9540009 for ; Wed, 23 Oct 2024 17:07:44 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mYGd4XV5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3di0ZZwYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3di0ZZwYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703174; a=rsa-sha256; cv=none; b=kRYo1kfuohtV7X3MUb5e5jL7EvELVn1VtlD3EqjQ+3R8ONzA32TSZXGufUpzQojISiQ+RF HO0Jn74V3rANnEC0+4M/Q/VDiyizIUQfIjrl91iw2Hq3EGG6mLIUAEnOcX6L/H6+SZgFFE VzPTO/h+uivxNMiZdewNieRdhTzQjos= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mYGd4XV5; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3di0ZZwYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3di0ZZwYKCHUlnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703174; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qzUZQuDC5qa2V/vpTubJjhmN2aD40kQ8oHuQLi+u1Z4=; b=vmBUdg7Ya3NnGxuXiG2o9zIVVN3qdZkQZzGE3y9Be5O6MOGIoZu3QtDjvCrJgY53Rdz5Ra WpqqK5nC3YmK2m0tA8HY0L2MRDJU05YC29OF3xUShxlKcEROBgtaUpGe08p0G57cJ6Xe4V M6t5g3hIByXbje7z8XHzsw+TlYrwj3A= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e2bd3e0a26cso69786276.0 for ; Wed, 23 Oct 2024 10:08:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703286; x=1730308086; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qzUZQuDC5qa2V/vpTubJjhmN2aD40kQ8oHuQLi+u1Z4=; b=mYGd4XV5l0KF49Hvlr1rg2RcNAenRc/BV5M5PQBz4Ii6Cfw05q8IVgF/Eot97APrg0 oZZ3lpqPshYHoTHHRJEjbEypwST9JXdU0IcVDOUT05Z46ncYhJiYiRv5tZTkXEZnfvuB yli0WkRE0DjaBmbAiHRj6xB6lWskzGb2VYaGh8OKX+BvFn195jPc6p4MLkWS5XJKzd27 Uq1nzJCJa3nMV9pe+V5+Ut2FlTw08APP8tr3Q340CRJVm9Hfb46G3/pvDWWFwcbGgzQF UdBir6FJYemPoPfI36gHIVLQ8Qp9sS326dnadAgrwTRqLsqeaZ0SAPAXx3JxdbtglO6A Sy7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703286; x=1730308086; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qzUZQuDC5qa2V/vpTubJjhmN2aD40kQ8oHuQLi+u1Z4=; b=MBiHogs2LutWLDTTQF0724x/WE+VcRNAnhejQNmRXM2zFMPnMRzP4X46vaIxlCSiDg 4tmV+c1pqq+HGhXTnK8KQVmpD4qpmvdaD8p4Df5ZnzdT9Q5tcsOS9vUdHXe6Rmz2YYyW d3b7Oyaaw5YsKvQVH8VSxspE02Jk1Urf7oBM1BUe3FeNm67fJBWG59DKFPa88Z7Bvo5S wKdejAbAuWXVnSC+lcegwSWJOgnGfyzVGG6wqAAaLD2cXjSQOkEG8YOu/RSJJWlhASD0 tTlgR6ukH4DwSxqHg++BYYjgW3BcNNxaFwdn0WaznksttPiHKryPqjUjdY5qqEBcXjWO hxAg== X-Forwarded-Encrypted: i=1; AJvYcCWWqvKsFY7Fm+iBzC6JRNpxYzSZ62Uj63O/N8Y/+tHeKpRR3GJFpUksLMjjiF0QDncNtSYha0Nb1g==@kvack.org X-Gm-Message-State: AOJu0YyWwxw8svjkTJrod7wOqhQelVpQMyITyGh3QMmSnxogoGe65xJW wGrOGMLWBhDAs2DG7pRxJhED4vaWRE6qqItFpkBMBWDR//Glyfgxr2Dn+YsqXRCU+4wrQxy1qTA gWQ== X-Google-Smtp-Source: AGHT+IEf9XVjhAQxbJksRoGGus3EwGVaSTpLfWlLEYi7zdwXtn6qn+dayeJKMPCmDwrDuVs0NH6hwtuuxI8= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:8289:0:b0:e2e:3401:ea0f with SMTP id 3f1490d57ef6-e2e3a6bbdffmr7278276.7.1729703286023; Wed, 23 Oct 2024 10:08:06 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:55 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-3-surenb@google.com> Subject: [PATCH v4 2/6] alloc_tag: introduce shutdown_mem_profiling helper function From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: C9C9540009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: fom4ay8sbgzh1gxqjww3hr7wiqqjfwkf X-HE-Tag: 1729703264-981072 X-HE-Meta: U2FsdGVkX197QxsqLbsVgAeLLUzS2cbI49bYGUI/0hf2ywidsbOaq7NaFxbtYP58ViJHcaYaqR5fFFuN2bmATsIWC5DW92MhvmqU7rr55/DiCRHj5ASWh8ymWqD44l5bqFMzyhQXkYYnubVrXr9Bf1KxBbeSSQzJ+IWPEN52Kf3ANVjKxtcMYujKYT9XiCa4JVVFeUycvrZcynvVWo1aSidWtt7KZ7Rm2iX3akcyjKz/Ummny+bmk6QbL+8QAWZp7z0HD8F9xGJQHfENoPGlSpoIr08JEzS9LFDvk2jV8UO3zAGkR36pm0x19jsPGaMxCzOx5wyTv0DmcnA0IUDamDBgKpQw3Vq42uS7VyvX+P8VRas/8OVztjZXu+pnwYuerQIOr0hti1VltB+m6h+LrdUvobA6Vg2TlR6gkqnhIG2O5udNvBy5tMyAjMC/80sDZ42pMk15z2elMHNMzya/2j+RZDxevH20PSh2AcctgMFBk/Wd9AtKyB/Tpg1ilOu5UT5Ndff84+UfuJUpdynBLFVn9qQYXBB8hBIFgGCknRWoXOthbkgEt/9DHbyR80sVzVrd3SvMHGFxeSlTjEGVs+AAPXXS97+EXJOixzxTC17+zF2+01/nX9au8RXoBNPq+1uLHz64vkJKwGqs96BcIGqksst7vnPjE7HGcUwlbbkcrrBHvo2hbAC87Ogv35lgi8gRw0RWQ+jEi3XXsDiGT59CbmFO+2wX4mxKBpnEDzLLliZ5xHb3Qo+Mi5QrdWHh5UcU9Q5npzf6YlntT7nP5BYD9OHxrXt8SHxGUgM6VyTzNS4GYtE0d8dUyPalGjnWq2ziMfVoJj3rRtoXBZFLQ8GaIhDss4ciVmfJS7LVO+h9PmPCIszjEj/5QyaVivcjwiDGzhfeyB/0i3F9b2peCFgTbyHOVEf++3QzHH+P51yOwy3+NXeqN1yZHivHvjckIWicI6vcIC873Gm4Fkn Np4ZPAuZ tSLQSbKsmSNtIEkrHgX+lau951cMxAbHYR6Ng0y+pHZ3AJA6Mzad0HhvBUx/HarDWZC79QRNqOwYts/EwoTDA7G4k6rIlQOyNFPn87lFpzz2MxriybRuP2cr20pFPwEt9g2LTP/wP3iWAr058bcvYHsHn+XmGXTttsclKP/87S9FrG8mQCU4Z0PNqk/vZj74y3I35ID0rxk95hq6f7UL6lo4zLOC+PLoWoIcZ+wp5BsbMJ/7smv5lQtH/H0SP2T7mkluiyFWGO0D1a51zQVJPYEOIDTj77qhwnxfOgtobCo/cUsfgH1heaAxe4k1SkraTLmX/vjCEBsQNO1AnpJRu3Ytr9fC8/N2mUE6e80ki+3+D2nUbTeOAGx1AUsqEdksmCW3dgccXtIvO0XUBcLkkVSp0ZEF7VpPL66GQyOmashynw5utZazNghP5vmOdqVj2vB5Fda/1YGNFEscUSsD2rRJaFV78IncPIBgs4vNRfwi+YvLd5PtHIkPvQ/7T2Ialoc8KxpyWv71jPo7R6Lw48eZVxXccL/37SdhINHTZnXmXHJJBDNRq1A5NYOwpa69OIyd2LCXV5RQX5Wv3F3TOzmxn7A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement a helper function to disable memory allocation profiling and use it when creation of /proc/allocinfo fails. Ensure /proc/allocinfo does not get created when memory allocation profiling is disabled. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- lib/alloc_tag.c | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..435aa837e550 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,6 +8,14 @@ #include #include +#define ALLOCINFO_FILE_NAME "allocinfo" + +#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT +static bool mem_profiling_support __meminitdata = true; +#else +static bool mem_profiling_support __meminitdata; +#endif + static struct codetag_type *alloc_tag_cttype; DEFINE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); @@ -144,9 +152,26 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl return nr; } +static void __init shutdown_mem_profiling(void) +{ + if (mem_alloc_profiling_enabled()) + static_branch_disable(&mem_alloc_profiling_key); + + if (!mem_profiling_support) + return; + + mem_profiling_support = false; +} + static void __init procfs_init(void) { - proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); + if (!mem_profiling_support) + return; + + if (!proc_create_seq(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op)) { + pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); + shutdown_mem_profiling(); + } } static bool alloc_tag_module_unload(struct codetag_type *cttype, @@ -174,12 +199,6 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } -#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT -static bool mem_profiling_support __meminitdata = true; -#else -static bool mem_profiling_support __meminitdata; -#endif - static int __init setup_early_mem_profiling(char *str) { bool enable; From patchwork Wed Oct 23 17:07:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5220CFA454 for ; Wed, 23 Oct 2024 17:08:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40C496B009C; Wed, 23 Oct 2024 13:08:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3651A6B009D; Wed, 23 Oct 2024 13:08:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19B376B00A1; Wed, 23 Oct 2024 13:08:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E2EFA6B009C for ; Wed, 23 Oct 2024 13:08:10 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D084BAC4B5 for ; Wed, 23 Oct 2024 17:07:35 +0000 (UTC) X-FDA: 82705499016.09.4076C35 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 1D38318000E for ; Wed, 23 Oct 2024 17:08:05 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pQNMU0Mz; spf=pass (imf24.hostedemail.com: domain of 3dy0ZZwYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3dy0ZZwYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703211; a=rsa-sha256; cv=none; b=ogHVKJf51ffJBIugyFkK1gFLpmvO/58Pxk7jNxG87Sh1CxR8KloLSxxLCE7ph4QtS9gdq7 +lR0H+VUHKtcbV9KqttGWWBBe+lbSZ6PPk2iI5bpWLKrOrw16IOrSW7/i2hMe/JS7yrMLy /ycbgVPY5MNgh3kior/Ri38pQd85vzw= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pQNMU0Mz; spf=pass (imf24.hostedemail.com: domain of 3dy0ZZwYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3dy0ZZwYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BDYzGeX72WULGwXnJI1q90ubF1+yTH7FuZO2allpANU=; b=C1Fw4ytDU8ck95NIz9bIHG87LEjb0aEG/7jeFuz9HWxatflWmxqA4GQuNfl6WEhsOO9Ls7 EqyVVy3L+uJxsrY7ZkxUvSFZnKV6otF5CgwtBywqmG3WMU+Qx1FeqxaGjzRJfDqHCAOIzw v69T2P+5FYgTUTuJkeDuHgnWTe09Y7M= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e293150c2c6so48142276.1 for ; Wed, 23 Oct 2024 10:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703288; x=1730308088; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BDYzGeX72WULGwXnJI1q90ubF1+yTH7FuZO2allpANU=; b=pQNMU0MzE9R4SN48w1KlVqMWAPjkbiSCD0RSk/ht/QlC50Szo1v3uOBWb7C7FOaoAU fDqJAj8i1AO72dX21Drb7t0aoPIkiiW8IvGLPg56ZxSchxBFVjpcRS50X9N4hjFfYuEV c6dDsDyBX7rid+Ys5+jc83Xavk7m/Bxh0/CN7rLivX/zFCJFljKlMYwaRuI7iT3YTcp3 v1ON8sk+u1vAfRY66nhg/PfBG3tZBhNKWWJF3uevcNBcz562FxnJ6oVuwqLUYgDYBOV8 LziZkaJ0WYtwO0O2AJQWGBDJgFn6Y0ArevECbvdKDyEilZJqM4b/wTDvyuQ67TwJblMB 4iCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703288; x=1730308088; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BDYzGeX72WULGwXnJI1q90ubF1+yTH7FuZO2allpANU=; b=SqT0zTTgv+xMLqS+L0bJ00z3uGzPIv4rf0q6B3IeLGytu287a/i41l060TGuPUEafW DYHYplX3YqSTqYZx+N90EKVSlzL8EgovuBcN/E5L1Cx8aBwhiXBvJgtY66eCR3lBSt6U lAP8e7QrBJ1EVlBhgmi5NkYPBv1mD93uFoe9PJNGX2hHU1OzgHCD/k0Xe/3fAh2hW6GK fn573uL82RXmjmoVjjYQGxfvXkl2Mxd0r/7xCs9APfNQg4odG76GggUwL+Mnq1x2NTeU wXndhyI4NjhoF5OToBYc+ZCminaEotTgaMjghQDwUbheu8zub3a9FNEk39HeekUqtwMI 1WgQ== X-Forwarded-Encrypted: i=1; AJvYcCVTwkcyH+5HM2tMnPq0aHarZdE6uXCZzTBCxdGTlN0X+c5mNgqiao9ykLhzMsUSfsWBnmkAlDGidg==@kvack.org X-Gm-Message-State: AOJu0Yym8ZhV1Mi+zTKXKYIj7qLntvq9TdXVibYOhmjR9z2juEpoe5Ud tUtH46umdUJ0V+dsRAc/rdjW0gt7nlxBtV955ibugThTq1FKpbbh1Ajcsn7Gn0tW5XAKO1ydLH7 +ng== X-Google-Smtp-Source: AGHT+IEUu6XihEhnY+t2e9Vxs/5iDORNtXg6aG1VSTYb7FdqL6TFnIMalLv7s/eH76D4Kj2pF+CEoHOIyug= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:868a:0:b0:e28:fc1b:66bb with SMTP id 3f1490d57ef6-e2e3a6b3017mr1697276.6.1729703287925; Wed, 23 Oct 2024 10:08:07 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:56 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-4-surenb@google.com> Subject: [PATCH v4 3/6] alloc_tag: load module tags into separate contiguous memory From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: q3eziwodc3sgmafdosrmoiqkkop1z8q8 X-Rspamd-Queue-Id: 1D38318000E X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1729703285-696397 X-HE-Meta: U2FsdGVkX1//QxaEYsJO14Hm9LhNEDHIL4SYnYuIB7JB1BZdmm24YnSOyjTmNt1e5FJ/9xRnC34W5Uph9fb1uWh3cTJfefZSkunderNx+w8RJGsQonYcNIZbaQQ9ebyiOsrd1Tsxd+U1HLP/bWtInQZl3+7eW3jjRSnMsFdqIdUNhNukM/CSVPIext0AATUgA99jeTzm5qO+yiFpEgsvgoRV9v3FQ3nHlywvNbhw/7TRljw0k6x/OPnZvnFEABzFbuzcodTkCnnuWZ08psGdLOKUnPtUaxDlMLiYjXbCxgn6RP/oz3iZRm8fF0IxzHql8XQcoyMjmtJ6I2fN6N8olTxbiGVwVPJbMODtAyhVdT9UWMQM5s9J0LNHDI2Nmm5wYr41CAtwRhXTNkBHW3LtITZWuS3fbMDEKP+8+jBvjd06iFLzPCfM3HHK3uMQ4E2FUEf/WQAjSUBpHr06CImOx4zBLMthBFw+xrdlbuu+K5Tm49Zs3aHFdcozMTA1m4lp/NjvgT//wmmwueT+EQ7nEbjpMvnLOgGcs7j2wPZNdd42ynBjOZR2QkUn22Bg5X7c/MthQ41AOpc7xuPgznbppu16Rt1MlTiqLhfQnsZ2du8S9Y/9nhvHUMBP7okHS0KDao25kyPPNYBDZr0euoLlivWUugGRRsRP3ZJJyKbgKMJ3RWc6bOj/1ApwMgyUoY6rwkVAw7wfIx84ntK34AB4kqcQZ7hQdWH+afKI2x3qZD8e+WEsHy41Fs76SVQ5a8AxTcPUbNhBI1kg439Kh/MCRYzGOGDHBrdQqrGynvWg9oZP+Duwz6BcKhkJ8+8ebZCQ2hCRrGT88Kkfp5gggfkvMVmQJmrTOKe+IAe7fzu5qbSKqeLej9u8hK2ekXb+NkYojlGM1BUdjQcYiQW0l9q+mnKTm0V7NSlVk1urcE4RTSXVz0u/r0Q90KBbeXGk42qH2UeAK55nhejlF7Lsmce HknpIcls GIijt9o27xUAabg2544Nw4K76fVQwByeH/E/b5yLb2Dwx4gqGCooqnDo9vUEwADrkSB340UJ9Jp737jeoEJMfhIgR+US/cwpOHlLkBQZVHIBWilQkp2iXrK24SDjJOsTmVtZTsggYs5HbIyWDstJXaLLeB3lzAOh+g0pykafMVN+iORC09bt3+BuzDgk8tTYOubZcRUzQyMjBYcXmmN4fyO0X68JK9j/n4LtHoBgZ+2kX+3/6/XjCV1QjGZtrA7T887LSi+zhgZuwHR2ZrqCIDMcw2iV4D9UILREe+8RhxsPT1UGWfqqxI0rc18wiKbHzR6ySd0nNqg1EshSsbXnlt9fZaq71fv/580QiHQpNsw6tW9uCtwOMgYUB2LqWVFYGqdeGf7Qdj9WIOBKzMWExVWfrh4DlMWW2JukosraT80OCLWmvZfDCKx1pXO+tF+27N7o7Q6FHqAtp9wQ9+8hFMKiZuV9abtdO75/u2nG5my2K+EtuX1yy1qvv3nu4O4J6W+Q9hG/Ojbod7jQ3r8fRCrAheiIA3Wovkr6jzxOd2quBq60gOMSOQMTu63Orwk5/qmQoO0XDIki1Bp0J1tEB4Z/u0A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a module gets unloaded there is a possibility that some of the allocations it made are still used and therefore the allocation tags corresponding to these allocations are still referenced. As such, the memory for these tags can't be freed. This is currently handled as an abnormal situation and module's data section is not being unloaded. To handle this situation without keeping module's data in memory, allow codetags with longer lifespan than the module to be loaded into their own separate memory. The in-use memory areas and gaps after module unloading in this separate memory are tracked using maple trees. Allocation tags arrange their separate memory so that it is virtually contiguous and that will allow simple allocation tag indexing later on in this patchset. The size of this virtually contiguous memory is set to store up to 100000 allocation tags. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/asm-generic/codetag.lds.h | 19 +++ include/linux/alloc_tag.h | 13 +- include/linux/codetag.h | 37 ++++- kernel/module/main.c | 80 ++++++---- lib/alloc_tag.c | 249 +++++++++++++++++++++++++++--- lib/codetag.c | 100 +++++++++++- scripts/module.lds.S | 5 +- 7 files changed, 441 insertions(+), 62 deletions(-) diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h index 64f536b80380..372c320c5043 100644 --- a/include/asm-generic/codetag.lds.h +++ b/include/asm-generic/codetag.lds.h @@ -11,4 +11,23 @@ #define CODETAG_SECTIONS() \ SECTION_WITH_BOUNDARIES(alloc_tags) +/* + * Module codetags which aren't used after module unload, therefore have the + * same lifespan as the module and can be safely unloaded with the module. + */ +#define MOD_CODETAG_SECTIONS() + +#define MOD_SEPARATE_CODETAG_SECTION(_name) \ + .codetag.##_name : { \ + SECTION_WITH_BOUNDARIES(_name) \ + } + +/* + * For codetags which might be used after module unload, therefore might stay + * longer in memory. Each such codetag type has its own section so that we can + * unload them individually once unused. + */ +#define MOD_SEPARATE_CODETAG_SECTIONS() \ + MOD_SEPARATE_CODETAG_SECTION(alloc_tags) + #endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 1f0a9ff23a2c..7431757999c5 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +struct alloc_tag_module_section { + unsigned long start_addr; + unsigned long end_addr; + /* used size */ + unsigned long size; +}; + #ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG #define CODETAG_EMPTY ((void *)1) @@ -54,6 +61,8 @@ static inline void set_codetag_empty(union codetag_ref *ref) {} #ifdef CONFIG_MEM_ALLOC_PROFILING +#define ALLOC_TAG_SECTION_NAME "alloc_tags" + struct codetag_bytes { struct codetag *ct; s64 bytes; @@ -76,7 +85,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { \ + __section(ALLOC_TAG_SECTION_NAME) = { \ .ct = CODE_TAG_INIT, \ .counters = &_shared_alloc_tag }; @@ -85,7 +94,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") = { \ + __section(ALLOC_TAG_SECTION_NAME) = { \ .ct = CODE_TAG_INIT, \ .counters = &_alloc_tag_cntr }; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c2a579ccd455..d10bd9810d32 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,8 +35,15 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - bool (*module_unload)(struct codetag_type *cttype, + void (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); +#ifdef CONFIG_MODULES + void (*module_replaced)(struct module *mod, struct module *new_mod); + bool (*needs_section_mem)(struct module *mod, unsigned long size); + void *(*alloc_section_mem)(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align); + void (*free_section_mem)(struct module *mod, bool used); +#endif }; struct codetag_iterator { @@ -71,11 +78,31 @@ struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) + +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size); +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align); +void codetag_free_module_sections(struct module *mod); +void codetag_module_replaced(struct module *mod, struct module *new_mod); void codetag_load_module(struct module *mod); -bool codetag_unload_module(struct module *mod); -#else +void codetag_unload_module(struct module *mod); + +#else /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ + +static inline bool +codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) { return false; } +static inline void * +codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) { return NULL; } +static inline void codetag_free_module_sections(struct module *mod) {} +static inline void codetag_module_replaced(struct module *mod, struct module *new_mod) {} static inline void codetag_load_module(struct module *mod) {} -static inline bool codetag_unload_module(struct module *mod) { return true; } -#endif +static inline void codetag_unload_module(struct module *mod) {} + +#endif /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index ef54733bd7d2..1787686e5cae 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1254,22 +1254,17 @@ static int module_memory_alloc(struct module *mod, enum mod_mem_type type) return 0; } -static void module_memory_free(struct module *mod, enum mod_mem_type type, - bool unload_codetags) +static void module_memory_free(struct module *mod, enum mod_mem_type type) { struct module_memory *mem = &mod->mem[type]; - void *ptr = mem->base; if (mem->is_rox) vfree(mem->rw_copy); - if (!unload_codetags && mod_mem_type_is_core_data(type)) - return; - - execmem_free(ptr); + execmem_free(mem->base); } -static void free_mod_mem(struct module *mod, bool unload_codetags) +static void free_mod_mem(struct module *mod) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1280,25 +1275,20 @@ static void free_mod_mem(struct module *mod, bool unload_codetags) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod, type, unload_codetags); + module_memory_free(mod, type); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod, MOD_DATA, unload_codetags); + module_memory_free(mod, MOD_DATA); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { - bool unload_codetags; - trace_module_free(mod); - unload_codetags = codetag_unload_module(mod); - if (!unload_codetags) - pr_warn("%s: memory allocation(s) from the module still alive, cannot unload cleanly\n", - mod->name); + codetag_unload_module(mod); mod_sysfs_teardown(mod); @@ -1341,7 +1331,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod, unload_codetags); + free_mod_mem(mod); } void *__symbol_get(const char *symbol) @@ -1606,6 +1596,20 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i if (WARN_ON_ONCE(type == MOD_INVALID)) continue; + /* + * Do not allocate codetag memory as we load it into + * preallocated contiguous memory. + */ + if (codetag_needs_module_section(mod, sname, s->sh_size)) { + /* + * s->sh_entsize won't be used but populate the + * type field to avoid confusion. + */ + s->sh_entsize = ((unsigned long)(type) & SH_ENTSIZE_TYPE_MASK) + << SH_ENTSIZE_TYPE_SHIFT; + continue; + } + s->sh_entsize = module_get_offset_and_type(mod, type, s, i); pr_debug("\t%s\n", sname); } @@ -2280,6 +2284,7 @@ static int move_module(struct module *mod, struct load_info *info) int i; enum mod_mem_type t = 0; int ret = -ENOMEM; + bool codetag_section_found = false; for_each_mod_mem_type(type) { if (!mod->mem[type].size) { @@ -2291,7 +2296,7 @@ static int move_module(struct module *mod, struct load_info *info) ret = module_memory_alloc(mod, type); if (ret) { t = type; - goto out_enomem; + goto out_err; } } @@ -2300,15 +2305,33 @@ static int move_module(struct module *mod, struct load_info *info) for (i = 0; i < info->hdr->e_shnum; i++) { void *dest; Elf_Shdr *shdr = &info->sechdrs[i]; - enum mod_mem_type type = shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; - unsigned long offset = shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK; + const char *sname; unsigned long addr; if (!(shdr->sh_flags & SHF_ALLOC)) continue; - addr = (unsigned long)mod->mem[type].base + offset; - dest = mod->mem[type].rw_copy + offset; + sname = info->secstrings + shdr->sh_name; + /* + * Load codetag sections separately as they might still be used + * after module unload. + */ + if (codetag_needs_module_section(mod, sname, shdr->sh_size)) { + dest = codetag_alloc_module_section(mod, sname, shdr->sh_size, + arch_mod_section_prepend(mod, i), shdr->sh_addralign); + if (IS_ERR(dest)) { + ret = PTR_ERR(dest); + goto out_err; + } + addr = (unsigned long)dest; + codetag_section_found = true; + } else { + enum mod_mem_type type = shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; + unsigned long offset = shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK; + + addr = (unsigned long)mod->mem[type].base + offset; + dest = mod->mem[type].rw_copy + offset; + } if (shdr->sh_type != SHT_NOBITS) { /* @@ -2320,7 +2343,7 @@ static int move_module(struct module *mod, struct load_info *info) if (i == info->index.mod && (WARN_ON_ONCE(shdr->sh_size != sizeof(struct module)))) { ret = -ENOEXEC; - goto out_enomem; + goto out_err; } memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size); } @@ -2336,9 +2359,12 @@ static int move_module(struct module *mod, struct load_info *info) } return 0; -out_enomem: +out_err: for (t--; t >= 0; t--) - module_memory_free(mod, t, true); + module_memory_free(mod, t); + if (codetag_section_found) + codetag_free_module_sections(mod); + return ret; } @@ -2459,6 +2485,8 @@ static struct module *layout_and_allocate(struct load_info *info, int flags) /* Module has been copied to its final place now: return it. */ mod = (void *)info->sechdrs[info->index.mod].sh_addr; kmemleak_load_module(mod, info); + codetag_module_replaced(info->mod, mod); + return mod; } @@ -2468,7 +2496,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod, true); + free_mod_mem(mod); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 435aa837e550..d9f51169ffeb 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include @@ -9,6 +10,7 @@ #include #define ALLOCINFO_FILE_NAME "allocinfo" +#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support __meminitdata = true; @@ -174,31 +176,226 @@ static void __init procfs_init(void) } } -static bool alloc_tag_module_unload(struct codetag_type *cttype, - struct codetag_module *cmod) +#ifdef CONFIG_MODULES + +static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +/* A dummy object used to indicate an unloaded module */ +static struct module unloaded_mod; +/* A dummy object used to indicate a module prepended area */ +static struct module prepend_mod; + +static struct alloc_tag_module_section module_tags; + +static bool needs_section_mem(struct module *mod, unsigned long size) { - struct codetag_iterator iter = codetag_get_ct_iter(cttype); - struct alloc_tag_counters counter; - bool module_unused = true; - struct alloc_tag *tag; - struct codetag *ct; + return size >= sizeof(struct alloc_tag); +} + +static struct alloc_tag *find_used_tag(struct alloc_tag *from, struct alloc_tag *to) +{ + while (from <= to) { + struct alloc_tag_counters counter; - for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { - if (iter.cmod != cmod) + counter = alloc_tag_read(from); + if (counter.bytes) + return from; + from++; + } + + return NULL; +} + +/* Called with mod_area_mt locked */ +static void clean_unused_module_areas_locked(void) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_for_each(&mas, val, module_tags.size) { + if (val != &unloaded_mod) continue; - tag = ct_to_alloc_tag(ct); - counter = alloc_tag_read(tag); + /* Release area if all tags are unused */ + if (!find_used_tag((struct alloc_tag *)(module_tags.start_addr + mas.index), + (struct alloc_tag *)(module_tags.start_addr + mas.last))) + mas_erase(&mas); + } +} + +/* Called with mod_area_mt locked */ +static bool find_aligned_area(struct ma_state *mas, unsigned long section_size, + unsigned long size, unsigned int prepend, unsigned long align) +{ + bool cleanup_done = false; + +repeat: + /* Try finding exact size and hope the start is aligned */ + if (!mas_empty_area(mas, 0, section_size - 1, prepend + size)) { + if (IS_ALIGNED(mas->index + prepend, align)) + return true; + + /* Try finding larger area to align later */ + mas_reset(mas); + if (!mas_empty_area(mas, 0, section_size - 1, + size + prepend + align - 1)) + return true; + } - if (WARN(counter.bytes, - "%s:%u module %s func:%s has %llu allocated at module unload", - ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) - module_unused = false; + /* No free area, try cleanup stale data and repeat the search once */ + if (!cleanup_done) { + clean_unused_module_areas_locked(); + cleanup_done = true; + mas_reset(mas); + goto repeat; } - return module_unused; + return false; +} + +static void *reserve_module_tags(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align) +{ + unsigned long section_size = module_tags.end_addr - module_tags.start_addr; + MA_STATE(mas, &mod_area_mt, 0, section_size - 1); + unsigned long offset; + void *ret = NULL; + + /* If no tags return NULL */ + if (size < sizeof(struct alloc_tag)) + return NULL; + + /* + * align is always power of 2, so we can use IS_ALIGNED and ALIGN. + * align 0 or 1 means no alignment, to simplify set to 1. + */ + if (!align) + align = 1; + + mas_lock(&mas); + if (!find_aligned_area(&mas, section_size, size, prepend, align)) { + ret = ERR_PTR(-ENOMEM); + goto unlock; + } + + /* Mark found area as reserved */ + offset = mas.index; + offset += prepend; + offset = ALIGN(offset, align); + if (offset != mas.index) { + unsigned long pad_start = mas.index; + + mas.last = offset - 1; + mas_store(&mas, &prepend_mod); + if (mas_is_err(&mas)) { + ret = ERR_PTR(xa_err(mas.node)); + goto unlock; + } + mas.index = offset; + mas.last = offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) { + mas.index = pad_start; + mas_erase(&mas); + ret = ERR_PTR(xa_err(mas.node)); + } + } else { + mas.last = offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) + ret = ERR_PTR(xa_err(mas.node)); + } +unlock: + mas_unlock(&mas); + + if (IS_ERR(ret)) + return ret; + + if (module_tags.size < offset + size) + module_tags.size = offset + size; + + return (struct alloc_tag *)(module_tags.start_addr + offset); } +static void release_module_tags(struct module *mod, bool used) +{ + MA_STATE(mas, &mod_area_mt, module_tags.size, module_tags.size); + struct alloc_tag *tag; + struct module *val; + + mas_lock(&mas); + mas_for_each_rev(&mas, val, 0) + if (val == mod) + break; + + if (!val) /* module not found */ + goto out; + + if (!used) + goto release_area; + + /* Find out if the area is used */ + tag = find_used_tag((struct alloc_tag *)(module_tags.start_addr + mas.index), + (struct alloc_tag *)(module_tags.start_addr + mas.last)); + if (tag) { + struct alloc_tag_counters counter = alloc_tag_read(tag); + + pr_info("%s:%u module %s func:%s has %llu allocated at module unload\n", + tag->ct.filename, tag->ct.lineno, tag->ct.modname, + tag->ct.function, counter.bytes); + } else { + used = false; + } +release_area: + mas_store(&mas, used ? &unloaded_mod : NULL); + val = mas_prev_range(&mas, 0); + if (val == &prepend_mod) + mas_store(&mas, NULL); +out: + mas_unlock(&mas); +} + +static void replace_module(struct module *mod, struct module *new_mod) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_lock(&mas); + mas_for_each(&mas, val, module_tags.size) { + if (val != mod) + continue; + + mas_store_gfp(&mas, new_mod, GFP_KERNEL); + break; + } + mas_unlock(&mas); +} + +static int __init alloc_mod_tags_mem(void) +{ + /* Allocate space to copy allocation tags */ + module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, + MODULE_ALLOC_TAG_VMAP_SIZE); + if (!module_tags.start_addr) + return -ENOMEM; + + module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; + + return 0; +} + +static void __init free_mod_tags_mem(void) +{ + execmem_free((void *)module_tags.start_addr); + module_tags.start_addr = 0; +} + +#else /* CONFIG_MODULES */ + +static inline int alloc_mod_tags_mem(void) { return 0; } +static inline void free_mod_tags_mem(void) {} + +#endif /* CONFIG_MODULES */ + static int __init setup_early_mem_profiling(char *str) { bool enable; @@ -274,14 +471,26 @@ static inline void sysctl_init(void) {} static int __init alloc_tag_init(void) { const struct codetag_type_desc desc = { - .section = "alloc_tags", - .tag_size = sizeof(struct alloc_tag), - .module_unload = alloc_tag_module_unload, + .section = ALLOC_TAG_SECTION_NAME, + .tag_size = sizeof(struct alloc_tag), +#ifdef CONFIG_MODULES + .needs_section_mem = needs_section_mem, + .alloc_section_mem = reserve_module_tags, + .free_section_mem = release_module_tags, + .module_replaced = replace_module, +#endif }; + int res; + + res = alloc_mod_tags_mem(); + if (res) + return res; alloc_tag_cttype = codetag_register_type(&desc); - if (IS_ERR(alloc_tag_cttype)) + if (IS_ERR(alloc_tag_cttype)) { + free_mod_tags_mem(); return PTR_ERR(alloc_tag_cttype); + } sysctl_init(); procfs_init(); diff --git a/lib/codetag.c b/lib/codetag.c index d1fbbb7c2ec3..654496952f86 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -207,6 +207,94 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) } #ifdef CONFIG_MODULES +#define CODETAG_SECTION_PREFIX ".codetag." + +/* Some codetag types need a separate module section */ +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) +{ + const char *type_name; + struct codetag_type *cttype; + bool ret = false; + + if (strncmp(name, CODETAG_SECTION_PREFIX, strlen(CODETAG_SECTION_PREFIX))) + return false; + + type_name = name + strlen(CODETAG_SECTION_PREFIX); + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) == 0) { + if (!cttype->desc.needs_section_mem) + break; + + down_write(&cttype->mod_lock); + ret = cttype->desc.needs_section_mem(mod, size); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) +{ + const char *type_name = name + strlen(CODETAG_SECTION_PREFIX); + struct codetag_type *cttype; + void *ret = NULL; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) == 0) { + if (WARN_ON(!cttype->desc.alloc_section_mem)) + break; + + down_write(&cttype->mod_lock); + ret = cttype->desc.alloc_section_mem(mod, size, prepend, align); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void codetag_free_module_sections(struct module *mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.free_section_mem) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.free_section_mem(mod, false); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + +void codetag_module_replaced(struct module *mod, struct module *new_mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.module_replaced) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.module_replaced(mod, new_mod); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + void codetag_load_module(struct module *mod) { struct codetag_type *cttype; @@ -220,13 +308,12 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -bool codetag_unload_module(struct module *mod) +void codetag_unload_module(struct module *mod) { struct codetag_type *cttype; - bool unload_ok = true; if (!mod) - return true; + return; /* await any module's kfree_rcu() operations to complete */ kvfree_rcu_barrier(); @@ -246,18 +333,17 @@ bool codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - if (!cttype->desc.module_unload(cttype, cmod)) - unload_ok = false; + cttype->desc.module_unload(cttype, cmod); cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); } up_write(&cttype->mod_lock); + if (found && cttype->desc.free_section_mem) + cttype->desc.free_section_mem(mod, true); } mutex_unlock(&codetag_lock); - - return unload_ok; } #endif /* CONFIG_MODULES */ diff --git a/scripts/module.lds.S b/scripts/module.lds.S index 3f43edef813c..711c6e029936 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -50,7 +50,7 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } .rodata : { @@ -59,9 +59,10 @@ SECTIONS { } #else .data : { - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } #endif + MOD_SEPARATE_CODETAG_SECTIONS() } /* bring in arch-specific sections */ From patchwork Wed Oct 23 17:07:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3F1ECFA452 for ; Wed, 23 Oct 2024 17:08:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73D206B009D; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64F0C6B00A1; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42A5C6B00A4; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1A5C56B009D for ; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B874A1C5BAC for ; Wed, 23 Oct 2024 17:07:52 +0000 (UTC) X-FDA: 82705499856.28.7845AEA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 4E9201A0024 for ; Wed, 23 Oct 2024 17:07:49 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SaCG1iU3; spf=pass (imf19.hostedemail.com: domain of 3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=dVgxTt569usgx4IFIns2QZI7M2pC6QYNyRiyTiGUEZOQLQkX/2aMywAB46HVaqAfalB5Kf YeLBwkPIcLT1agXdT1bMcrf0YFagNDVdv5LnU11rDxVCInDL/7M0BhBM1iWxyP674QyK15 UOUi77lsWxSOieC1EgqFYDAvdYuUtS0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703138; a=rsa-sha256; cv=none; b=nf3ealrI1Nm+zjlg+o6dxYoCHQ2aXHcOYxVaTf29wcDGgwFpGK+0bfYqFmt43wN1bLoefL cjxFD05dx751so6v6hf18MKLrgTvHfF7UUmFm1iLz4u9N6TdXuXZqWdqawNmeYP3AvkQG7 TOJ2rC3yyxGVOd9HP2C6gX3E0XZ+KyY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SaCG1iU3; spf=pass (imf19.hostedemail.com: domain of 3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e3204db795so104968097b3.2 for ; Wed, 23 Oct 2024 10:08:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703290; x=1730308090; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=SaCG1iU3/EULCq4+ouPpEWK7we2fhH4R0V661QuDhyPAbYrLWnqtJ2dCM1HLfcnsOX ZJ0/SqncxBkFPGID+9iVgPErkTV2SBqIEdLTEla9rY4vSwTbdhdQun2R9aDMZ9ZF8ddN Gl/BnJS1ss84qjaa5QNp06JLINQreVmKKqL8uk1m7Uhrf1tBYWv57Hs75dxJK1jq6Stl XkjIAtxp0OoYNkn3K8oqd45Xpf0D51XYy6hRBEyDAfFSz1w+xutlgsp2MojsI4iMuuLT sxf2uyYy7N1/7v5iMZyuKI5TFDBluEP4iG6QhHz2PY0MyFVWg6KEU0q5dGolzDuWwfzv 2wig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703290; x=1730308090; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=r5veWWagqD/Td12mM/uh1y8Ad+KQV6akBPIs3hVnmUS3edyF6EEkv7GOsYaceo0y8H 1kSzsj8dFmOTBhIkjrWixmHqM/vS4uIJbiMCMAYFacRUUeB7KOZ8F+FUtD4vrhvqKfjs Zg+VzrvDdOGUH0Np2ZcgQjcYhxzY4QDW/GAJIcCqXdSAVMOJoFy06PqXSsB4t+F55b5u pj2RwJE/toIpVRo5c394fzWgM6p27S7CR6RNWH9yvD8sGsd4CfPLNsKAWchmfFFkcCyE ytSFmaH0pgHg3PSsbziO9+0yvqosJJOUFY5Ew48bsy/sp+O6XNEc+EzUZIuHw6v/Zsvr lwyg== X-Forwarded-Encrypted: i=1; AJvYcCWlBjlEr1UydBVKDrXaEyYtDhpyUGY4xWE/1SX42kUSm4qfBPvRBjiE7WZLYQxwvwm1cd8fXC4lRw==@kvack.org X-Gm-Message-State: AOJu0Yx4EhMz2Meu39mUv4kvt7LSHduUV7NzNP7oebjfas+3QM5x0Kzj MDbJrxSZCDwhT4dou2p8JaartviPN6tDSC1LzAgrkuV+f20mMMLK0jAmObUlgFKfJDGmieDlSRz qEg== X-Google-Smtp-Source: AGHT+IGGx5cwUMRbaaaRLBx9ozQm5nYkieFYXts2XJ8fpTDFGg4nxvLC1BvVBA4+mhBqSvprWgkBELkrSzs= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a05:690c:4b03:b0:6b0:d571:3540 with SMTP id 00721157ae682-6e7f0f97164mr929077b3.6.1729703289927; Wed, 23 Oct 2024 10:08:09 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:57 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-5-surenb@google.com> Subject: [PATCH v4 4/6] alloc_tag: populate memory for module tags as needed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4E9201A0024 X-Stat-Signature: xopgx8abfhckgzxy499cppes8xf9i4bu X-HE-Tag: 1729703269-911162 X-HE-Meta: U2FsdGVkX1+xYYifc07chbmfo5keOBvAvILHAvBX/YH6IlHMjYkXa6eq4ofBNVlBHGtolI5bXWYP4k+v9gHbum9L/awi9nP+ilMB1XznpDqRYNXtf6qDNXaleQOICvUGIReHU0dNQKJGx6/qvltg3fOdSkGjCngkNB441rl5eb9motyQfoQ7PHIteBKaW87/sdlRmtlHBtRPQCZpZmQUpmH+ZrqBrbSckNpgDKmvR6IB6mr3oQAqwUYKY2yKVkQXCjdCigapgG1qYAZJwPZDpHOtSaitzb6yWF+usr8yJCnAGO1AXetTJEkjD+t4YRyq+gpixz/wWs0ZPrIsL/4z3tnNyshyYaZpA31a1Rey664bG6Om5xduszDkybghww2K1Fb5WHFGn28oBxYiYRlbUk6sFRgxoso5kLeWyBTbU+X4BP5ewBDo1/X0iFbZWNOiHEFGpmJpeDIgqbTJF8TxjFwmI2i+6BHqBj7MFCiaKH6V4YPKaY5J1DmQ/X1uO08ttdtr/Awn9u9BW6CcxECi7ra1caFnIB+X54442OBt+8RtFF1V61xw/EUVlsMDfrNA2OM1HuhKvFbrAhdPmD25Ovodcy9/gKqFMZ2QEK3j9YbVr/YPeB4/CMCCZMC4H01wsDqF8z7DtN5HFUWUozrWCBsG4iLKZIhzrAoH35l2VoeSBQSxTIPWs9PerblvJJ7nRSzJg8d3fnAcsu+l9/q8Icwq1+3teTAajGuijpoQD46yhPUTrgTA8FM7luSGdQdjMKQnHZ8u56/pWcfOIWQWUq81S6scZ64gK6YyV2zUOkr4zKnWqhWYDSo137+ewlHa7N+O23JQTSwjbtuUco+knz3qgMOxbv/kwgwRlpptAkbe9F/40etZiKj2HCoFtR8/5sI8v7oqSYhV/qmH9zFySqDUILnX4kViNgoXZuyR3TpM+fdJR0aeGi/7sILPzliDoi8fQte33erw9Wnt9WF w1H4OpDl nRvHyevt8JKVUvk/e0Y6HJDDosEckat1ZrmiHVdjSZ6Sfcy1SJ/3F43xs4uGKmRXMZw/6iSqetp5OFPWFTgIdKWSBK+Cxru6SeTn73JYGOnY0qQzGfvArQzasP1wFl0TOH2is6kRQKzbaZlts5+RulkzgN4FSSMKJjUhH+pGeiMWTVKuKMyeppAE/lMdppXXf3b+G71na6VWun9FWhTtqHJLohWVaKKVyi3ARAwn4L8/wIUm4iKd6DDCD2eO/5bTcnMzKJro/l0zbJTfWQXF2LpaHf6+Z91dd2mKkCN9GqFcR4+ZhJPqxLJ97LGts+UIXaU0vAQwoFmH41vixH73gVtuNjIHuU4thNgMW8hozWk/8XdX4e5KQw8Rc85Hy46vD4PehC85/Z5k4ziq58Z0jIohnvhdZqTXSXT3+e2HpXzCvC1FBTzUcGkg5zv4CvNNuT7+8WEaml58tWlGbXmveiPhvtfLkS1bqlJsX0KktQmhbfJRZGIid1nRSKmcyFO2cgb/tonzQ6/1RaIo/fNprTcvJSnpxTk8/MFn34/9WT+s2DqhW0qy53/5wlLWOKIBTvJzO8+Pq3vTqnlFvDlsKlKgwdA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/linux/execmem.h | 10 ++++++ include/linux/vmalloc.h | 3 ++ lib/alloc_tag.c | 73 ++++++++++++++++++++++++++++++++++++----- mm/execmem.c | 16 +++++++++ mm/internal.h | 6 ++++ mm/vmalloc.c | 4 +-- 6 files changed, 101 insertions(+), 11 deletions(-) diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 1517fa196bf7..5a5e2917f870 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -139,6 +139,16 @@ void *execmem_alloc(enum execmem_type type, size_t size); */ void execmem_free(void *ptr); +/** + * execmem_vmap - create virtual mapping for EXECMEM_MODULE_DATA memory + * @size: size of the virtual mapping in bytes + * + * Maps virtually contiguous area in the range suitable for EXECMEM_MODULE_DATA. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *execmem_vmap(size_t size); + /** * execmem_update_copy - copy an update to executable memory * @dst: destination address to update diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 27408f21e501..31e9ffd936e3 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -202,6 +202,9 @@ extern int remap_vmalloc_range_partial(struct vm_area_struct *vma, extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, unsigned long pgoff); +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, + struct page **pages, unsigned int page_shift); + /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values * and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index d9f51169ffeb..061e43196247 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,14 +8,15 @@ #include #include #include +#include #define ALLOCINFO_FILE_NAME "allocinfo" #define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT -static bool mem_profiling_support __meminitdata = true; +static bool mem_profiling_support = true; #else -static bool mem_profiling_support __meminitdata; +static bool mem_profiling_support; #endif static struct codetag_type *alloc_tag_cttype; @@ -154,7 +155,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl return nr; } -static void __init shutdown_mem_profiling(void) +static void shutdown_mem_profiling(void) { if (mem_alloc_profiling_enabled()) static_branch_disable(&mem_alloc_profiling_key); @@ -179,6 +180,7 @@ static void __init procfs_init(void) #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +static struct vm_struct *vm_module_tags; /* A dummy object used to indicate an unloaded module */ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ @@ -252,6 +254,33 @@ static bool find_aligned_area(struct ma_state *mas, unsigned long section_size, return false; } +static int vm_module_tags_populate(void) +{ + unsigned long phys_size = vm_module_tags->nr_pages << PAGE_SHIFT; + + if (phys_size < module_tags.size) { + struct page **next_page = vm_module_tags->pages + vm_module_tags->nr_pages; + unsigned long addr = module_tags.start_addr + phys_size; + unsigned long more_pages; + unsigned long nr; + + more_pages = ALIGN(module_tags.size - phys_size, PAGE_SIZE) >> PAGE_SHIFT; + nr = alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_NOWARN, + NUMA_NO_NODE, more_pages, next_page); + if (nr < more_pages || + vmap_pages_range(addr, addr + (nr << PAGE_SHIFT), PAGE_KERNEL, + next_page, PAGE_SHIFT) < 0) { + /* Clean up and error out */ + for (int i = 0; i < nr; i++) + __free_page(next_page[i]); + return -ENOMEM; + } + vm_module_tags->nr_pages += nr; + } + + return 0; +} + static void *reserve_module_tags(struct module *mod, unsigned long size, unsigned int prepend, unsigned long align) { @@ -310,8 +339,18 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, if (IS_ERR(ret)) return ret; - if (module_tags.size < offset + size) + if (module_tags.size < offset + size) { + int grow_res; + module_tags.size = offset + size; + grow_res = vm_module_tags_populate(); + if (grow_res) { + shutdown_mem_profiling(); + pr_err("Failed to allocate memory for allocation tags in the module %s. Memory allocation profiling is disabled!\n", + mod->name); + return ERR_PTR(grow_res); + } + } return (struct alloc_tag *)(module_tags.start_addr + offset); } @@ -372,12 +411,23 @@ static void replace_module(struct module *mod, struct module *new_mod) static int __init alloc_mod_tags_mem(void) { - /* Allocate space to copy allocation tags */ - module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, - MODULE_ALLOC_TAG_VMAP_SIZE); - if (!module_tags.start_addr) + /* Map space to copy allocation tags */ + vm_module_tags = execmem_vmap(MODULE_ALLOC_TAG_VMAP_SIZE); + if (!vm_module_tags) { + pr_err("Failed to map %lu bytes for module allocation tags\n", + MODULE_ALLOC_TAG_VMAP_SIZE); + module_tags.start_addr = 0; return -ENOMEM; + } + vm_module_tags->pages = kmalloc_array(get_vm_area_size(vm_module_tags) >> PAGE_SHIFT, + sizeof(struct page *), GFP_KERNEL | __GFP_ZERO); + if (!vm_module_tags->pages) { + free_vm_area(vm_module_tags); + return -ENOMEM; + } + + module_tags.start_addr = (unsigned long)vm_module_tags->addr; module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; return 0; @@ -385,8 +435,13 @@ static int __init alloc_mod_tags_mem(void) static void __init free_mod_tags_mem(void) { - execmem_free((void *)module_tags.start_addr); + int i; + module_tags.start_addr = 0; + for (i = 0; i < vm_module_tags->nr_pages; i++) + __free_page(vm_module_tags->pages[i]); + kfree(vm_module_tags->pages); + free_vm_area(vm_module_tags); } #else /* CONFIG_MODULES */ diff --git a/mm/execmem.c b/mm/execmem.c index 576a57e2161f..5c0f9f2d6f83 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -368,6 +368,22 @@ void execmem_free(void *ptr) vfree(ptr); } +struct vm_struct *execmem_vmap(size_t size) +{ + struct execmem_range *range = &execmem_info->ranges[EXECMEM_MODULE_DATA]; + struct vm_struct *area; + + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->start, range->end, NUMA_NO_NODE, + GFP_KERNEL, __builtin_return_address(0)); + if (!area && range->fallback_start) + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->fallback_start, range->fallback_end, + NUMA_NO_NODE, GFP_KERNEL, __builtin_return_address(0)); + + return area; +} + void *execmem_update_copy(void *dst, const void *src, size_t size) { return text_poke_copy(dst, src, size); diff --git a/mm/internal.h b/mm/internal.h index 508f7802dd2b..f1ce0e10bed8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1219,6 +1219,12 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_folio(struct folio *folio); +struct vm_struct *__get_vm_area_node(unsigned long size, + unsigned long align, unsigned long shift, + unsigned long flags, unsigned long start, + unsigned long end, int node, gfp_t gfp_mask, + const void *caller); + /* * mm/gup.c */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 74c0a5eae210..7ed39d104201 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -653,7 +653,7 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; @@ -3106,7 +3106,7 @@ static void clear_vm_uninitialized_flag(struct vm_struct *vm) vm->flags &= ~VM_UNINITIALIZED; } -static struct vm_struct *__get_vm_area_node(unsigned long size, +struct vm_struct *__get_vm_area_node(unsigned long size, unsigned long align, unsigned long shift, unsigned long flags, unsigned long start, unsigned long end, int node, gfp_t gfp_mask, const void *caller) From patchwork Wed Oct 23 17:07:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DADE2CFA454 for ; Wed, 23 Oct 2024 17:08:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C90446B00A1; Wed, 23 Oct 2024 13:08:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C18B36B00A4; Wed, 23 Oct 2024 13:08:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6EEC6B00A6; Wed, 23 Oct 2024 13:08:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 830F96B00A1 for ; Wed, 23 Oct 2024 13:08:15 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 963581A0BC3 for ; Wed, 23 Oct 2024 17:07:43 +0000 (UTC) X-FDA: 82705499310.03.26EC302 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 6031F20024 for ; Wed, 23 Oct 2024 17:07:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=swBVk8SK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3ey0ZZwYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ey0ZZwYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703241; a=rsa-sha256; cv=none; b=t7tGUEOOFa1SJWuEaX6COxZ2pU9Q+RHHIckBQuqvDl28HjjmaK0Tw5p/QLV67AvgysV/tf Zk7bPmZGetU4VBOm5HDST/3UlZ53wCNzN1fo8yq26qCxvpvGT004bRjU3f6gb/0YJxrmRC bkW72hmjQlcpRyFRxVfbj0budIKo63U= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=swBVk8SK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3ey0ZZwYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ey0ZZwYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703241; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8wnMIe1iCPC4H6xxK8PR5xB7BjuNdLAE9wA+yh8Gbwc=; b=8BS8XP99Awvhs7nQPX+DMBY6eG7naQRuNcAwC50hU7iKIPPAt1fs41oqZ8MtEr55ibXSXw qNfa4l0Jn8CS94RnlgtJEqRHkZ9FM3DdXiWcJDmVLtxKTbFA69yb8g4eoDJAC+8mIghoZs t8zCxqlpTP/0haNGFX7naFA0iRSdEIY= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e297a366304so21237276.2 for ; Wed, 23 Oct 2024 10:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703292; x=1730308092; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8wnMIe1iCPC4H6xxK8PR5xB7BjuNdLAE9wA+yh8Gbwc=; b=swBVk8SK5NZp+bQpQMWbW0HCR5sSCWS8N0gF5cVtMc1vvGA/+5R3DM9DSHq+3c2j0v Nqss0biYWa48hkUk5d1CqhM78CqgdDdllTtde0WFpykwPxIwEVojSnURrwexo4viohmX 7hij1KLqer2SEbNcSYOQlcUMAjUPcM1WtpCf/jqGihTlJgK5r9mdmCfTBJJpHAYkOo49 zG5hhw8kATY4OdPwNEi8gk20vmQqhSx8cXV09+o4BeYD/AJsVIZzo6r9yrVUxfoKGm2L jnOAiWpA7yGt34Rg1R4ujjcjoECfWSxdryat6yyrraK5X6iOirLfYXD+GE3v0102iCyB pHNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703292; x=1730308092; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8wnMIe1iCPC4H6xxK8PR5xB7BjuNdLAE9wA+yh8Gbwc=; b=GnV+KI70LM84NUrRrwMgaVHOweBiO6ZONwV7i4vtd+IDW7yDfFD60B7AGa6nmp2B3T ZjaPDjZGLFM4sQUTH827pn3DZ+/o195saRLmtXIvUAQUKd0ysrrqtH+9h1yK/3nydQpw yvtjw6l7vVDcqJDDTFSmz6DgYIVKdF5ryyfPjUKA/eTH2zAlqX7mLmB9LzMTWJ20Fobu tq8HSAW/TntpSTi8Gepvf9sa3iDYW3cfRS9W6ub2YyZbyNFRgpkccFbq/ocIevfosnuh GWwzifkb9JtL8hAeLegf5j4vNbUNmK9b9/imaDD/8hgFX18tp0UfAwX/MK/6yjYEXfZT cggw== X-Forwarded-Encrypted: i=1; AJvYcCW+eDyc2TVnEG50PluGdGuAdOs+eiC7JGgOFnRYi2Ed6q0gQydne+RXoWxDSnV3rC3no3THI87vyA==@kvack.org X-Gm-Message-State: AOJu0YwrRW9c8ki3xTBKfYvv7b91w7tTSB/DBGs1Yvelm9RPesIvL1mI 7N+dvkJ6jiM6fqVQ2L/72q2HtGxRd1TH+fag6F0LZCuyZuO66u1XXv6/6GD19WU+Cg6jkHSiBao AkQ== X-Google-Smtp-Source: AGHT+IHWTSefrY2mOEgvtjZrHcKSjJbm4CbIxnZltjP19nac0+xcMzP1s41Uk58cU65/uvtLl+YlVZeSEX4= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a5b:70b:0:b0:e28:e4a7:3206 with SMTP id 3f1490d57ef6-e2e3a6d62a7mr4485276.8.1729703291999; Wed, 23 Oct 2024 10:08:11 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:58 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-6-surenb@google.com> Subject: [PATCH v4 5/6] alloc_tag: introduce pgtag_ref_handle to abstract page tag references From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 6031F20024 X-Rspamd-Server: rspam01 X-Stat-Signature: wbbesoed8hapcdbagzchew7uizuhgioh X-HE-Tag: 1729703275-4511 X-HE-Meta: U2FsdGVkX19DMNQEvB9XHulPmhhnnyqRZ+lSyXB6IxUjZDEisXW8Sqv9TloMUDHJsQevYqwpxcQi2lxDXabDvX0QmYdjJn1ofinOSz6aJRYPMgCfeiLE5pBVIpuCbUSMe/qDhOG50hwFrBV4LTl/qj3EqK3HS9fmxOpfqif851x8eG6yYrxyphopdHDDGnAdGUJ6NJsCX+UY0QwFUVjtaEjXnZzHaw3ZL7QTyM/6OCik5EjXKwQY8xgMCYMGoPznngNotfeCic8Cr+g69iAA3ye7K5Wy3YCFNIJcmMlqSMOZikAynlB87laN1KiGI7brUMoqpO70DUKpzz1QqKMb4tl0se4NrKU2kbgGtc7wLmw/FRfZCOlAyscP7ILu0yPK3J6SPWwTJq9hFiy47OXY00rb1QndV57VdMbZ6wLSuYRmfiPAZ1jGiPn9aUHnLne7JAAjuPOJFb5jPVNfLo6wZ3Qf5NPiaDHHG157aKeEahzSQdIlZGqeN2eceIc6rejn8LuVLFrx4hFgcRJgPzm+f+MT9EjU9Fef/pCtp1fFo+pNX6EbPYdsp/ZjIGBu0EKQxu15V/wnik6ovRalbK0OuHGiP55ht2px4rySTPqTfHpLymMfpzz9KZkQaiSctI03QldCedQcubfllD8z+z8bfGSvIZIqcX/N/CLHfdf/4+IjmEjeGE+XD+0H++upgrasVMPWUd8+zhL0laeBmqKBO1s1FSMizf3QiKqC4tmibYtlXI/K/YcCSQlOubq6nuNBe2y7djxNMjJNzC5YRROgyUR143b7d3kvwzDLaEeS1f8pfd7WTcuFcwIvj3x4oZET+Rs4bgAHOjpzIdyzcFKzltmw2r0Sm4F8cdlBGdWRE7ICbI3iby4iDcgAz0U4yX/3VTgIAGn6doJvDJ+U5+7n0Zp5xDMhfS4eAEdtMUfjppWxPWDAw8DgppNsURvGZW5cLQ7T1f8gW3pYppYNwk3 80tJvrem SldgPhLp7QkxCwQzrL942GAMq+p36/e7dlTq4Z9E9yR9HJTokNRhyKgF8GZy+YySTOasMwYV30yHVlyMQw4upWw11uC4fQUx9pZoepIDoYTBvagBqYTtqTgxWl0/j/GjvssR2TOTQ7FRhZ6MPBH9SjeGgx3/FViT7VteIImdg4QDOIck23zL4VpV0tncOA393n2PwBOVU6Inpqw4UkXKeRQvvLtIp3D+Mq7ot1DGJzeyFdtTe7TaqUem+CC+U7nUC6QLTVQq2GR+5RFM1yOZrII/0VxPnfk1FJ2uVc3moPGPVcT5rUYoz2v3yyGs24A/G2a4Whmfx5FW2KpoGjvQkozVVw6s5Z913enyAGTtuaXxBaJ5GCL918Rpg+36Aj27APoLjTLbiJHEx7R6iM3Nom6ZsNqhuznFjSIDokMJPXVcdg2ycH2kHynoG04OkwAhAo0cLArDie8Bpc1xZ35nAoiVR4iaY+vn7p2Sdac4d4JYQ/ccd3x1NSYbNBFR7IA6BpUshFhRy/m8t/GUMOmZO1Whry9dZ2YEyHwHvzHbDJrRkl+KOqtnBVOr56cFAfM3kWA+h4YXiHAZEQXTwATGFnb0XNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To simplify later changes to page tag references, introduce new pgtag_ref_handle type. This allows easy replacement of page_ext as a storage of page allocation tags. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/linux/mm.h | 25 +++++----- include/linux/pgalloc_tag.h | 92 ++++++++++++++++++++++--------------- 2 files changed, 67 insertions(+), 50 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5cd22303fbc0..8efb4a6a1a70 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4180,37 +4180,38 @@ static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new return; for (i = nr_pages; i < (1 << old_order); i += nr_pages) { - union codetag_ref *ref = get_page_tag_ref(folio_page(folio, i)); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { + if (get_page_tag_ref(folio_page(folio, i), &ref, &handle)) { /* Set new reference to point to the original tag */ - alloc_tag_ref_set(ref, tag); - put_page_tag_ref(ref); + alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) { + union pgtag_ref_handle handle; + union codetag_ref ref; struct alloc_tag *tag; - union codetag_ref *ref; tag = pgalloc_tag_get(&old->page); if (!tag) return; - ref = get_page_tag_ref(&new->page); - if (!ref) + if (!get_page_tag_ref(&new->page, &ref, &handle)) return; /* Clear the old ref to the original allocation tag. */ clear_page_tag_ref(&old->page); /* Decrement the counters of the tag on get_new_folio. */ - alloc_tag_sub(ref, folio_nr_pages(new)); - - __alloc_tag_ref_set(ref, tag); - - put_page_tag_ref(ref); + alloc_tag_sub(&ref, folio_nr_pages(new)); + __alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } #else /* !CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 59a3deb792a8..b13cd3313a88 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -11,46 +11,59 @@ #include +union pgtag_ref_handle { + union codetag_ref *ref; /* reference in page extension */ +}; + extern struct page_ext_operations page_alloc_tagging_ops; -static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, + union pgtag_ref_handle *handle) { - return (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); -} + struct page_ext *page_ext; + union codetag_ref *tmp; -static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) -{ - return (void *)ref - page_alloc_tagging_ops.offset; + if (!page) + return false; + + page_ext = page_ext_get(page); + if (!page_ext) + return false; + + tmp = (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); + ref->ct = tmp->ct; + handle->ref = tmp; + return true; } -/* Should be called only if mem_alloc_profiling_enabled() */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) +static inline void put_page_tag_ref(union pgtag_ref_handle handle) { - if (page) { - struct page_ext *page_ext = page_ext_get(page); + if (WARN_ON(!handle.ref)) + return; - if (page_ext) - return codetag_ref_from_page_ext(page_ext); - } - return NULL; + page_ext_put((void *)handle.ref - page_alloc_tagging_ops.offset); } -static inline void put_page_tag_ref(union codetag_ref *ref) +static inline void update_page_tag_ref(union pgtag_ref_handle handle, + union codetag_ref *ref) { - if (WARN_ON(!ref)) + if (WARN_ON(!handle.ref || !ref)) return; - page_ext_put(page_ext_from_codetag_ref(ref)); + handle.ref->ct = ref->ct; } static inline void clear_page_tag_ref(struct page *page) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { - set_codetag_empty(ref); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -59,11 +72,13 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { - alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE * nr); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -71,11 +86,13 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; - if (ref) { - alloc_tag_sub(ref, PAGE_SIZE * nr); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -85,13 +102,14 @@ static inline struct alloc_tag *pgalloc_tag_get(struct page *page) struct alloc_tag *tag = NULL; if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref = get_page_tag_ref(page); - - alloc_tag_sub_check(ref); - if (ref) { - if (ref->ct) - tag = ct_to_alloc_tag(ref->ct); - put_page_tag_ref(ref); + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub_check(&ref); + if (ref.ct) + tag = ct_to_alloc_tag(ref.ct); + put_page_tag_ref(handle); } } @@ -106,8 +124,6 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } -static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void clear_page_tag_ref(struct page *page) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} From patchwork Wed Oct 23 17:07:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4680CFA452 for ; Wed, 23 Oct 2024 17:08:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E15FD6B00A6; Wed, 23 Oct 2024 13:08:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9DC16B00A8; Wed, 23 Oct 2024 13:08:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF3376B00A9; Wed, 23 Oct 2024 13:08:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 999266B00A6 for ; Wed, 23 Oct 2024 13:08:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DA6B7140D03 for ; Wed, 23 Oct 2024 17:07:58 +0000 (UTC) X-FDA: 82705500066.14.F1C3C38 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 1603BC0025 for ; Wed, 23 Oct 2024 17:07:53 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="GOksAK/S"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3fi0ZZwYKCH0tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fi0ZZwYKCH0tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703171; a=rsa-sha256; cv=none; b=YoLZK7l4qlvXxzVeYBnQmfBZySohnSmhDB0H5mk/eutD2WgPJItyVGihvpCgpfQJ9Tq6o0 WZ2VznKszBlRRJoxng2r3bTdTdQbX53ifcyrqAQGCTfHmPXjzJneEk8QRpAUGhlQzV8DFV 0TqPptTxNewc6E/NWFbgFsjM3+n59oI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="GOksAK/S"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3fi0ZZwYKCH0tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fi0ZZwYKCH0tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703171; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hUAPSrGkP1fndYxBbEXwkdRLC1ZGIzlRm3shVAxWj1o=; b=djzg5NlWz5o6Bi8RmwxbMldymy+MFWkl5kUbjIGVDmTPTet5D4Zfv/8rA1iWSD66BX9ycU kSNknXU7bAGrVq3N/V/d6cP6RewqsCxYgDnhYMPAdDtyaWCaacH2XqQOOViX1K8284oRzO NTduDPaTcSxogX/CdqBr2p/RYvNNjHM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e293150c2c6so48779276.1 for ; Wed, 23 Oct 2024 10:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703294; x=1730308094; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hUAPSrGkP1fndYxBbEXwkdRLC1ZGIzlRm3shVAxWj1o=; b=GOksAK/SJM/x4Vvb0hSMnxSWr2WBFozBFW7VTwDYfgJGA2C/skf7nD9Zj7nz3XzGSu qz5GMwPKSOpJi8NWG/K5XCbxOpu2IBBWTFUlc/kFz2t7jxCccdUPSsXXXlA6P7ktWrXD WkdahneLYY7//ObmgzSchna0H2cLk5I5TVd2bjdN4vnfjtBK/ZYfx8VMdgA3zaj4bpk0 WnIofDii+B7i4RTBV+BaDPlW/Ex88c/Q+6xY6S2yCyAhkrtZMfNSzKQY5SygLylRTpVa 5uPjq09P/h2lCSYwFkKWFh2oa7h7a7251v5x9G+/MrPF45Eb6Ksq0TH+CcexlIzmIQTx vsEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703294; x=1730308094; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hUAPSrGkP1fndYxBbEXwkdRLC1ZGIzlRm3shVAxWj1o=; b=B7c5C0rb0Tie68+IQvDnzhckvIPyuFcLUwJOHtxkjtjba0+0fDxm+aSbBd4UEJ3Pps 3wfBK6unjP6qpSt3rAPSffBFb3mzyumszDmxe3RDdVu5vUR+g1qpGmIbZQZjx2w8wpyW x7mtZ2Ow9SIcX5Gzg+qiq14fn5KFws0D+EfDhShzO+Qpz6+V8gUoLBBGLlhM0Py5NETS Ab/ThyFsuUztW7CXhoL+4kfvp0jYQqmmfm1KnHMo9ovOSA+i2JHbtHB8jZY0CBgt4uEW ia5G75JcIlMEROlqkNVILR/tMiW7Py81P3GG9eXucgpGMdl76HahqhZVTC2Z8X94nv0I caJA== X-Forwarded-Encrypted: i=1; AJvYcCUBJmHqWo1iPMJ4ZR91H3EfjQJjr1f2S7IO851cabNSdqJsJo6nXeB7XHjbbm8T4+vThDOR7NTlGw==@kvack.org X-Gm-Message-State: AOJu0YyFFJo0UbomEoalY5YFWSZ/LuckBawVpQzr2stg3TJ2BMqROnsN GTbWgOKpkZ3tzeS+WVorRy35EyTZje09AAFj4k4/eB5fBzpgkgCdLCq5fcgsNm2noecVG4nCQKH j/A== X-Google-Smtp-Source: AGHT+IGRE52J3Q/FMmtRptFBkvIdYNYKY+K4lu8CvvfN06S+6RGcE++e1CnN4bokrfVHywZPohucB5wHvBc= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:9787:0:b0:e29:7cfa:9fbc with SMTP id 3f1490d57ef6-e2e3a6dc4d5mr1689276.11.1729703294360; Wed, 23 Oct 2024 10:08:14 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:59 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-7-surenb@google.com> Subject: [PATCH v4 6/6] alloc_tag: support for page allocation tag compression From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1603BC0025 X-Stat-Signature: n9mwm9rh9inmrug6kd1z51ychnxqinqp X-Rspam-User: X-HE-Tag: 1729703273-625308 X-HE-Meta: U2FsdGVkX19MOYYDYabXQviFVPqIvQpnhTu0ukpYgQ42zdEQGrLBXg/hAs3HHipHmYv7s7WgW4NvtFhEeNWMi33swyRr+hh9a3tWQCubAA/1lXGkJEfXGqRKSN98yYU/ZBmiDs3A8Z7b7b+An+Fb0lzBk2Rx1uyLgmFHgKqiF3vC/e0CywD+oc2XUWeIh/UW9S7nkXvKdSA2zVx5IBK4cToc5J2fl7Z8WWxB52wduKNg+9yAjPbX1xXIS1bFwEOM5jsiGToP/2DTYrEFuQb2dg4WqQfVv00WynCWb69jujO6Xdnckcg4iWt0enIOQUCL7Q++wNpxpBYHQr/LATy2C6lL+pBoSW/zesE9KWZrRifePTaraqGtrXvu02V/vyL/F4TgO1yHU8XRxuFfuqIdqL5WNMfRT7FNu2nUp5RfRALwAPmgARj8aaA4AU1k6pSjNOcaM39w6biqaRMovmGuWx2DwX/Syr2D9XBV8a8J/QOx8hQnDnjKasE9B9F2SnsN00IOE1QGn/GmdeWVIsUU81yrXlOZG5WjDMkI8ggJ4Yiph5uFPb8GB2mTuD2MOiw0giORQW3V4F9YdPw7xgSVx/fJEzbqj4J3zVIXf7rV0iI3SUHZV/e0GvBWBxSh/4XE0KO3wUzCeZoO5WfZOA+GvX8KW45MIIvZuTSCc3ugPeFDbfCuJ4KU6KEqTCEz7No5NzS9Ea4SJM4HWa3A/Ylb5XVNiIhPQMM1pEJ358iRq7W0Ze6VGFqpgHJBPkOdcbe+6TLTDtzwanCfzLncMIcmcLQYId78WvrVKX+j5a0oBVU5S4BlSkuV1V2ugTMZiBMfzGvky3FM2BVkNEoxPttqyH58yG/CvLbZ3sk9CzQarh2ZTmdpilX1cGlCn559SjlDNqJNuOke9g/RWOVullApOFQ1m+4zK4G1FMBwg+bsWA/reAyVN5CMU60KJQH7RLRdAnlklviMyQBOmTag7Ie 4EMcasYD u50DBgUCIpeBM+gxiyLfw6KcF8jepKyolgOkBLFn0OXdx779LqFDnCrWyJ9fvdls+Na6QpMsHLzdYxl6RRf64DTdV7YjXQ3XNtVVqdY3zJFCPdZMc/Eove4jFVVZtMweKrQlp4znf3dmgx/R0C2bAE4sfp43pcxXmI1MsAGnmUPUQrjHscvQKaR/UrTuQrbUASlB45W7kim3+pBXsdakKYiVr6ed2eLG01Wc/8hDNQt1YTy1aga8tA4xuETt7Gcto/iYO1BX6UtVfv2aKlqeQOk0hr+xFXqLTD49e1yb3LioFr6EWZ7uc41A7bj8kLNuXtB/5+62/QfuOnQuVf8sbmxdHiFr24GEXpHFHcan/kyug/w4sTdNHsMottCpOaZuhwf6kGrraTcIf0QXfp4Fq/oDg1zRIW7RwZqnodOOj5uiaRQjmvDjl6cpfbPL5bUfvHtpNf1xq3UkddkQp4CNma5Mq0cZV5WkwC0w2M/W2TbULwR7Wvntd4MYoXHmRutnMcs8wILB1j2u5ix/qEjgKqL7YTCdE/X+ddpToPNIAbIm0jUP7H6Iuccm102jqhyZDT3ZsUVLxrylGlEiiu/aSBJ2TPA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement support for storing page allocation tag references directly in the page flags instead of page extensions. sysctl.vm.mem_profiling boot parameter it extended to provide a way for a user to request this mode. Enabling compression eliminates memory overhead caused by page_ext and results in better performance for page allocations. However this mode will not work if the number of available page flag bits is insufficient to address all kernel allocations. Such condition can happen during boot or when loading a module. If this condition is detected, memory allocation profiling gets disabled with an appropriate warning. By default compression mode is disabled. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- Documentation/mm/allocation-profiling.rst | 7 +- include/linux/alloc_tag.h | 10 +- include/linux/codetag.h | 3 + include/linux/page-flags-layout.h | 7 ++ include/linux/pgalloc_tag.h | 145 +++++++++++++++++++--- lib/alloc_tag.c | 142 +++++++++++++++++++-- lib/codetag.c | 4 +- mm/mm_init.c | 5 +- 8 files changed, 290 insertions(+), 33 deletions(-) diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst index ffd6655b7be2..316311240e6a 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -18,12 +18,17 @@ kconfig options: missing annotation Boot parameter: - sysctl.vm.mem_profiling=0|1|never + sysctl.vm.mem_profiling={0|1|never}[,compressed] When set to "never", memory allocation profiling overhead is minimized and it cannot be enabled at runtime (sysctl becomes read-only). When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y, default value is "1". When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, default value is "never". + "compressed" optional parameter will try to store page tag references in a + compact format, avoiding page extensions. This results in improved performance + and memory consumption, however it might fail depending on system configuration. + If compression fails, a warning is issued and memory allocation profiling gets + disabled. sysctl: /proc/sys/vm/mem_profiling diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 7431757999c5..4f811ec0ffe0 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,8 +30,16 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +struct alloc_tag_kernel_section { + struct alloc_tag *first_tag; + unsigned long count; +}; + struct alloc_tag_module_section { - unsigned long start_addr; + union { + unsigned long start_addr; + struct alloc_tag *first_tag; + }; unsigned long end_addr; /* used size */ unsigned long size; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index d10bd9810d32..d14dbd26b370 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -13,6 +13,9 @@ struct codetag_module; struct seq_buf; struct module; +#define CODETAG_SECTION_START_PREFIX "__start_" +#define CODETAG_SECTION_STOP_PREFIX "__stop_" + /* * An instance of this structure is created in a special ELF section at every * code location being tagged. At runtime, the special section is treated as diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h index 7d79818dc065..4f5c9e979bb9 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -111,5 +111,12 @@ ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH) +#define NR_NON_PAGEFLAG_BITS (SECTIONS_WIDTH + NODES_WIDTH + ZONES_WIDTH + \ + LAST_CPUPID_SHIFT + KASAN_TAG_WIDTH + \ + LRU_GEN_WIDTH + LRU_REFS_WIDTH) + +#define NR_UNUSED_PAGEFLAG_BITS (BITS_PER_LONG - \ + (NR_NON_PAGEFLAG_BITS + NR_PAGEFLAGS)) + #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index b13cd3313a88..1fe63b52e5e5 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -11,29 +11,118 @@ #include +extern struct page_ext_operations page_alloc_tagging_ops; +extern unsigned long alloc_tag_ref_mask; +extern int alloc_tag_ref_offs; +extern struct alloc_tag_kernel_section kernel_tags; + +DECLARE_STATIC_KEY_FALSE(mem_profiling_compressed); + +typedef u16 pgalloc_tag_idx; + union pgtag_ref_handle { union codetag_ref *ref; /* reference in page extension */ + struct page *page; /* reference in page flags */ }; -extern struct page_ext_operations page_alloc_tagging_ops; +/* Reserved indexes */ +#define CODETAG_ID_NULL 0 +#define CODETAG_ID_EMPTY 1 +#define CODETAG_ID_FIRST 2 + +#ifdef CONFIG_MODULES + +extern struct alloc_tag_module_section module_tags; + +static inline struct alloc_tag *module_idx_to_tag(pgalloc_tag_idx idx) +{ + return &module_tags.first_tag[idx - kernel_tags.count]; +} + +static inline pgalloc_tag_idx module_tag_to_idx(struct alloc_tag *tag) +{ + return CODETAG_ID_FIRST + kernel_tags.count + (tag - module_tags.first_tag); +} + +#else /* CONFIG_MODULES */ + +static inline struct alloc_tag *module_idx_to_tag(pgalloc_tag_idx idx) +{ + pr_warn("invalid page tag reference %lu\n", (unsigned long)idx); + return NULL; +} + +static inline pgalloc_tag_idx module_tag_to_idx(struct alloc_tag *tag) +{ + pr_warn("invalid page tag 0x%lx\n", (unsigned long)tag); + return CODETAG_ID_NULL; +} + +#endif /* CONFIG_MODULES */ + +static inline void idx_to_ref(pgalloc_tag_idx idx, union codetag_ref *ref) +{ + switch (idx) { + case (CODETAG_ID_NULL): + ref->ct = NULL; + break; + case (CODETAG_ID_EMPTY): + set_codetag_empty(ref); + break; + default: + idx -= CODETAG_ID_FIRST; + ref->ct = idx < kernel_tags.count ? + &kernel_tags.first_tag[idx].ct : + &module_idx_to_tag(idx)->ct; + break; + } +} + +static inline pgalloc_tag_idx ref_to_idx(union codetag_ref *ref) +{ + struct alloc_tag *tag; + + if (!ref->ct) + return CODETAG_ID_NULL; + + if (is_codetag_empty(ref)) + return CODETAG_ID_EMPTY; + + tag = ct_to_alloc_tag(ref->ct); + if (tag >= kernel_tags.first_tag && tag < kernel_tags.first_tag + kernel_tags.count) + return CODETAG_ID_FIRST + (tag - kernel_tags.first_tag); + + return module_tag_to_idx(tag); +} + + /* Should be called only if mem_alloc_profiling_enabled() */ static inline bool get_page_tag_ref(struct page *page, union codetag_ref *ref, union pgtag_ref_handle *handle) { - struct page_ext *page_ext; - union codetag_ref *tmp; - if (!page) return false; - page_ext = page_ext_get(page); - if (!page_ext) - return false; + if (static_key_enabled(&mem_profiling_compressed)) { + pgalloc_tag_idx idx; + + idx = (page->flags >> alloc_tag_ref_offs) & alloc_tag_ref_mask; + idx_to_ref(idx, ref); + handle->page = page; + } else { + struct page_ext *page_ext; + union codetag_ref *tmp; + + page_ext = page_ext_get(page); + if (!page_ext) + return false; + + tmp = (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); + ref->ct = tmp->ct; + handle->ref = tmp; + } - tmp = (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); - ref->ct = tmp->ct; - handle->ref = tmp; return true; } @@ -42,16 +131,35 @@ static inline void put_page_tag_ref(union pgtag_ref_handle handle) if (WARN_ON(!handle.ref)) return; - page_ext_put((void *)handle.ref - page_alloc_tagging_ops.offset); + if (!static_key_enabled(&mem_profiling_compressed)) + page_ext_put((void *)handle.ref - page_alloc_tagging_ops.offset); } -static inline void update_page_tag_ref(union pgtag_ref_handle handle, - union codetag_ref *ref) +static inline void update_page_tag_ref(union pgtag_ref_handle handle, union codetag_ref *ref) { - if (WARN_ON(!handle.ref || !ref)) - return; - - handle.ref->ct = ref->ct; + if (static_key_enabled(&mem_profiling_compressed)) { + struct page *page = handle.page; + unsigned long old_flags; + unsigned long flags; + unsigned long idx; + + if (WARN_ON(!page || !ref)) + return; + + idx = (unsigned long)ref_to_idx(ref); + idx = (idx & alloc_tag_ref_mask) << alloc_tag_ref_offs; + do { + old_flags = READ_ONCE(page->flags); + flags = old_flags; + flags &= ~(alloc_tag_ref_mask << alloc_tag_ref_offs); + flags |= idx; + } while (unlikely(!try_cmpxchg(&page->flags, &old_flags, flags))); + } else { + if (WARN_ON(!handle.ref || !ref)) + return; + + handle.ref->ct = ref->ct; + } } static inline void clear_page_tag_ref(struct page *page) @@ -122,6 +230,8 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); } +void __init alloc_tag_sec_init(void); + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void clear_page_tag_ref(struct page *page) {} @@ -130,6 +240,7 @@ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} +static inline void alloc_tag_sec_init(void) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 061e43196247..a6f6f014461e 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -12,6 +13,8 @@ #define ALLOCINFO_FILE_NAME "allocinfo" #define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) +#define SECTION_START(NAME) (CODETAG_SECTION_START_PREFIX NAME) +#define SECTION_STOP(NAME) (CODETAG_SECTION_STOP_PREFIX NAME) #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support = true; @@ -26,6 +29,11 @@ EXPORT_SYMBOL(_shared_alloc_tag); DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, mem_alloc_profiling_key); +DEFINE_STATIC_KEY_FALSE(mem_profiling_compressed); + +struct alloc_tag_kernel_section kernel_tags = { NULL, 0 }; +unsigned long alloc_tag_ref_mask; +int alloc_tag_ref_offs; struct allocinfo_private { struct codetag_iterator iter; @@ -155,7 +163,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl return nr; } -static void shutdown_mem_profiling(void) +static void shutdown_mem_profiling(bool remove_file) { if (mem_alloc_profiling_enabled()) static_branch_disable(&mem_alloc_profiling_key); @@ -163,6 +171,8 @@ static void shutdown_mem_profiling(void) if (!mem_profiling_support) return; + if (remove_file) + remove_proc_entry(ALLOCINFO_FILE_NAME, NULL); mem_profiling_support = false; } @@ -173,10 +183,40 @@ static void __init procfs_init(void) if (!proc_create_seq(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op)) { pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); - shutdown_mem_profiling(); + shutdown_mem_profiling(false); } } +void __init alloc_tag_sec_init(void) +{ + struct alloc_tag *last_codetag; + + if (!mem_profiling_support) + return; + + if (!static_key_enabled(&mem_profiling_compressed)) + return; + + kernel_tags.first_tag = (struct alloc_tag *)kallsyms_lookup_name( + SECTION_START(ALLOC_TAG_SECTION_NAME)); + last_codetag = (struct alloc_tag *)kallsyms_lookup_name( + SECTION_STOP(ALLOC_TAG_SECTION_NAME)); + kernel_tags.count = last_codetag - kernel_tags.first_tag; + + /* Check if kernel tags fit into page flags */ + if (kernel_tags.count > (1UL << NR_UNUSED_PAGEFLAG_BITS)) { + shutdown_mem_profiling(false); /* allocinfo file does not exist yet */ + pr_err("%lu allocation tags cannot be references using %d available page flag bits. Memory allocation profiling is disabled!\n", + kernel_tags.count, NR_UNUSED_PAGEFLAG_BITS); + return; + } + + alloc_tag_ref_offs = (LRU_REFS_PGOFF - NR_UNUSED_PAGEFLAG_BITS); + alloc_tag_ref_mask = ((1UL << NR_UNUSED_PAGEFLAG_BITS) - 1); + pr_debug("Memory allocation profiling compression is using %d page flag bits!\n", + NR_UNUSED_PAGEFLAG_BITS); +} + #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); @@ -186,10 +226,59 @@ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ static struct module prepend_mod; -static struct alloc_tag_module_section module_tags; +struct alloc_tag_module_section module_tags; + +static inline unsigned long alloc_tag_align(unsigned long val) +{ + if (!static_key_enabled(&mem_profiling_compressed)) { + /* No alignment requirements when we are not indexing the tags */ + return val; + } + + if (val % sizeof(struct alloc_tag) == 0) + return val; + return ((val / sizeof(struct alloc_tag)) + 1) * sizeof(struct alloc_tag); +} + +static bool ensure_alignment(unsigned long align, unsigned int *prepend) +{ + if (!static_key_enabled(&mem_profiling_compressed)) { + /* No alignment requirements when we are not indexing the tags */ + return true; + } + + /* + * If alloc_tag size is not a multiple of required alignment, tag + * indexing does not work. + */ + if (!IS_ALIGNED(sizeof(struct alloc_tag), align)) + return false; + + /* Ensure prepend consumes multiple of alloc_tag-sized blocks */ + if (*prepend) + *prepend = alloc_tag_align(*prepend); + + return true; +} + +static inline bool tags_addressable(void) +{ + unsigned long tag_idx_count; + + if (!static_key_enabled(&mem_profiling_compressed)) + return true; /* with page_ext tags are always addressable */ + + tag_idx_count = CODETAG_ID_FIRST + kernel_tags.count + + module_tags.size / sizeof(struct alloc_tag); + + return tag_idx_count < (1UL << NR_UNUSED_PAGEFLAG_BITS); +} static bool needs_section_mem(struct module *mod, unsigned long size) { + if (!mem_profiling_support) + return false; + return size >= sizeof(struct alloc_tag); } @@ -300,6 +389,13 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, if (!align) align = 1; + if (!ensure_alignment(align, &prepend)) { + shutdown_mem_profiling(true); + pr_err("%s: alignment %lu is incompatible with allocation tag indexing. Memory allocation profiling is disabled!\n", + mod->name, align); + return ERR_PTR(-EINVAL); + } + mas_lock(&mas); if (!find_aligned_area(&mas, section_size, size, prepend, align)) { ret = ERR_PTR(-ENOMEM); @@ -343,9 +439,15 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, int grow_res; module_tags.size = offset + size; + if (mem_alloc_profiling_enabled() && !tags_addressable()) { + shutdown_mem_profiling(true); + pr_warn("With module %s there are too many tags to fit in %d page flag bits. Memory allocation profiling is disabled!\n", + mod->name, NR_UNUSED_PAGEFLAG_BITS); + } + grow_res = vm_module_tags_populate(); if (grow_res) { - shutdown_mem_profiling(); + shutdown_mem_profiling(true); pr_err("Failed to allocate memory for allocation tags in the module %s. Memory allocation profiling is disabled!\n", mod->name); return ERR_PTR(grow_res); @@ -429,6 +531,8 @@ static int __init alloc_mod_tags_mem(void) module_tags.start_addr = (unsigned long)vm_module_tags->addr; module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; + /* Ensure the base is alloc_tag aligned when required for indexing */ + module_tags.start_addr = alloc_tag_align(module_tags.start_addr); return 0; } @@ -451,8 +555,10 @@ static inline void free_mod_tags_mem(void) {} #endif /* CONFIG_MODULES */ +/* See: Documentation/mm/allocation-profiling.rst */ static int __init setup_early_mem_profiling(char *str) { + bool compressed = false; bool enable; if (!str || !str[0]) @@ -461,22 +567,37 @@ static int __init setup_early_mem_profiling(char *str) if (!strncmp(str, "never", 5)) { enable = false; mem_profiling_support = false; + pr_info("Memory allocation profiling is disabled!\n"); } else { - int res; + char *token = strsep(&str, ","); + + if (kstrtobool(token, &enable)) + return -EINVAL; - res = kstrtobool(str, &enable); - if (res) - return res; + if (str) { + if (strcmp(str, "compressed")) + return -EINVAL; + + compressed = true; + } mem_profiling_support = true; + pr_info("Memory allocation profiling is enabled %s compression and is turned %s!\n", + compressed ? "with" : "without", enable ? "on" : "off"); } - if (enable != static_key_enabled(&mem_alloc_profiling_key)) { + if (enable != mem_alloc_profiling_enabled()) { if (enable) static_branch_enable(&mem_alloc_profiling_key); else static_branch_disable(&mem_alloc_profiling_key); } + if (compressed != static_key_enabled(&mem_profiling_compressed)) { + if (compressed) + static_branch_enable(&mem_profiling_compressed); + else + static_branch_disable(&mem_profiling_compressed); + } return 0; } @@ -484,6 +605,9 @@ early_param("sysctl.vm.mem_profiling", setup_early_mem_profiling); static __init bool need_page_alloc_tagging(void) { + if (static_key_enabled(&mem_profiling_compressed)) + return false; + return mem_profiling_support; } diff --git a/lib/codetag.c b/lib/codetag.c index 654496952f86..4949511b4933 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -149,8 +149,8 @@ static struct codetag_range get_section_range(struct module *mod, const char *section) { return (struct codetag_range) { - get_symbol(mod, "__start_", section), - get_symbol(mod, "__stop_", section), + get_symbol(mod, CODETAG_SECTION_START_PREFIX, section), + get_symbol(mod, CODETAG_SECTION_STOP_PREFIX, section), }; } diff --git a/mm/mm_init.c b/mm/mm_init.c index 4ba5607aaf19..1c205b0a86ed 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -83,8 +83,7 @@ void __init mminit_verify_pageflags_layout(void) unsigned long or_mask, add_mask; shift = BITS_PER_LONG; - width = shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - - LAST_CPUPID_SHIFT - KASAN_TAG_WIDTH - LRU_GEN_WIDTH - LRU_REFS_WIDTH; + width = shift - NR_NON_PAGEFLAG_BITS; mminit_dprintk(MMINIT_TRACE, "pageflags_layout_widths", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Flags %d\n", SECTIONS_WIDTH, @@ -2639,7 +2638,7 @@ void __init mm_core_init(void) BUILD_BUG_ON(MAX_ZONELISTS > 2); build_all_zonelists(NULL); page_alloc_init_cpuhp(); - + alloc_tag_sec_init(); /* * page_ext requires contiguous pages, * bigger than MAX_PAGE_ORDER unless SPARSEMEM.