From patchwork Wed Oct 23 17:07:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46D57CFA452 for ; Wed, 23 Oct 2024 17:08:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 848E06B0092; Wed, 23 Oct 2024 13:08:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F8BE6B0096; Wed, 23 Oct 2024 13:08:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 698DE6B0098; Wed, 23 Oct 2024 13:08:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4B4096B0092 for ; Wed, 23 Oct 2024 13:08:05 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7C65D1A0BA0 for ; Wed, 23 Oct 2024 17:07:33 +0000 (UTC) X-FDA: 82705498806.09.260005F Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf09.hostedemail.com (Postfix) with ESMTP id D1BC5140002 for ; Wed, 23 Oct 2024 17:07:50 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FBfBuzHq; spf=pass (imf09.hostedemail.com: domain of 3cS0ZZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3cS0ZZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703130; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=p/SgywtZx7rb7/mEn3XyHfH3Cb2SyryihrWMC3ba6F8=; b=4oYu/x84hQait20BiSesfSzCLU6KCRuvx33GgJpvrbW2qX1UUWb+2SwAmOZ5Pf8MdJ9G39 z63PF90eD3ndIjPMCAiZV1oIQqZRFKH4IOxDmVyXTUTZ6vtQosMijRdsr+Z1k9nHSDDGUc fOP/+dywqAI78HxCYUvNUMcTf8qqokE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703130; a=rsa-sha256; cv=none; b=BraInk1XxsX4vQxpH02qE7Ks6ZKmjgeW6hN/MBNv1ZFigXvOKl8EbTphWZoYbODaGGOSNt F863oxajN7NUaklUaAi2Am/FERsbfYQE9wLVFou0YITWXqjME8z49AxBm/bHBKgvZLNX93 YrrLx5VV3WSM1S64GilGaxd11U+AXdE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FBfBuzHq; spf=pass (imf09.hostedemail.com: domain of 3cS0ZZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3cS0ZZwYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e292d801e59so59066276.0 for ; Wed, 23 Oct 2024 10:08:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703282; x=1730308082; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=p/SgywtZx7rb7/mEn3XyHfH3Cb2SyryihrWMC3ba6F8=; b=FBfBuzHqMY0Pp2p8wblPJzffmfDqDamftJwEq9zIkyOL4bEbpQLCwg1WCEf+xmY1o/ VHGJ8hWWLYUXPU5v3w8OzaeHN+474ugY8kROH2qVNMOSTO9HJ/ESho5WhYCHljye2oZv fYcLK74fTXItfZ16Pupldlz3Jw9uqwKu6kcn5mgb6ElCZqJQcPuO78C9WoCRNEzEDJPQ JZlkwvBtwlxyFsGJzTgcO4KMAtTv3vA66Pre82m7wBbhtLe8ivINaPo6LCiMLqh2ZMoz zpAScbmLjODcusVF4wtFXt5c4jzYiYBzMFhujr2gQyoJeIYj8vNxYASi3CyMRQ8rP2Qs 5mUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703282; x=1730308082; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=p/SgywtZx7rb7/mEn3XyHfH3Cb2SyryihrWMC3ba6F8=; b=lEWfObSBkVcf4+MMJRU7FuykyNDtZV1vWojNwcs7zDfjFpojspoS/QoIFuWpEsPHdE hKMEL5UykKxPMGhTNYUCtRlRT2+rdH3BlpUKR6z1H7Zbb/NBGdScjSynsJrGvBgoxLBy m3lVG3leN5bnnemDEKdVXveukscQiBRblBT69OntbgC4nO61ceIXk1Co17vV5KeJnHkL OB3N9QIAgqt/T0VT4ShOkLhX+8frFo6bkqLARH4QNs5up+n+oheCKIxpOORLMBkHHGJo EfTDdcvIIcgrO6C/Gnaq2A+wQ7ZHPA1ZjjgcFYmTYSv3uMUXmOlFpNXaCGJ+jgD6mCIm XChA== X-Forwarded-Encrypted: i=1; AJvYcCWsnOFaZhdMuza/ViZ18ciXBHDe52uIhKPdn+g5zrCehnDqD5ccwfvta5RLKezBTFJOCfQQNFTq1A==@kvack.org X-Gm-Message-State: AOJu0YzIK0Hoki4cSjwmNKE1O3mixEPnmBD2GUre6EC0iFLX7Cl87Mil WbmPj3QfcnYNxOjZPIg98yVeNNU9gQStSs3AL9t8dlMnpqkzKqyv+O4lkOsN4Ue47/IfcHh7W5y ofg== X-Google-Smtp-Source: AGHT+IEsK5kZcuCRBJ2lcZigYshTAvQXhOVGbGNjlRgIh/suKcwgOyvPk5eRcakI/OtXpbirCYYTcQ44vPI= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a05:6902:50d:b0:e2e:44ae:111a with SMTP id 3f1490d57ef6-e2e44ae12f1mr899276.8.1729703281826; Wed, 23 Oct 2024 10:08:01 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:53 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-1-surenb@google.com> Subject: [PATCH v4 0/6] page allocation tag compression From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 6d8dbbginhoddpuzzoxywzd1kkiww8z5 X-Rspamd-Queue-Id: D1BC5140002 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1729703270-562964 X-HE-Meta: U2FsdGVkX1+qP7bEH+oEJqmwYW75uoUv1HMREjbYxxb77zHk0wEfQUKgKedwCZsuGjGM3yiU5qSyEcvB2rkGn3djQpZUDAXSDloVTiuC9ypg6sdGIkJrr+N06mZV9WrA4aRi24OsjkggkToR9lYTNk45vIMOrsG1s+dmJUSlYhCr9AHlV3tP6m5kwLATpJZ1RChzc5HsPhHJNNVc/6N4S2cYzJATPSLJhu4ibz2CIQRRgquiVvVDtyOpF6tUAJ6Af+BCZDfkfRe3IPZgvXoR3C5DYraT+b3Ex3WAeOBDIZqDu69tDRpFeY+FIqPhLmM4bZBedboSmv3t4Nxl4p3ja8QgfWWUivJ73rSWN1gcDG+paeu6Gxq3pZGqyPJNUL8dFrKXAzEENX5qd7cgnEf0ElfTz7vuRELEY9jAvn9LGLxPg7UcXLfrQJWY/+HNeGTFRzokIPPHCLoPF05WkYpyLaEOfoyYPd3i6rBgAXwkxX0ucX0uJwEnss3QusjshnDSjCdHYJf1giDwrQ6gKfP+7pGeKGN+MzQ94NkFNpfk71+/cj2lMTubHkwxQNEJrQpYpP4EnmiFoWg2b3ck2YC067Wwj9hfeIC+QRftLsLiH74ui9iWvCj6vufKIp+aFIg/dgtepzvymBxzVhitpdqGl2puGbzEQxSWW8mBjQQPwbTIE0UmBBlSIp6uCUeZCayNlRdzuYxSUTYXyEGeZbcxGimTIZD/oBUHjiKJvhPISc6sFRotgM3tmwggcZskRPsDV8FuHOaf5wFt/SIH4BnFLIYorIiPdmC6pAb71TELcAQfmTOCbH2y0HFQ9mKVhk8KFv01j5LGcdFWjNAFYCK+13/kWDZxjIO1ZjDSk/xmMwoC+URxLS+zetR3HqTezkUvyOFjalKgIA+a7OtoYdmwyjzuaJ6/IUqe9sZpgVDbnaVpe+lyqcZDmm+ZMjaqZJ/wRyyxNKUpBpko3jk/YVD jei3Umjr tR8MEczL3mpKPJ5EBrlQ/grCXgfOSqVVcQGOFSQXMNSAmCGp3rJPrfXr8Gji2kAY2gEiXtQZ2Bu52j0Vi/xzEhG7nGgpkb+WWz5JeDfX3KzFRoPMMl7xIVPc7uA6zrUspHzzZeWCRdJR7vcZI3CZuJhEQE4ZTjK4SzcQP0G6RcxA8AaKdS5lVwAf81ZTE03tGHtQPwpBIxxUERR1S+t8Rg19KzbJ8Uxr07SRL1TFzdFcYW3WV5EF6XmmqCAkRnJfZFfxtGIzvj0igrJqZakMWmfJhZQ1RbdEjr624GOcTHPku7zNF5AQeP6RPzGc+PQvwPLRDk6RrKnoIuOTLR/bkZHPjayxZzLc/TvjnJUs7Gu/9VqwlbHePjwCj4BekfUsl3gDFzuJO4Wf46JZ/2ampYNrB9F4UvHgUD0gnsp7cvT7EB3Wt237e6KWAkyDUshjoc8azftyD4K4V2b+zJXJpdA797O7byVbqaOl8fOPO24OZ0D/fDly1rsy1A2xRqJFCOGLWSPCj5a0Hg72Fw8pteXgQ/7X2JSesgLHffeSsjA70LoTVbDXFHUPigUKWEaN/c9yH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset implements several improvements: 1. Gracefully handles module unloading while there are used allocations allocated from that module; 2. Provides an option to store page allocation tag references in the page flags, removing dependency on page extensions and eliminating the memory overhead from storing page allocation references (~0.2% of total system memory). This also improves page allocation performance when CONFIG_MEM_ALLOC_PROFILING is enabled by eliminating page extension lookup. Page allocation performance overhead is reduced from 41% to 5.5%. Patch #1 introduces mas_for_each_rev() helper function. Patch #2 introduces shutdown_mem_profiling() helper function to be used when disabling memory allocation profiling. Patch #3 copies module tags into virtually contiguous memory which serves two purposes: - Lets us deal with the situation when module is unloaded while there are still live allocations from that module. Since we are using a copy version of the tags we can safely unload the module. Space and gaps in this contiguous memory are managed using a maple tree. - Enables simple indexing of the tags in the later patches. Patch #4 changes the way we allocate virtually contiguous memory for module tags to reserve only vitrual area and populate physical pages only as needed at module load time. Patch #5 abstracts page allocation tag reference to simplify later changes. Patch #6 adds compression option to the sysctl.vm.mem_profiling boot parameter for storing page allocation tag references inside page flags if they fit. If the number of available page flag bits is insufficient to address all kernel allocations, memory allocation profiling gets disabled with an appropriate warning. Patchset applies to mm-unstable. Changes since v3 [1]: - rebased over Mike's patchset in mm-unstable - added Reviewed-by, per Liam Howlett - limited execmem_vmap to work with EXECMEM_MODULE_DATA only, per Mike Rapoport - moved __get_vm_area_node() declaration into mm/internal.h, per Mike Rapoport - split parts of reserve_module_tags() into helper functions to make it more readable, per Mike Rapoport - introduced shutdown_mem_profiling() to be used when disabling memory allocation profiling - replaced CONFIG_PGALLOC_TAG_USE_PAGEFLAGS with a new boot parameter option, per Michal Hocko - minor code cleanups and refactoring to make the code more readable - added VMALLOC and MODULE SUPPORT reviewers I missed before [1] https://lore.kernel.org/all/20241014203646.1952505-1-surenb@google.com/ Suren Baghdasaryan (6): maple_tree: add mas_for_each_rev() helper alloc_tag: introduce shutdown_mem_profiling helper function alloc_tag: load module tags into separate contiguous memory alloc_tag: populate memory for module tags as needed alloc_tag: introduce pgtag_ref_handle to abstract page tag references alloc_tag: support for page allocation tag compression Documentation/mm/allocation-profiling.rst | 7 +- include/asm-generic/codetag.lds.h | 19 + include/linux/alloc_tag.h | 21 +- include/linux/codetag.h | 40 +- include/linux/execmem.h | 10 + include/linux/maple_tree.h | 14 + include/linux/mm.h | 25 +- include/linux/page-flags-layout.h | 7 + include/linux/pgalloc_tag.h | 197 +++++++-- include/linux/vmalloc.h | 3 + kernel/module/main.c | 80 ++-- lib/alloc_tag.c | 467 ++++++++++++++++++++-- lib/codetag.c | 104 ++++- mm/execmem.c | 16 + mm/internal.h | 6 + mm/mm_init.c | 5 +- mm/vmalloc.c | 4 +- scripts/module.lds.S | 5 +- 18 files changed, 903 insertions(+), 127 deletions(-) base-commit: b5d43fad926a3f542cd06f3c9d286f6f489f7129