From patchwork Wed Oct 23 17:07:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13847638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3F1ECFA452 for ; Wed, 23 Oct 2024 17:08:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73D206B009D; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64F0C6B00A1; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42A5C6B00A4; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1A5C56B009D for ; Wed, 23 Oct 2024 13:08:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B874A1C5BAC for ; Wed, 23 Oct 2024 17:07:52 +0000 (UTC) X-FDA: 82705499856.28.7845AEA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 4E9201A0024 for ; Wed, 23 Oct 2024 17:07:49 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SaCG1iU3; spf=pass (imf19.hostedemail.com: domain of 3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729703138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=dVgxTt569usgx4IFIns2QZI7M2pC6QYNyRiyTiGUEZOQLQkX/2aMywAB46HVaqAfalB5Kf YeLBwkPIcLT1agXdT1bMcrf0YFagNDVdv5LnU11rDxVCInDL/7M0BhBM1iWxyP674QyK15 UOUi77lsWxSOieC1EgqFYDAvdYuUtS0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729703138; a=rsa-sha256; cv=none; b=nf3ealrI1Nm+zjlg+o6dxYoCHQ2aXHcOYxVaTf29wcDGgwFpGK+0bfYqFmt43wN1bLoefL cjxFD05dx751so6v6hf18MKLrgTvHfF7UUmFm1iLz4u9N6TdXuXZqWdqawNmeYP3AvkQG7 TOJ2rC3yyxGVOd9HP2C6gX3E0XZ+KyY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SaCG1iU3; spf=pass (imf19.hostedemail.com: domain of 3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eS0ZZwYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e3204db795so104968097b3.2 for ; Wed, 23 Oct 2024 10:08:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703290; x=1730308090; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=SaCG1iU3/EULCq4+ouPpEWK7we2fhH4R0V661QuDhyPAbYrLWnqtJ2dCM1HLfcnsOX ZJ0/SqncxBkFPGID+9iVgPErkTV2SBqIEdLTEla9rY4vSwTbdhdQun2R9aDMZ9ZF8ddN Gl/BnJS1ss84qjaa5QNp06JLINQreVmKKqL8uk1m7Uhrf1tBYWv57Hs75dxJK1jq6Stl XkjIAtxp0OoYNkn3K8oqd45Xpf0D51XYy6hRBEyDAfFSz1w+xutlgsp2MojsI4iMuuLT sxf2uyYy7N1/7v5iMZyuKI5TFDBluEP4iG6QhHz2PY0MyFVWg6KEU0q5dGolzDuWwfzv 2wig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703290; x=1730308090; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=r5veWWagqD/Td12mM/uh1y8Ad+KQV6akBPIs3hVnmUS3edyF6EEkv7GOsYaceo0y8H 1kSzsj8dFmOTBhIkjrWixmHqM/vS4uIJbiMCMAYFacRUUeB7KOZ8F+FUtD4vrhvqKfjs Zg+VzrvDdOGUH0Np2ZcgQjcYhxzY4QDW/GAJIcCqXdSAVMOJoFy06PqXSsB4t+F55b5u pj2RwJE/toIpVRo5c394fzWgM6p27S7CR6RNWH9yvD8sGsd4CfPLNsKAWchmfFFkcCyE ytSFmaH0pgHg3PSsbziO9+0yvqosJJOUFY5Ew48bsy/sp+O6XNEc+EzUZIuHw6v/Zsvr lwyg== X-Forwarded-Encrypted: i=1; AJvYcCWlBjlEr1UydBVKDrXaEyYtDhpyUGY4xWE/1SX42kUSm4qfBPvRBjiE7WZLYQxwvwm1cd8fXC4lRw==@kvack.org X-Gm-Message-State: AOJu0Yx4EhMz2Meu39mUv4kvt7LSHduUV7NzNP7oebjfas+3QM5x0Kzj MDbJrxSZCDwhT4dou2p8JaartviPN6tDSC1LzAgrkuV+f20mMMLK0jAmObUlgFKfJDGmieDlSRz qEg== X-Google-Smtp-Source: AGHT+IGGx5cwUMRbaaaRLBx9ozQm5nYkieFYXts2XJ8fpTDFGg4nxvLC1BvVBA4+mhBqSvprWgkBELkrSzs= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a05:690c:4b03:b0:6b0:d571:3540 with SMTP id 00721157ae682-6e7f0f97164mr929077b3.6.1729703289927; Wed, 23 Oct 2024 10:08:09 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:57 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-5-surenb@google.com> Subject: [PATCH v4 4/6] alloc_tag: populate memory for module tags as needed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4E9201A0024 X-Stat-Signature: xopgx8abfhckgzxy499cppes8xf9i4bu X-HE-Tag: 1729703269-911162 X-HE-Meta: U2FsdGVkX1+xYYifc07chbmfo5keOBvAvILHAvBX/YH6IlHMjYkXa6eq4ofBNVlBHGtolI5bXWYP4k+v9gHbum9L/awi9nP+ilMB1XznpDqRYNXtf6qDNXaleQOICvUGIReHU0dNQKJGx6/qvltg3fOdSkGjCngkNB441rl5eb9motyQfoQ7PHIteBKaW87/sdlRmtlHBtRPQCZpZmQUpmH+ZrqBrbSckNpgDKmvR6IB6mr3oQAqwUYKY2yKVkQXCjdCigapgG1qYAZJwPZDpHOtSaitzb6yWF+usr8yJCnAGO1AXetTJEkjD+t4YRyq+gpixz/wWs0ZPrIsL/4z3tnNyshyYaZpA31a1Rey664bG6Om5xduszDkybghww2K1Fb5WHFGn28oBxYiYRlbUk6sFRgxoso5kLeWyBTbU+X4BP5ewBDo1/X0iFbZWNOiHEFGpmJpeDIgqbTJF8TxjFwmI2i+6BHqBj7MFCiaKH6V4YPKaY5J1DmQ/X1uO08ttdtr/Awn9u9BW6CcxECi7ra1caFnIB+X54442OBt+8RtFF1V61xw/EUVlsMDfrNA2OM1HuhKvFbrAhdPmD25Ovodcy9/gKqFMZ2QEK3j9YbVr/YPeB4/CMCCZMC4H01wsDqF8z7DtN5HFUWUozrWCBsG4iLKZIhzrAoH35l2VoeSBQSxTIPWs9PerblvJJ7nRSzJg8d3fnAcsu+l9/q8Icwq1+3teTAajGuijpoQD46yhPUTrgTA8FM7luSGdQdjMKQnHZ8u56/pWcfOIWQWUq81S6scZ64gK6YyV2zUOkr4zKnWqhWYDSo137+ewlHa7N+O23JQTSwjbtuUco+knz3qgMOxbv/kwgwRlpptAkbe9F/40etZiKj2HCoFtR8/5sI8v7oqSYhV/qmH9zFySqDUILnX4kViNgoXZuyR3TpM+fdJR0aeGi/7sILPzliDoi8fQte33erw9Wnt9WF w1H4OpDl nRvHyevt8JKVUvk/e0Y6HJDDosEckat1ZrmiHVdjSZ6Sfcy1SJ/3F43xs4uGKmRXMZw/6iSqetp5OFPWFTgIdKWSBK+Cxru6SeTn73JYGOnY0qQzGfvArQzasP1wFl0TOH2is6kRQKzbaZlts5+RulkzgN4FSSMKJjUhH+pGeiMWTVKuKMyeppAE/lMdppXXf3b+G71na6VWun9FWhTtqHJLohWVaKKVyi3ARAwn4L8/wIUm4iKd6DDCD2eO/5bTcnMzKJro/l0zbJTfWQXF2LpaHf6+Z91dd2mKkCN9GqFcR4+ZhJPqxLJ97LGts+UIXaU0vAQwoFmH41vixH73gVtuNjIHuU4thNgMW8hozWk/8XdX4e5KQw8Rc85Hy46vD4PehC85/Z5k4ziq58Z0jIohnvhdZqTXSXT3+e2HpXzCvC1FBTzUcGkg5zv4CvNNuT7+8WEaml58tWlGbXmveiPhvtfLkS1bqlJsX0KktQmhbfJRZGIid1nRSKmcyFO2cgb/tonzQ6/1RaIo/fNprTcvJSnpxTk8/MFn34/9WT+s2DqhW0qy53/5wlLWOKIBTvJzO8+Pq3vTqnlFvDlsKlKgwdA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/linux/execmem.h | 10 ++++++ include/linux/vmalloc.h | 3 ++ lib/alloc_tag.c | 73 ++++++++++++++++++++++++++++++++++++----- mm/execmem.c | 16 +++++++++ mm/internal.h | 6 ++++ mm/vmalloc.c | 4 +-- 6 files changed, 101 insertions(+), 11 deletions(-) diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 1517fa196bf7..5a5e2917f870 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -139,6 +139,16 @@ void *execmem_alloc(enum execmem_type type, size_t size); */ void execmem_free(void *ptr); +/** + * execmem_vmap - create virtual mapping for EXECMEM_MODULE_DATA memory + * @size: size of the virtual mapping in bytes + * + * Maps virtually contiguous area in the range suitable for EXECMEM_MODULE_DATA. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *execmem_vmap(size_t size); + /** * execmem_update_copy - copy an update to executable memory * @dst: destination address to update diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 27408f21e501..31e9ffd936e3 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -202,6 +202,9 @@ extern int remap_vmalloc_range_partial(struct vm_area_struct *vma, extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, unsigned long pgoff); +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, + struct page **pages, unsigned int page_shift); + /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED values * and let generic vmalloc and ioremap code know when arch_sync_kernel_mappings() diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index d9f51169ffeb..061e43196247 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,14 +8,15 @@ #include #include #include +#include #define ALLOCINFO_FILE_NAME "allocinfo" #define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT -static bool mem_profiling_support __meminitdata = true; +static bool mem_profiling_support = true; #else -static bool mem_profiling_support __meminitdata; +static bool mem_profiling_support; #endif static struct codetag_type *alloc_tag_cttype; @@ -154,7 +155,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl return nr; } -static void __init shutdown_mem_profiling(void) +static void shutdown_mem_profiling(void) { if (mem_alloc_profiling_enabled()) static_branch_disable(&mem_alloc_profiling_key); @@ -179,6 +180,7 @@ static void __init procfs_init(void) #ifdef CONFIG_MODULES static struct maple_tree mod_area_mt = MTREE_INIT(mod_area_mt, MT_FLAGS_ALLOC_RANGE); +static struct vm_struct *vm_module_tags; /* A dummy object used to indicate an unloaded module */ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ @@ -252,6 +254,33 @@ static bool find_aligned_area(struct ma_state *mas, unsigned long section_size, return false; } +static int vm_module_tags_populate(void) +{ + unsigned long phys_size = vm_module_tags->nr_pages << PAGE_SHIFT; + + if (phys_size < module_tags.size) { + struct page **next_page = vm_module_tags->pages + vm_module_tags->nr_pages; + unsigned long addr = module_tags.start_addr + phys_size; + unsigned long more_pages; + unsigned long nr; + + more_pages = ALIGN(module_tags.size - phys_size, PAGE_SIZE) >> PAGE_SHIFT; + nr = alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_NOWARN, + NUMA_NO_NODE, more_pages, next_page); + if (nr < more_pages || + vmap_pages_range(addr, addr + (nr << PAGE_SHIFT), PAGE_KERNEL, + next_page, PAGE_SHIFT) < 0) { + /* Clean up and error out */ + for (int i = 0; i < nr; i++) + __free_page(next_page[i]); + return -ENOMEM; + } + vm_module_tags->nr_pages += nr; + } + + return 0; +} + static void *reserve_module_tags(struct module *mod, unsigned long size, unsigned int prepend, unsigned long align) { @@ -310,8 +339,18 @@ static void *reserve_module_tags(struct module *mod, unsigned long size, if (IS_ERR(ret)) return ret; - if (module_tags.size < offset + size) + if (module_tags.size < offset + size) { + int grow_res; + module_tags.size = offset + size; + grow_res = vm_module_tags_populate(); + if (grow_res) { + shutdown_mem_profiling(); + pr_err("Failed to allocate memory for allocation tags in the module %s. Memory allocation profiling is disabled!\n", + mod->name); + return ERR_PTR(grow_res); + } + } return (struct alloc_tag *)(module_tags.start_addr + offset); } @@ -372,12 +411,23 @@ static void replace_module(struct module *mod, struct module *new_mod) static int __init alloc_mod_tags_mem(void) { - /* Allocate space to copy allocation tags */ - module_tags.start_addr = (unsigned long)execmem_alloc(EXECMEM_MODULE_DATA, - MODULE_ALLOC_TAG_VMAP_SIZE); - if (!module_tags.start_addr) + /* Map space to copy allocation tags */ + vm_module_tags = execmem_vmap(MODULE_ALLOC_TAG_VMAP_SIZE); + if (!vm_module_tags) { + pr_err("Failed to map %lu bytes for module allocation tags\n", + MODULE_ALLOC_TAG_VMAP_SIZE); + module_tags.start_addr = 0; return -ENOMEM; + } + vm_module_tags->pages = kmalloc_array(get_vm_area_size(vm_module_tags) >> PAGE_SHIFT, + sizeof(struct page *), GFP_KERNEL | __GFP_ZERO); + if (!vm_module_tags->pages) { + free_vm_area(vm_module_tags); + return -ENOMEM; + } + + module_tags.start_addr = (unsigned long)vm_module_tags->addr; module_tags.end_addr = module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_SIZE; return 0; @@ -385,8 +435,13 @@ static int __init alloc_mod_tags_mem(void) static void __init free_mod_tags_mem(void) { - execmem_free((void *)module_tags.start_addr); + int i; + module_tags.start_addr = 0; + for (i = 0; i < vm_module_tags->nr_pages; i++) + __free_page(vm_module_tags->pages[i]); + kfree(vm_module_tags->pages); + free_vm_area(vm_module_tags); } #else /* CONFIG_MODULES */ diff --git a/mm/execmem.c b/mm/execmem.c index 576a57e2161f..5c0f9f2d6f83 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -368,6 +368,22 @@ void execmem_free(void *ptr) vfree(ptr); } +struct vm_struct *execmem_vmap(size_t size) +{ + struct execmem_range *range = &execmem_info->ranges[EXECMEM_MODULE_DATA]; + struct vm_struct *area; + + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->start, range->end, NUMA_NO_NODE, + GFP_KERNEL, __builtin_return_address(0)); + if (!area && range->fallback_start) + area = __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->fallback_start, range->fallback_end, + NUMA_NO_NODE, GFP_KERNEL, __builtin_return_address(0)); + + return area; +} + void *execmem_update_copy(void *dst, const void *src, size_t size) { return text_poke_copy(dst, src, size); diff --git a/mm/internal.h b/mm/internal.h index 508f7802dd2b..f1ce0e10bed8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1219,6 +1219,12 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_folio(struct folio *folio); +struct vm_struct *__get_vm_area_node(unsigned long size, + unsigned long align, unsigned long shift, + unsigned long flags, unsigned long start, + unsigned long end, int node, gfp_t gfp_mask, + const void *caller); + /* * mm/gup.c */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 74c0a5eae210..7ed39d104201 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -653,7 +653,7 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; @@ -3106,7 +3106,7 @@ static void clear_vm_uninitialized_flag(struct vm_struct *vm) vm->flags &= ~VM_UNINITIALIZED; } -static struct vm_struct *__get_vm_area_node(unsigned long size, +struct vm_struct *__get_vm_area_node(unsigned long size, unsigned long align, unsigned long shift, unsigned long flags, unsigned long start, unsigned long end, int node, gfp_t gfp_mask, const void *caller)