From patchwork Mon Jul 10 03:27:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng Zhang X-Patchwork-Id: 13306203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2578EB64DC for ; Mon, 10 Jul 2023 03:27:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DD2D6B0075; Sun, 9 Jul 2023 23:27:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 68C596B0078; Sun, 9 Jul 2023 23:27:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 553E86B007B; Sun, 9 Jul 2023 23:27:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 458CA6B0075 for ; Sun, 9 Jul 2023 23:27:38 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 107881A0284 for ; Mon, 10 Jul 2023 03:27:38 +0000 (UTC) X-FDA: 80994267396.30.ACF0735 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf04.hostedemail.com (Postfix) with ESMTP id 6DD1B40002 for ; Mon, 10 Jul 2023 03:27:35 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T0Dkrsab; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf04.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688959656; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=z+IV/3EU3GY4vUZv6vrcuErb2vDPlc1Isf1U0nOs3EE=; b=vNEi5Z+SnSi1wpZXRgulPQ1g+/nuN1sDyMPCbhd3R1D5Cs2bFe8hCSE00iIoKoYGTG53/K 4c6i6foah+uEiXOyM6e1HyuLS6HpW+nxCwme0TvpWYNnhmu43JcVFesEuOCX4ygMPylSlG zysQ7x1He51qRcP2rz5OT6mHO2LI7uc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=T0Dkrsab; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf04.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688959656; a=rsa-sha256; cv=none; b=S7VAqDMW4ZAUpPiGd3e10aU8StxmWHEzOPV2I4HEIhoCnJyrAOf/B4fhWRq1LyGWNSrxCP 0iX1b+4PZyy86rJgTRI/+Gf7t1IZcLrOZ1MbqdafN8pUUblGaTfoYuqLUHj7DVilCYhXOP uaWEEK27BnJyVMKCRJZPNS4u9yPyHE0= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-666ecf9a081so3485329b3a.2 for ; Sun, 09 Jul 2023 20:27:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1688959654; x=1691551654; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=z+IV/3EU3GY4vUZv6vrcuErb2vDPlc1Isf1U0nOs3EE=; b=T0DkrsabQei0/XIIO4dbC8R+HefLxd2T0hbWqqISpHzkWcVXf6rFLJrihNvSzxwoNL y8OL10sicociCQXP4eGVvGHpBvzWf65vEDtKQj3iZGoYIDjHp3D7icwIwk6uh5TVF9F9 qi8v3BrfFvlhUy+vOtLb+Be9OYoyaDDnED8Ubl5i7m90FFpvOTcnRTjV6z5YSV25ZiyT XlRSrV7GmtV1YHpef7qd0LaiCRZqJ2nBX0RlP1brg+K7Y3BiMkpbC11kgdQTJhDXEekN jvx1AaHdQnL3/hwtGFEeqR2pvIy5CiITrk6Xbn2DGXkORVJJF0SdbhWRoZfTKBesUNRY W6PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688959654; x=1691551654; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=z+IV/3EU3GY4vUZv6vrcuErb2vDPlc1Isf1U0nOs3EE=; b=PAr4fMTA9HqIhHN6zY02aGgrO+V/tRLwZqT9/thsL5ZwgzUx0G6FT6hW1ZgZgNEAPY YE6xVuYbY0p33XVfEdspq+vkd1/4jP03CecgFr0XjbFi7RCGlJqD5RcAz4AmevFV732B UDaO+e/oEvFC9q3R854ri7AOska4RHK74pUaJ5sjgS0kaM2ZDkqTjawBDEg3b8m4grJd cn2LdR2EMYhOLZspPTiRrsQCY73beVfvdmvEnHuqjkzmtNu+CJLIk7S9dATaNKP8ZExq 745tSHl3dabMEWTm4hnv/C41491Wzw7D3ZOVl1xaF3eznG20SFP01DUY4X+UdOG06MAS JKwg== X-Gm-Message-State: ABy/qLYnOe5qQe9zX0r4/8zYiylTp0NjVsvEnBg/BYDcRks/JRx+ibqR /2P8dBCDeQxtvBM4J67al+E2Qw== X-Google-Smtp-Source: APBJJlHK/NBb4uu/gdhAvc3gFctQlQEEpQnHzVEJ2KN/gCiA6SP14wS++9kHd3JIOelROmaBKGrQnA== X-Received: by 2002:a05:6a00:1704:b0:682:2fea:39f0 with SMTP id h4-20020a056a00170400b006822fea39f0mr13853334pfc.5.1688959653872; Sun, 09 Jul 2023 20:27:33 -0700 (PDT) Received: from GL4FX4PXWL.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id j15-20020aa7800f000000b00682c864f35bsm6279748pfi.140.2023.07.09.20.27.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 09 Jul 2023 20:27:33 -0700 (PDT) From: Peng Zhang To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, muchun.song@linux.dev, Peng Zhang Subject: [PATCH] mm: kfence: allocate kfence_metadata at runtime Date: Mon, 10 Jul 2023 11:27:14 +0800 Message-Id: <20230710032714.26200-1-zhangpeng.00@bytedance.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) MIME-Version: 1.0 X-Rspamd-Queue-Id: 6DD1B40002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: xmfn1znqgbuy5p9eo1wzwon5wc6u61fe X-HE-Tag: 1688959655-754554 X-HE-Meta: U2FsdGVkX180hUI7/dqNgcZAdcgDzH4iomfMcBwf+rMpqWHYOULD64XzLHUSsmH3ei0C5p8ya2C56bnF4FfuaPIbDBeOW46GRuzt4a3dr4Ecn4/0TgH2nJKx1iCRLBL871Jv2obzbBZa3gu4eaYFCcrki6cR0GsiF/KYup58Yx/Wsq4D5jqmpYA8oZGHAzC/cbYgnAArHHunq6mwWIr+wUYAUcufpJNNuQOFrL955hLODfls0/ef//nRLN9fUsuPQ4WYReQb6VLCrfC544LCpVuT3Yhc/Mjt/nWUqGmDcAY4fxOEE40cTO0KnfOjMVc5P4mW9eF2UDrbHZcwyZyj3HSKx3yLBvRZBqeK3oWwn2ORO8ft4WgDAQ5JnkkrDL3zT1m/+FbXQX2Tn/nJkJTrHjkRRPvA8402ZdFToYsL9J9f+A1rpTbtc/IIBX0bY3wlWFImxXulilIru22h28tXNcWTX2Cv7z0BA9+UqnIx2h6pbyTD5gmhi+FiA3Me/+oDsCDZdjV0z/12ANPFouEDA9C8kDG+Zh8Hsap+57efD4Y/8+l1BGh2X2ZG4UTCRewK3Xv1+ksAon124YqTHHzYr0OTZ1g0j+YF0dhQFKVpYLlXMBsJ/zAw9ZHVh5hZ69WV4Fk2zgPPY9QhIIsj3iHyeTbemd2or/EoQfTnT0dnswOhkdX27t18cXoSYDrH+y/EoefP5ciauqjvb5hJZxPjA7VHO+nvSWHTcy1MiUk+FGWIUnZ2IHK8Rlbd1KSwbT5Iz/0lCeQUU4Kk8D+fJrqDEGnJZO/B6Fpu61++WlaQn6uBRhLxNi6dvPdLaA083BKg9KrsrqV23qcwVxy3dTXLbDasCWbGPGPhDe6D6m1UFegngy4tF6yJ7RqwKiQVDoci8gVNn1fVYzNUlqA6WnEa4zurb04FHB29WIvSILEPP9E1HQB3gTQxZWn5EXVWnjhBgrcsNeTxYea6oAdpgon VoaHgcXh UeniIK5DHB7GJLafblCcDpFxGmluJgAATyf78PzgCgSbtKo5bvgZLm+Hpt22rU9TuN6xZsjCEshdVgBM+89pf+lulHlKzriMsoDSPVAMn35BSK1eG9RdxqRbVVdZQjV04lH7nBzogVUWPAEfd17DMK1IBa5S29IsME/rqguUIaWhdwi1kJygjfKw/IAXgs/jYG7EgcQ2fz9vrsWZLzclxZU1NXZlJMu2plkMix8BM9GzmbERlXTbxLgAFn5dIH5I9A6Q7qOADSwpf4V8XvgQeW/G1Y/d/wwXwkv9C/iN1j8ZdEo8XfViQ+AnbDY/fsFTOyhbpHh8IdXOCxNET3ODN4KBl1OrNRHxaBQeTaN3k5yMI3YSoDy4viyRcqmzh8OvAXVJIsRM+DtxbDefw9W4agshjf2xzVB7iIyiuMqJERcWdD/in7Bkr3AVu3v/m09iWRn/xe1AlyeQ71ww= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kfence_metadata is currently a static array. For the purpose of allocating scalable __kfence_pool, we first change it to runtime allocation of metadata. Since the size of an object of kfence_metadata is 1160 bytes, we can save at least 72 pages (with default 256 objects) without enabling kfence. Below is the numbers obtained in qemu (with default 256 objects). before: Memory: 8134692K/8388080K available (3668K bss) after: Memory: 8136740K/8388080K available (1620K bss) More than expected, it saves 2MB memory. Signed-off-by: Peng Zhang --- mm/kfence/core.c | 102 ++++++++++++++++++++++++++++++++------------- mm/kfence/kfence.h | 5 ++- 2 files changed, 78 insertions(+), 29 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index dad3c0eb70a0..b9fec1c46e3d 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -116,7 +116,7 @@ EXPORT_SYMBOL(__kfence_pool); /* Export for test modules. */ * backing pages (in __kfence_pool). */ static_assert(CONFIG_KFENCE_NUM_OBJECTS > 0); -struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS]; +struct kfence_metadata *kfence_metadata; /* Freelist with available objects. */ static struct list_head kfence_freelist = LIST_HEAD_INIT(kfence_freelist); @@ -643,13 +643,56 @@ static unsigned long kfence_init_pool(void) return addr; } +static int kfence_alloc_metadata(void) +{ + unsigned long nr_pages = KFENCE_METADATA_SIZE / PAGE_SIZE; + +#ifdef CONFIG_CONTIG_ALLOC + struct page *pages; + + pages = alloc_contig_pages(nr_pages, GFP_KERNEL, first_online_node, + NULL); + if (pages) + kfence_metadata = page_to_virt(pages); +#else + if (nr_pages > MAX_ORDER_NR_PAGES) { + pr_warn("KFENCE_NUM_OBJECTS too large for buddy allocator\n"); + return -EINVAL; + } + kfence_metadata = alloc_pages_exact(KFENCE_METADATA_SIZE, + GFP_KERNEL); +#endif + + if (!kfence_metadata) + return -ENOMEM; + + memset(kfence_metadata, 0, KFENCE_METADATA_SIZE); + return 0; +} + +static void kfence_free_metadata(void) +{ + if (WARN_ON(!kfence_metadata)) + return; +#ifdef CONFIG_CONTIG_ALLOC + free_contig_range(page_to_pfn(virt_to_page((void *)kfence_metadata)), + KFENCE_METADATA_SIZE / PAGE_SIZE); +#else + free_pages_exact((void *)kfence_metadata, KFENCE_METADATA_SIZE); +#endif + kfence_metadata = NULL; +} + static bool __init kfence_init_pool_early(void) { - unsigned long addr; + unsigned long addr = (unsigned long)__kfence_pool; if (!__kfence_pool) return false; + if (!kfence_alloc_metadata()) + goto free_pool; + addr = kfence_init_pool(); if (!addr) { @@ -663,6 +706,7 @@ static bool __init kfence_init_pool_early(void) return true; } + kfence_free_metadata(); /* * Only release unprotected pages, and do not try to go back and change * page attributes due to risk of failing to do so as well. If changing @@ -670,31 +714,12 @@ static bool __init kfence_init_pool_early(void) * fails for the first page, and therefore expect addr==__kfence_pool in * most failure cases. */ +free_pool: memblock_free_late(__pa(addr), KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool)); __kfence_pool = NULL; return false; } -static bool kfence_init_pool_late(void) -{ - unsigned long addr, free_size; - - addr = kfence_init_pool(); - - if (!addr) - return true; - - /* Same as above. */ - free_size = KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool); -#ifdef CONFIG_CONTIG_ALLOC - free_contig_range(page_to_pfn(virt_to_page((void *)addr)), free_size / PAGE_SIZE); -#else - free_pages_exact((void *)addr, free_size); -#endif - __kfence_pool = NULL; - return false; -} - /* === DebugFS Interface ==================================================== */ static int stats_show(struct seq_file *seq, void *v) @@ -896,6 +921,10 @@ void __init kfence_init(void) static int kfence_init_late(void) { const unsigned long nr_pages = KFENCE_POOL_SIZE / PAGE_SIZE; + unsigned long addr = (unsigned long)__kfence_pool; + unsigned long free_size = KFENCE_POOL_SIZE; + int ret; + #ifdef CONFIG_CONTIG_ALLOC struct page *pages; @@ -913,15 +942,29 @@ static int kfence_init_late(void) return -ENOMEM; #endif - if (!kfence_init_pool_late()) { - pr_err("%s failed\n", __func__); - return -EBUSY; + ret = kfence_alloc_metadata(); + if (!ret) + goto free_pool; + + addr = kfence_init_pool(); + if (!addr) { + kfence_init_enable(); + kfence_debugfs_init(); + return 0; } - kfence_init_enable(); - kfence_debugfs_init(); + pr_err("%s failed\n", __func__); + kfence_free_metadata(); + free_size = KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool); + ret = -EBUSY; - return 0; +free_pool: +#ifdef CONFIG_CONTIG_ALLOC + free_contig_range(page_to_pfn(virt_to_page((void *)addr)), free_size / PAGE_SIZE); +#else + free_pages_exact((void *)addr, free_size); +#endif + return ret; } static int kfence_enable_late(void) @@ -941,6 +984,9 @@ void kfence_shutdown_cache(struct kmem_cache *s) struct kfence_metadata *meta; int i; + if (!__kfence_pool) + return; + for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++) { bool in_use; diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index 392fb273e7bd..f46fbb03062b 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -102,7 +102,10 @@ struct kfence_metadata { #endif }; -extern struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS]; +#define KFENCE_METADATA_SIZE PAGE_ALIGN(sizeof(struct kfence_metadata) * \ + CONFIG_KFENCE_NUM_OBJECTS) + +extern struct kfence_metadata *kfence_metadata; static inline struct kfence_metadata *addr_to_metadata(unsigned long addr) {