From patchwork Tue Mar 28 09:58:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 742F2C76195 for ; Tue, 28 Mar 2023 09:58:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E4C06B007D; Tue, 28 Mar 2023 05:58:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06FDE6B007E; Tue, 28 Mar 2023 05:58:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2A366B0080; Tue, 28 Mar 2023 05:58:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D1E966B007D for ; Tue, 28 Mar 2023 05:58:57 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A620E1608F8 for ; Tue, 28 Mar 2023 09:58:57 +0000 (UTC) X-FDA: 80617858314.11.13C19FA Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf13.hostedemail.com (Postfix) with ESMTP id DBE0420008 for ; Tue, 28 Mar 2023 09:58:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cRycGiJJ; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IMHBZhrpM5z4k2ZtEf9hxQYrO1Qyd30oWmty+49rOpw=; b=W3uVAaas25PnBvku/a8moBzH+WDCsIaC80KkeLMQs0T9bn7OS1gPZb8IRqsq/f3EQO9tki 8YMkQKHPGmJHzYvpz57aipJEpp+rQIqET/+KuRxw4p+D5phfeRZglqS6kY5dTuK9gsqPbh LMBr0X2Fngk+bW+vaYKM98rWC6zpGe4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cRycGiJJ; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997535; a=rsa-sha256; cv=none; b=TYpRIpHHcboEE4MvptDmWuUhG03Lrg/qn49+7EO6S63FDRx4ZETVwWeea1EvqAOGczZAuH JWtbYxXrd4pvzGkOyEDtX04O/D/zcP2uLSSWnZ6bLhq5Ezuqde/QQh6/k+KdpEF7QdmaTo AEXLLMPy0zptDSYyxsY1NjJKSWqZLL4= Received: by mail-pl1-f175.google.com with SMTP id ja10so11134819plb.5 for ; Tue, 28 Mar 2023 02:58:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IMHBZhrpM5z4k2ZtEf9hxQYrO1Qyd30oWmty+49rOpw=; b=cRycGiJJnkVXyhfrk/dJRNSCMoFpZNGGzl7dazDlUieFTT+oQ+yoOI0svPNXNgtJqi ueJI4DHM5wT7gQ5RfXmNTIxsoHKaTtsJl7K7gn59uLOroQOiZT59BCAtb8NqmrT1Mh3k Gmp9KRHfZU2HK7LFnjWVQlIsu8uap2Y3Occ0QtUuXKCCiQ/RNy7ku8loyA64bCWAh8nu YarRVEK3zgJVrD1NAyG+Rw6Yp31+UQjj/8IL6k6ldDxLShPxNG7chryjPAjYipTl6RtD e5Kw7dStq3e/3/leQwjmSZgKAQc77VNntEv25vM1Y9n+FcT/YOolm+LYWL6kjyf2byr7 cH1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IMHBZhrpM5z4k2ZtEf9hxQYrO1Qyd30oWmty+49rOpw=; b=yo9b8baILrvf2rfdNAJGPWURcTckVXL74IRa0ZMWgYc41usrAZKWzkNruHwUWN+F62 h5ZEodRPg/0arY+lYGpqf/yUFHgVJBWr22Cwx5DznyXvrWFnyvOqhhA8etvoY0KguVVL RLJP8+5Zmo4OMZOc4jRerjOAEgy8c9L8rX6/zMuBCsvQ8I+fUURKlW5sF81pnfV+Xbm2 4E1axldcgyq/+9rCkTMvup6qzPI6GnKLj25zb/R4Se20AQRBEx72+kZ7lDVyGUMKovW6 VoUhsPEKOwDRDXKyA6r5HLBFO6OFNL/LoSI+0SpH6RCa2+3DzXhM/AMZGBOQAkbCFfUI s2/g== X-Gm-Message-State: AO0yUKX2CnHM9N0lynuSv+/VTvK7otblx767AruCP4eiYyuht+9iCq0B JV3Lu0eqRc8n+sA0+CJX1UG0/A== X-Google-Smtp-Source: AK7set932jr2dvDlZkZ6I1j9l5nVAPzrKihlTnIBzV515yMv5dukdFImzzt3Wkm14B5fhfxAnCRdyA== X-Received: by 2002:a05:6a20:6baf:b0:da:1e1:3f46 with SMTP id bu47-20020a056a206baf00b000da01e13f46mr13145721pzb.31.1679997534793; Tue, 28 Mar 2023 02:58:54 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:54 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 5/6] mm: kfence: change kfence pool page layout Date: Tue, 28 Mar 2023 17:58:06 +0800 Message-Id: <20230328095807.7014-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DBE0420008 X-Rspam-User: X-Stat-Signature: hm3yrdu8rg8i8hec7myakooczr5gd8ka X-HE-Tag: 1679997535-482058 X-HE-Meta: U2FsdGVkX18uhV1EEiIMjFkeDq99mqUnoW7r5CK4AickNZydhaQs1hMKS7mnTcZhgQCh7FEd2q+ke70ai1EjNEL6XPkRBC8qa9iUZMdUGJD8e2mKK/Un78F8hpgPARLtBXoxb5pHFU9yvqGrqq3OvVsVyC+vMsQZPwl3JQZ1UMT+H/2FBEhyHP03Q2JHF9llwuO+Ui5Ho+Fx7ekJY3qc3rTYP4UQIC/ISpc1TDru+DuYIbqAsLnUw6/OmqYqpHRjs7wtnGbOvnaEX2P9lnNJceKpH/NQFGd57vZt5Gc2bVhTF9HvIl90uxd6sBgHvy41T6OdZBJ6jIf7vZKvrVMBNOF3xX0w+byo46E0+dcPWWvzB8Nw+YZBekBcfr2Vw1UMTLtBRxMiqojfofZD2UU6WzjWGgrD7TMMQFb4Mw8R5LhS5VTN3xXN9XudJ32jg7J2UQuAnIQQrWI/gNEI4gINcpGcBITguNwdzZ1NkjIo9hTt0HQYk+r5Ws6A/SNRHpL4gqeHIg0frTsBk0fnmfuAC29VX2TvimdafzoNtbv89he1StjAjT0V6shCcUsRb4rlyds+YtHYJ4hG05vx/fnq3rs+MMxqFPhbp37cqkcSVpBNge19JyctEr5iHzMC0d8Di/bz6/05GbXVFocsu7nMV1Yl3npMnYAeNBvNatKrcW2VIb39ObvShVyYC0/Rvdl7M2UfMVdHjZzoSxbkwLBIkQN4M24+ZuspruCof4qUMgZN11BJi/tnMGy3aKTY5UA0i1kGzZOgo9fg7t4c3nIT+ttOJI4C2OHSLKHVlbC2i2eRbrJSiUmG2jChneUk+QDR70GJl5tk2nuzgkteV1ZI6yo52Cm5g1jQg4G81mX4vD0mDKjI18EuwoGnrkJErh4275CF1JpxZV/TnHa+7zXouajmty5QWYibiDs1txyh0u30vh9/dTKx2SkI0NarFMnNuJR8udTVFO5rnkp0J+2 5ZhM2OwO z0c7MvNOD6u9vtyE9YUFTGfcTxIe/pDmy9toyS/E1tbrInfeylNlNwP6tDmcwBuebMiRzFbCHgpo5SGhtdftZjUfE0/Z7ilZiRcLsFiaEHgsdZksumcfCfCjwDdbXbtDeHVd3s3ml92MU13YYlnHXyqo/jxQhRMBnkVdmQS2+zGzNcccgyUkwf9XZjLqA4CKvjbBkCv4zOy2po8rDBkgduwmrTcEUMRiOE90lDV/Je3B3AeBc31PS/mINO+k6vwBxN9Z8iEB7mSsil7jeRW3Q0ljgOppHb7NG7ZLcFNokoyGH9uCoWpiwMvlFj5sWtbc0NIIXC4jGH0JOmy8PPkHGRQ33RRy4ZDrsrULKHbO0P8vj9HZXqJjfvYbAQcgbFqkJkRWN6PtCITXcUzk+wAzSoX8zEwWEyUAQbeKcT5Y8u+LnUEzBEeP09OsAgQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The original kfence pool layout (Given a layout with 2 objects): +------------+------------+------------+------------+------------+------------+ | guard page | guard page | object | guard page | object | guard page | +------------+------------+------------+------------+------------+------------+ | | | +----kfence_metadata[0]---+----kfence_metadata[1]---+ The comment says "the additional page in the beginning gives us an even number of pages, which simplifies the mapping of address to metadata index". However, removing the additional page does not complicate any mapping calculations. So changing it to the new layout to save a page. And remmove the KFENCE_ERROR_INVALID test since we cannot test this case easily. The new kfence pool layout (Given a layout with 2 objects): +------------+------------+------------+------------+------------+ | guard page | object | guard page | object | guard page | +------------+------------+------------+------------+------------+ | | | +----kfence_metadata[0]---+----kfence_metadata[1]---+ Signed-off-by: Muchun Song --- include/linux/kfence.h | 8 ++------ mm/kfence/core.c | 40 ++++++++-------------------------------- mm/kfence/kfence.h | 2 +- mm/kfence/kfence_test.c | 14 -------------- 4 files changed, 11 insertions(+), 53 deletions(-) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 726857a4b680..25b13a892717 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -19,12 +19,8 @@ extern unsigned long kfence_sample_interval; -/* - * We allocate an even number of pages, as it simplifies calculations to map - * address to metadata indices; effectively, the very first page serves as an - * extended guard page, but otherwise has no special purpose. - */ -#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE) +/* The last page serves as an extended guard page. */ +#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS * 2 + 1) * PAGE_SIZE) extern char *__kfence_pool; DECLARE_STATIC_KEY_FALSE(kfence_allocation_key); diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 41befcb3b069..f205b860f460 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -240,24 +240,7 @@ static inline void kfence_unprotect(unsigned long addr) static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *meta) { - unsigned long offset = (meta - kfence_metadata + 1) * PAGE_SIZE * 2; - unsigned long pageaddr = (unsigned long)&__kfence_pool[offset]; - - /* The checks do not affect performance; only called from slow-paths. */ - - /* Only call with a pointer into kfence_metadata. */ - if (KFENCE_WARN_ON(meta < kfence_metadata || - meta >= kfence_metadata + CONFIG_KFENCE_NUM_OBJECTS)) - return 0; - - /* - * This metadata object only ever maps to 1 page; verify that the stored - * address is in the expected range. - */ - if (KFENCE_WARN_ON(ALIGN_DOWN(meta->addr, PAGE_SIZE) != pageaddr)) - return 0; - - return pageaddr; + return ALIGN_DOWN(meta->addr, PAGE_SIZE); } /* @@ -535,34 +518,27 @@ static void kfence_init_pool(void) unsigned long addr = (unsigned long)__kfence_pool; int i; - /* - * Protect the first 2 pages. The first page is mostly unnecessary, and - * merely serves as an extended guard page. However, adding one - * additional page in the beginning gives us an even number of pages, - * which simplifies the mapping of address to metadata index. - */ - for (i = 0; i < 2; i++, addr += PAGE_SIZE) - kfence_protect(addr); - for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++, addr += 2 * PAGE_SIZE) { struct kfence_metadata *meta = &kfence_metadata[i]; - struct slab *slab = page_slab(virt_to_page(addr)); + struct slab *slab = page_slab(virt_to_page(addr + PAGE_SIZE)); /* Initialize metadata. */ INIT_LIST_HEAD(&meta->list); raw_spin_lock_init(&meta->lock); meta->state = KFENCE_OBJECT_UNUSED; - meta->addr = addr; /* Initialize for validation in metadata_to_pageaddr(). */ + meta->addr = addr + PAGE_SIZE; list_add_tail(&meta->list, &kfence_freelist); - /* Protect the right redzone. */ - kfence_protect(addr + PAGE_SIZE); + /* Protect the left redzone. */ + kfence_protect(addr); __folio_set_slab(slab_folio(slab)); #ifdef CONFIG_MEMCG slab->memcg_data = (unsigned long)&meta->objcg | MEMCG_DATA_OBJCGS; #endif } + + kfence_protect(addr); } static bool __init kfence_init_pool_early(void) @@ -1043,7 +1019,7 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs atomic_long_inc(&counters[KFENCE_COUNTER_BUGS]); - if (page_index % 2) { + if (page_index % 2 == 0) { /* This is a redzone, report a buffer overflow. */ struct kfence_metadata *meta; int distance = 0; diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index 600f2e2431d6..249d420100a7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -110,7 +110,7 @@ static inline struct kfence_metadata *addr_to_metadata(unsigned long addr) * __kfence_pool, in which case we would report an "invalid access" * error. */ - index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2) - 1; + index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2); if (index < 0 || index >= CONFIG_KFENCE_NUM_OBJECTS) return NULL; diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c index b5d66a69200d..d479f9c8afb1 100644 --- a/mm/kfence/kfence_test.c +++ b/mm/kfence/kfence_test.c @@ -637,19 +637,6 @@ static void test_gfpzero(struct kunit *test) KUNIT_EXPECT_FALSE(test, report_available()); } -static void test_invalid_access(struct kunit *test) -{ - const struct expect_report expect = { - .type = KFENCE_ERROR_INVALID, - .fn = test_invalid_access, - .addr = &__kfence_pool[10], - .is_write = false, - }; - - READ_ONCE(__kfence_pool[10]); - KUNIT_EXPECT_TRUE(test, report_matches(&expect)); -} - /* Test SLAB_TYPESAFE_BY_RCU works. */ static void test_memcache_typesafe_by_rcu(struct kunit *test) { @@ -787,7 +774,6 @@ static struct kunit_case kfence_test_cases[] = { KUNIT_CASE(test_kmalloc_aligned_oob_write), KUNIT_CASE(test_shrink_memcache), KUNIT_CASE(test_memcache_ctor), - KUNIT_CASE(test_invalid_access), KUNIT_CASE(test_gfpzero), KUNIT_CASE(test_memcache_typesafe_by_rcu), KUNIT_CASE(test_krealloc),