From patchwork Mon Jul 3 14:38:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matteo Rizzo X-Patchwork-Id: 13300197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC4A6C0015E for ; Mon, 3 Jul 2023 14:38:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D066F280007; Mon, 3 Jul 2023 10:38:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB5E0280001; Mon, 3 Jul 2023 10:38:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA505280007; Mon, 3 Jul 2023 10:38:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id AAC41280001 for ; Mon, 3 Jul 2023 10:38:47 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 65DDCA08F0 for ; Mon, 3 Jul 2023 14:38:47 +0000 (UTC) X-FDA: 80970557094.18.019BF54 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 758AF40017 for ; Mon, 3 Jul 2023 14:38:44 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=G20YFXXd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3ct2iZAsKCOMRFYYJTWNeeTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--matteorizzo.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ct2iZAsKCOMRFYYJTWNeeTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--matteorizzo.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688395124; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=kl2jCAqNCPSkEP0NABub10B8q2+JV/gXjY8VZw9Pkgk=; b=1ALoqXOC8FnXfP+SCcy0EdwHgdj4V3jj2vFMkbO81pfBDhGPLwUZXyGtc5XbKevqD2G9Pm a1wmnDWrOepwQ2cth6ltGHvliYfALJi0WJg8Apxwqk+Z8cpmLW5zoOKhZDZZsU4bBCDKyC NbA/0Xep4WnALep4cwh2QA7EclST/A4= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=G20YFXXd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3ct2iZAsKCOMRFYYJTWNeeTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--matteorizzo.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ct2iZAsKCOMRFYYJTWNeeTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--matteorizzo.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688395124; a=rsa-sha256; cv=none; b=3S5MzoLhCCeB1zUlikwf49rmMfLm8y6IU0o1qvhljw0nwwcGSjHsdKIBTB4SYzKx8PSatI bhs5Ry3/SvY7qs01Gy+FDKZp5Bh/9IOB9uQ0RahsXW1osh4gXSYweaF0ZmSXdqvYXP2Li8 YqdZPBilp5IeJEMLUCt85+w9f5JKxCQ= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-c413d8224f4so4954443276.1 for ; Mon, 03 Jul 2023 07:38:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688395123; x=1690987123; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=kl2jCAqNCPSkEP0NABub10B8q2+JV/gXjY8VZw9Pkgk=; b=G20YFXXdRv/mQ0oqlg19IV+d2kRvg3amgtHZR5NyPDqxiGLxd+3LiNemu3w212yCvH l/E8x3ZK4DBfpYr4ihiZP++/4fm2OV6TBqGsa0eWB8/cpWph9DIsdmoQgMHnuhNYwz83 UcG6OiEXh+2TYCdRmcwb9K1TtjTf4XE1V1gbWrLsIKnl1QM5Ztc2gI2mK+vYfETODso2 OolO6OspMpmDwUSp8i5vgnPa6Qx4sRa+7ZcvD/H7OX3Myit/g4/R8v+nDr4VbVJQVyB+ Om+kgoascbCgjyvHI6Xgz41trUhM2g2Ykfx1YcGmFgZ75W6vl5HRV9obNgZOGS5G6Lsg SCrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688395123; x=1690987123; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kl2jCAqNCPSkEP0NABub10B8q2+JV/gXjY8VZw9Pkgk=; b=A+o7RHaxPWA/OXMm2gIz6GsUWhWE31L5dG0K7YWXsAlGP+rAjJlFKKCSi9AcGxlbDp WKqYI8LDUhmd0270Y4fARzGrVrix+JTUJTtEe9G2aW9VewPM/occgdLmpZhr9o8IbY1m gqA2Aa9VrjZ0/YUrccnqfHHS6CUluhpUj5l1PF0WULQyQESjTSsuva2vKVc0AF6MdXec KHGZ65A4GBWgIkAPTngac4xunBWLiySvcMvoS+YFPszrGyt+xugczV6d482aO/9lIG/8 T73EYAAF1XZJe0HkI7id8f1Duq9KwpZme6W2dcI9AcDnOAhIOT/PiOhNvieTEFtwm4mW 8OJw== X-Gm-Message-State: ABy/qLa2tD7/bXrSkkpqhEhR1312+Evz9MPn40OPPAHSOslLKaEN6iDQ PfPpnFwO759B/M2QiRgmPJNQNIsvM/mEJ7s3r3hn4DufxTnHgmfAqoYsZsm98Hg1v2yjQ+R1SaK mhXA0l5intwOmQapNKsNNrZB7XIzWDzyA86nVfWx+jg51YBqAwVf83+EkKJNbavRcgKtDlA== X-Google-Smtp-Source: APBJJlErQjF8xVj4u2JEy0UYELOqnNZuWsnTZZV7UShF56diCA1LFrEiwfZhbfPmUmPY8M4bEzXkHKhEfXry7ojHlg== X-Received: from mr-cloudtop2.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:fb5]) (user=matteorizzo job=sendgmr) by 2002:a25:6901:0:b0:c17:b9e7:4c2c with SMTP id e1-20020a256901000000b00c17b9e74c2cmr121848ybc.6.1688395122873; Mon, 03 Jul 2023 07:38:42 -0700 (PDT) Date: Mon, 3 Jul 2023 14:38:20 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230703143820.152479-1-matteorizzo@google.com> Subject: [PATCH] mm/slub: refactor freelist to use custom type From: Matteo Rizzo To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz Cc: roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, jannh@google.com, matteorizzo@google.com X-Rspamd-Queue-Id: 758AF40017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ftwtya7xztbyy48j7k53wnywyyq1wb5y X-HE-Tag: 1688395124-234981 X-HE-Meta: U2FsdGVkX18I5ke5Ugg3LJ6HxBQJnGGAaFlRia868djoHTm+Wh4dHFgKpnWLCPqN5JAJcD/08wElRCYaaMYN/lcpQDfYSPlKph7vKsKpMR5jIQKpT2CdEM+2zOfBx+wzTV/OfRTPzQ0ENVupZkaSilIJEerhpgLRus8/J80Td5K6UQP35zNjoep7JQWHbPP6cmvOF6P/ZhiZ9oXWO4BZ8YUC9qMO0W28kee8VptJDe6rxEAn/N+TV2W+ch87jBx8Q8qVG1yazwjHzm6sU3FfP7mWnm6rXoua/8F3SBphskWUOBes60sRJKK4qhaygq6JH/iZpYglEyrXaC363vvd4k9mKqJhTdrWbd8UWAw447/H31k5+fzakPzg/7TR0QdPse0wj+FAaXA4KhNLu7hH15wyh0REJJTOYclJaY7Wo45LcNfxWXrKOw//91WZKToAyg/vQ5QLq3wRhXalh/2hD6fkJ8yzAsPXFUK4Nc0d1ZmWWbGBbswv0QHMkSQYiUVkCiAWwOgXkM5XNs3K8A0hrIuEJoA0DhB+B5eedNRca2CninkRwIbn9cziPL+z2sE7pb4Nl7BQe5qdDemsVSSBhRrtr0NhZeKFLA24j9laZS6kg321xJ5tUoQZe1blKlQbov1TEPrIfi4zuww1phBzVbg1Sbk7X1gwCIsx1AIrQ7DVPSAyZEEJAAf6majFUH0y+4t9g9J8vkCddLBNMozPbY719UnSVpiWrVwssIdnRZ7HkmfgErSARfKjFzZje8L88LBQ+z0PuSMLzfCyyx/Sa+9YrxKn5/u8glHBhdAsVUAn8BrsblJEgYUvYyiBATS1x6lDk415554jhOzmmZcdoJRvRz0Kkd2LQi6e+rVPZ0t9K5WX5jzp/3gw2vz3yWsYMMr9FhXJXw1D2TUskyGSF5zUTG+5t7v/AiY8YfE5MF6PazjwRX6fdIaAFakX4qY0yCRgt8GjbfSxF3GfjrX 5Q4cZU+J FGZNcc5ztMScDr+Hdzrq6OztmqFq2AUN9Q0QPS/Bn+vH93q1SiKYa5HVDld21z6G5NPx0fni5URsiJtP0Jg8SE2vHAJ1043ivn34l4Urbht8IYdRjVkVqdOqK5vQgGRLgrFWQqHe2ZxNavstyRyKOYmjM8WFMCflLxGoMdLPGTaMIV/4AzRpu9j+PVKqaA0RJILQVp5zdbCJ18Ej49t6xIa5aaogZFd7/E0xYSPPIVCvbfHdVB00i28UjOV4BNxTuyNjejGydvTuf4px+uHeJHJSR6OKGs5McL/Gr2YchkvknWt8qRWVOBor1F6P5gTRhU5WfC5UUYXUcUP8c1tBfrndsZ5sr/YNjC20pSASCk8pF3etdUKrdYFqgp1iFfkH65JqI71/fJYWSm+u4vL0Y6BFjUhKGxocZDnleLgcrngFr/qtUn6WbRhyt9EZQnG4cmg4HEoTz3EBI8pwSK3DZixCpRl+C1WL44s9pKWz1x2/JTAfDL19ROx9ngp9t27wie7HaeBwJeNpE2xTEu74wP8AVzE5elVLjLnYSsoFM3Uzsi+c0hK69a/GI9pzESMBBb3U/d90WRdKQn9KfTwE1Uj7vNjQ0PK41Z8bp65QG/UJmK19v/ABWUfRqFKUVZb7L9Z1bmy1dU3tcADsZ+3E1Z6Xe0KEhYVirTleAhMHAYXA4qdumHVRFX9b2IfzOwM6HFHbkPqFUQbImEuyg3Cq/3Hh00Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jann Horn Currently the SLUB code represents encoded freelist entries as "void*". That's misleading, those things are encoded under CONFIG_SLAB_FREELIST_HARDENED so that they're not actually dereferencable. Give them their own type, and split freelist_ptr() into one function per direction (one for encoding, one for decoding). Signed-off-by: Jann Horn Co-developed-by: Matteo Rizzo Signed-off-by: Matteo Rizzo --- include/linux/slub_def.h | 6 ++++++ mm/slub.c | 37 ++++++++++++++++++++++++++----------- 2 files changed, 32 insertions(+), 11 deletions(-) base-commit: a901a3568fd26ca9c4a82d8bc5ed5b3ed844d451 diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index deb90cf4bffb..c747820a55b4 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -43,6 +43,12 @@ enum stat_item { }; #ifndef CONFIG_SLUB_TINY +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * When changing the layout, make sure freelist and tid are still compatible * with this_cpu_cmpxchg_double() alignment requirements. diff --git a/mm/slub.c b/mm/slub.c index e3b5d5c0eb3a..26d0ca02b61d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -365,8 +365,8 @@ static struct workqueue_struct *flushwq; * with an XOR of the address where the pointer is held and a per-cache * random number. */ -static inline void *freelist_ptr(const struct kmem_cache *s, void *ptr, - unsigned long ptr_addr) +static inline freeptr_t freelist_ptr_encode(const struct kmem_cache *s, + void *ptr, unsigned long ptr_addr) { #ifdef CONFIG_SLAB_FREELIST_HARDENED /* @@ -379,25 +379,40 @@ static inline void *freelist_ptr(const struct kmem_cache *s, void *ptr, * calls get_freepointer() with an untagged pointer, which causes the * freepointer to be restored incorrectly. */ - return (void *)((unsigned long)ptr ^ s->random ^ - swab((unsigned long)kasan_reset_tag((void *)ptr_addr))); + return (freeptr_t){.v = (unsigned long)ptr ^ s->random ^ + swab((unsigned long)kasan_reset_tag((void *)ptr_addr))}; #else - return ptr; + return (freeptr_t){.v = (unsigned long)ptr}; #endif } +static inline void *freelist_ptr_decode(const struct kmem_cache *s, + freeptr_t ptr, unsigned long ptr_addr) +{ + void *decoded; + +#ifdef CONFIG_SLAB_FREELIST_HARDENED + /* See the comment in freelist_ptr_encode */ + decoded = (void *)(ptr.v ^ s->random ^ + swab((unsigned long)kasan_reset_tag((void *)ptr_addr))); +#else + decoded = (void *)ptr.v; +#endif + return decoded; +} + /* Returns the freelist pointer recorded at location ptr_addr. */ static inline void *freelist_dereference(const struct kmem_cache *s, void *ptr_addr) { - return freelist_ptr(s, (void *)*(unsigned long *)(ptr_addr), + return freelist_ptr_decode(s, *(freeptr_t *)(ptr_addr), (unsigned long)ptr_addr); } static inline void *get_freepointer(struct kmem_cache *s, void *object) { object = kasan_reset_tag(object); - return freelist_dereference(s, object + s->offset); + return freelist_dereference(s, (freeptr_t *)(object + s->offset)); } #ifndef CONFIG_SLUB_TINY @@ -421,15 +436,15 @@ __no_kmsan_checks static inline void *get_freepointer_safe(struct kmem_cache *s, void *object) { unsigned long freepointer_addr; - void *p; + freeptr_t p; if (!debug_pagealloc_enabled_static()) return get_freepointer(s, object); object = kasan_reset_tag(object); freepointer_addr = (unsigned long)object + s->offset; - copy_from_kernel_nofault(&p, (void **)freepointer_addr, sizeof(p)); - return freelist_ptr(s, p, freepointer_addr); + copy_from_kernel_nofault(&p, (freeptr_t *)freepointer_addr, sizeof(p)); + return freelist_ptr_decode(s, p, freepointer_addr); } static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) @@ -441,7 +456,7 @@ static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) #endif freeptr_addr = (unsigned long)kasan_reset_tag((void *)freeptr_addr); - *(void **)freeptr_addr = freelist_ptr(s, fp, freeptr_addr); + *(freeptr_t *)freeptr_addr = freelist_ptr_encode(s, fp, freeptr_addr); } /* Loop over all objects in a slab */