From patchwork Fri Sep 15 10:59:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matteo Rizzo X-Patchwork-Id: 13386880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BC5CEE6456 for ; Fri, 15 Sep 2023 10:59:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2ABAA6B0352; Fri, 15 Sep 2023 06:59:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 232996B0354; Fri, 15 Sep 2023 06:59:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D2CC6B0355; Fri, 15 Sep 2023 06:59:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EDA156B0352 for ; Fri, 15 Sep 2023 06:59:50 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BD5FD120684 for ; Fri, 15 Sep 2023 10:59:50 +0000 (UTC) X-FDA: 81238536540.24.510F8BF Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 0E01214000A for ; Fri, 15 Sep 2023 10:59:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MsOVwWi8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3JDkEZQsKCNUD1KK5FI9QQF7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--matteorizzo.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3JDkEZQsKCNUD1KK5FI9QQF7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--matteorizzo.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694775589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xbAGox+1BQp1J2SCvFRdM4+gVemYBYzERzmxGZTZFIk=; b=MrFGc6J3HuSBGqMCbggXa5K5w3phi6g+3lKgUMGLDXTnBCe7W4ne5NUXk7soUBR/9ZZnRd D/9i0W4z1T20WFXCUTlMe6A55afuGk/CZab0a4OSchmmGuNJ2qf1Ciij0whux9vE9gtX+b UwYUhMq+efJjJx50FnKlb35otuMbjtc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MsOVwWi8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3JDkEZQsKCNUD1KK5FI9QQF7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--matteorizzo.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3JDkEZQsKCNUD1KK5FI9QQF7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--matteorizzo.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694775589; a=rsa-sha256; cv=none; b=kegFIea9t2sgw58Q8PaJnEALuu7oOEaE5N61ew769a8plNjdaVnCWTswCf0Pg4UEB0ETal n2k2lwwDryjgv87A9eYO3R+JodY3Pbl+40gSgO7TYXLRy0sYauN6nNCDgf9S7jWhNw4Wt3 5a+i5XDJHWAqWzq/tub1JFvjjKkcNVM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-c647150c254so3818467276.1 for ; Fri, 15 Sep 2023 03:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694775588; x=1695380388; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xbAGox+1BQp1J2SCvFRdM4+gVemYBYzERzmxGZTZFIk=; b=MsOVwWi8K9XjU8Soa8amD+qMUDwf4NQ/Sd4UvxBHkeLlFRwURFDo0bcbWnBYB92krH heAhvuN7PmBYbLrZj5tN+0SM7aGWu1fBC7OUehbX4NL7gUMBlSFwamTnj07JuKgTmmiY /S8mOrAmFgKtj4aDFBdLN6B6A714aRu3dhkwVMu/lcFGgH+1P7ToWAXZxJLK19pAYCm2 DPZJQBIrfPslNO+YzGnjbNOxZrgAQWJpfnKFyhTV0hAjtonfTXgR0luDs2mSPnOsGCPg Lb7Zl0ihzD/0OdAnK1vDSK+XnBrXhOJQkQn6NiZv4UiNflD4hFpbcZx2H+frf9u1ANgp 9MXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694775588; x=1695380388; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xbAGox+1BQp1J2SCvFRdM4+gVemYBYzERzmxGZTZFIk=; b=vM1TvsxINWYUMPAQ/Eo7pHZlKK3mow6+XB7BaeW/C2fK3y1ET048R/X99UKnmCMwtc zJvKfn55nXwXbMm510wbn2aZ2V1dQKbTSR2B6AXDOP6WX+gmJWnRceWtXczdxy+HPIDu +xr6AW5vhcyKat8L5allyWI+B0MA5UtbaA4vA4iWeQFBgwamA+k+IcgGQQG71EjwYnD4 CJqsS6eCU1c04x8vlZLwk+fpIpbLV7HI3+o+K1BdHiPDVWi9A+NiMFNanik6FvdZtrvs tyehOFqJlpfrn8aQ3EyFNB5fH3fF/qitqVN9ABZ7D1MerchBOEOqH+mYPTq5WfcJunj+ /XGA== X-Gm-Message-State: AOJu0YzK4R3cgDVq4XpLX+Qz1x5Tr/9048ElOmA6j2sSIjt05fFD4r35 xNFI0FB/CBqacbJCQniihNdECwf5LYhh9/fbUg== X-Google-Smtp-Source: AGHT+IFkQIG3OgVqqlEN6/c9lGi9RDx6q7XJsfPR2EVN/OdFKbEkhspIMm/2iOfMAd3xJyQxw6Wg1f3WaLFs8ikxSg== X-Received: from mr-cloudtop2.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:2a6]) (user=matteorizzo job=sendgmr) by 2002:a05:6902:11cd:b0:d81:5c03:df99 with SMTP id n13-20020a05690211cd00b00d815c03df99mr40387ybu.3.1694775588301; Fri, 15 Sep 2023 03:59:48 -0700 (PDT) Date: Fri, 15 Sep 2023 10:59:23 +0000 In-Reply-To: <20230915105933.495735-1-matteorizzo@google.com> Mime-Version: 1.0 References: <20230915105933.495735-1-matteorizzo@google.com> X-Mailer: git-send-email 2.42.0.459.ge4e396fd5e-goog Message-ID: <20230915105933.495735-5-matteorizzo@google.com> Subject: [RFC PATCH 04/14] mm: use virt_to_slab instead of folio_slab From: Matteo Rizzo To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, corbet@lwn.net, luto@kernel.org, peterz@infradead.org Cc: jannh@google.com, matteorizzo@google.com, evn@google.com, poprdi@google.com, jordyzomer@google.com X-Rspamd-Queue-Id: 0E01214000A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 55gh6u53g4s7tua5amxc6k7d3jrg89eq X-HE-Tag: 1694775588-34253 X-HE-Meta: U2FsdGVkX19zFkIPrK0aHbkM7W4Y43tDMrYBj/Z7sNcwh5gsW5EoXekiGSPVJv9ssUCifuaajSSNAoBqqj/tskbfi+FwjSzdp7qfoFRIBwvD3db/CBnnxV0mujr58nw825BxuKdDoaWtm8ey9FX1HLGTAodBUERbTQc2Sru33ixe4R4RTj39KX5k5BI7qmZ5qs3XeJABW3EQwPKmwG45FS7IdHTw83a9iULu/Qmu5DgVrjIQCs/u1XkGFNvwFtIE423fvUcjkHAU3s+08zuVKeobAzYngBHk/xJhN4b01mcSS8/mdXJpL8zbX2sSKKCTXyrNPXFCLaMFbeGw28uQa1zx1vxGSFYrpncoMKQJY0mr9RJt8VXUiEXi7UEJu/HmwxjPqmlGPAuSxqVrfU4nSKvCUqqIFLP1mjKXUaqC6LJznnHSBh0Ded4ecQFHBe/gkiQ53O8zrpObizGZC4pcAxZ58YpwNp2nBmMhlSGTbktKttpo7q8kUMdeRbY1QZsIwLTedRRN7TaWs80rvT3IXNyJGCRsJz2tCO6uc9RCT57v+PvSX7j8ohL+i6WIMOMg511qbxbmCHXxFXTgRNLdne6jBuOvZke7DcVmM5KWKVk73beEPCEBjwmqMUlGXxbRP4kSgtsoFzE4TaMlOJgT+ig1p6KTMAdGTdBIZSdeYNUaARbJAWsd3gXQq55eESN4cIsQGGOTvzD8jk3hLKog7UFk4dCMa1gbq1BD3Lx4cybsEz8NMCdzppJ7YvxGhB9MS17m1h/rpwaI20qExntmI3zjKrK3EvZLc1ak3vCfti5s9E6y3XgNUBP09oducft4hvDw6s/sOLal0f9xSmIlhmNWCnTAifc6Jr1o5E/1fwUFzw8+06C04wPdF54FIsK2GNV9KNSgzyPT1e2livUiGV+OLnxlO7Nnu+7sJlRICbkhYG+dyYYsLDoAOB2rdg7J9AuXuY9kyysoW7Jr3Il RE+sdsSl VibMhVQEjflqOtggRPyw61JKkG58yT97WLLQOnc83Sc/Mx6AF4sDK6yTW1U5iBuB/ePJKt3TXe9SNVzX/S7lpvVrak1haQ/Ldyzr98shBGiN6hTpyAr26XJxJKi21gpa35fROiZ4HR7REr4ZeW/OVGwjbnKaZc4SkgfEoe2no/eRwaHC+lWyhpDVkQb+Xs5JRL3HwfMBbTp0pChN/NOC4i4PlDU7U0UqoMKiUst5Z41GKA4QhwGQFT82rty5W49zI/rY6RCkmggksEckcBkYY1t9lCZxabRF35NGt4iKI7vUZUb35Q7rui0vMi6eINpu7gEklHyysxcilFg5S2kLAFKFlFzyVTy9LTes9VgucF6/79TwylP36AU82LjAXQL8KI17zQp8Cvw/MzakvxDKIkNKiBNBEo53xE7lnI9XP7UDMMQhKIkX/gojkMkCvlx6sk7EjG45XrQLjHQBtjZ/PpkCSXp/nmpOcFVrlW5U75ElQJ4tPqoqKU3CFF6gayyybf0Kw7OZ7MrEnzZ96UWTKTvZ8+9S1XSNzU4+9LxnYnMd7hwL/4GMGDiXUXGER80Y+c4MvaC3tFBvYZq4eKqWrmDBY7pWYV1iDTu3HNF6mVevQFH+KYudFoApkntoqw2Q9yegbeeBqIwx42XmYsvAkPzCbyam92VicCTb8NfGkMVwH/VeNZPjtSt6aCgcmmNpaq6wZOfWSV1u9GoyEgSLMkOunY5NKnQpnL9sd9nrmFOtm2HF/uOo1OfLURFdYlF++Z4UIKZAutCZp9UA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jann Horn This is refactoring in preparation for the introduction of SLAB_VIRTUAL which does not implement folio_slab. With SLAB_VIRTUAL there is no longer a 1:1 correspondence between slabs and pages of physical memory used by the slab allocator. There is no way to look up the slab which corresponds to a specific page of physical memory without iterating over all slabs or over the page tables. Instead of doing that, we can look up the slab starting from its virtual address which can still be performed cheaply with both SLAB_VIRTUAL enabled and disabled. Signed-off-by: Jann Horn Co-developed-by: Matteo Rizzo Signed-off-by: Matteo Rizzo Reviewed-by: Kees Cook --- mm/memcontrol.c | 2 +- mm/slab_common.c | 12 +++++++----- mm/slub.c | 14 ++++++-------- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..0ab9f5323db7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2936,7 +2936,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) struct slab *slab; unsigned int off; - slab = folio_slab(folio); + slab = virt_to_slab(p); objcgs = slab_objcgs(slab); if (!objcgs) return NULL; diff --git a/mm/slab_common.c b/mm/slab_common.c index 79102d24f099..42ceaf7e9f47 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1062,13 +1062,13 @@ void kfree(const void *object) if (unlikely(ZERO_OR_NULL_PTR(object))) return; - folio = virt_to_folio(object); if (unlikely(!is_slab_addr(object))) { + folio = virt_to_folio(object); free_large_kmalloc(folio, (void *)object); return; } - slab = folio_slab(folio); + slab = virt_to_slab(object); s = slab->slab_cache; __kmem_cache_free(s, (void *)object, _RET_IP_); } @@ -1089,12 +1089,13 @@ EXPORT_SYMBOL(kfree); size_t __ksize(const void *object) { struct folio *folio; + struct kmem_cache *s; if (unlikely(object == ZERO_SIZE_PTR)) return 0; - folio = virt_to_folio(object); if (unlikely(!is_slab_addr(object))) { + folio = virt_to_folio(object); if (WARN_ON(folio_size(folio) <= KMALLOC_MAX_CACHE_SIZE)) return 0; if (WARN_ON(object != folio_address(folio))) @@ -1102,11 +1103,12 @@ size_t __ksize(const void *object) return folio_size(folio); } + s = virt_to_slab(object)->slab_cache; #ifdef CONFIG_SLUB_DEBUG - skip_orig_size_check(folio_slab(folio)->slab_cache, object); + skip_orig_size_check(s, object); #endif - return slab_ksize(folio_slab(folio)->slab_cache); + return slab_ksize(s); } void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) diff --git a/mm/slub.c b/mm/slub.c index df2529c03bd3..ad33d9e1601d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3848,25 +3848,23 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, { int lookahead = 3; void *object; - struct folio *folio; + struct slab *slab; size_t same; object = p[--size]; - folio = virt_to_folio(object); + slab = virt_to_slab(object); if (!s) { /* Handle kalloc'ed objects */ - if (unlikely(!folio_test_slab(folio))) { - free_large_kmalloc(folio, object); + if (unlikely(slab == NULL)) { + free_large_kmalloc(virt_to_folio(object), object); df->slab = NULL; return size; } - /* Derive kmem_cache from object */ - df->slab = folio_slab(folio); - df->s = df->slab->slab_cache; + df->s = slab->slab_cache; } else { - df->slab = folio_slab(folio); df->s = cache_from_obj(s, object); /* Support for memcg */ } + df->slab = slab; /* Start new detached freelist */ df->tail = object;