From patchwork Tue Oct 12 18:01:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BDB1C433FE for ; Tue, 12 Oct 2021 18:01:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9FC1360D07 for ; Tue, 12 Oct 2021 18:01:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9FC1360D07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 3E498900002; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3475C940007; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C219900005; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 09E76900002 for ; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id BAD4539487 for ; Tue, 12 Oct 2021 18:01:54 +0000 (UTC) X-FDA: 78688553748.09.E0C3934 Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by imf20.hostedemail.com (Postfix) with ESMTP id 2E552D0000B9 for ; Tue, 12 Oct 2021 18:01:54 +0000 (UTC) Received: by mail-qv1-f43.google.com with SMTP id v2so171391qve.11 for ; Tue, 12 Oct 2021 11:01:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RbJgcnawwGjXwGJ+K39oDahwdZcJV0i7dw3hiC/LbPg=; b=5QzNpr2BLUJljDOokSOrPvDtFpuVc2sQczyen4O28zLyi/SZAeeUF4PZj3EHu1Ejh9 6og8K5GbRhy0s+5frTMlywctpmIm/awkwIE+1yomNj0vZKSHX6Ykd7HM8741HuQ+Y4Ym N/XcCd5KEM05IRe9srxI+6keDwm0vLRzuxMweb1LwgLKsc6qMs9pg0wSz3vmByeUZZMA miMRtC7mZqhHfFSUCPkMCn4WmMInn0NwC/CHd8syZKbLSPREr1uhq4EQTmPcS6qVya8g zebI/wMDknFkZfUs5J4RUw+pDv8zVzF0bQzTlSwdT+jjL1l2qUuVOQbluBQUqReRXvr4 w71A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RbJgcnawwGjXwGJ+K39oDahwdZcJV0i7dw3hiC/LbPg=; b=6k9bRjoJ0DABbT+wHUoBNDmYhDnlkEg+sg7/gABeQWgGhr2O/CQr4kJkdW0MKaYtLM bPrh5BEVjllQ8skq11SL9xl9Q3+ltzyqEPwWvWwNajLJpbVIEYH3LD4pO8MId0JGxqU9 aWwm4UsE7WWh+N2qag90fndKn2uJMahz2cPKinvq3uk956fUhHPBrpS7xM0NNT13C7LQ rB3TP0TVaeNCz/LhHA+5ETbX25Ady+2PcRkDJK4HoCY/pWzfdaAFgGAKJ99BThGvov5a Dd1w4e3BQPWbEY03G5+Cg1LSRQJm77dJqVXPAtGFnxS3ea4IIbcje45dxeQVyQ2AKzX/ n/4A== X-Gm-Message-State: AOAM531A4J4u48CxKrytYGSc+GtOUT3hnx5SLjJX8R+uugYUV+Yd0qqb 3GhWQfNxsWmBFyrLIuhng/645HjqyqE+hg== X-Google-Smtp-Source: ABdhPJzTe2asDGWWbqqNCni7taALl8el2xqT2c/oJLsadygVsrY+qHQZwzxlTiPhVhwsEAu/G5f6FQ== X-Received: by 2002:ad4:470e:: with SMTP id k14mr30980031qvz.55.1634061712890; Tue, 12 Oct 2021 11:01:52 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id o130sm3304642qke.123.2021.10.12.11.01.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:52 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 01/11] PageSlab: bubble compound_head() into callsites Date: Tue, 12 Oct 2021 14:01:38 -0400 Message-Id: <20211012180148.1669685-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2E552D0000B9 X-Stat-Signature: c7dchb35m3zk7s9dae6ap5zbc443anee Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=5QzNpr2B; spf=pass (imf20.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspamd-Server: rspam06 X-HE-Tag: 1634061714-376294 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to be safe to call on tail pages, PageSlab() currently does an unconditional compound_head() lookup. This adds overhead to many contexts in which tail pages cannot occur. To have tailpage resolution only in places that need it, move the compound_head() call from PageSlab() into all current callsites. This is a mechanical replacement with no change in behavior or overhead. Subsequent patches will be able to eliminate the compound_head() calls from contexts in which they are not needed. Signed-off-by: Johannes Weiner --- arch/ia64/kernel/mca_drv.c | 2 +- drivers/ata/libata-sff.c | 2 +- fs/proc/page.c | 6 ++++-- include/linux/memcontrol.h | 6 +++--- include/linux/net.h | 2 +- include/linux/page-flags.h | 10 +++++----- kernel/resource.c | 2 +- mm/debug.c | 2 +- mm/kasan/common.c | 4 ++-- mm/kasan/generic.c | 2 +- mm/kasan/report.c | 2 +- mm/kasan/report_tags.c | 2 +- mm/memory-failure.c | 6 +++--- mm/memory.c | 3 ++- mm/nommu.c | 2 +- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 4 ++-- mm/slob.c | 4 ++-- mm/slub.c | 12 ++++++------ mm/usercopy.c | 2 +- mm/util.c | 2 +- 22 files changed, 44 insertions(+), 40 deletions(-) diff --git a/arch/ia64/kernel/mca_drv.c b/arch/ia64/kernel/mca_drv.c index 5bfc79be4cef..903e7c26b63e 100644 --- a/arch/ia64/kernel/mca_drv.c +++ b/arch/ia64/kernel/mca_drv.c @@ -136,7 +136,7 @@ mca_page_isolate(unsigned long paddr) return ISOLATE_NG; /* kick pages having attribute 'SLAB' or 'Reserved' */ - if (PageSlab(p) || PageReserved(p)) + if (PageSlab(compound_head(p)) || PageReserved(p)) return ISOLATE_NG; /* add attribute 'Reserved' and register the page */ diff --git a/drivers/ata/libata-sff.c b/drivers/ata/libata-sff.c index b71ea4a680b0..3a46d305616e 100644 --- a/drivers/ata/libata-sff.c +++ b/drivers/ata/libata-sff.c @@ -647,7 +647,7 @@ static void ata_pio_xfer(struct ata_queued_cmd *qc, struct page *page, qc->ap->ops->sff_data_xfer(qc, buf + offset, xfer_size, do_write); kunmap_atomic(buf); - if (!do_write && !PageSlab(page)) + if (!do_write && !PageSlab(compound_head(page))) flush_dcache_page(page); } diff --git a/fs/proc/page.c b/fs/proc/page.c index 9f1077d94cde..2c249f84e1fd 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -66,7 +66,8 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf, */ ppage = pfn_to_online_page(pfn); - if (!ppage || PageSlab(ppage) || page_has_type(ppage)) + if (!ppage || + PageSlab(compound_head(ppage)) || page_has_type(ppage)) pcount = 0; else pcount = page_mapcount(ppage); @@ -126,7 +127,7 @@ u64 stable_page_flags(struct page *page) * Note that page->_mapcount is overloaded in SLOB/SLUB/SLQB, so the * simple test in page_mapped() is not enough. */ - if (!PageSlab(page) && page_mapped(page)) + if (!PageSlab(compound_head(page)) && page_mapped(page)) u |= 1 << KPF_MMAP; if (PageAnon(page)) u |= 1 << KPF_ANON; @@ -152,6 +153,7 @@ u64 stable_page_flags(struct page *page) else if (PageTransCompound(page)) { struct page *head = compound_head(page); + /* XXX: misses isolated file THPs */ if (PageLRU(head) || PageAnon(head)) u |= 1 << KPF_THP; else if (is_huge_zero_page(head)) { diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3096c9a0ee01..02394f802698 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -397,7 +397,7 @@ static inline struct mem_cgroup *__page_memcg(struct page *page) { unsigned long memcg_data = page->memcg_data; - VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(PageSlab(compound_head(page)), page); VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); @@ -418,7 +418,7 @@ static inline struct obj_cgroup *__page_objcg(struct page *page) { unsigned long memcg_data = page->memcg_data; - VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(PageSlab(compound_head(page)), page); VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); VM_BUG_ON_PAGE(!(memcg_data & MEMCG_DATA_KMEM), page); @@ -466,7 +466,7 @@ static inline struct mem_cgroup *page_memcg_rcu(struct page *page) { unsigned long memcg_data = READ_ONCE(page->memcg_data); - VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(PageSlab(compound_head(page)), page); WARN_ON_ONCE(!rcu_read_lock_held()); if (memcg_data & MEMCG_DATA_KMEM) { diff --git a/include/linux/net.h b/include/linux/net.h index ba736b457a06..79767ae262ef 100644 --- a/include/linux/net.h +++ b/include/linux/net.h @@ -299,7 +299,7 @@ do { \ */ static inline bool sendpage_ok(struct page *page) { - return !PageSlab(page) && page_count(page) >= 1; + return !PageSlab(compound_head(page)) && page_count(page) >= 1; } int kernel_sendmsg(struct socket *sock, struct msghdr *msg, struct kvec *vec, diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a558d67ee86f..e96c9cb5bf8b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -344,7 +344,7 @@ PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD) TESTCLEARFLAG(Active, active, PF_HEAD) PAGEFLAG(Workingset, workingset, PF_HEAD) TESTCLEARFLAG(Workingset, workingset, PF_HEAD) -__PAGEFLAG(Slab, slab, PF_NO_TAIL) +__PAGEFLAG(Slab, slab, PF_ONLY_HEAD) __PAGEFLAG(SlobFree, slob_free, PF_NO_TAIL) PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */ @@ -776,7 +776,7 @@ __PAGEFLAG(Isolated, isolated, PF_ANY); */ static inline int PageSlabPfmemalloc(struct page *page) { - VM_BUG_ON_PAGE(!PageSlab(page), page); + VM_BUG_ON_PAGE(!PageSlab(compound_head(page)), page); return PageActive(page); } @@ -791,19 +791,19 @@ static inline int __PageSlabPfmemalloc(struct page *page) static inline void SetPageSlabPfmemalloc(struct page *page) { - VM_BUG_ON_PAGE(!PageSlab(page), page); + VM_BUG_ON_PAGE(!PageSlab(compound_head(page)), page); SetPageActive(page); } static inline void __ClearPageSlabPfmemalloc(struct page *page) { - VM_BUG_ON_PAGE(!PageSlab(page), page); + VM_BUG_ON_PAGE(!PageSlab(compound_head(page)), page); __ClearPageActive(page); } static inline void ClearPageSlabPfmemalloc(struct page *page) { - VM_BUG_ON_PAGE(!PageSlab(page), page); + VM_BUG_ON_PAGE(!PageSlab(compound_head(page)), page); ClearPageActive(page); } diff --git a/kernel/resource.c b/kernel/resource.c index ca9f5198a01f..a363211fda99 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -151,7 +151,7 @@ static void free_resource(struct resource *res) if (!res) return; - if (!PageSlab(virt_to_head_page(res))) { + if (!PageSlab(compound_head(virt_to_head_page(res)))) { spin_lock(&bootmem_resource_lock); res->sibling = bootmem_resource_free; bootmem_resource_free = res; diff --git a/mm/debug.c b/mm/debug.c index fae0f81ad831..500f5adce00e 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -84,7 +84,7 @@ static void __dump_page(struct page *page) * page->_mapcount space in struct page is used by sl[aou]b pages to * encode own info. */ - mapcount = PageSlab(head) ? 0 : page_mapcount(page); + mapcount = PageSlab(compound_head(head)) ? 0 : page_mapcount(page); pr_warn("page:%p refcount:%d mapcount:%d mapping:%p index:%#lx pfn:%#lx\n", page, page_ref_count(head), mapcount, mapping, diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 2baf121fb8c5..b5e81273fc6b 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -411,7 +411,7 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * !PageSlab() when the size provided to kmalloc is larger than * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ - if (unlikely(!PageSlab(page))) { + if (unlikely(!PageSlab(compound_head(page)))) { if (____kasan_kfree_large(ptr, ip)) return; kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE, false); @@ -575,7 +575,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag page = virt_to_head_page(object); /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ - if (unlikely(!PageSlab(page))) + if (unlikely(!PageSlab(compound_head(page)))) return __kasan_kmalloc_large(object, size, flags); else return ____kasan_kmalloc(page->slab_cache, object, size, flags); diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index c3f5ba7a294a..94c0c86c79d9 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -335,7 +335,7 @@ void kasan_record_aux_stack(void *addr) struct kasan_alloc_meta *alloc_meta; void *object; - if (is_kfence_address(addr) || !(page && PageSlab(page))) + if (is_kfence_address(addr) || !(page && PageSlab(compound_head(page)))) return; cache = page->slab_cache; diff --git a/mm/kasan/report.c b/mm/kasan/report.c index 884a950c7026..7cdcf968f43f 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -256,7 +256,7 @@ static void print_address_description(void *addr, u8 tag) dump_stack_lvl(KERN_ERR); pr_err("\n"); - if (page && PageSlab(page)) { + if (page && PageSlab(compound_head(page))) { struct kmem_cache *cache = page->slab_cache; void *object = nearest_obj(cache, page, addr); diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c index 8a319fc16dab..32f955d98e76 100644 --- a/mm/kasan/report_tags.c +++ b/mm/kasan/report_tags.c @@ -21,7 +21,7 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) tag = get_tag(info->access_addr); addr = kasan_reset_tag(info->access_addr); page = kasan_addr_to_page(addr); - if (page && PageSlab(page)) { + if (page && PageSlab(compound_head(page))) { cache = page->slab_cache; object = nearest_obj(cache, page, (void *)addr); alloc_meta = kasan_get_alloc_meta(cache, object); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 3e6449f2102a..0d214f800a4e 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -131,7 +131,7 @@ static int hwpoison_filter_dev(struct page *p) /* * page_mapping() does not accept slab pages. */ - if (PageSlab(p)) + if (PageSlab(compound_head(p))) return -EINVAL; mapping = page_mapping(p); @@ -289,7 +289,7 @@ void shake_page(struct page *p) if (PageHuge(p)) return; - if (!PageSlab(p)) { + if (!PageSlab(compound_head(p))) { lru_add_drain_all(); if (PageLRU(p) || is_free_buddy_page(p)) return; @@ -1285,7 +1285,7 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn, * Here we are interested only in user-mapped pages, so skip any * other types of pages. */ - if (PageReserved(p) || PageSlab(p)) + if (PageReserved(p) || PageSlab(compound_head(p))) return true; if (!(PageLRU(hpage) || PageHuge(p))) return true; diff --git a/mm/memory.c b/mm/memory.c index adf9b9ef8277..a789613af270 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1738,7 +1738,8 @@ pte_t *__get_locked_pte(struct mm_struct *mm, unsigned long addr, static int validate_page_before_insert(struct page *page) { - if (PageAnon(page) || PageSlab(page) || page_has_type(page)) + if (PageAnon(page) || + PageSlab(compound_head(page)) || page_has_type(page)) return -EINVAL; flush_dcache_page(page); return 0; diff --git a/mm/nommu.c b/mm/nommu.c index 02d2427b8f9e..c233126dd476 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -87,7 +87,7 @@ unsigned int kobjsize(const void *objp) * If the allocator sets PageSlab, we know the pointer came from * kmalloc(). */ - if (PageSlab(page)) + if (PageSlab(compound_head(page))) return ksize(objp); /* diff --git a/mm/slab.c b/mm/slab.c index d0f725637663..829f2b6d4af7 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1396,7 +1396,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - BUG_ON(!PageSlab(page)); + BUG_ON(!PageSlab(compound_head(page))); __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page_mapcount_reset(page); diff --git a/mm/slab.h b/mm/slab.h index 58c01a34e5b8..0446948c9c4e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -410,8 +410,9 @@ static inline struct kmem_cache *virt_to_cache(const void *obj) struct page *page; page = virt_to_head_page(obj); - if (WARN_ONCE(!PageSlab(page), "%s: Object is not a Slab page!\n", - __func__)) + if (WARN_ONCE(!PageSlab(compound_head(page)), + "%s: Object is not a Slab page!\n", + __func__)) return NULL; return page->slab_cache; } diff --git a/mm/slab_common.c b/mm/slab_common.c index ec2bb0beed75..5f7063797f0e 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -564,7 +564,7 @@ bool kmem_valid_obj(void *object) if (object < (void *)PAGE_SIZE || !virt_addr_valid(object)) return false; page = virt_to_head_page(object); - return PageSlab(page); + return PageSlab(compound_head(page)); } EXPORT_SYMBOL_GPL(kmem_valid_obj); @@ -594,7 +594,7 @@ void kmem_dump_obj(void *object) if (WARN_ON_ONCE(!virt_addr_valid(object))) return; page = virt_to_head_page(object); - if (WARN_ON_ONCE(!PageSlab(page))) { + if (WARN_ON_ONCE(!PageSlab(compound_head(page)))) { pr_cont(" non-slab memory.\n"); return; } diff --git a/mm/slob.c b/mm/slob.c index 74d3f6e60666..4115788227fb 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -553,7 +553,7 @@ void kfree(const void *block) kmemleak_free(block); sp = virt_to_page(block); - if (PageSlab(sp)) { + if (PageSlab(compound_head(sp))) { int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN); unsigned int *m = (unsigned int *)(block - align); slob_free(m, *m + align); @@ -579,7 +579,7 @@ size_t __ksize(const void *block) return 0; sp = virt_to_page(block); - if (unlikely(!PageSlab(sp))) + if (unlikely(!PageSlab(compound_head(sp)))) return page_size(sp); align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN); diff --git a/mm/slub.c b/mm/slub.c index 3d2025f7163b..37a4cc1e73a7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1089,7 +1089,7 @@ static int check_slab(struct kmem_cache *s, struct page *page) { int maxobj; - if (!PageSlab(page)) { + if (!PageSlab(compound_head(page))) { slab_err(s, page, "Not a valid slab page"); return 0; } @@ -1295,7 +1295,7 @@ static noinline int alloc_debug_processing(struct kmem_cache *s, return 1; bad: - if (PageSlab(page)) { + if (PageSlab(compound_head(page))) { /* * If this is a slab page then lets do the best we can * to avoid issues in the future. Marking all objects @@ -1325,7 +1325,7 @@ static inline int free_consistency_checks(struct kmem_cache *s, return 0; if (unlikely(s != page->slab_cache)) { - if (!PageSlab(page)) { + if (!PageSlab(compound_head(page))) { slab_err(s, page, "Attempt to free object(0x%p) outside of slab", object); } else if (!page->slab_cache) { @@ -3554,7 +3554,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, page = virt_to_head_page(object); if (!s) { /* Handle kalloc'ed objects */ - if (unlikely(!PageSlab(page))) { + if (unlikely(!PageSlab(compound_head(page)))) { free_nonslab_page(page, object); p[size] = NULL; /* mark object processed */ return size; @@ -4516,7 +4516,7 @@ size_t __ksize(const void *object) page = virt_to_head_page(object); - if (unlikely(!PageSlab(page))) { + if (unlikely(!PageSlab(compound_head(page)))) { WARN_ON(!PageCompound(page)); return page_size(page); } @@ -4536,7 +4536,7 @@ void kfree(const void *x) return; page = virt_to_head_page(x); - if (unlikely(!PageSlab(page))) { + if (unlikely(!PageSlab(compound_head(page)))) { free_nonslab_page(page, object); return; } diff --git a/mm/usercopy.c b/mm/usercopy.c index b3de3c4eefba..924e236522da 100644 --- a/mm/usercopy.c +++ b/mm/usercopy.c @@ -235,7 +235,7 @@ static inline void check_heap_object(const void *ptr, unsigned long n, */ page = compound_head(kmap_to_page((void *)ptr)); - if (PageSlab(page)) { + if (PageSlab(compound_head(page))) { /* Check slab allocator for flags and size. */ __check_heap_object(ptr, n, page, to_user); } else { diff --git a/mm/util.c b/mm/util.c index bacabe446906..6e6abdc9f62e 100644 --- a/mm/util.c +++ b/mm/util.c @@ -712,7 +712,7 @@ struct address_space *page_mapping(struct page *page) page = compound_head(page); /* This happens if someone calls flush_dcache_page on slab page */ - if (unlikely(PageSlab(page))) + if (unlikely(PageSlab(compound_head(page)))) return NULL; if (unlikely(PageSwapCache(page))) { From patchwork Tue Oct 12 18:01:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B17C433F5 for ; Tue, 12 Oct 2021 18:01:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DFE9060D07 for ; Tue, 12 Oct 2021 18:01:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DFE9060D07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 914BF940008; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C48A940007; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 764E7940008; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id 699DF940007 for ; Tue, 12 Oct 2021 14:01:55 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 09AD039B23 for ; Tue, 12 Oct 2021 18:01:55 +0000 (UTC) X-FDA: 78688553790.19.2DB8F65 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf22.hostedemail.com (Postfix) with ESMTP id B085C1907 for ; Tue, 12 Oct 2021 18:01:54 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id z40so18804406qko.7 for ; Tue, 12 Oct 2021 11:01:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Lyh5VszI4R7zeybDzYqsIPSiDXABCH8mlDeTJtPXHYo=; b=uW83OlcYmd9OYcOM7gdtBuDzsXhWtXx1xhFC2d6WrbzMz8avtsJakLyys2WcAwd1id t+puB5jH0LxZZN+VR7IXNhLdGUeCFeEIot2al+WyrrFsMFcrwu/yKJzyNNyFE6+K/rKi 2i8FnA1fOa4exo5V9P4Ilme1WXxoywz7C8xWoWiBw921RNFjH7kfo2PtgutGdCKrBRoY 4pa505He3m0IqeodsA/Z8+ifYbmV6SxZwfNnQ7d3nuryD9A0w2z3+PWk4fe+8BG/B4sA ndwiMUfixeU3DYgvLWGursDymyc5jmBd7KUfoSEpqAOnFOYviOf1xQ3dPZsjVAHQT1JF zuDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Lyh5VszI4R7zeybDzYqsIPSiDXABCH8mlDeTJtPXHYo=; b=djZGlYk26VxlYQwxDXd3nsrj7/r/dvyinykuJkkWMtAdWfSme7VZLa8uzlXLVhMSW5 s/7lz4ilbL90jQ7Og06Fr82oNOnM4w3ISrM72C2CTumZGqSeIsWvFRAfUwzGwgmUKnum v98KQtV1e/wlHaENdNVIiZYwqJVRx4bMI0riN/xSZaLTyym9PyLJzTjW2BBy/gH0G2qn dWuOfSqOZRP+ZPUkd1bL+JAmO502qDG2341R5GPBUWZ3MugKD4DIpgDk1V8QF4zH8FpV AdHyxFx3HAAS6yGpijio2QdtIa0JW5S708Ml85w01weLf2MnWXzBA7sF8+eYfRfu2v8y LWuw== X-Gm-Message-State: AOAM531XHRjMI8AI+yYYvMzMQAyNCzc94ww2a5xhEkVmIXT/TvoobqdI 7jAlzXwQc++LYL/YV0uE+A0QVCa2GwP4yg== X-Google-Smtp-Source: ABdhPJzrK8bQDC7NK5HHGHuo4A32b4Tvsk/5CgUECMVqu9ckf3I8IoJVXNxz+cQGMAGQgU4EnksWpA== X-Received: by 2002:a37:2743:: with SMTP id n64mr8926545qkn.394.1634061713973; Tue, 12 Oct 2021 11:01:53 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id a4sm5526589qtm.12.2021.10.12.11.01.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:53 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 02/11] PageSlab: eliminate unnecessary compound_head() calls in fs/proc/page Date: Tue, 12 Oct 2021 14:01:39 -0400 Message-Id: <20211012180148.1669685-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B085C1907 X-Stat-Signature: 95gzprczz1eos5rn8se8ef7ikmxsmeko Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=uW83OlcY; spf=pass (imf22.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-HE-Tag: 1634061714-841772 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cache multiple lookups in a local variable. Signed-off-by: Johannes Weiner --- fs/proc/page.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/proc/page.c b/fs/proc/page.c index 2c249f84e1fd..37d95309e1c1 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -108,6 +108,7 @@ static inline u64 kpf_copy_bit(u64 kflags, int ubit, int kbit) u64 stable_page_flags(struct page *page) { + struct page *head; u64 k; u64 u; @@ -118,6 +119,7 @@ u64 stable_page_flags(struct page *page) if (!page) return 1 << KPF_NOPAGE; + head = compound_head(page); k = page->flags; u = 0; @@ -127,7 +129,7 @@ u64 stable_page_flags(struct page *page) * Note that page->_mapcount is overloaded in SLOB/SLUB/SLQB, so the * simple test in page_mapped() is not enough. */ - if (!PageSlab(compound_head(page)) && page_mapped(page)) + if (!PageSlab(head) && page_mapped(page)) u |= 1 << KPF_MMAP; if (PageAnon(page)) u |= 1 << KPF_ANON; @@ -151,8 +153,6 @@ u64 stable_page_flags(struct page *page) * to make sure a given page is a thp, not a non-huge compound page. */ else if (PageTransCompound(page)) { - struct page *head = compound_head(page); - /* XXX: misses isolated file THPs */ if (PageLRU(head) || PageAnon(head)) u |= 1 << KPF_THP; @@ -185,7 +185,7 @@ u64 stable_page_flags(struct page *page) u |= kpf_copy_bit(k, KPF_LOCKED, PG_locked); u |= kpf_copy_bit(k, KPF_SLAB, PG_slab); - if (PageTail(page) && PageSlab(compound_head(page))) + if (PageTail(page) && PageSlab(head)) u |= 1 << KPF_SLAB; u |= kpf_copy_bit(k, KPF_ERROR, PG_error); From patchwork Tue Oct 12 18:01:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0247C433FE for ; Tue, 12 Oct 2021 18:02:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 46A0F60D07 for ; Tue, 12 Oct 2021 18:02:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 46A0F60D07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5E2E9940009; Tue, 12 Oct 2021 14:01:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 590B5940007; Tue, 12 Oct 2021 14:01:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4311A940009; Tue, 12 Oct 2021 14:01:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 34E8C940007 for ; Tue, 12 Oct 2021 14:01:56 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E851C183E6016 for ; Tue, 12 Oct 2021 18:01:55 +0000 (UTC) X-FDA: 78688553790.21.40D2E0D Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf25.hostedemail.com (Postfix) with ESMTP id 95BE8B000187 for ; Tue, 12 Oct 2021 18:01:55 +0000 (UTC) Received: by mail-qk1-f171.google.com with SMTP id z40so18804451qko.7 for ; Tue, 12 Oct 2021 11:01:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0e0rL00kLadi2lyvM6hOzWlirSZ5oBt2j8X8F7bJZSM=; b=chQcxN4bRHoM7gj2TinQmcPE3Y4oJRxOkiEHEqowVxhCJo6D+Op8aYj8LkysJoHjDt Ayh2luHwOdYCu+ecHB4EonMFD7nG5YVFsuzIR1Non/doeQFyR3rwSnBvD2tgIX719L5D NtJIuo/6VjEkihK9V8OZrEvY7+8xrBHkeKCUeVFHJFwICJ0/U2v4UXYBhUBwo6I0W7t1 MrxJ0Bx8nlpqDdR6T2tOncI4YqW7cw53JU2/mObdD/bJBfEqeA6EihjNw5M/h991Zvgp r/mHE8Kxz8P9RbBBsKrnLUEyV6JbZInyDW84XQIr+1R9nW/jlR07K3uM55hJPtBeMTFB T0MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0e0rL00kLadi2lyvM6hOzWlirSZ5oBt2j8X8F7bJZSM=; b=7wuZtJdWyWePI9DkWc9cM42G38z2KCp2qVRr9JKQQABnwe3OQQBjVUaelB39FZyZFH lEVrVaGr2fs0rBY0RkA+7Q08E3JAlnsWfcoo+Z2wfGaSyBKonnjZ2SFXdb8cyqU0gK8e EU2dtc70CwGMlWq/F+zDKTgBEJO4P8k0ihD5HhqGuNpJ0o77o2gq3DvxBqUM6wNS29X2 AlAylOXWMQuOOqQqTDjZBeQTVJXMEZH971W06QnwX9yp3rC1rcx8pTLpAjuB9TK1xdEU 9UBdhDce1tQesI+SrD+1+mtCDlBlu769l6omBwmx2PcFZhcc8hhhfuZjHcNMBw4yGOb2 d4tg== X-Gm-Message-State: AOAM531xsIlwZ+azHT3xWDwDfEtzh6+qOy97Vi42VW3kX+tNqJmFpofH h+3o8Dh3GLscI6ZdPp59B/AY5IK8OkDv2g== X-Google-Smtp-Source: ABdhPJza8yelISHYV2A2JNlOPcudILx0Sa5Dcpdsaxzffy0q9fqNcu137J6I1K7iZBHRbBmYnVVrFw== X-Received: by 2002:a37:a041:: with SMTP id j62mr21265322qke.423.1634061714932; Tue, 12 Oct 2021 11:01:54 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id a16sm5992748qkn.16.2021.10.12.11.01.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:54 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 03/11] PageSlab: eliminate unnecessary compound_head() calls in kernel/resource Date: Tue, 12 Oct 2021 14:01:40 -0400 Message-Id: <20211012180148.1669685-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 95BE8B000187 X-Stat-Signature: 71cprw8cju4cgxuyutxyza3qmfyigeee Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=chQcxN4b; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.171 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspamd-Server: rspam06 X-HE-Tag: 1634061715-835960 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: virt_to_head_page() implies it. Signed-off-by: Johannes Weiner --- kernel/resource.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/resource.c b/kernel/resource.c index a363211fda99..ca9f5198a01f 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -151,7 +151,7 @@ static void free_resource(struct resource *res) if (!res) return; - if (!PageSlab(compound_head(virt_to_head_page(res)))) { + if (!PageSlab(virt_to_head_page(res))) { spin_lock(&bootmem_resource_lock); res->sibling = bootmem_resource_free; bootmem_resource_free = res; From patchwork Tue Oct 12 18:01:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D5C7C433FE for ; Tue, 12 Oct 2021 18:02:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2BDE6109E for ; Tue, 12 Oct 2021 18:02:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B2BDE6109E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 45A6A94000A; Tue, 12 Oct 2021 14:01:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36D28940007; Tue, 12 Oct 2021 14:01:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EAB094000A; Tue, 12 Oct 2021 14:01:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0108.hostedemail.com [216.40.44.108]) by kanga.kvack.org (Postfix) with ESMTP id 06C84940007 for ; Tue, 12 Oct 2021 14:01:57 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B9D6C82499A8 for ; Tue, 12 Oct 2021 18:01:56 +0000 (UTC) X-FDA: 78688553832.14.46AB2C3 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf16.hostedemail.com (Postfix) with ESMTP id 76DADF00009E for ; Tue, 12 Oct 2021 18:01:56 +0000 (UTC) Received: by mail-qt1-f176.google.com with SMTP id r1so120111qta.12 for ; Tue, 12 Oct 2021 11:01:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=003A3RpbxJpia6h8OJVwVmN8N2rifd/MJ8qrrv8J8WI=; b=HNR7OnYKCsaKVe/66ccJJ3qRoY0/5QYlZMwMRph64x/N/Q5ppPrii/KqTzqfJyKdku KkoTJCIgr+Mfe6kFyiWDVWYyx4iQco4EWDdAO7H7mrg4HmI8DIh/+JcuBPbcoVGuO4BI b+NZqM+Xax0KCiR4sp+coYDJHbAH31t+u2YL51HLfkVmFkZxxJsiOzDUSvAGX7pyQpaU SC/LjIPC8lu8gyhIbM0Cz0sOAYVrfjDPXwEiAiT7QaOzhMRz00gzW+iHfmP3801XAJb2 ojN8A3QDAt0roW9h05jyXE8hjKZXI6Jrw64Uz7Smw19dyyZAo6EPHDmAY53Vf0G9tkHz O/5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=003A3RpbxJpia6h8OJVwVmN8N2rifd/MJ8qrrv8J8WI=; b=1fk3BLkN+otyW7yTI4MlnUqSB8Olu/xRil0bsfQQh7ovF8jTfbxD7ETmD9FEvHW2Z+ 1/4cr+wAsJEk1DTqT5AL+zrNw5dS62cQmJS1Ng1aw5flYVIXX+QUzhbDGk/9vX0dmyUY VJnkDv8Ygwx11dVnC/TNVWZSeCNAFcG0Qsu6yNDNnDtfSD2qypiogTVL9SOlYHOvCA0N A21uf9Ms9NzPijEsUVeZCtzsevxiZ1cajRGrj890eMSxBOf0lVOSdIOhyr5BlK4Zv2tn dwQAibIUW96Fbn4Y61kbbxXMvL3WykjQGP8ufOKMoGSZzyPA9Br/NDhgJmZifPpJ0F/g HShw== X-Gm-Message-State: AOAM530DWqHz348VIpvEQ343MD6ZymEeexsqozkhj/IrxytZAhIJ+0+Y qctHIZv1HMd6cUHXdWjgSBsfmS8bk5u7Ew== X-Google-Smtp-Source: ABdhPJyX5Fvu+U8cqav7fytlJ4DiQHHGf32yxspbjIFqLp/AdEMNeikDRJKkSjWl/VNUdIMXzGBEfg== X-Received: by 2002:a05:622a:1444:: with SMTP id v4mr24079249qtx.298.1634061715795; Tue, 12 Oct 2021 11:01:55 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id t24sm6273030qkj.38.2021.10.12.11.01.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:55 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 04/11] PageSlab: eliminate unnecessary compound_head() calls in mm/debug Date: Tue, 12 Oct 2021 14:01:41 -0400 Message-Id: <20211012180148.1669685-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 76DADF00009E X-Stat-Signature: yban9o4ua3cjnt3dsdqse7z8gfiaiozd Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=HNR7OnYK; spf=pass (imf16.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspamd-Server: rspam06 X-HE-Tag: 1634061716-82665 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: head is resolved at the beginning of the function. Signed-off-by: Johannes Weiner --- mm/debug.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/debug.c b/mm/debug.c index 500f5adce00e..fae0f81ad831 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -84,7 +84,7 @@ static void __dump_page(struct page *page) * page->_mapcount space in struct page is used by sl[aou]b pages to * encode own info. */ - mapcount = PageSlab(compound_head(head)) ? 0 : page_mapcount(page); + mapcount = PageSlab(head) ? 0 : page_mapcount(page); pr_warn("page:%p refcount:%d mapcount:%d mapping:%p index:%#lx pfn:%#lx\n", page, page_ref_count(head), mapcount, mapping, From patchwork Tue Oct 12 18:01:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5C90C433FE for ; Tue, 12 Oct 2021 18:02:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B4F560295 for ; Tue, 12 Oct 2021 18:02:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5B4F560295 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5E5F694000B; Tue, 12 Oct 2021 14:01:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 51B2A940007; Tue, 12 Oct 2021 14:01:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36EA594000B; Tue, 12 Oct 2021 14:01:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0001.hostedemail.com [216.40.44.1]) by kanga.kvack.org (Postfix) with ESMTP id 23EF0940007 for ; Tue, 12 Oct 2021 14:01:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C87D23207F for ; Tue, 12 Oct 2021 18:01:57 +0000 (UTC) X-FDA: 78688553874.16.4621649 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf04.hostedemail.com (Postfix) with ESMTP id 7F4CF50000BB for ; Tue, 12 Oct 2021 18:01:57 +0000 (UTC) Received: by mail-qt1-f175.google.com with SMTP id c20so175951qtb.2 for ; Tue, 12 Oct 2021 11:01:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iIQ3XB2hRjxutOXFzANVm4hFlprKhy5560F/z1rM1rA=; b=lGVH1y+a9cFDJqGIq5ZIU7pxSbOjZgxVsVpH3EeNfdNc9s2Vk+TWGrZcVHRdgO0L9y mYi/q0IsMML8uZBfPR7nR4KRN+4T8MCDXSc0NFImIK2u0iAo3UFG1pexA2BqiKErNUGx e8mpkqXXF5mzFiNINkf8NDcJqwyCQ83VR8istJXW110leDrBuk/fDyerrNNw68S/QaYN boGMZ4gFu3KkAqquhkrXlA+sR/+XsSAtI7PBg39+ATPS6bqQd57T2sf/u9ABk6ehlyz/ p30ZG12lzmBiFszdQ34vQRLklLf2G6aSG84DOBJSmuHKgiKLMdZqwDF12oQ9kd6ObmB0 +eRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iIQ3XB2hRjxutOXFzANVm4hFlprKhy5560F/z1rM1rA=; b=YvjO/Zrc7F1ClDig4YE/0O2FSTj3iQxp+ypNUfvtIjyLpZhXTDLQ+isHRcVYsqGinA RoOiaW4/FwcrY8achKLU04IcXIGeSrJeGynnPXHq+dsfTxuw3/ITyMZQ4jXXvC1zvlD1 NK0wQblLBaOyyZUj0aHeAis1+MRKywRU8Ws9Od06yDX7+MKsR/Sn+wWTYt69rHmWTGA/ quFHCJo9oIHnoahoNbum5W6aO3B36fEmMb/6m4O52/mec87mxaePeXLjgZNYSATxNDF6 72TcDl3FqC2FF8v0PbAGL5AqvSZolykI/PHKO0rTkDuGPGLpXzoGewfOXrA2GNxJkHl/ 5NoA== X-Gm-Message-State: AOAM533qZVIgtlSNHTITBrxUuLLCEwc5KNM5ktVfu8+SNZCKOM7ed/wp I2aBsLqiD6UxoNRSzIoc8cUmLdPwpYYXaQ== X-Google-Smtp-Source: ABdhPJwsgMHV/mOuI2+fp4J9rrrUqJwZVRic6cwsPkyPIm+tLDhrpKr8zek80w4mZnoRaY0jtif5tg== X-Received: by 2002:ac8:5a41:: with SMTP id o1mr23493208qta.21.1634061716703; Tue, 12 Oct 2021 11:01:56 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id n79sm6172567qke.97.2021.10.12.11.01.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:56 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 05/11] PageSlab: eliminate unnecessary compound_head() calls in mm/kasan Date: Tue, 12 Oct 2021 14:01:42 -0400 Message-Id: <20211012180148.1669685-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=lGVH1y+a; spf=pass (imf04.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7F4CF50000BB X-Stat-Signature: kbd953atbceoduohy49k9fj3ox7pm15p X-HE-Tag: 1634061717-168453 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All tested pages come from virt_to_head_page(): either directly or through kasan_addr_to_page(). Remove the redundant compound_head() calls. Rename the helper to kasan_addr_to_head_page() to clarify. Signed-off-by: Johannes Weiner --- mm/kasan/common.c | 4 ++-- mm/kasan/generic.c | 4 ++-- mm/kasan/kasan.h | 2 +- mm/kasan/report.c | 6 +++--- mm/kasan/report_tags.c | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index b5e81273fc6b..2baf121fb8c5 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -411,7 +411,7 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * !PageSlab() when the size provided to kmalloc is larger than * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ - if (unlikely(!PageSlab(compound_head(page)))) { + if (unlikely(!PageSlab(page))) { if (____kasan_kfree_large(ptr, ip)) return; kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE, false); @@ -575,7 +575,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag page = virt_to_head_page(object); /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ - if (unlikely(!PageSlab(compound_head(page)))) + if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else return ____kasan_kmalloc(page->slab_cache, object, size, flags); diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index 94c0c86c79d9..d4303a6722ab 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -330,12 +330,12 @@ DEFINE_ASAN_SET_SHADOW(f8); void kasan_record_aux_stack(void *addr) { - struct page *page = kasan_addr_to_page(addr); + struct page *page = kasan_addr_to_head_page(addr); struct kmem_cache *cache; struct kasan_alloc_meta *alloc_meta; void *object; - if (is_kfence_address(addr) || !(page && PageSlab(compound_head(page)))) + if (is_kfence_address(addr) || !(page && PageSlab(page))) return; cache = page->slab_cache; diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 8bf568a80eb8..fe39eeee6b59 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -249,7 +249,7 @@ bool kasan_report(unsigned long addr, size_t size, bool is_write, unsigned long ip); void kasan_report_invalid_free(void *object, unsigned long ip); -struct page *kasan_addr_to_page(const void *addr); +struct page *kasan_addr_to_head_page(const void *addr); depot_stack_handle_t kasan_save_stack(gfp_t flags); void kasan_set_track(struct kasan_track *track, gfp_t flags); diff --git a/mm/kasan/report.c b/mm/kasan/report.c index 7cdcf968f43f..405ecf3c9301 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -151,7 +151,7 @@ static void print_track(struct kasan_track *track, const char *prefix) } } -struct page *kasan_addr_to_page(const void *addr) +struct page *kasan_addr_to_head_page(const void *addr) { if ((addr >= (void *)PAGE_OFFSET) && (addr < high_memory)) @@ -251,12 +251,12 @@ static inline bool init_task_stack_addr(const void *addr) static void print_address_description(void *addr, u8 tag) { - struct page *page = kasan_addr_to_page(addr); + struct page *page = kasan_addr_to_head_page(addr); dump_stack_lvl(KERN_ERR); pr_err("\n"); - if (page && PageSlab(compound_head(page))) { + if (page && PageSlab(page)) { struct kmem_cache *cache = page->slab_cache; void *object = nearest_obj(cache, page, addr); diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c index 32f955d98e76..5ae9df06ed44 100644 --- a/mm/kasan/report_tags.c +++ b/mm/kasan/report_tags.c @@ -20,8 +20,8 @@ const char *kasan_get_bug_type(struct kasan_access_info *info) tag = get_tag(info->access_addr); addr = kasan_reset_tag(info->access_addr); - page = kasan_addr_to_page(addr); - if (page && PageSlab(compound_head(page))) { + page = kasan_addr_to_head_page(addr); + if (page && PageSlab(page)) { cache = page->slab_cache; object = nearest_obj(cache, page, (void *)addr); alloc_meta = kasan_get_alloc_meta(cache, object); From patchwork Tue Oct 12 18:01:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 736F0C433F5 for ; Tue, 12 Oct 2021 18:02:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 27ADF60D07 for ; Tue, 12 Oct 2021 18:02:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 27ADF60D07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 1778E94000C; Tue, 12 Oct 2021 14:01:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FDE3940007; Tue, 12 Oct 2021 14:01:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB74D94000C; Tue, 12 Oct 2021 14:01:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0113.hostedemail.com [216.40.44.113]) by kanga.kvack.org (Postfix) with ESMTP id DC764940007 for ; Tue, 12 Oct 2021 14:01:58 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 90C4F32637 for ; Tue, 12 Oct 2021 18:01:58 +0000 (UTC) X-FDA: 78688553916.25.8638F20 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf05.hostedemail.com (Postfix) with ESMTP id 484B6506F3EA for ; Tue, 12 Oct 2021 18:01:58 +0000 (UTC) Received: by mail-qt1-f175.google.com with SMTP id r17so129984qtx.10 for ; Tue, 12 Oct 2021 11:01:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J8ymrs8BqWOXJaCIl/vcEAutu2VS1jf5LIfnMpirIno=; b=api8DVz7bx2oVQ9f/ldYVg6zIqaVgDK7XC9AFr+oN86gJUIaU00fus2enBiUFdASXt 9z97UHOo927hwXvSGCqTP+3e5Iw95QOfW8fVJS/47yZSekiS0TdW1Pc8iMeEjE9Nuyi+ Pi7Cji+zNPmltNfU5w+6HHF8PdI6yIQH87MkiQcaOZGv0hjr7XPQJTkrU/cUMXhMKfef QEAQk/jeArS7E8NqSbtWRxnWH3BdQwv2AgQcrUsE4Tf1Yed9OouT6Kmah7HPNjLgsV3q vgMFftQyqsTPtygeQr5esj1j1xfCcPWCPKt1B4XgzkbuXHYcoYGGJN4TzKlfZ31yVoOd zTSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J8ymrs8BqWOXJaCIl/vcEAutu2VS1jf5LIfnMpirIno=; b=uw0GedPTLGGao5H3uKfvqIb4VNJW4Rb3/w52eb3YR087JWvc+3UDVe2EwVp7ehHaPl hjaZG653ky5sH5maoexorp/JMa0bj33qeDPTZzZvXJ7WteRI9dN73Xc4OUWCYCJgdrRs WLG7a35F2mWeQGSH8hozK850K4Jy0ejtluN5CGip6yXeyPTriNNwfb0bRder+T3JnkZB 3LAHM9zpOh/HgofPkm2eR5xUUSA+z8o2/XDbhtSuBZMYs+/803tpJWhTzABnLNsXM08S xqKgK1LkZWO52zYSR8Ur7X3q+BoqIiY0yy5xQYjGYj674LuoIHPBcxO17dYXbt3YQm/z 1QbQ== X-Gm-Message-State: AOAM531dg57BhE1heFqdwp5Q4s746brd4r7tHFNHYQXjj9HHgHPnmh9i MIZmSusdqa4kZBB8U0wDu/GxJg3+po4l7w== X-Google-Smtp-Source: ABdhPJwex0JUkya50iE19scsh2CAYyRUyxGrtV8kBBtZ1MSZyKRxk+YCwqYGdOyf580wHjeqTNFzFQ== X-Received: by 2002:ac8:1e10:: with SMTP id n16mr24325914qtl.86.1634061717671; Tue, 12 Oct 2021 11:01:57 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id 6sm3417977qko.88.2021.10.12.11.01.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:57 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 06/11] PageSlab: eliminate unnecessary compound_head() call in mm/nommu Date: Tue, 12 Oct 2021 14:01:43 -0400 Message-Id: <20211012180148.1669685-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 484B6506F3EA X-Stat-Signature: yj1cqfdp7fmwoa5dz63sp8wa5tkcwndf Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=api8DVz7; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-HE-Tag: 1634061718-612238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page comes from virt_to_head_page(). Signed-off-by: Johannes Weiner --- mm/nommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/nommu.c b/mm/nommu.c index c233126dd476..02d2427b8f9e 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -87,7 +87,7 @@ unsigned int kobjsize(const void *objp) * If the allocator sets PageSlab, we know the pointer came from * kmalloc(). */ - if (PageSlab(compound_head(page))) + if (PageSlab(page)) return ksize(objp); /* From patchwork Tue Oct 12 18:01:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 168C3C433F5 for ; Tue, 12 Oct 2021 18:02:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B4773604DA for ; Tue, 12 Oct 2021 18:02:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B4773604DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E6BEE94000D; Tue, 12 Oct 2021 14:01:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D815E940007; Tue, 12 Oct 2021 14:01:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1F4494000D; Tue, 12 Oct 2021 14:01:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0093.hostedemail.com [216.40.44.93]) by kanga.kvack.org (Postfix) with ESMTP id B34BC940007 for ; Tue, 12 Oct 2021 14:01:59 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 73D712D4A7 for ; Tue, 12 Oct 2021 18:01:59 +0000 (UTC) X-FDA: 78688553958.27.A2D5943 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf22.hostedemail.com (Postfix) with ESMTP id 364691903 for ; Tue, 12 Oct 2021 18:01:59 +0000 (UTC) Received: by mail-qt1-f170.google.com with SMTP id r17so130028qtx.10 for ; Tue, 12 Oct 2021 11:01:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kIr58qV5WpRay8Un4aCE59erJsFKjxl5xtHUFfszgvw=; b=APFcJCMwEwhUnWDhbjJl/47EYYCRSL66MqFK5gwiy/c+CsUcYNGXOxd9bxZdBy26Qq sQBYLREgxDAgtnn0OoS24xevjiCKpekpRPs9pcOAZVH+hHl3EShtl87yRMhzVt2xtPSD jPyEooMAjk1TTSDeYX5cvzvKmzcNYKSKE6mnhyVeDoeYYodo+YxTLbkAx78hS9pSGW7j fCKOPfVbm2QDM/7D5Odtb04yT6PJdR3tUXYXxv7GmkmzXV8+NIleM1elr4SukPwBYMnE owD6GHSShAXaSVgW+947pD2JY8B5PPv+oF1csX0yijtlTIO2sMHmRgJnxgMBKbg+lzn/ Yflw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kIr58qV5WpRay8Un4aCE59erJsFKjxl5xtHUFfszgvw=; b=FPLMqQCxabKYV6A5WffpUVDzHNialwoHOYS1MdanslWnfRpZQMoMQcmDZUYJtbglom RZk3caK/ZkM817E0CYZKCDcx030rYgDh3TyzQ72OUqfnmRX5zGzS858srtuZ5uNOx/Ma xsxTez72S3t4TJPrXde40MH+5ovsCEBz/6ruAq1ufAmf+h5+wqqkLFjLwxW6j4Si8CzJ HOY0q+Ib7l+k8i/2a4vh5sl6Kyw13hUZ7M14M0+nxKGoE6mnOFLWgdJXzO4DgsVEDpOZ 0qXTD0IWnAwTk1RlKnjcGBGIxJ0SvIpXMWy4CQiw95CcYjJylguqUGIdfIA26Re3RIpM Rc1A== X-Gm-Message-State: AOAM530LTN1dfig1Mdjdm5PZQR8aCMtHlwkvsw5qHK2WGQPkwxPhDPr3 dCnTBEOC7n+NFSOlLhrtwtw2XWFhUb5brQ== X-Google-Smtp-Source: ABdhPJwASH6CVRSFTgQo/VwOt0e0q8DY9ofHco7CvmtZqHAxJ1tq9C/Qvq6u9Bbcz1hr7c2/HMKJeQ== X-Received: by 2002:ac8:5e49:: with SMTP id i9mr1670456qtx.145.1634061718567; Tue, 12 Oct 2021 11:01:58 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id i67sm6157794qkd.90.2021.10.12.11.01.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:58 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 07/11] PageSlab: eliminate unnecessary compound_head() call in mm/slab Date: Tue, 12 Oct 2021 14:01:44 -0400 Message-Id: <20211012180148.1669685-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 364691903 X-Stat-Signature: 6u6ymkb1ewai7ojqics5zk48qcoksmce Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=APFcJCMw; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf22.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-HE-Tag: 1634061719-605159 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kmem_freepages() is only ever called on the full slab, i.e. the headpage. Further, after the PageSlab() check, the callsite continues with a series of operations that are not legal on tailpages and would trigger asserts on their own already. Signed-off-by: Johannes Weiner --- mm/slab.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slab.c b/mm/slab.c index 829f2b6d4af7..d0f725637663 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1396,7 +1396,7 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - BUG_ON(!PageSlab(compound_head(page))); + BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page_mapcount_reset(page); From patchwork Tue Oct 12 18:01:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54816C433F5 for ; Tue, 12 Oct 2021 18:02:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0D434610C9 for ; Tue, 12 Oct 2021 18:02:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0D434610C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 1F04C94000E; Tue, 12 Oct 2021 14:02:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14BDD940007; Tue, 12 Oct 2021 14:02:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F069494000E; Tue, 12 Oct 2021 14:02:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id DCCB2940007 for ; Tue, 12 Oct 2021 14:02:00 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A29DB1838F83E for ; Tue, 12 Oct 2021 18:02:00 +0000 (UTC) X-FDA: 78688554000.08.175E115 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf21.hostedemail.com (Postfix) with ESMTP id 51265D035A60 for ; Tue, 12 Oct 2021 18:02:00 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id r15so12015468qkp.8 for ; Tue, 12 Oct 2021 11:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=II+k1bERRgejUzmo9A3RKta99347TW8VHEIFP/mwtwE=; b=QcbIN7EkYNq6duRE+TluPBPUoFAfrstVRWNKeU8/xHunVY2sJpjcRkudHjbdT+pyxS ZwzDNYTsoPYxsAZKWf79iepWQUvL9lR67DdpvvuoL9wTvqiBwXyskfCNRk6Z0+q7LAN3 RyM1ovet4u4Jd8bv7NVZKmZxR+qHPGVX6D2UTrgu09vkY/wFfVkssEPY7HHBd5FCRWne EbRu9qvRCOHdtbB9Z+YefmyMxM7GZbsGjRXM3MmDmZnYg+XYYMx4gqAJJyIGRS1J/ZBL B5WSIa+SMX52bUfI36yPJji10ZAZxvpGpOuSGyI2i7BQ768SJ1mX8+aifvQRoBNcPHf0 giGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=II+k1bERRgejUzmo9A3RKta99347TW8VHEIFP/mwtwE=; b=6pa9zhrWLXXi4SSwhj/V0X7+GERvNzL31kAdmSP5pQRt2Ec131V2DqvzyfDwJJieBW dZmWl1qZ8V/xg7/8VxNXtfjo6l9LHMTE+sruJx67YCTdOr1guxPUWKyDJ/GjGe4Izt2m 4OVnxxFxavwlDMeELuS7fnQyVwkXOtpwy/ogqoYsCuc4smvAFTj/OZpnvBin+OYdhjdo ZBxASf/zgQzhDobOvXK2W4wqtDDx5ZU2wOdOVUh65lUw/PPReyKU5ullqpbWd3l2uSA8 DzOZq35ELDpe7vhn6xjbaaZtOjusD+Jnye9vaHbcrF831jzw851uvMlrKwNcoV2vtXx/ 8Huw== X-Gm-Message-State: AOAM533qu0eO+xxbdVXy10kboGmFeuOmjbVoPayqY8zrshlRJX7+B9ck /rT1XClGx9C+7123YF/P77txDTBCbBHKAw== X-Google-Smtp-Source: ABdhPJxtS3LWPGdPrw8j47cnxN2+xJsYFJyJ6FYuzADbfbBoVrm6CrU69C5MuqoTyUGlQPtBghN5Hw== X-Received: by 2002:a37:4593:: with SMTP id s141mr20569385qka.368.1634061719532; Tue, 12 Oct 2021 11:01:59 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id b8sm1927583qtr.82.2021.10.12.11.01.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:01:59 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 08/11] PageSlab: eliminate unnecessary compound_head() calls in mm/slab_common Date: Tue, 12 Oct 2021 14:01:45 -0400 Message-Id: <20211012180148.1669685-9-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 51265D035A60 X-Stat-Signature: ceouqt6p81zfg1gsk85t3qhj91djxz94 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=QcbIN7Ek; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf21.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-HE-Tag: 1634061720-547507 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: virt_to_head_page() implies it. Signed-off-by: Johannes Weiner --- mm/slab.h | 3 +-- mm/slab_common.c | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 0446948c9c4e..4bcaa08320e6 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -410,8 +410,7 @@ static inline struct kmem_cache *virt_to_cache(const void *obj) struct page *page; page = virt_to_head_page(obj); - if (WARN_ONCE(!PageSlab(compound_head(page)), - "%s: Object is not a Slab page!\n", + if (WARN_ONCE(!PageSlab(page), "%s: Object is not a Slab page!\n", __func__)) return NULL; return page->slab_cache; diff --git a/mm/slab_common.c b/mm/slab_common.c index 5f7063797f0e..ec2bb0beed75 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -564,7 +564,7 @@ bool kmem_valid_obj(void *object) if (object < (void *)PAGE_SIZE || !virt_addr_valid(object)) return false; page = virt_to_head_page(object); - return PageSlab(compound_head(page)); + return PageSlab(page); } EXPORT_SYMBOL_GPL(kmem_valid_obj); @@ -594,7 +594,7 @@ void kmem_dump_obj(void *object) if (WARN_ON_ONCE(!virt_addr_valid(object))) return; page = virt_to_head_page(object); - if (WARN_ON_ONCE(!PageSlab(compound_head(page)))) { + if (WARN_ON_ONCE(!PageSlab(page))) { pr_cont(" non-slab memory.\n"); return; } From patchwork Tue Oct 12 18:01:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C08BAC433F5 for ; Tue, 12 Oct 2021 18:02:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 753EB610E7 for ; Tue, 12 Oct 2021 18:02:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 753EB610E7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id DAA8494000F; Tue, 12 Oct 2021 14:02:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D57C0940007; Tue, 12 Oct 2021 14:02:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF80394000F; Tue, 12 Oct 2021 14:02:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id AE4BD940007 for ; Tue, 12 Oct 2021 14:02:01 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6ABD0183BCDA0 for ; Tue, 12 Oct 2021 18:02:01 +0000 (UTC) X-FDA: 78688554042.29.48D9189 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf06.hostedemail.com (Postfix) with ESMTP id 1219C801A89C for ; Tue, 12 Oct 2021 18:02:00 +0000 (UTC) Received: by mail-qk1-f176.google.com with SMTP id z40so18804661qko.7 for ; Tue, 12 Oct 2021 11:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EJleHbVtWFR6gqUGI4pMK3bP4aI2cTYgy4hO4rU19aY=; b=SzHk97+39EXkttB7Lcw0JFRnbjxuOzQa4QMcS+7Pnak9LUL8400Ch6IeQEO4X9RQYC SmUsHsHiEjQqsphLd2x9Jh3Y1GZBSxR57bLxQ93NhuCHeuPz5C8HlCXC/KAUylUm4jPG uJhs+iEHiloK/6bKOXSgO0ii+g4sdLCzkrSeh+a6LChSoa2c/Hv+oSChwRKqcNzb+wqF otALpvGvHBEq27rFOqPBpe+d1Wu68gC15DM+a/UgIfnIyLe0WpcAEUZLJBE63nD9EyZE ++nx+P0QcnjPPMhsIAYtUYtpvgmV+bwzHTtLCD9ds4ZI5yTFDjT7Mi0BQzG1w473Im5P MSOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EJleHbVtWFR6gqUGI4pMK3bP4aI2cTYgy4hO4rU19aY=; b=tyH9G3hQcwk5w170mv0GVmT+LS7hWRhe18+kYKagrmM8r+WYF4JX5OFVxOMgIvxrll u3tKfHLfTcYKBXR0Scp10AXgwBE0z1x9w7QfiM5HoGrcqnIW1BBZCrup+nAwUB/NEKWw Myy8ezaMvRwV6eXPaKh3431Eqe6pG88udf4ed7ddFwgpoc/qVbhzSDrZb4LFD72fSyLr MuBGcjC8Ol2GWsim0SuNJzZp+liE6AaFSajdznfh3jBVPsDf+nq8fE3o8tT9KAujJ2cy J3BCPWIOJVOGRITNFA59HZWRJ5S7p5fzyo36/q4cY63AEtethIQxDaNnfvkZdY3KS11V XgRg== X-Gm-Message-State: AOAM533CjXhEhmW7W5b7yhOEJmoLC6vlbmdMIkuIUNUrgq8e9l77BwNt RbJy9RfeEYPQ1H6Vli5bsrkRrk4cMPjPLg== X-Google-Smtp-Source: ABdhPJyTqn/1OfEZHmHubXFqDcHcCvvI6aiY+JnAN16rKv0iWgpmjvHcrtWJma04shZz9kCbKhhbWg== X-Received: by 2002:ae9:de84:: with SMTP id s126mr21370083qkf.350.1634061720380; Tue, 12 Oct 2021 11:02:00 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id t9sm7261733qtx.47.2021.10.12.11.02.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:02:00 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 09/11] PageSlab: eliminate unnecessary compound_head() calls in mm/slub Date: Tue, 12 Oct 2021 14:01:46 -0400 Message-Id: <20211012180148.1669685-10-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1219C801A89C X-Stat-Signature: rha38wmzne9zdmjmajxbo68cqqu8npax Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=SzHk97+3; spf=pass (imf06.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspamd-Server: rspam06 X-HE-Tag: 1634061720-362790 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: virt_to_head_page() implies it. Signed-off-by: Johannes Weiner --- mm/slub.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 37a4cc1e73a7..d1bd004c8c2f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3554,7 +3554,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, page = virt_to_head_page(object); if (!s) { /* Handle kalloc'ed objects */ - if (unlikely(!PageSlab(compound_head(page)))) { + if (unlikely(!PageSlab(page))) { free_nonslab_page(page, object); p[size] = NULL; /* mark object processed */ return size; @@ -4516,7 +4516,7 @@ size_t __ksize(const void *object) page = virt_to_head_page(object); - if (unlikely(!PageSlab(compound_head(page)))) { + if (unlikely(!PageSlab(page))) { WARN_ON(!PageCompound(page)); return page_size(page); } @@ -4536,7 +4536,7 @@ void kfree(const void *x) return; page = virt_to_head_page(x); - if (unlikely(!PageSlab(compound_head(page)))) { + if (unlikely(!PageSlab(page))) { free_nonslab_page(page, object); return; } From patchwork Tue Oct 12 18:01:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8238DC433EF for ; Tue, 12 Oct 2021 18:02:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 31FC960D07 for ; Tue, 12 Oct 2021 18:02:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 31FC960D07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C48D2940010; Tue, 12 Oct 2021 14:02:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD2D3940007; Tue, 12 Oct 2021 14:02:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FC72940010; Tue, 12 Oct 2021 14:02:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id 9150D940007 for ; Tue, 12 Oct 2021 14:02:02 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 588A21834336F for ; Tue, 12 Oct 2021 18:02:02 +0000 (UTC) X-FDA: 78688554084.25.7676ED8 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf10.hostedemail.com (Postfix) with ESMTP id 067FF6001999 for ; Tue, 12 Oct 2021 18:02:01 +0000 (UTC) Received: by mail-qt1-f174.google.com with SMTP id w8so163894qts.4 for ; Tue, 12 Oct 2021 11:02:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5lvw5Qkq1aYtQrdDyynEw7+1fb+LBtUMWle5QZBxmmc=; b=ZRxAa3dCUIQ/2t2A5bfPUBO8+qsrLWcdjgBysmSz2hHYKrWWA4F0s4IpOR036raCEC 9NXiSvCs1tTgg7nSbEwvLytFT3p8aeclNnkTRgSYbvuGXeOI5RggN9r2XlJBapTCy6k6 wflt913blZ8EukxCzpHwbZfU0RjgzaERvpGVArHj+ZVzfRWLLMGKWqWQz6VdHYO3yRRb RsyJDpGVkl88h4tLRrtNYBzQJK8hZEeVFfkQrCrySOBcj75PPMeMv0eQUF02ScpNIFlX IhVkvf+hBAf10AJv+XiKYEfzsd3TLuSXS90GMuAkx+qwVECwOSYxozbpL8MTPxjqffoX QUXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5lvw5Qkq1aYtQrdDyynEw7+1fb+LBtUMWle5QZBxmmc=; b=gES9K5x+Zuy7zPgNo3aXHripRkUr5ZSZ9ZKHf9JbvnD3oW9UbJ5cyI7Q1iGsMHMEbO zLsbuxCmAeoyDZ3Aef5qYstp0W7DIybEk/e+FZG6iI6sbvFan8YvvJbgAYYyzF+ODPZC 1OhqzqbNNYItb76eeZv4KOP6Muk8ZmUdeic5wh/zOCItBpsjUby8Fk+4pjjstDhn8G03 P6M7QzO57UYnCsKYKB/XfFI3bp6crIg03jo+UkF8D0vqLduvBs2gU3hosR8fHTJIZMJK B3BNILtWwMN2VKrbrMOC7En440OtjoYleIbcVl8cAkx/o0oU3OI9BScciyYkyD81Qpm4 GTkQ== X-Gm-Message-State: AOAM530957mP/GwntqlAYvChfrpeHewkZJYYC1aEJLR5AYsWoJTR5WbF EenD3mhmTasw2WT/g2f1a1X5JAb3uX7gqQ== X-Google-Smtp-Source: ABdhPJyN0BoLusx+W7Z8FkZAXpS+c+bgMpYmaXYb4NUw/SQe981ABNR1OzmuWw0cbmEILaHzL0SOQA== X-Received: by 2002:ac8:705d:: with SMTP id y29mr24210129qtm.73.1634061721311; Tue, 12 Oct 2021 11:02:01 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id m17sm6876285qtx.62.2021.10.12.11.02.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:02:01 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 10/11] PageSlab: eliminate unnecessary compound_head() call in mm/usercopy Date: Tue, 12 Oct 2021 14:01:47 -0400 Message-Id: <20211012180148.1669685-11-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 067FF6001999 X-Stat-Signature: ka5ikjef6pqqbjjp8ti9sj9zmki5trp3 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=ZRxAa3dC; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.174 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspamd-Server: rspam06 X-HE-Tag: 1634061721-135010 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Johannes Weiner --- mm/usercopy.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/usercopy.c b/mm/usercopy.c index 924e236522da..b3de3c4eefba 100644 --- a/mm/usercopy.c +++ b/mm/usercopy.c @@ -235,7 +235,7 @@ static inline void check_heap_object(const void *ptr, unsigned long n, */ page = compound_head(kmap_to_page((void *)ptr)); - if (PageSlab(compound_head(page))) { + if (PageSlab(page)) { /* Check slab allocator for flags and size. */ __check_heap_object(ptr, n, page, to_user); } else { From patchwork Tue Oct 12 18:01:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 12553433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F28ACC433F5 for ; Tue, 12 Oct 2021 18:02:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A84F1610CC for ; Tue, 12 Oct 2021 18:02:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A84F1610CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0890B940011; Tue, 12 Oct 2021 14:02:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0394A940007; Tue, 12 Oct 2021 14:02:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF715940011; Tue, 12 Oct 2021 14:02:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0076.hostedemail.com [216.40.44.76]) by kanga.kvack.org (Postfix) with ESMTP id CF203940007 for ; Tue, 12 Oct 2021 14:02:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 604CD2D011 for ; Tue, 12 Oct 2021 18:02:03 +0000 (UTC) X-FDA: 78688554126.03.D53D175 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by imf10.hostedemail.com (Postfix) with ESMTP id E9E5F6001987 for ; Tue, 12 Oct 2021 18:02:02 +0000 (UTC) Received: by mail-qv1-f54.google.com with SMTP id g14so227407qvb.0 for ; Tue, 12 Oct 2021 11:02:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=a7XABMOoymgkbcoo6ZtkRDMSDt06eIQo5rFsIuxFmeE=; b=m5wmVkJMh33Z9cOp79n7VVYuLnyUoX7yOoJUKNLNg4xJqPTZj4Ia1mLGb5Pf3vbkuM pcJuVSqiwxXq8xiJ7jvfyne8fnCj8hcS+7/9/9bHPWhYmHEayMlxPoGFfZiS62GKc7Bp OOJaJFO8wiOtXuh1Rya0oeNfC7gU3WJsxLCurYqX4grXeqdgpQ9lgMAwFAr8bsZH+Okb aLdPzVMPy+EpA5O1YbEyKYPt/x/gZeEzrUUpt8Oei07UKQg4U1f3hptZBPlNQicpqcXP L2T6ja/boBEdOcQaonHA2XU3k5LoU/VmQFuk+45B/qAY4Pr4ymC1gpMBvLutTiGZdgfU xrbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a7XABMOoymgkbcoo6ZtkRDMSDt06eIQo5rFsIuxFmeE=; b=NIy5QJHmcG2ynE2DB+XRbC2vtIP6zpga1xqfBA7ZeMhOxDIEep4yU1K+A3YGIMW1qZ tH9ZsiBuRuuprQoSDzjFO7cHoDZ07EwSg12HqyTfmM5dkisnWJrtGWd5FBQKq89Jj64o kEyTbBJBBtsdsqRyR9IdPgUA7UpLLgmJJalJ3WraCLCqlOCY8yczKj+R8ocrfBU7SobY ySVigxa4AjQJJzdACt5+zFQVf/3t+W7Oy3jPSIicbVloMogngn3rw3r+vlTF07gqjPqM et9b7U9OlAq3j7vs6NPOnh+9CxeEw60pt5ZwfprJMjWxkgrS3zhO3vZErnnVy7L6knJ/ QRqA== X-Gm-Message-State: AOAM53110p5ltCh1sDPrbdvU1oS0U4pB0gJ0KiYOQ7y6JN91AY2qbFSO DPTneBV5xHeCLmIVsfsO4nM52Da0OskyZg== X-Google-Smtp-Source: ABdhPJwl5SDEbPaff1ENR4HVvlY5tr2GgtyFZjAxERPqQ4NzNIGb7/CniB5M5sALLSfvwuMi6AoxSw== X-Received: by 2002:ad4:54c8:: with SMTP id j8mr31830935qvx.45.1634061722202; Tue, 12 Oct 2021 11:02:02 -0700 (PDT) Received: from localhost (cpe-98-15-154-102.hvc.res.rr.com. [98.15.154.102]) by smtp.gmail.com with ESMTPSA id b20sm7191158qtx.89.2021.10.12.11.02.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 11:02:02 -0700 (PDT) From: Johannes Weiner To: linux-mm@kvack.org Cc: Matthew Wilcox , Kent Overstreet , "Kirill A. Shutemov" , Vlastimil Babka , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 11/11] PageSlab: eliminate unnecessary compound_head() calls in mm/memcontrol Date: Tue, 12 Oct 2021 14:01:48 -0400 Message-Id: <20211012180148.1669685-12-hannes@cmpxchg.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012180148.1669685-1-hannes@cmpxchg.org> References: <20211012180148.1669685-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E9E5F6001987 X-Stat-Signature: iwmsgt8bptdddg6t8g93yaysnw1go8dm Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=m5wmVkJM; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.54 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-HE-Tag: 1634061722-168652 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page->memcg_data is generally not valid on tailpages, so the accessor functions don't need to worry about encountering them outside of bugs. The only place where the memory controller may encounter tail pages is in mem_cgroup_from_obj(), but that uses virt_to_head_page() first thing to resolve slab objects to slab pages. Signed-off-by: Johannes Weiner --- include/linux/memcontrol.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 02394f802698..3096c9a0ee01 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -397,7 +397,7 @@ static inline struct mem_cgroup *__page_memcg(struct page *page) { unsigned long memcg_data = page->memcg_data; - VM_BUG_ON_PAGE(PageSlab(compound_head(page)), page); + VM_BUG_ON_PAGE(PageSlab(page), page); VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); @@ -418,7 +418,7 @@ static inline struct obj_cgroup *__page_objcg(struct page *page) { unsigned long memcg_data = page->memcg_data; - VM_BUG_ON_PAGE(PageSlab(compound_head(page)), page); + VM_BUG_ON_PAGE(PageSlab(page), page); VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); VM_BUG_ON_PAGE(!(memcg_data & MEMCG_DATA_KMEM), page); @@ -466,7 +466,7 @@ static inline struct mem_cgroup *page_memcg_rcu(struct page *page) { unsigned long memcg_data = READ_ONCE(page->memcg_data); - VM_BUG_ON_PAGE(PageSlab(compound_head(page)), page); + VM_BUG_ON_PAGE(PageSlab(page), page); WARN_ON_ONCE(!rcu_read_lock_held()); if (memcg_data & MEMCG_DATA_KMEM) {