From patchwork Wed Feb 21 19:40:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AD87C5478A for ; Wed, 21 Feb 2024 19:41:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F06AD6B0085; Wed, 21 Feb 2024 14:41:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E6F1B6B0087; Wed, 21 Feb 2024 14:41:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C69786B0087; Wed, 21 Feb 2024 14:41:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ADC656B0083 for ; Wed, 21 Feb 2024 14:41:01 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7F4911A0AB6 for ; Wed, 21 Feb 2024 19:41:01 +0000 (UTC) X-FDA: 81816829122.14.8843C6E Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 88A1B20005 for ; Wed, 21 Feb 2024 19:40:59 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Jq7Ng9Tf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3ylHWZQYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ylHWZQYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YsTKwVJ6O4OlJiNRRNjRyUU9LCYieAfH/reETwxdtW4=; b=L9OgA5MgCXmvQIejHVF69msNwK1bvLn5pbcPV0JxIjBHjqHLUCHNeczq9o87QMyC8e31Hd fZmkjq40aZ4+ItDRAIturHNJk1OStikfX8grnZNObGvFpo9YmjAytT1dCnAHKm/IK2Lg1Q 43ewXQBMC+t2PPwTryzejfZNLlKCblY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Jq7Ng9Tf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3ylHWZQYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ylHWZQYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544459; a=rsa-sha256; cv=none; b=FckcIKuKpA07W7o8pG3O8d79M0PSpVLV+rQpFJXlHA7SSvozReSY7jXn4NtbYiCbnWT8bv txQZTeTxthc4pzR9ANTa0TqbeC9R9bYzjvk34Aayb8AwO6uCLz6hABZLYnWfrCYS7l/HHf ThPV4JvVq0DMSQSCyFRmPulbzRMeviE= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60895686ddbso3455137b3.2 for ; Wed, 21 Feb 2024 11:40:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544458; x=1709149258; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=YsTKwVJ6O4OlJiNRRNjRyUU9LCYieAfH/reETwxdtW4=; b=Jq7Ng9TfSJpY7hp4RFGYv8OGbHseO5uQmU+7WOg3vFkQQPwfLNmd117QL5H9HTqRSD HVlnZ+Q+yJ13NAXSBB6Jf3XRs/scGViWgTr63WpNXcLys5YHp25g9tMvKmZmKo7XE9j9 rCnvXZv2rFZBUUffy5QU8ZDmfIhR615GESc96vngyK3mPRAa77llbuo1OohXlQhhpsPh 4ugOjceArWgkj+/ft861R7yXSNaionZh/1rmGZIymm99+52uDp6a5Mt9H3C/65Lp1seX mFGMWEeA6ahcmtiEJ6Vs7wwvzxsPtWA5H1lZ93xPUbBKsMAjuktp1Gh+2wNO26UE7485 yqnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544458; x=1709149258; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=YsTKwVJ6O4OlJiNRRNjRyUU9LCYieAfH/reETwxdtW4=; b=UQbeHU/b18Mv5816uEE77dsNsmVoNME2IK6qY0t9MSwHFXsTZchGOpPFSFnttBu4lg doHIY8ioTxnijM0f+Vp6lt52m0EtTRd9PRCrb+bqxCa14MRdGr7+8WOEu4PrO5R6hP9c +heExe2ZVnNXJaSGrQOq9YH68JHyaupS6lTLQnC0t6Ft6vC5g6Z7CsxxvS89arphIQYE QNaVpQwyxNVmzVIyqLW2msHxU4nZ+5obw37KEcZNfAs8q5sGifX3r89kajWSRivrlho4 ncqZfCND4evMlY1nhduOfdApKZZg7BnyL6HC1p8LxniZPWB5w+YL8eBx5VsVMGVlYNRN S8kA== X-Forwarded-Encrypted: i=1; AJvYcCXS0Twp/hrWHTbCFkJM+aPLbbtFXRstxSVNjFWZxNC9Rd4VcKCI8t5yYM/ccT1s8JoHFdQ2L1vyyfUWBHOCXcXwhfQ= X-Gm-Message-State: AOJu0Yy9D4xV1xySZafiO1cOe9nfdFLYaKvGidd84u5eo+KViagTeax+ GEFiDbKBopB1A9r4p2kmg0zgdCXjW067yyUZPxkAlSvkWaODa+DIkgTQF+3aVXW9F2feXWY9Jjv sew== X-Google-Smtp-Source: AGHT+IG1D+sHkF+2zF3Zt3iL4+KZm2P1RuJvk9otZrc+38habPphSCRkeBnVCDyXAiBY0Czobu1XSofb9DE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:154f:b0:dc6:d890:1a97 with SMTP id r15-20020a056902154f00b00dc6d8901a97mr17436ybu.9.1708544458467; Wed, 21 Feb 2024 11:40:58 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:14 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-2-surenb@google.com> Subject: [PATCH v4 01/36] fix missing vmalloc.h includes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 88A1B20005 X-Stat-Signature: i5f5t3cpq8uccmomy7gkr86zrsbsqugi X-HE-Tag: 1708544459-852593 X-HE-Meta: U2FsdGVkX19fDUcDUWHiJPYrgDQkJa3TyS8/WZLFJC2Y3Zt2uStzCi4msT+SOWTA1vaxUwfwIOh3BwzOENJZhwjSI6+lhmmjjsi8Lafc02E91U38epCkr/l7TmdDanXXWMvHPd389abbSNxJhg6D/XGRmCTou5SvTSgt4eHoUS6YKk6XL7K1S+CwjXjIJdz7vPavFfmic/6JA559iYTsndLwdBHkokihLpXKRv1uoxQEtxoQWjWlmC2eZiJm74+kyFUXmr3IXRRqigj40oWfeSMGWgZylMvR6VOu7hUtJfg/wHYylNIcCCt9dtNzq6OUwly2L5qDxYKLdNReZSH9CwE+aj9kSsmpQEQJzfA4vz+EzOxTDHI5K/Glm3upFlzlGvGrvaoWEWM7kBfYmEPxqykSXSBo8z6QurKtntqXWQ1hRCMJ+mic+SCe3LIRNQd7yAEA0f+wqnSEHZGnzXeR9t2Gyfv/u+6HI6rMDDGRvkq4mQhkU2pArLYVfXZwJdmRsyZFJ7zTyJ98ekYO7p1BbB8IMzkFQq302hbEgJyJa5LxDZeEI8QyUWLpohPU1T3h+P5dL7f/7wYfqY8RrUMVlDWxEuHcqwGtCS4rny6/Hm2iEVTKLXa1p/N0foqw6eQ8LmcafPra/eJ4E8tbDwqusMYC3h/5ZXK/BW3jLbXPNcOPuztQ8kkCIthz5yUdoMyjmFtqbGarxr3M16TZR3kEtYzGQG1IdMwd3yr39QxzlYlD8V0PI3SKilj0ZHxDwV0UhXQ5R6T7KLrBS2kVj/HvCbhlt1oGQTEKlhTDSs3LrrVYVCJWkz76pk/aAP7cIoszDaJ19pgq3TVZANufky0NVk1DRHnXMmKOHZiTvXYQ5Bg7DEmYAhHhdc6lqCnd41x2dCp6a29+B3PKHhZavS4uZXyZ0UdemgPlzq/B/isIeEA/F7aL/lwQ3G/vwpjkLZlW3pJ23g1SkbTo2+Su7gu LtOtb6av HuY4g7Gi/+y88pHYVEwUtMJ4MO5lySPy4ci1VoKUqzmlHq3BiCA7vx6b2fmW06PDj2f5PMW4T/4T+ewR7M5FznyHClqy8CURSJ0F83cX7+/ESUAuO4csj5OsQ/1eL6g+MKpSazsa4tkVyIJkabWl2CvIVxloVmBkzeEz3+l/aI8KwUyWdUjeS73V1I+TC4Ok7zqwc1euHG3u2FOPIwxuz6sklgRyodRZok2dWQstN5v12bqP/OKSq8wcMFNHCc5rpN81pSGQLr8BgUejpUsip4qP9wO9GaJScuaFS22+mEwiq6KgqdBuc3xAxd35cKwzsNYjJBolWrij+CwWwxbGdghnAQJtCbQl6lBUiVWCuWU8v+LajhhxgLuX+h/9p6BGRNuX2qRYtqdaKWatUZ91fxuKEJhM5/GE9/m3IQQ7LfKNMWbVpTAC2ZRN+hEIgWBQtk/s5FmNok20aELaORu6SF+6y4tr8celxEzW3oupgrQdDejcmvf2KGZL6vUokvm815XhRJ7Hk2Tfc23zoDclZdUZVDX2Kd5oFhCNaUA5XDE8NqACdN7YXuP/r3GR03WiqRpSZUXEO6Z1x4yZMZ6DxAMQpXrgxRNY5Y8/Xcu2BZq7ixHWvsIG9I3xrGAXGBS1Wtb72XgvBeAKMYsMz/UqOzIOAW1DU0ev4icHVKWLlZDjoulc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The next patch drops vmalloc.h from a system header in order to fix a circular dependency; this adds it to all the files that were pulling it in implicitly. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- arch/alpha/lib/checksum.c | 1 + arch/alpha/lib/fpreg.c | 1 + arch/alpha/lib/memcpy.c | 1 + arch/arm/kernel/irq.c | 1 + arch/arm/kernel/traps.c | 1 + arch/arm64/kernel/efi.c | 1 + arch/loongarch/include/asm/kfence.h | 1 + arch/powerpc/kernel/iommu.c | 1 + arch/powerpc/mm/mem.c | 1 + arch/riscv/kernel/elf_kexec.c | 1 + arch/riscv/kernel/probes/kprobes.c | 1 + arch/s390/kernel/cert_store.c | 1 + arch/s390/kernel/ipl.c | 1 + arch/x86/include/asm/io.h | 1 + arch/x86/kernel/cpu/sgx/main.c | 1 + arch/x86/kernel/irq_64.c | 1 + arch/x86/mm/fault.c | 1 + drivers/accel/ivpu/ivpu_mmu_context.c | 1 + drivers/gpu/drm/gma500/mmu.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_pages.c | 1 + drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c | 1 + drivers/gpu/drm/i915/gt/shmem_utils.c | 1 + drivers/gpu/drm/i915/gvt/firmware.c | 1 + drivers/gpu/drm/i915/gvt/gtt.c | 1 + drivers/gpu/drm/i915/gvt/handlers.c | 1 + drivers/gpu/drm/i915/gvt/mmio.c | 1 + drivers/gpu/drm/i915/gvt/vgpu.c | 1 + drivers/gpu/drm/i915/intel_gvt.c | 1 + drivers/gpu/drm/imagination/pvr_vm_mips.c | 1 + drivers/gpu/drm/mediatek/mtk_drm_gem.c | 1 + drivers/gpu/drm/omapdrm/omap_gem.c | 1 + drivers/gpu/drm/v3d/v3d_bo.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_binding.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c | 1 + drivers/gpu/drm/xen/xen_drm_front_gem.c | 1 + drivers/hwtracing/coresight/coresight-trbe.c | 1 + drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c | 1 + drivers/net/ethernet/microsoft/mana/hw_channel.c | 1 + drivers/platform/x86/uv_sysfs.c | 1 + drivers/scsi/mpi3mr/mpi3mr_transport.c | 2 ++ drivers/vfio/pci/pds/dirty.c | 1 + drivers/virt/acrn/mm.c | 1 + drivers/virtio/virtio_mem.c | 1 + include/linux/pds/pds_common.h | 2 ++ include/rdma/rdmavt_qp.h | 1 + mm/debug_vm_pgtable.c | 1 + sound/pci/hda/cs35l41_hda.c | 1 + 51 files changed, 53 insertions(+) diff --git a/arch/alpha/lib/checksum.c b/arch/alpha/lib/checksum.c index 3f35c3ed6948..c29b98ef9c82 100644 --- a/arch/alpha/lib/checksum.c +++ b/arch/alpha/lib/checksum.c @@ -14,6 +14,7 @@ #include #include +#include static inline unsigned short from64to16(unsigned long x) { diff --git a/arch/alpha/lib/fpreg.c b/arch/alpha/lib/fpreg.c index 7c08b225261c..3d32165043f8 100644 --- a/arch/alpha/lib/fpreg.c +++ b/arch/alpha/lib/fpreg.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #if defined(CONFIG_ALPHA_EV6) || defined(CONFIG_ALPHA_EV67) diff --git a/arch/alpha/lib/memcpy.c b/arch/alpha/lib/memcpy.c index cbac3dc6d963..0e536a1a39ff 100644 --- a/arch/alpha/lib/memcpy.c +++ b/arch/alpha/lib/memcpy.c @@ -18,6 +18,7 @@ #include #include +#include /* * This should be done in one go with ldq_u*2/mask/stq_u. Do it diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index fe28fc1f759d..dab42d066d06 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 3bad79db5d6e..27addbf0f98c 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 0228001347be..a0dc6b88b11b 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include diff --git a/arch/loongarch/include/asm/kfence.h b/arch/loongarch/include/asm/kfence.h index 6c82aea1c993..54062656dc7b 100644 --- a/arch/loongarch/include/asm/kfence.h +++ b/arch/loongarch/include/asm/kfence.h @@ -10,6 +10,7 @@ #define _ASM_LOONGARCH_KFENCE_H #include +#include #include #include diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c index a9bebfd56b3b..25782d361884 100644 --- a/arch/powerpc/kernel/iommu.c +++ b/arch/powerpc/kernel/iommu.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index 3a440004b97d..a197d4c2244b 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include diff --git a/arch/riscv/kernel/elf_kexec.c b/arch/riscv/kernel/elf_kexec.c index 5bd1ec3341fe..92b1e16f99c4 100644 --- a/arch/riscv/kernel/elf_kexec.c +++ b/arch/riscv/kernel/elf_kexec.c @@ -19,6 +19,7 @@ #include #include #include +#include #include int arch_kimage_file_post_load_cleanup(struct kimage *image) diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 2f08c14a933d..71a8b8945b26 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/s390/kernel/cert_store.c b/arch/s390/kernel/cert_store.c index 554447768bdd..bf983513dd33 100644 --- a/arch/s390/kernel/cert_store.c +++ b/arch/s390/kernel/cert_store.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c index ba75f6bee774..0854a8450a6e 100644 --- a/arch/s390/kernel/ipl.c +++ b/arch/s390/kernel/ipl.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index 3814a9263d64..c6b799d28126 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -42,6 +42,7 @@ #include #include #include +#include #define build_mmio_read(name, size, type, reg, barrier) \ static inline type name(const volatile void __iomem *addr) \ diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 166692f2d501..27892e57c4ef 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "driver.h" #include "encl.h" diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c index fe0c859873d1..ade0043ce56e 100644 --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 679b09cfe241..af223e57aa63 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -20,6 +20,7 @@ #include /* efi_crash_gracefully_on_page_fault()*/ #include #include /* find_and_lock_vma() */ +#include #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ diff --git a/drivers/accel/ivpu/ivpu_mmu_context.c b/drivers/accel/ivpu/ivpu_mmu_context.c index fe6161299236..128aef8e5a19 100644 --- a/drivers/accel/ivpu/ivpu_mmu_context.c +++ b/drivers/accel/ivpu/ivpu_mmu_context.c @@ -6,6 +6,7 @@ #include #include #include +#include #include diff --git a/drivers/gpu/drm/gma500/mmu.c b/drivers/gpu/drm/gma500/mmu.c index a70b01ccdf70..4d78b33eaa82 100644 --- a/drivers/gpu/drm/gma500/mmu.c +++ b/drivers/gpu/drm/gma500/mmu.c @@ -5,6 +5,7 @@ **************************************************************************/ #include +#include #include "mmu.h" #include "psb_drv.h" diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c index 0ba955611dfb..8780aa243105 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c @@ -5,6 +5,7 @@ */ #include +#include #include "gt/intel_gt.h" #include "gt/intel_tlb.h" diff --git a/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c b/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c index b2a5882b8f81..075657018739 100644 --- a/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c @@ -4,6 +4,7 @@ * Copyright © 2016 Intel Corporation */ +#include #include "mock_dmabuf.h" static struct sg_table *mock_map_dma_buf(struct dma_buf_attachment *attachment, diff --git a/drivers/gpu/drm/i915/gt/shmem_utils.c b/drivers/gpu/drm/i915/gt/shmem_utils.c index bccc3a1200bc..1fb6ff77fd89 100644 --- a/drivers/gpu/drm/i915/gt/shmem_utils.c +++ b/drivers/gpu/drm/i915/gt/shmem_utils.c @@ -7,6 +7,7 @@ #include #include #include +#include #include "i915_drv.h" #include "gem/i915_gem_object.h" diff --git a/drivers/gpu/drm/i915/gvt/firmware.c b/drivers/gpu/drm/i915/gvt/firmware.c index 4dd52ac2043e..d800d267f0e9 100644 --- a/drivers/gpu/drm/i915/gvt/firmware.c +++ b/drivers/gpu/drm/i915/gvt/firmware.c @@ -30,6 +30,7 @@ #include #include +#include #include "i915_drv.h" #include "gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 094fca9b0e73..58cca4906f41 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -39,6 +39,7 @@ #include "trace.h" #include "gt/intel_gt_regs.h" +#include #if defined(VERBOSE_DEBUG) #define gvt_vdbg_mm(fmt, args...) gvt_dbg_mm(fmt, ##args) diff --git a/drivers/gpu/drm/i915/gvt/handlers.c b/drivers/gpu/drm/i915/gvt/handlers.c index efcb00472be2..ea9c30092767 100644 --- a/drivers/gpu/drm/i915/gvt/handlers.c +++ b/drivers/gpu/drm/i915/gvt/handlers.c @@ -52,6 +52,7 @@ #include "display/skl_watermark_regs.h" #include "display/vlv_dsi_pll_regs.h" #include "gt/intel_gt_regs.h" +#include /* XXX FIXME i915 has changed PP_XXX definition */ #define PCH_PP_STATUS _MMIO(0xc7200) diff --git a/drivers/gpu/drm/i915/gvt/mmio.c b/drivers/gpu/drm/i915/gvt/mmio.c index 5b5def6ddef7..780762f28aa4 100644 --- a/drivers/gpu/drm/i915/gvt/mmio.c +++ b/drivers/gpu/drm/i915/gvt/mmio.c @@ -33,6 +33,7 @@ * */ +#include #include "i915_drv.h" #include "i915_reg.h" #include "gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/vgpu.c b/drivers/gpu/drm/i915/gvt/vgpu.c index 08ad1bd651f1..63c751ca4119 100644 --- a/drivers/gpu/drm/i915/gvt/vgpu.c +++ b/drivers/gpu/drm/i915/gvt/vgpu.c @@ -34,6 +34,7 @@ #include "i915_drv.h" #include "gvt.h" #include "i915_pvinfo.h" +#include void populate_pvinfo_page(struct intel_vgpu *vgpu) { diff --git a/drivers/gpu/drm/i915/intel_gvt.c b/drivers/gpu/drm/i915/intel_gvt.c index 9b6d87c8b583..5a01d60e5186 100644 --- a/drivers/gpu/drm/i915/intel_gvt.c +++ b/drivers/gpu/drm/i915/intel_gvt.c @@ -28,6 +28,7 @@ #include "gt/intel_context.h" #include "gt/intel_ring.h" #include "gt/shmem_utils.h" +#include /** * DOC: Intel GVT-g host support diff --git a/drivers/gpu/drm/imagination/pvr_vm_mips.c b/drivers/gpu/drm/imagination/pvr_vm_mips.c index b7fef3c797e6..6563dcde109c 100644 --- a/drivers/gpu/drm/imagination/pvr_vm_mips.c +++ b/drivers/gpu/drm/imagination/pvr_vm_mips.c @@ -14,6 +14,7 @@ #include #include #include +#include /** * pvr_vm_mips_init() - Initialise MIPS FW pagetable diff --git a/drivers/gpu/drm/mediatek/mtk_drm_gem.c b/drivers/gpu/drm/mediatek/mtk_drm_gem.c index 4f2e3feabc0f..3e519869b632 100644 --- a/drivers/gpu/drm/mediatek/mtk_drm_gem.c +++ b/drivers/gpu/drm/mediatek/mtk_drm_gem.c @@ -4,6 +4,7 @@ */ #include +#include #include #include diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c index 3421e8389222..9ea0c64c26b5 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem.c +++ b/drivers/gpu/drm/omapdrm/omap_gem.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include diff --git a/drivers/gpu/drm/v3d/v3d_bo.c b/drivers/gpu/drm/v3d/v3d_bo.c index 1bdfac8beafd..bd078852cd60 100644 --- a/drivers/gpu/drm/v3d/v3d_bo.c +++ b/drivers/gpu/drm/v3d/v3d_bo.c @@ -21,6 +21,7 @@ #include #include +#include #include "v3d_drv.h" #include "uapi/drm/v3d_drm.h" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c b/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c index ae2de914eb89..2731f6ded1c2 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c @@ -54,6 +54,7 @@ #include "vmwgfx_drv.h" #include "vmwgfx_binding.h" #include "device_include/svga3d_reg.h" +#include #define VMW_BINDING_RT_BIT 0 #define VMW_BINDING_PS_BIT 1 diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c b/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c index 195ff8792e5a..dd4ca6a9c690 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c @@ -31,6 +31,7 @@ #include #include +#include bool vmw_supports_3d(struct vmw_private *dev_priv) { diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c b/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c index 829df395c2ed..6e6beff9e262 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c @@ -25,6 +25,7 @@ * **************************************************************************/ +#include #include "vmwgfx_devcaps.h" #include "vmwgfx_drv.h" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index d3e308fdfd5b..7a451410ad77 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -53,6 +53,7 @@ #include #include #include +#include #define VMWGFX_DRIVER_DESC "Linux drm driver for VMware graphics devices" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c index 36987ef3fc30..4ce22843015e 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c @@ -35,6 +35,7 @@ #include #include +#include /* * Helper macro to get dx_ctx_node if available otherwise print an error diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c b/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c index a1da5678c731..835d1eed8dd9 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c @@ -31,6 +31,7 @@ #include #include +#include int vmw_getparam_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv) diff --git a/drivers/gpu/drm/xen/xen_drm_front_gem.c b/drivers/gpu/drm/xen/xen_drm_front_gem.c index 3ad2b4cfd1f0..63112ed975c4 100644 --- a/drivers/gpu/drm/xen/xen_drm_front_gem.c +++ b/drivers/gpu/drm/xen/xen_drm_front_gem.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 6136776482e6..96a32b213669 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -17,6 +17,7 @@ #include #include +#include #include "coresight-self-hosted-trace.h" #include "coresight-trbe.h" diff --git a/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c b/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c index 2e2c3be8a0b4..e6eb98d70f3c 100644 --- a/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c +++ b/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "octep_config.h" #include "octep_main.h" diff --git a/drivers/net/ethernet/microsoft/mana/hw_channel.c b/drivers/net/ethernet/microsoft/mana/hw_channel.c index 2729a2c5acf9..11021c34e47e 100644 --- a/drivers/net/ethernet/microsoft/mana/hw_channel.c +++ b/drivers/net/ethernet/microsoft/mana/hw_channel.c @@ -3,6 +3,7 @@ #include #include +#include static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id) { diff --git a/drivers/platform/x86/uv_sysfs.c b/drivers/platform/x86/uv_sysfs.c index 38d1b692d3c0..40e010877189 100644 --- a/drivers/platform/x86/uv_sysfs.c +++ b/drivers/platform/x86/uv_sysfs.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/scsi/mpi3mr/mpi3mr_transport.c b/drivers/scsi/mpi3mr/mpi3mr_transport.c index c0c8ab586957..408a4023406b 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_transport.c +++ b/drivers/scsi/mpi3mr/mpi3mr_transport.c @@ -7,6 +7,8 @@ * */ +#include + #include "mpi3mr.h" /** diff --git a/drivers/vfio/pci/pds/dirty.c b/drivers/vfio/pci/pds/dirty.c index 8ddf4346fcd5..0a161becd646 100644 --- a/drivers/vfio/pci/pds/dirty.c +++ b/drivers/vfio/pci/pds/dirty.c @@ -3,6 +3,7 @@ #include #include +#include #include #include diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index fa5d9ca6be57..c088ee1f1180 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "acrn_drv.h" diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index 8e3223294442..e8355f55a8f7 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -21,6 +21,7 @@ #include #include #include +#include #include diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h index 30581e2e04cc..5802e1deef24 100644 --- a/include/linux/pds/pds_common.h +++ b/include/linux/pds/pds_common.h @@ -4,6 +4,8 @@ #ifndef _PDS_COMMON_H_ #define _PDS_COMMON_H_ +#include + #define PDS_CORE_DRV_NAME "pds_core" /* the device's internal addressing uses up to 52 bits */ diff --git a/include/rdma/rdmavt_qp.h b/include/rdma/rdmavt_qp.h index 2e58d5e6ac0e..d67892944193 100644 --- a/include/rdma/rdmavt_qp.h +++ b/include/rdma/rdmavt_qp.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Atomic bit definitions for r_aflags. */ diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 5662e29fe253..d711246929aa 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include diff --git a/sound/pci/hda/cs35l41_hda.c b/sound/pci/hda/cs35l41_hda.c index d3fa6e136744..990b5bd717a1 100644 --- a/sound/pci/hda/cs35l41_hda.c +++ b/sound/pci/hda/cs35l41_hda.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "hda_local.h" #include "hda_auto_parser.h" #include "hda_jack.h" From patchwork Wed Feb 21 19:40:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55CE4C48BEB for ; Wed, 21 Feb 2024 19:41:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7FB26B0080; Wed, 21 Feb 2024 14:41:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D05146B0087; Wed, 21 Feb 2024 14:41:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B58826B0088; Wed, 21 Feb 2024 14:41:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 98ABB6B0080 for ; Wed, 21 Feb 2024 14:41:03 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 67289160AF9 for ; Wed, 21 Feb 2024 19:41:03 +0000 (UTC) X-FDA: 81816829206.26.DDA5298 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf03.hostedemail.com (Postfix) with ESMTP id D21E320018 for ; Wed, 21 Feb 2024 19:41:01 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pSkQ4ShW; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3zFHWZQYKCP0xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zFHWZQYKCP0xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544461; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eOix7cIZ9OpjtRmnOCb5oo8qhMGRBISTNpiHk2ice5Y=; b=2ekAuDGHOs4jvgzQ8q6QgF/ej4VYdktkp2HGBXfhclKtqPzUZGaFcnuzb5qFuPGgNLDjhJ diNBhBp3EHqaMbYZxTlNKbLMeA1BEiPnb2G5qkYBKMtXU1Z46kz/M17847hZUaepI9d4W+ i/8Y80xb2yZdZCaXUfmkKvHvB03774I= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pSkQ4ShW; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3zFHWZQYKCP0xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zFHWZQYKCP0xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544461; a=rsa-sha256; cv=none; b=5VQhTinLgcXDxjaMMRp5EwAOWNWrwHlE4wXf0GSAJXGQx9KrH/o+n9yn1pN20rNWtRUMYe wK7dRE0FljsvOtVaB2g9rWraSz9ToWp9AkYvsNB8kFinFrciC5UETItfc0iu5WBi/L1Nzg yqd7LEYNRwA1TlGeocP6SqFu3loSMj4= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf618042daso11731478276.0 for ; Wed, 21 Feb 2024 11:41:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544461; x=1709149261; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eOix7cIZ9OpjtRmnOCb5oo8qhMGRBISTNpiHk2ice5Y=; b=pSkQ4ShW1k4wkdeEHS+Ma0JhqEiJxXv9KriXluFF7YcQWeB9PuqRcQOnhkEz58ZqBO 79eAwAGPjJWYyZROwZ+X/WQxW9yjSlZ123FJX2li/DFaPGNsZHNnF2XLDHtdLFsBP2FQ 3t4kFLImCGQDT9S5Ol3P9Hxl+Y1m+JbyioWTKLfiJlPt8zfZu1vHLzntvbv+/oSO/IsA AP0fiuC/eVA1bo2rvyjKroYpTN3OjA6Ga+wX72EopwyKfWs/V4FrfG6JxV3rPwszGVi9 MtRZrKSixZQiUoeB397PX7HcJIT2oKGlVPX6hFfN6e8jl8VoaKKr/b84SzhzX3Q7PwRf Yv1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544461; x=1709149261; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eOix7cIZ9OpjtRmnOCb5oo8qhMGRBISTNpiHk2ice5Y=; b=bnELazc05vElYcmfV6qnDxNMDdVHuvU/JWHCDEjUP8lSpL6XwYkfAD+/NTz2hTO5v3 4RDpx0i9S4le3qVbYnimltPz6OXQz14r8A8dWZcEofKyRzqjM2/wL55qMTBfdHWIrY3Y tHUlSOJhzWowj7ciYKSyB+Bbnrn5ZocDZvVQ1Q1oKrUvUS2dbu2bNFsNjqc6Cshm6RIJ OqCMCYlNFL6Mr3YQ35TJAjsFtEIxwO+223nNOExiKHYb/iUcQoVfrsX5RLnnkrDg8qij GTroVcMO4pzD6VIfs+C28LMoRuYFvWy+5soVlP1rRv9c2J7FRhI511hCBwJy1ImD/voz Uqcw== X-Forwarded-Encrypted: i=1; AJvYcCUsoREZXST1vq1WExLwwLWA0C80nwBndUHiMMlhlh8p42rVyTOIAcbFJkJ8NrEQBhjENj4JLhm3I3Wk4oodgxVELSU= X-Gm-Message-State: AOJu0YwtL9zSCe7FZwHRtpmiwaiMmeQsGhl6Tl47Rn7Z4pnPO8U1PnKE z0N+/LEB8wPS0MNE0K6UIanorlKx9a3F3Lwvb8LXo9rh8kUls/XtPGK9m0ebZQRi1SraJ1O0NDm BpQ== X-Google-Smtp-Source: AGHT+IFQED8r11XAmHCvTPIM6osrIYpquXv216eXmkF7EFZy4gkeT8fImMy3GTq6A3JUXD3VP8Bk7+J6aLM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:120b:b0:dbe:d0a9:2be8 with SMTP id s11-20020a056902120b00b00dbed0a92be8mr7992ybu.0.1708544460699; Wed, 21 Feb 2024 11:41:00 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:15 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-3-surenb@google.com> Subject: [PATCH v4 02/36] asm-generic/io.h: Kill vmalloc.h dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 6qzb5iiciq9qciu8mf1jkdn9gyyem6cx X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D21E320018 X-HE-Tag: 1708544461-757140 X-HE-Meta: U2FsdGVkX1+NLbWg/PbEuE1HK3e2IRCSqjYy0YmDtPxy96Ycb8aRmJu/hMoaDdt1IbJP6CoUKuHlzWdb8MVwrOqw5EgHH7OXhzQS7pCddYF3bjyc0wJ+6JvT+UDT16UurEudKoafFN8TmJjeHKAv3NqCxkPIKfONS3v+bZWzI95claASdsLBHy0bTvbbNJDWbTsiVnAd8T2ODCBWJHq5MS/VOjjjPUWBMEPc7GvXCzuHBTzGmt+q18xgoRVs6UMaxP4loKSaZLk0xAybmHVap1Nw4MNqtCpjNxQmtFn+gfLv6b81MUCNxD3T6aedN9ICIc9OEnbqqZVlTEiA1MfTMT5nCfLhy2y2GQg9juH5By8SFAdTlnQE224bKEzaHOXd2b+JJiDIWmvdcv+2S01KwkpqVkhAe1hlv7fL0BfgLV7al0cllB+CKAeYutN/hxhPCRM64eHGOSx6img/kFNdFOLRt0JvFtg2LqNE7z05Kw2eA0hclAp4JzzZEGJM6E3pz/vRRVnqnJ9+HqIGCZtWeNdHYhx0Iuxl+rE7KFDy1dvU2ohMXWkWZUIEUfEPguWOJmEJddYAHBCTKn1nEvfdYZsVipjhrgaMnA2vEyT4Ot5e1bcxZusqbaF6E+AjnEhSlW14xR1QG3erdvEQq/dnptsv9HpI+vk7xvELuhFGr/adjzgcLAVrwPwf8a3rxEBHG9XDTQW+AAOHCn9Z9uNf0+qIb65gc55TxbpUx5BPz8RZ36YazdSmC36nDDxZJ9lloZkRV1yyUMHiF9DI7adEN6FwA7pM48bzrDap6ODAuNMLWD3+2JUmnAul5uy+K2TXQh04MHPkbqq9wGZYpmUNpUvSgWZwDcMiGU8EsBj1icABhbc8LZzLtTOfXJ61kGrjM9tUo4jcRBarbbR8e5BDstQqRazxazpSZMJ6959yv/tEinOsJdQEt7tJKWwjQFYlA1sFPCJQuWFfyszyNm7 N5C0r970 lMUIDKIPryhvnzL3k5hDzyt3xrPsHBDSUHjozmXmIKrg7LyGnUKst3pI3kBhU/UMXbM/OwqWimaQsAm6ZFkhLuvZogrdwqeL/fzrp4J+AyPICTkb8acuCFnIOd96NyBvNLhMQGle20SzBDVoRhQKwFvL/9QBX4XsmQGlNKjA08fXboWlWIkfZ9CdTLBTf20vkxk8idOi6WlajX9D5LcqEnrbvuhyGeAc56Qx4qJUupuzVt5l1MD0Gz7MqfKRaUiPbvydf0/yiWMjNdCj7O55SFygXavLyEFv+DctVzVGwXAAorZFr0Rucv5YcefLf76abxP/xHawgZbdTVVCxLc4e3SOq4A6B9iP3H2vkcRm6o16ZMDpMsm82x6sgQUooZItQ8lkuhSe793ngxQK53YYiEGCkMXzNm9ratEoH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Needed to avoid a new circular dependency with the memory allocation profiling series. Naturally, a whole bunch of files needed to include vmalloc.h that were previously getting it implicitly. Signed-off-by: Kent Overstreet Reviewed-by: Pasha Tatashin --- include/asm-generic/io.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h index bac63e874c7b..c27313414a82 100644 --- a/include/asm-generic/io.h +++ b/include/asm-generic/io.h @@ -991,7 +991,6 @@ static inline void iowrite64_rep(volatile void __iomem *addr, #ifdef __KERNEL__ -#include #define __io_virt(x) ((void __force *)(x)) /* From patchwork Wed Feb 21 19:40:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C60F8C48BEB for ; Wed, 21 Feb 2024 19:41:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DAD26B0089; Wed, 21 Feb 2024 14:41:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 564516B008A; Wed, 21 Feb 2024 14:41:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C1626B008C; Wed, 21 Feb 2024 14:41:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 250646B0089 for ; Wed, 21 Feb 2024 14:41:07 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0675A160AF9 for ; Wed, 21 Feb 2024 19:41:07 +0000 (UTC) X-FDA: 81816829374.20.A98BAE5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 41935140023 for ; Wed, 21 Feb 2024 19:41:04 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=AqFlrnpH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3z1HWZQYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3z1HWZQYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544464; a=rsa-sha256; cv=none; b=ADgzatfzV4kgATi2TWwTA7/Pn+4cAIsmumhzHaN4ScjwnZBgD1BGLSpeC/bF036k3e6R7B wCq6kEHAAjYFGdSCl+hQxJqBqj3RLjab7NUgHOa9zQyolH64smHYaOus6IUjukv16F1fGy wtkqdAZyEmQacE59Z60qBZQDz+dseLg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=AqFlrnpH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3z1HWZQYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3z1HWZQYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544464; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aDZXV5wClyo81E7oQNbL7t5gFNRFpKZi8JXsyoMDgnM=; b=5RBddI26BIsIOX6CGu6IWeQY48b73rWtfcj0wBrIfvsPAFPVFN2foP6ikxeqt68cnIon3e vHckBi4DzYSJg6PQBAF2xNpvvM9OguURzQdk23jWRHLmSFoKWwfTwFLorKrCbIdNQFhC16 cWdmlG1moM/6HKQP6G99XLjUQ7MPoz0= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcf22e5b70bso551453276.1 for ; Wed, 21 Feb 2024 11:41:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544463; x=1709149263; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aDZXV5wClyo81E7oQNbL7t5gFNRFpKZi8JXsyoMDgnM=; b=AqFlrnpH8CHghMdeBb6dzOZLCM34jv4PHX3aoWXIP7eKR0sd5pOznVKDC9ZpBNoOXC zpxau3P3SjytotMB1Yesgchupt/MITNuvdzHaDGGntEcAQI0a52meNmdI4gtCiR7UGVl ACxBxxHbrytLKLF0WraIQsyJ2XHNVCYjlLmaq4nIZLae0O27Gk36kyGcB2miUZjvN6wx waDqGNPk2I/1umbBykWWwhp/A7IffnfiN7vaoucjvew0qBHDg3cODGqTH4sskqI2KIxy MPy7KmMfEXnB8Y3ihCgRMxFiBn82N4nazDZ8J4acspm0MOs82ivLYfb5anCaHWOcXNXb tiLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544463; x=1709149263; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aDZXV5wClyo81E7oQNbL7t5gFNRFpKZi8JXsyoMDgnM=; b=vccjEVNuriwa50UXGhEcPNOS7XXET93C2LD3fnbpeouxBZeetvbh1WASZTdHYth7KO kZKiSQNdVXPCXnO6fMxi1UR2krWRYhCLt8vKMCxQh734jYsxqjSS7BLx6IinGOFh5sdP pGb05DwXpWZDHIi+EiZwdxsoMkL6Ai4kisstu1vUMOdz6Er3cFSl9bxdaIhNiEWcP1VJ VWnjpXPqcT7ceEOscQzwIk5/NRYjInrbcI8wwat66qVr/i40ms7w2ZNx4LjufhSUwluI 5mXbzS5ZHUANk0w1TEoa3lH7hXMas2mUQPMk/HDKoO0dWObi8qjpw/cRl4RogJ6bKJMx psmw== X-Forwarded-Encrypted: i=1; AJvYcCVQ7RqpEzCesy0hA7eXWeFEJTcGSP3FwrrNzHlu5BFerGY8uNrkPMIC/Scmmigs8+QMjdx1sr7urAKcYNT6mrmywDs= X-Gm-Message-State: AOJu0YwFahHusmp/9QC4Cv1+mzJJaXgXHzp3/8NpQ9EluhRb23GV22UN ai56meTDnuPld8cxw0j9EItVnDNVAGY5kP1QgfqtI9t2kQimYlCg3/SgX37QiE6uIWtCT2J+BaM sDg== X-Google-Smtp-Source: AGHT+IHYTX89gbhdUEyuW7HvamV6v1+4UrLx/voxHAa1LwjOId1uGP/0Vg+lc29zmz5uf9mDtRdIligkWrQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:1008:b0:dc6:d9eb:6422 with SMTP id w8-20020a056902100800b00dc6d9eb6422mr17397ybt.10.1708544463028; Wed, 21 Feb 2024 11:41:03 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:16 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-4-surenb@google.com> Subject: [PATCH v4 03/36] mm/slub: Mark slab_free_freelist_hook() __always_inline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 41935140023 X-Stat-Signature: b1iakt65eeuc19494w19zbn8k3wukfsh X-HE-Tag: 1708544464-300124 X-HE-Meta: U2FsdGVkX1+2DdAv77pyeT5smFXoaZ9nLCAfEyAzu23Yrhk5+SEVjI8+NWTZYmAUeu8K95Yw2QsW5JPAZNs0JEucv5b3yXvi9UgbtKetU1SdRqaCl1h44Lfnvr6ic8VmSxokE9r60KcoDXQaSXyyTHNSZv0w0RtkWWL34BQoK1Q5X7KB09sLpWVjbuoyHz5d1Q+PoYAd/5jYOx7e8RoTnJXe1A95Grp/tn3n9ngyhXZiFquP7TXfzRRusbjiZyKY+TGYrd1svAL30xSrPdZ0pKoAeDmG5Nq9OJi+u7MnxybAyAhVUvXFaeOm7qg8ADTs0AXK+MiY2gpuoiWnL3BjCBVvHhUQHta/HWzdX7qXsG2fVM6YYNJtviLhX/teg825oLgYLgj1b6LcGYQdnH3MliewnNKXZxED5oToUlN+OA/ZdTQB+dtCx2o/pKHQeuikfJvXYoBoIM2w1dOu4fvIH/EujwN/F97xADjfnHmDaOaDmUOuGix8jQIgB18+flPCaIYJb9Vvjh7J0NlyXd7+FOFE9z2YtMVh2rCKJtLBQbvqe/RuwR0f9ZqvYhtihSCV/cYiy/g+zYw/7lAoBnH3thIOH8EAiHgfkQ8oz3x+/qtjXIMI0dBtGGOVCA7wpPJ+YYMdS+Ie2HlksrPQHT5QuQXpRXf7qqE3YmEAksIZlIFlUQFlqL1rZvwDC9WuED3RIHxZE5uH5lyuGPLfs9qsSD971safnB0FqEdV1XpCMyr2XqQ9Bi5oNRSm+aB5gClTHTsq7PB7OZIxTJ8mU6xtFo2QMJ7uOi9tTw2sEZShGzx33PPejN/AOyCxHfN4oftT7+kg632yTSjh8DJycroKPCOiKBJ8NG4O2cwlAEixscnnn9gWGAjyvhLpwbQUuQPsyGbHJaZE0LYHROw71NipvG56MDC+2T7fpwFRjv+F6+I9gU37G8CRK2Zaojl5J9T/10HHxcvJC+CwiZmxka0 9Vbm0nck wvTe5tYiNHVX7P+6WjuT5QxFcwmmQqp4wGw4QCoQ7F31+tYcb/beSWCx3BECJZeBtDZaIB/hRNTg2z2oBAc/iFtm4ulhrrQOfe0fYdPs4DXNYkxc+dg0nI3k1a/7C6rQdud5xBTC9lxMEtubf5dnrfyhRLiCLWYoH9M2UWeWQm1UqlFbuyB4E7ncNlf9d2ZGzoVbmL4UABjJiwfsJFh7HDyPqg/5mIPu6Dj+IKGA09F1JBv3fySxp4AZbzvWoCtYItR4RKHoHzOxEzAthEfl7oebnMgH7eyYtSyzNEMUh8sWifg2QWRFYbGx+xKwplXyFL05/AV7pNyO5uWKF6yYJiMmWmke8cRwPghFYQQr0HWhcKyzsPrI9A6n+kHPxkoyu3rGucuVa62IY2NlEGPdlYlOZ0nF6Bd7AJuxKpzVTWVvKlF7m/ctWYhLiSIt4i0AcKCd784J+DN8vsNwxSVu9i7P2Ew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Vlastimil Babka Reviewed-by: Pasha Tatashin --- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..d31b03a8d9d5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2121,7 +2121,7 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init) return !kasan_slab_free(s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, +static __always_inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail, int *cnt) { From patchwork Wed Feb 21 19:40:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BC77C5478B for ; Wed, 21 Feb 2024 19:41:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 604FE6B008A; Wed, 21 Feb 2024 14:41:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F2816B008C; Wed, 21 Feb 2024 14:41:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CC7D6B0092; Wed, 21 Feb 2024 14:41:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 12D5A6B008A for ; Wed, 21 Feb 2024 14:41:08 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DEC6B1A072F for ; Wed, 21 Feb 2024 19:41:07 +0000 (UTC) X-FDA: 81816829374.13.BAE602D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf26.hostedemail.com (Postfix) with ESMTP id 48E89140013 for ; Wed, 21 Feb 2024 19:41:06 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e6CL7XXR; spf=pass (imf26.hostedemail.com: domain of 30VHWZQYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=30VHWZQYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nNOJH11gO16gI1CGV9wD1JhhI3G0pl89I4/43bWDXAE=; b=AmeIz05WXz0ipAlRtIMtbEz/po2phifyxpeP44bJKrwCqixkp3XWSp5ySRVYCBfbib3VIs k7cBD8TlN74NZjHM2jQvyX9WHYJoHe5iSkv8EnydaGZtKlCQo/LISpmdmkrwZtmnx6jI85 iDFGqvDv7tdFzrvBDN4Fo+ZRrMl0r2U= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e6CL7XXR; spf=pass (imf26.hostedemail.com: domain of 30VHWZQYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=30VHWZQYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544466; a=rsa-sha256; cv=none; b=jbgThjfY6XtqlCkbwcoto8P5PBJvxm+rtUflBSqGO59CpY+pwxlFfXBzubHzk8MaTkcopr LrPRdtcogYwiZuowet9hP7xoBxta+07bKl792p54WnvlLy0OxrWBhZtdDZsQLYU1h4n+L4 mwceDh3IOADnzlDzUieRsZvAnGO0AkM= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608405e0340so17672687b3.0 for ; Wed, 21 Feb 2024 11:41:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544465; x=1709149265; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nNOJH11gO16gI1CGV9wD1JhhI3G0pl89I4/43bWDXAE=; b=e6CL7XXRWaDMIcsZbfWxowCYb0fnHNlbvRZ3v2euqlVcNXwH7MY0s0UlLshQJQKEQp bLjLLxeEmHW3zw0PzJq0B6lwXPr90Gq5kSbmnFrMo3K42lMzDWzWbi+UKRNU3Lnu7zY7 +iorZahfmpqMS8ft4eEHoNKNjPHTi3Vw5xYDVS2bSesMKoBQcVpGBesgIt1NbbWuzbbH S7IiMhw+PM+jgwCpRpa2P0rp5h/++SmeU6vlYL/ef7AH5wwaDFSoy+i/2Qg3XUIlIdl1 Gpe7Z+AfcKVaqH6+XMco3NpEZxLXQmbiP+8jmBLPu+hEB8X1cDk0839iqY6pQ48Muo7G UhVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544465; x=1709149265; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nNOJH11gO16gI1CGV9wD1JhhI3G0pl89I4/43bWDXAE=; b=ITumXPlAU98jQlo6rsSqTrd0JCGdO1xVuYCz08YRT1wZHHGtsa6J8RO36LjCJfeuye UN02ylIrS65rnVQSoXIL3/HJjddG82M0CrazxEIKrkgrAv2ohGhNC0fiylCw9a2Ys3+4 QvDtg7jVPEwZyDz+uukyKPLMCPlAOXgmCx9qKPAcePkKgv7xHIcCnw8Lk5vAlW5UtJd+ GL80BAaSNEqHD7FBvdcJiWxoClN9E60M14cFhSxSu9wosi5aUrAC8G0OKYUgLgCO44gL pcUqIpjrChkw42TMlLaJHw4lky9zAGOgK+3yNEhhc7PhqggV+I+QNXvho/5L4VqH6pFy RNYQ== X-Forwarded-Encrypted: i=1; AJvYcCUERNWKDzBEmu7hdDSl632iGNkF/150KmmeeaGDbcEZ0ua4P79DAVvhAwrkjJY1Tm58PX/STOIrcQJ9iUPx5de/7sQ= X-Gm-Message-State: AOJu0YxEIAt0CqLXRX3/jSWtmC1R9DFk4tGbfE2m9mllpTm2qgVnfIXM nMDXK6Qu0JIw1JOqY7VqBHEvvYRChCvQOcK1QdIa+g4vBEhCk1nCob9l5i05CQpPSTWtdckoLG0 EDg== X-Google-Smtp-Source: AGHT+IFw1HW6rh2zXq6oVisO+flBpM8PZrPAxomjgm47RtsfgtCbzompT9Q8/PbN9Jjx8wnIh7u82HkUv1c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a81:528e:0:b0:608:94cb:6f6 with SMTP id g136-20020a81528e000000b0060894cb06f6mr174090ywb.7.1708544465320; Wed, 21 Feb 2024 11:41:05 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:17 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-5-surenb@google.com> Subject: [PATCH v4 04/36] scripts/kallysms: Always include __start and __stop symbols From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 48E89140013 X-Rspam-User: X-Stat-Signature: p3sq4fgsjfgufkt14d8mxhsq774omskt X-Rspamd-Server: rspam01 X-HE-Tag: 1708544466-262808 X-HE-Meta: U2FsdGVkX1/nFJF0XiGXZ/5b98sOYIdsZ5lp+y8d650UjDi9ylYQS2MC8dolHB38u+/q3gzKEkJu/zZ2PAAvPbkmIpbmXAMOsBFXXg2vnvNEovgIpx0nqRJEtnv+3nguUHygO9m5RF5l6LjsVijHtrbYVTZJOM6PM/lsaVPMpMCf3/eSvXYK93fRwCoRP7brbKFW4OKmq5K13x6eVWGMWPPRV+3WmQLE7VmJPOvCUUwZmjjf/6/lhvm0DD698tGNQhedUJQUu/u5NmdWdnnlVKiiE90R/nJ/moZZLH74PRSszlr9LRz/5rC2ItK9KQ2jnFWPTF/9I5+iP44nhk9oGYyKMxb1l74VOR/J1CcFZxPxF4HhjBSIL2xVKRTb+1rQKA9MhzSjZBii1wTra0ytekyt4g+6awitU8IlWHOOl5HG8pSOlO6CS26kFdIJX3NFrHpeBcmHZ5+MRjMUbGVQSy1s6qbtlgERZfVgAJmJ8MLa9AYlapD1Qd4E0+YZ2x9/ErE1B5N7Uv16jc/HRdMYpx/AMae9LoyY/1NyH3osst9coB1sG4RVZi/fy3/K5/MuB8+b39g9bg8eDMXxmBKl/5yaZN8TQI+2diiYTtLp8F/pzFekDyhCqOMqynNGt6pBEB1RdQlV2amzwOJSdWz5wDSrVweoIKhnyktpas5wijCppGrJ4Ktsm3i5Mxe57wGOYG+L9vkgbmGElcaDyrGVMiS/IuKptYAmriXyI7MeJLji9D0ube0J1qX4XZ/8IRg6rfNaLWGDlvBK+xwP469GS+x/HUZ3mjEul4indkWp2AFni2yzAzS2rBJECs6WkKct5yCNod/eQAd1MR6JvplVYL/oWIxLq3gPvDVXy1jtJbsHYx2m10xgJz5ETJa/2dQlVd9FZ8RvKmI5mIIT0rQVePt4t0el3Zw7QgzAKEtiECuYqPBzTWZY2isDMgpybldsX1+KPO42a9sGbS1ucMp tO3gVBEH /RCo8+JOCyRhfGOfMZFP2Zj9TFmWWFCtw5Z48RQBSDhgtJzWKpX26sDioKFlSPf9emOWpQORbc8QoiEaP4w2py1JKh+O8QKGgNP8YTe9uKeQ5sThe9E1XgUqmGIfkAndetmMGEv7lgpOlLeR2UZ0TNnL1cZKFpeVjmjs3SqOiH01iq7Fr9mlgHZc8l5CRrFnl/R3QeyqqhQYl/yTThWeHSOhzIcI0Hs4+bBqqEQ1HaRVmsKYbFGTRK2Gcb6MQ+weq6zg7QzGMHfAJ2e4E3z4Kp8gAw3TFwNoMjMuj/BvNeeZkuN4I5uLgy/xJ/Eh6mCXg66/df0xqpmoM+DG2MFu5L0FPKrc9RNT8W3cmrRz12MeTe78v+k4ZIOqf+wKxlGt8fHo3rbxpQU1TnFQyWu75pBAmj8oLJuhlHnhvuNbPfUO3712yG4qk19KLvVYTTGdfMpJhrnODRb9BldoqZ8/itRmNvg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin --- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 653b92f6d4c8..47978efe4797 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -204,6 +204,11 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } +static bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + static int symbol_valid(const struct sym_entry *s) { const char *name = sym_name(s); @@ -211,6 +216,14 @@ static int symbol_valid(const struct sym_entry *s) /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { + /* + * Symbols starting with __start and __stop are used to denote + * section boundaries, and should always be included: + */ + if (string_starts_with(name, "__start_") || + string_starts_with(name, "__stop_")) + return 1; + if (symbol_in_range(s, text_ranges, ARRAY_SIZE(text_ranges)) == 0) return 0; From patchwork Wed Feb 21 19:40:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF418C48BEB for ; Wed, 21 Feb 2024 19:41:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4BA66B0092; Wed, 21 Feb 2024 14:41:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D2BC6B0093; Wed, 21 Feb 2024 14:41:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D7666B0095; Wed, 21 Feb 2024 14:41:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6054C6B0092 for ; Wed, 21 Feb 2024 14:41:10 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1D69D160AC1 for ; Wed, 21 Feb 2024 19:41:10 +0000 (UTC) X-FDA: 81816829500.23.7522559 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 6F9C61A000F for ; Wed, 21 Feb 2024 19:41:08 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TJuSI6HQ; spf=pass (imf19.hostedemail.com: domain of 301HWZQYKCAYy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=301HWZQYKCAYy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544468; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=14sostfgXdBtkigrgclSIrhsv4NljUYVKVXEohyvFzI=; b=yNxxS6ZHtP/SScj8ONxLVhG6C8J3hbboDG8HmooeyDEg2RE6+eqLHjavQZnVwbjIc7o2MS fpo7zFO2JfYIAXgSi+1IDXzV7/dAtXolcqO1ckc8OlwctMOxpqAkzqWOCWjYzerDp8I2p/ OrpjAdKc6B+ZX87dBSPyfX7D14xBsYk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TJuSI6HQ; spf=pass (imf19.hostedemail.com: domain of 301HWZQYKCAYy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=301HWZQYKCAYy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544468; a=rsa-sha256; cv=none; b=5qVJ6sOVPX8GvHg7zxbJeOZc6fSDlSXvofF8NNVnGc99V7chwE4lMHnefuBWBlJWPwZs4t NDWoH7k9jv7UXxiiuHzTi3WKdWJF8uKK1g6mgqOfNSJ4S1erUew8kWZGckA5YcaBY7JTu6 J4Wf5RfFssPQpHuSJBeoylaZynb/nYI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6047fed0132so108263307b3.1 for ; Wed, 21 Feb 2024 11:41:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544467; x=1709149267; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=14sostfgXdBtkigrgclSIrhsv4NljUYVKVXEohyvFzI=; b=TJuSI6HQKTGRCDQhKLRnA9hN8hrDUlPstYih4VwTTLO/wP49a8hyrVJ5vlVlFRShRa igRNhNuifBiYDUgJWoVMDdmup+a4x6Kp1kcSc5HCtgrCgohT7ZTTkFh61c4TmKIyR5Y8 ZGZtgeAAZnh5I77vvsQAkBnOYtVGmQGPUnwlcZFwyLprFDQYO1rNwCMmXb8h4MCRyYEO GXtd8D/xOoM0Fc9iFlWUOYstjpDkWajqpHycEPuDGEIKbia6FtEiXC2S0mMm6PCyiJLG /J9GmFubyuagPOpLXwQGsoOtqMyAPOeDCHhyohlcPBTHFMZPPPQFt7HBEJ36Q38Qc/TE Ad4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544467; x=1709149267; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=14sostfgXdBtkigrgclSIrhsv4NljUYVKVXEohyvFzI=; b=BbZwGaayNDpQy6jM4KqaN8OOUjrOMTxeQZaaaGPQGGUnaOC3XLU9nKFFPd2Xyuyc/C vnFeTYwKJRfxbv6H9nhYZtqJ+gzCKzqO6HUu1ITdkZyz8sAoV3TkNS5G56QkvShB0QXH +8U3NBF5HZuSUoBoCOHVk4uUIwivwzuqKnmrcMf30yp/4gyejZl2gvgnw+LFoX0WN6oz d3ITr84NvOZn1AS6G8C3qt6GbrPhMRit8TTMzWK19tpFVxv4MLh6RV6fzPMR2xByyvRx nVeIBNYcHCJ0eeeLuhHoXgvzF4Qivl0JaW8ZioNxid7b0PBQubFFuJ7UUxF3JVRyv4Um cPhQ== X-Forwarded-Encrypted: i=1; AJvYcCV8fzFriuJjvHTAUCK7Yh6OnGi+gsn8wqnOgMpPDWM2K7TjmyoeThD5Cy47ksZX6WwhOVuM3iWVhqG0GL9DmQa90Os= X-Gm-Message-State: AOJu0Yx945KBOUGBJ6OD6p2o3qLZWO9+xYuxs+ba5mx6ozeHEJbv7dYi cLLR+8oGy/0LuYqZSYOAK0lbrp8ghA1sPn1KW1LuWygWxxkITdYrykrO95N6JD6/DL9NvFd2d+t ZTQ== X-Google-Smtp-Source: AGHT+IEg8J3/Tgw3rQ0HmvocmCN1KEaF26PclNShE/JXdkinKLyGnMcr7oBUIgKaZkEgV9b6wmo43w0h+Rc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:4c2:b0:dcc:4785:b51e with SMTP id v2-20020a05690204c200b00dcc4785b51emr10314ybs.12.1708544467498; Wed, 21 Feb 2024 11:41:07 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:18 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-6-surenb@google.com> Subject: [PATCH v4 05/36] fs: Convert alloc_inode_sb() to a macro From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Alexander Viro X-Rspamd-Queue-Id: 6F9C61A000F X-Rspam-User: X-Stat-Signature: m6t5sgxsha7wutzcgp9o7nuf7tuqymh3 X-Rspamd-Server: rspam01 X-HE-Tag: 1708544468-921273 X-HE-Meta: U2FsdGVkX1/OV9ideRMTOqYbDQmHgvf0rmFa8d0SloW7ETucp6L1YLjo01Pk8WP3+lwfP/STFgXd7HOH3jI3UqwrE5wAQkZtH5Go7xM7YSR454IfDGcX25bEK9iZ97r5KrK1CoyIvUaaHSBGB1NNKbVDTk1H1q3L5n7TqfuqCHzdO+QRKdjYDcezVPi6eXL0kYgbA5F1inZT/lrD6/zIBTvFA9jbxkvsIoQT6EMEaUw7IEUJsGmPtKCitvGXoIX7YHk2Ffy3yVXleRzqe78LDn/xtvvrETEYWra1ooaH3Xh/XIPruVwkJBxlqqRPXiSdnj+Wk/abafC+4pGYxT+cpUdAHGOrYseoWWhVwz/nc0u4zWbIaU/vAdvcIgZLcJMFfxD0mks2FzOTEDrwa/Uq0zJMkvT/vh9GjgDzh5F0uUDG9fDGVDvfzxtvUBmDa03UrMBHTrvKjKS+/Xs/Epf3dwDpXYqSe1vvGKOejy4EQiJgi6wpBK4VFhE0z/KI9QSEy4EqBzbSD4Bs1hkxwlwJ3vaX0JTTuUjA/9nTzXnLsguu8uLFTXZpima59wyBNqjcpBo0UZZWXQzOuun+M44sL5+hZA0vHLV3IyFU8IPCJgxqwshCnYlmk6CrEiRnSyMkXk2K245TQP0pRkHB7HDZ0g+/tKD6+J5v7H2hj3CrLF4fbQK2kGBwEpo+lXV1kYugxn7vlMgy//RmAdkI/XUsBVb4vP9lnkujeAU+YwktbsayvtoCXViZIoEEQGYfke5Wc7qcQe28aiKlqEI8C4g9nw1TRO2yIXHdgbZU97vqiE/N8zkQ3LGrYc1hJOuE/MqqfpOavm9y3VHKltrUZJeXsIdN+B9RuvfMZFdbHWkkzgU5iAry6XCyqWvlhCHQq/JggD7kf1cQK3Wiq5KUz7I+bAMqtbK1UXApfFc3gQ8Kgre6utLuicLWLrDJBalHoqkBqOk60Tiu9cgAnIt25DK C98Paeci H6P1BXYNrYtmV9NPW3imtEtnoJHn/e5bPlk+CgnOBWVCguqOGZPrBmALoC2zw+I8D0arWhxaxwUNXUtoGQU0NDKUVz/o0GsBWOqviyrar3jH+EW4nzvpQxNbfovjRSPqlnC5vmRQGTNQIFCjjCc/jsGPkvoHgTSKgvUs7Mu4U7p8Eczh4Nwjh8+FsufMVsjU/Lc09f4eNQQN3ItIlmSWFbLUEuGRN/rFVZ5D6vpiEiHBZ32MSGG5S/k+CNlkshEeGFpyD4WPiRnbVLlQZIORzUqqGkAfJQUys0170OmCpL38OPl1xD94NQIu1PmgMYyKt0ee4CkMtq1hGBsbrVW5uDP/6fm3xFmqEIyUjFf6DRGEFfOcQslBjALKgYu5TJOw1gz0hT4zkcszdQtRJPAuDJ7GIMZCeQyIn8aBXsA8tdauwJSeBEt2dbadduhCMmRbpehyaFGicf6AlOAKlNEdHKdID4g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Alexander Viro Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin --- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 023f37c60709..08d8246399c3 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3010,11 +3010,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap, * This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) -{ - return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb(_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp) extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) From patchwork Wed Feb 21 19:40:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA3F3C48BEB for ; Wed, 21 Feb 2024 19:41:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C46866B0095; Wed, 21 Feb 2024 14:41:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BCE436B0096; Wed, 21 Feb 2024 14:41:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9ABC16B0098; Wed, 21 Feb 2024 14:41:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7FDB86B0095 for ; Wed, 21 Feb 2024 14:41:12 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5C2CAA0F8A for ; Wed, 21 Feb 2024 19:41:12 +0000 (UTC) X-FDA: 81816829584.08.96BDC54 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf20.hostedemail.com (Postfix) with ESMTP id A14BD1C000A for ; Wed, 21 Feb 2024 19:41:10 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Up2wyhnL; spf=pass (imf20.hostedemail.com: domain of 31VHWZQYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31VHWZQYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QzsUMX2xNFk2gXovx8vmt7vEJ3oqQRqP0H0Mn26pLn0=; b=tlOSLOIR6++/QuehYGjYvlG90ftvnl/H/dTBRmfuPVvC/WgTK9HIrg4b7NeZL8zXHrPM2t kKT0q3P4ZlTFXApgVmyZw47E58oauN0jGrx2xKPfF5OVAxLMnX8ZJ2/rpZ2MCtQJ+IxVNb 5pvyGwTXntOcgz50XV1A4nmQZ83MMNc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544470; a=rsa-sha256; cv=none; b=vLCzM8QqYV/s/inRo4gXoywzu7ga36LFZ2uXe/6XZcPDd6Osq14Q8r10eIqvX5GntgP9dA lkhD98lgfBJUWxwEmK1czRLQY6UncOb1qg+CgTXbzWNNE5wgGEyo1jjOdMDYHQxvJ92b+o S0mzjcLBzKDmPkMQMXrRWL4ODHtvXAI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Up2wyhnL; spf=pass (imf20.hostedemail.com: domain of 31VHWZQYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31VHWZQYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608575317f8so43595487b3.3 for ; Wed, 21 Feb 2024 11:41:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544470; x=1709149270; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=QzsUMX2xNFk2gXovx8vmt7vEJ3oqQRqP0H0Mn26pLn0=; b=Up2wyhnLDFMs4ShVNnl9vBgO/8MKOCLGytkygJt4k4pE38ZjhwlQADxT/dV2Bx3LHd 8yCGU9tg8q/ln2mW3MPeIKwxDR7U9w2qP5CivP9B5gaugppvn5QahI65DGeqvg9yCEz4 y4LULLnjUQaruLEY5WF/AWw9/U+4KujBAMgCtz+2SQywxTQU6K9cZCQHnNsHJ4b0PV9r Fenx++1HevbC3Sonvu5BRJMTEq0Fw+kqFWRAuf2CHkwYbhFZORhqe5whmZ8hu01YFvca S41AFJpCpROaKmJo3/6tYE94gKgw/BSASleNYYZ9ug+lo1FahmxzcxvEAhRJ8+h+pGJ8 bJnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544470; x=1709149270; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=QzsUMX2xNFk2gXovx8vmt7vEJ3oqQRqP0H0Mn26pLn0=; b=YPEykuyQDMU25gyyn6JR3jZIKTprFKG3nObr2LytwTfKVupEKznp7xuv47JT7mN44c U+ZloaFRPctHrs0fMAl9oYjHyCRSi+tmeUE/NbUlITIgnzRJUbYSqWhyyvoO7RQEAoHO /Sw0MEB2AgTrdjBrSBAj0qczskM1YX6OdBVLDgk43RnKuL5uDzG5rxIEayn9/qwsy/jA RY+DrnE+f9vtElh64p+c1RcYtpEU/i0rnRYeGKSW9KDO5z8v5Dr5ZOe9p4UBX1+ZY/yb m7UT4ka9jfkVLE82YjoKYokpoSuyB0mCKSQjVq6m+koA6DPmxIwtP6/D1n/J109qqBQe QF7g== X-Forwarded-Encrypted: i=1; AJvYcCVirkFzcAZjQAhuYRoCgxBnYsDikfCoko8wh5i7yF8YWZl1ZJOI61IoS5JxvWKEAL/ZfOgv7AAV/TwBZw5EjqKpHO4= X-Gm-Message-State: AOJu0YzA9xAgGzq1AoD4q5iw1FpIIqno9y9Ft/Qmb/1mAT7OBAgTYODe ofrOqbEApT2/0E9KqEAVheL0R+rQq5T/iXG8NnHmqEfL9JLzj3sWN97EACMCfMWApcfjSq7hzfq abw== X-Google-Smtp-Source: AGHT+IF7qn4ITZM6B47DxNzFWv3QLjl0M2+lztN5UUvu7bxl5t2P16qOZECK10ItUf59ZcZs0KUc+Z86KCU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:e6c5:0:b0:608:801a:e66e with SMTP id p188-20020a0de6c5000000b00608801ae66emr474072ywe.3.1708544469656; Wed, 21 Feb 2024 11:41:09 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:19 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-7-surenb@google.com> Subject: [PATCH v4 06/36] mm: enumerate all gfp flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, " =?utf-8?b?UGV0ciBU?= =?utf-8?b?ZXNhxZnDrWs=?= " X-Stat-Signature: y4j6u1ey638hwtw6yh8d9exg3d3ozeoq X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: A14BD1C000A X-Rspam-User: X-HE-Tag: 1708544470-782240 X-HE-Meta: U2FsdGVkX19uyvjJEao3dBQWEMHtHaeK2TBUrVO8hIJdDY64CLAI4aNmvz5g3sApcwLCDpLWIk2zXM8qSX+E6HZwt07OsuJUq2h3FA6wy8yqwN7tzvscDA5uho2lZHKp2txtGkjvv5vLRBGnu9tQp+DbMtHkfEBDcFlf63dRzvQGBWhB098xO88sb/s2ok1ehmQhm9TUsRA+QOqblGwFOMfadOTH8ILNg2ecyyn3zxizw9fAI883UbE6UCBDVqPdx6gyzMWSXuRbcDimMTqWqtlgJBIX7cHit2npxkFL1SgLyiGEwa/rIuK3ImcQAfH/AZmJkIk1NZ40Bg6uA240U8qrLL7qbrTCBKeB70UL+lRsrBLEbtsCKzCDBU7e31lEWo1xlRaywk+rPSpsg5qV3hDiV4ps4ke2+k9bbzaBWKljO3mC9qoTiU3V/jsiffEGY5GOVlShIDVJS0fdRlmRSoPQ8wqImFM4yH07RWCVjEVCk8c7zcC+xwpAdkS6KloxpWzGSkrtn6x0+vJZbnCEOGuhc14mUlmPdmeejgy89vSR/kCRSfx/k0qk/X/mu61+542QspqBfyWEjwzRjie1es28TLaf8dXjH/tjuhEfxTCthHtNxtr1ayUlEJJOEiYrilr7WW77CCfuZ2/XTgk1KsluSt2mckX9uslVHEcQwgx0R2skMHjnKKApz6bal+79GpNF9Ra06MhJJIIjE+YJeIqM0eQANyjJUb2x1d9jIM2E53fv7/Rn5yR7roak6sG+juv1UhYDQGC9bIInoshxp3olQoGvHumMGAbjJPMgO0PWjvpf0082d9Pvh1fKIEx7lq8RiWKGRch0lJ7naGB1r/XX34/ogCABHERmV3xEwxkzqDKHJgYQG7X53vD742VTmoJhddYt79HHZuem5H0OMNZ8Tzt4laRPynAgKnfbV0aCYYM6FaHIziwhC+nSZwtKtErh9eL7FEYeN8Gz1zZ 8vWdsUCm LGE4vKJcaBLHCv4k3DtvHiM6Tb4tiIjYlgWi33c7P0uq8gXx6uL1AQ7EFM3UfKzrnwCiwk0ekvUh8QPW3+U4j1k1pMtzXtLYT5pqE+uFYgzEuLDC2JO/GszOyYTcki2H5Ax8vjGgWZbmqXgkKmUIDQLyN26iSjp/apMAFS6iE89FDbzYOMRTA2egr9/tTRjsk0VNtalO5f9qB7nOtuD5pMpiFvFAk9Tmo47/KmHrIWzB/uis2UZr3S2h+oVDI6QBU2LUVtoWnBTdlzJkpFOpxEHzrktOisHOFElK4PycjwQLtyNbKtuEu//Hg8XMS2VpyfY9CC31w4CqA3KrIatfC2R75CaDDK2cIYw21zKMDZNeSdul+tM8l6u7eoebEgW5cZCeeXwWnQX62YGypSSRcJ3aRr3YmYx5qWPMU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce GFP bits enumeration to let compiler track the number of used bits (which depends on the config options) instead of hardcoding them. That simplifies __GFP_BITS_SHIFT calculation. Suggested-by: Petr Tesařík Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Acked-by: Michal Hocko --- include/linux/gfp_types.h | 90 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 28 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 1b6053da8754..868c8fb1bbc1 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -21,44 +21,78 @@ typedef unsigned int __bitwise gfp_t; * include/trace/events/mmflags.h and tools/perf/builtin-kmem.c */ +enum { + ___GFP_DMA_BIT, + ___GFP_HIGHMEM_BIT, + ___GFP_DMA32_BIT, + ___GFP_MOVABLE_BIT, + ___GFP_RECLAIMABLE_BIT, + ___GFP_HIGH_BIT, + ___GFP_IO_BIT, + ___GFP_FS_BIT, + ___GFP_ZERO_BIT, + ___GFP_UNUSED_BIT, /* 0x200u unused */ + ___GFP_DIRECT_RECLAIM_BIT, + ___GFP_KSWAPD_RECLAIM_BIT, + ___GFP_WRITE_BIT, + ___GFP_NOWARN_BIT, + ___GFP_RETRY_MAYFAIL_BIT, + ___GFP_NOFAIL_BIT, + ___GFP_NORETRY_BIT, + ___GFP_MEMALLOC_BIT, + ___GFP_COMP_BIT, + ___GFP_NOMEMALLOC_BIT, + ___GFP_HARDWALL_BIT, + ___GFP_THISNODE_BIT, + ___GFP_ACCOUNT_BIT, + ___GFP_ZEROTAGS_BIT, +#ifdef CONFIG_KASAN_HW_TAGS + ___GFP_SKIP_ZERO_BIT, + ___GFP_SKIP_KASAN_BIT, +#endif +#ifdef CONFIG_LOCKDEP + ___GFP_NOLOCKDEP_BIT, +#endif + ___GFP_LAST_BIT +}; + /* Plain integer GFP bitmasks. Do not use this directly. */ -#define ___GFP_DMA 0x01u -#define ___GFP_HIGHMEM 0x02u -#define ___GFP_DMA32 0x04u -#define ___GFP_MOVABLE 0x08u -#define ___GFP_RECLAIMABLE 0x10u -#define ___GFP_HIGH 0x20u -#define ___GFP_IO 0x40u -#define ___GFP_FS 0x80u -#define ___GFP_ZERO 0x100u +#define ___GFP_DMA BIT(___GFP_DMA_BIT) +#define ___GFP_HIGHMEM BIT(___GFP_HIGHMEM_BIT) +#define ___GFP_DMA32 BIT(___GFP_DMA32_BIT) +#define ___GFP_MOVABLE BIT(___GFP_MOVABLE_BIT) +#define ___GFP_RECLAIMABLE BIT(___GFP_RECLAIMABLE_BIT) +#define ___GFP_HIGH BIT(___GFP_HIGH_BIT) +#define ___GFP_IO BIT(___GFP_IO_BIT) +#define ___GFP_FS BIT(___GFP_FS_BIT) +#define ___GFP_ZERO BIT(___GFP_ZERO_BIT) /* 0x200u unused */ -#define ___GFP_DIRECT_RECLAIM 0x400u -#define ___GFP_KSWAPD_RECLAIM 0x800u -#define ___GFP_WRITE 0x1000u -#define ___GFP_NOWARN 0x2000u -#define ___GFP_RETRY_MAYFAIL 0x4000u -#define ___GFP_NOFAIL 0x8000u -#define ___GFP_NORETRY 0x10000u -#define ___GFP_MEMALLOC 0x20000u -#define ___GFP_COMP 0x40000u -#define ___GFP_NOMEMALLOC 0x80000u -#define ___GFP_HARDWALL 0x100000u -#define ___GFP_THISNODE 0x200000u -#define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT) +#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT) +#define ___GFP_WRITE BIT(___GFP_WRITE_BIT) +#define ___GFP_NOWARN BIT(___GFP_NOWARN_BIT) +#define ___GFP_RETRY_MAYFAIL BIT(___GFP_RETRY_MAYFAIL_BIT) +#define ___GFP_NOFAIL BIT(___GFP_NOFAIL_BIT) +#define ___GFP_NORETRY BIT(___GFP_NORETRY_BIT) +#define ___GFP_MEMALLOC BIT(___GFP_MEMALLOC_BIT) +#define ___GFP_COMP BIT(___GFP_COMP_BIT) +#define ___GFP_NOMEMALLOC BIT(___GFP_NOMEMALLOC_BIT) +#define ___GFP_HARDWALL BIT(___GFP_HARDWALL_BIT) +#define ___GFP_THISNODE BIT(___GFP_THISNODE_BIT) +#define ___GFP_ACCOUNT BIT(___GFP_ACCOUNT_BIT) +#define ___GFP_ZEROTAGS BIT(___GFP_ZEROTAGS_BIT) #ifdef CONFIG_KASAN_HW_TAGS -#define ___GFP_SKIP_ZERO 0x1000000u -#define ___GFP_SKIP_KASAN 0x2000000u +#define ___GFP_SKIP_ZERO BIT(___GFP_SKIP_ZERO_BIT) +#define ___GFP_SKIP_KASAN BIT(___GFP_SKIP_KASAN_BIT) #else #define ___GFP_SKIP_ZERO 0 #define ___GFP_SKIP_KASAN 0 #endif #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x4000000u +#define ___GFP_NOLOCKDEP BIT(___GFP_NOLOCKDEP_BIT) #else #define ___GFP_NOLOCKDEP 0 #endif -/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -249,7 +283,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (26 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** From patchwork Wed Feb 21 19:40:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1A1FC5478A for ; Wed, 21 Feb 2024 19:41:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 541866B0098; Wed, 21 Feb 2024 14:41:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CDA06B0099; Wed, 21 Feb 2024 14:41:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E4EB6B009A; Wed, 21 Feb 2024 14:41:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 01EF86B0098 for ; Wed, 21 Feb 2024 14:41:14 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C6D6B80BD3 for ; Wed, 21 Feb 2024 19:41:14 +0000 (UTC) X-FDA: 81816829668.16.8F0F4ED Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 2CE6C18001D for ; Wed, 21 Feb 2024 19:41:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p+ltIFrJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 32FHWZQYKCAs352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=32FHWZQYKCAs352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WxhfLGit1QGZvv88CDz2b9TMcGpnJbteJyGythFonXg=; b=gPJvYPBg96vA+me+HWDcpzK0xWzByitHGvI/o1y+PfSW+UzgBT2RoOpnaDO+0pMjOuiz2V pG9a7Iwn94s3wjzNtGwfpDNx8KrkEmI33lJTXT8A+BcpsYl57221HVIkejZ1ty5OyO+PCE QOyizSWdS0AsSlPLT7HzIkJl7hV5ZaY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p+ltIFrJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 32FHWZQYKCAs352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=32FHWZQYKCAs352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544473; a=rsa-sha256; cv=none; b=bHgqngxChghSzwZozUuG16rpzDGQCmy00eweKDRfmfQRcOQBN6WAFxJGbXy6F9fEglO73c tu07ahZ4Ad3hV7ig4ZLjNwDUZilPZKSYUl9joxEerwKw65opfgPKushWigux8deWOwgq+S opT7OgB4UGmcue3Wlu6mkBbmcwYrFBQ= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b5d1899eso213215276.0 for ; Wed, 21 Feb 2024 11:41:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544472; x=1709149272; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WxhfLGit1QGZvv88CDz2b9TMcGpnJbteJyGythFonXg=; b=p+ltIFrJzSupYx8jHsKQp8/pa6d2cXe9Xa9sSpbrKeAVuAOdThTqGK7wcjarih1WBP uliMjmf7Y3hQacnJGBE8TdaH7r1KaXbgg5Xk+kslO/dp4fyqA7K0bU2caq3a8qsfyIWg B9oBEev/tcErASV3nptbaZMgesN0l36e+3rI+TqK9BQYZDbtnw8vo5bZRDw4PitiaGAD lxX2p3AgcKTS109ihNjGdqIolS+6ha0n98rSyaXANxwOcmDr7EpXB5mFgFOE/Tr+HFVD SV8M+Nr5kQkedsdNm5YDwJqsZK9xLDHVFF1ph32GkxAzilxkTCl0dpORExkbT3+IuHtk bYlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544472; x=1709149272; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WxhfLGit1QGZvv88CDz2b9TMcGpnJbteJyGythFonXg=; b=Uq5mYbi8+cylZ5UhKpiUQ/jx/i4RqRPtqXC0wK7psJIH/7bwMLLesT31IBT3aA2P+o 88oq+T7duIxUEQ/gjzIzfSQL+UBx7P0Gn68S6vh2S8pjO6hUQAzMhengo0c4VelucEO8 ocRsY3B74hYYkj82BKLD8CGTB013tTJQM5ptYsdcrHT45NJd61dQwvcuV8HkkzDqE4PU pVPURt0UNoo+6+26VkdqSR7ybzF9laJ6Mq5rDaZRBsHA0B5YauMRWhwgJUSsf5P7TYBy uz6elzMkRvYa2HNExqJ0ujrlPpoZ4vf/D9nQmX/H6z3vAeW0IA6ne29B1yaevNiswB5+ MHbQ== X-Forwarded-Encrypted: i=1; AJvYcCUrNFAZJbDBIVTVjp8HcWLSJ/AaOOrnU+2bgbpGTNPLpPJDL3h4/WhST15VUMfCiDgtRnbu59jCVvQagP971kUu3cI= X-Gm-Message-State: AOJu0YyJiFcZbhLChQ1wunOUXopWs8xdJ0ccL6rET2P0B69kUs/ARZ3Z oINPaSO1Fc754dGQ/dZfLUjALUudB9LVa4VUkSZesWSlIAqxoPHmdBKY1i21UHXv4P8JbMTRMjx Mlw== X-Google-Smtp-Source: AGHT+IFHcVmtqPOIGTS9q5X2yZq2RumAgxDeMWYEbPcJp3ccn54KA7yETMVtGjEab6EzCxk/TJs7FGlLDMQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:124b:b0:dc7:7655:46ce with SMTP id t11-20020a056902124b00b00dc7765546cemr24975ybu.2.1708544472027; Wed, 21 Feb 2024 11:41:12 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:20 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-8-surenb@google.com> Subject: [PATCH v4 07/36] mm: introduce slabobj_ext to support slab object extensions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 2CE6C18001D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: inranrqhko8teeb1p4wotmkb33bemiof X-HE-Tag: 1708544472-227939 X-HE-Meta: U2FsdGVkX1+yfX85oG5/2ViWqjAIBkNn7HKgaX8BprTpiIoPRJjkZb/eSk/EBEj5I/j0iosuo4yrwqtvJAy5YfLJh0Kz8p8ll3eFTpbfzrSPOSWk/Tmfz8tHuryNtxPkB9tbghDsigp93EC1qmvd+yi8I2VbwnoKQvHvBTOZlpbKB+d6tztRcBrwwCilzs8wCs7qNHiMeoSMILRKeDTBFDSNyk0cjcxHX02beLu1zSXTmEi3XMawmDkcZAGshcmVd6SAjIEJuxfxTV9abT/Y6BO0UI/fkQLgmxcSTAvA4H25RbmtySlqQLkIPQHydkpQD3Who3xTHWhm61w/UefCzJGm+4IqMMTkF5Khc6ZS+m+lwQwMCuMfZAmuWWXZFPGDMmJy+PBgeAwBQdtzhlKwC09XRdLHkWi1lYTAOd7CmP1z8MgXo+FdojcNLfutioaC9x+eLD4wT/PZlrTE4v4OtPIiB1QMywIHOL6bBkoUNA6u9LM/QbYjGBV75dON6Rx4NWRQminjHaQ9vov+KAbf3FIGxZ6tN9EEcQuLtMPGMTUQke1IrOnwYrj1+1fGvLXqyQwTw6TEZY5fY9tmAmyIf2Rg3DhUlxW9vnLxJ5DQr0znMAM6IGdzj1klLq7G+7j1lnxm9tQkuNXohXmhSe+cqbEon4mwEDPGS+eVx76eyM9haRYs1kZ7Sqyu/+oY9i5c47T4xQjA7yCPn0kFllGpJ8AcW8eAmLf/w7ptX+pDc/DcyYOQfTGuKL/G4WsR8fkF6RFGP8g3+UZ4mOF3752kIEbQku0V9TTRiAeYn8sDwAJrgMZKqJNwO+8zblVW9iPjaCk0lsd+VASYRijDxo37qGvYtK4ADJBVT0bMs1jRDa4tBSKVzD5sTj9gKsCJmfyhVlTpjfN3FQW+I7XY4Wb3ilpMkzpzW3fsaKUaJPFC+JeId40ugK48JeFVVitS2LGBk/u/hq9enj9/7xfTwEP J+50Pzi1 rsV9nznqBN762+KF+Wmf0PxZY1AIgIniMqvJS/vmKGtS3nkUbmvc8dicLffri8UG08T2xRsw0SHj8E2ASfS+7CU4dit3BrtiMX+yuyueGrhmt1Wd0P5hFW60dDD1LQkK27U57odEeb1TXScktZND9EfE1qV8NSAETqoptZmwNhJIYi6UONZGWb+joq9uvdEt7NarL/IUWpBKggHefnYpD12znekq0C+7W6p5ow8KkOrpOr4KI8ML4c7uFisR/335pmcR3HlY3KFxVlnIj/G7hhNsvftb+FkEOyNbwmDyHvcn215n5YqiHNp8rYHYPWoFVl7q3H3SgDj5qcTRPV4WkzAoVVSpw6W+6XoQBtx78/uNuD02+uyOs+IQKbBvXEf1kD+aLRXvHU9nAfQ2+NhbZwrQnaX/wJ0nb9/friSiTdhTb00S/+SSQKf4jHK46Y16ezvGhQB81HJrSeQdyjMbdR6WFVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/linux/memcontrol.h | 20 +++++-- include/linux/mm_types.h | 4 +- init/Kconfig | 4 ++ mm/kfence/core.c | 14 ++--- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 +++---------------- mm/page_owner.c | 2 +- mm/slab.h | 62 +++++++++++++-------- mm/slub.c | 109 +++++++++++++++++++++++++++---------- 9 files changed, 156 insertions(+), 119 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 20ff87f8e001..eb1dc181e412 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -348,8 +348,8 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; enum page_memcg_data_flags { - /* page->memcg_data is a pointer to an objcgs vector */ - MEMCG_DATA_OBJCGS = (1UL << 0), + /* page->memcg_data is a pointer to an slabobj_ext vector */ + MEMCG_DATA_OBJEXTS = (1UL << 0), /* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL << 1), /* the next bit after the last actual flag */ @@ -387,7 +387,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -408,7 +408,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -505,7 +505,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; if (memcg_data & MEMCG_DATA_KMEM) { @@ -551,7 +551,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob static inline bool folio_memcg_kmem(struct folio *folio) { VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); - VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJEXTS, folio); return folio->memcg_data & MEMCG_DATA_KMEM; } @@ -1633,6 +1633,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +/* + * Extended information for slab objects stored as an array in page->memcg_data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { + struct obj_cgroup *objcg; +} __aligned(8); + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8b611e13153e..9ff97f4e74c5 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -169,7 +169,7 @@ struct page { /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -306,7 +306,7 @@ struct folio { }; atomic_t _mapcount; atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif #if defined(WANT_PAGE_VIRTUAL) diff --git a/init/Kconfig b/init/Kconfig index 8426d59cc634..fe5f5e75bd3f 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -929,6 +929,9 @@ config NUMA_BALANCING_DEFAULT_ENABLED If set, automatic NUMA balancing will be enabled if running on a NUMA machine. +config SLAB_OBJ_EXT + bool + menuconfig CGROUPS bool "Control Group support" select KERNFS @@ -962,6 +965,7 @@ config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup. diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 8350f5c06f2e..964b8482275b 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -595,9 +595,9 @@ static unsigned long kfence_init_pool(void) continue; __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata_init[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts | + MEMCG_DATA_OBJEXTS; #endif } @@ -645,8 +645,8 @@ static unsigned long kfence_init_pool(void) if (!i || (i % 2)) continue; -#ifdef CONFIG_MEMCG - slab->memcg_data = 0; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = 0; #endif __folio_clear_slab(slab_folio(slab)); } @@ -1139,8 +1139,8 @@ void __kfence_free(void *addr) { struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); -#ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->objcg); +#ifdef CONFIG_MEMCG_KMEM + KFENCE_WARN_ON(meta->obj_exts.objcg); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index f46fbb03062b..084f5f36e8e7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -97,8 +97,8 @@ struct kfence_metadata { struct kfence_track free_track; /* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM + struct slabobj_ext obj_exts; #endif }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1ed40f9d3a27..7021639d2a6f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2977,13 +2977,6 @@ void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) } #ifdef CONFIG_MEMCG_KMEM -/* - * The allocated objcg pointers array is not accounted directly. - * Moreover, it should not come from DMA buffer and is not readily - * reclaimable. So those GFP bits should be masked off. - */ -#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ - __GFP_ACCOUNT | __GFP_NOFAIL) /* * mod_objcg_mlstate() may be called with irq enabled, so @@ -3003,62 +2996,27 @@ static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab) -{ - unsigned int objects = objs_per_slab(s, slab); - unsigned long memcg_data; - void *vec; - - gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); - if (!vec) - return -ENOMEM; - - memcg_data = (unsigned long) vec | MEMCG_DATA_OBJCGS; - if (new_slab) { - /* - * If the slab is brand new and nobody can yet access its - * memcg_data, no synchronization is required and memcg_data can - * be simply assigned. - */ - slab->memcg_data = memcg_data; - } else if (cmpxchg(&slab->memcg_data, 0, memcg_data)) { - /* - * If the slab is already in use, somebody can allocate and - * assign obj_cgroups in parallel. In this case the existing - * objcg vector should be reused. - */ - kfree(vec); - return 0; - } - - kmemleak_not_leak(vec); - return 0; -} - static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { /* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in - * slab->memcg_data. + * slab->obj_exts. */ if (folio_test_slab(folio)) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; struct slab *slab; unsigned int off; slab = folio_slab(folio); - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return NULL; off = obj_to_index(slab->slab_cache, slab, p); - if (objcgs[off]) - return obj_cgroup_memcg(objcgs[off]); + if (obj_exts[off].objcg) + return obj_cgroup_memcg(obj_exts[off].objcg); return NULL; } @@ -3066,7 +3024,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) /* * folio_memcg_check() is used here, because in theory we can encounter * a folio where the slab flag has been cleared already, but - * slab->memcg_data has not been freed yet + * slab->obj_exts has not been freed yet * folio_memcg_check() will guarantee that a proper memory * cgroup pointer or NULL will be returned. */ diff --git a/mm/page_owner.c b/mm/page_owner.c index 5634e5d890f8..262aa7d25f40 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -377,7 +377,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..7f19b0a2acd8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -87,8 +87,8 @@ struct slab { unsigned int __unused; atomic_t __page_refcount; -#ifdef CONFIG_MEMCG - unsigned long memcg_data; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long obj_exts; #endif }; @@ -97,8 +97,8 @@ struct slab { SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); -#ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, memcg_data); +#ifdef CONFIG_SLAB_OBJ_EXT +SLAB_MATCH(memcg_data, obj_exts); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); @@ -541,42 +541,60 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_SLAB_OBJ_EXT + /* - * slab_objcgs - get the object cgroups vector associated with a slab + * slab_obj_exts - get the pointer to the slab object extension vector + * associated with a slab. * @slab: a pointer to the slab struct * - * Returns a pointer to the object cgroups vector associated with the slab, + * Returns a pointer to the object extension vector associated with the slab, * or NULL if no such vector has been associated yet. */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { - unsigned long memcg_data = READ_ONCE(slab->memcg_data); + unsigned long obj_exts = READ_ONCE(slab->obj_exts); - VM_BUG_ON_PAGE(memcg_data && !(memcg_data & MEMCG_DATA_OBJCGS), +#ifdef CONFIG_MEMCG + VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, slab_page(slab)); + VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); +#else + return (struct slabobj_ext *)obj_exts; +#endif } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab); -void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr); -#else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab); + +#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { return NULL; } -static inline int memcg_alloc_slab_cgroups(struct slab *slab, - struct kmem_cache *s, gfp_t gfp, - bool new_slab) +static inline int alloc_slab_obj_exts(struct slab *slab, + struct kmem_cache *s, gfp_t gfp, + bool new_slab) { return 0; } -#endif /* CONFIG_MEMCG_KMEM */ + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + return NULL; +} + +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + enum node_stat_item idx, int nr); +#endif size_t __ksize(const void *objp); diff --git a/mm/slub.c b/mm/slub.c index d31b03a8d9d5..76fb600fbc80 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -683,10 +683,10 @@ static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *sla if (s->flags & __CMPXCHG_DOUBLE) { ret = __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); } else { ret = __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); } if (likely(ret)) return true; @@ -710,13 +710,13 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, if (s->flags & __CMPXCHG_DOUBLE) { ret = __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); } else { unsigned long flags; local_irq_save(flags); ret = __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); + freelist_new, counters_new); local_irq_restore(flags); } if (likely(ret)) @@ -1881,13 +1881,72 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; } -#ifdef CONFIG_MEMCG_KMEM -static inline void memcg_free_slab_cgroups(struct slab *slab) +#ifdef CONFIG_SLAB_OBJ_EXT + +/* + * The allocated objcg pointers array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ + __GFP_ACCOUNT | __GFP_NOFAIL) + +int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) { - kfree(slab_objcgs(slab)); - slab->memcg_data = 0; + unsigned int objects = objs_per_slab(s, slab); + unsigned long obj_exts; + void *vec; + + gfp &= ~OBJCGS_CLEAR_MASK; + vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + slab_nid(slab)); + if (!vec) + return -ENOMEM; + + obj_exts = (unsigned long)vec; +#ifdef CONFIG_MEMCG + obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + if (new_slab) { + /* + * If the slab is brand new and nobody can yet access its + * obj_exts, no synchronization is required and obj_exts can + * be simply assigned. + */ + slab->obj_exts = obj_exts; + } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + /* + * If the slab is already in use, somebody can allocate and + * assign slabobj_exts in parallel. In this case the existing + * objcg vector should be reused. + */ + kfree(vec); + return 0; + } + + kmemleak_not_leak(vec); + return 0; } +static inline void free_slab_obj_exts(struct slab *slab) +{ + struct slabobj_ext *obj_exts; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + kfree(obj_exts); + slab->obj_exts = 0; +} +#else /* CONFIG_SLAB_OBJ_EXT */ +static inline void free_slab_obj_exts(struct slab *slab) +{ +} +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM static inline size_t obj_full_size(struct kmem_cache *s) { /* @@ -1966,15 +2025,15 @@ static void __memcg_slab_post_alloc_hook(struct kmem_cache *s, if (likely(p[i])) { slab = virt_to_slab(p[i]); - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { + if (!slab_obj_exts(slab) && + alloc_slab_obj_exts(slab, s, flags, false)) { obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; + slab_obj_exts(slab)[off].objcg = objcg; mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); } else { @@ -1995,18 +2054,18 @@ void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, - struct obj_cgroup **objcgs) + struct slabobj_ext *obj_exts) { for (int i = 0; i < objects; i++) { struct obj_cgroup *objcg; unsigned int off; off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; + objcg = obj_exts[off].objcg; if (!objcg) continue; - objcgs[off] = NULL; + obj_exts[off].objcg = NULL; obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), -obj_full_size(s)); @@ -2018,16 +2077,16 @@ static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; if (!memcg_kmem_online()) return; - objcgs = slab_objcgs(slab); - if (likely(!objcgs)) + obj_exts = slab_obj_exts(slab); + if (likely(!obj_exts)) return; - __memcg_slab_free_hook(s, slab, p, objects, objcgs); + __memcg_slab_free_hook(s, slab, p, objects, obj_exts); } static inline @@ -2038,15 +2097,6 @@ void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, obj_cgroup_uncharge(objcg, objects * obj_full_size(s)); } #else /* CONFIG_MEMCG_KMEM */ -static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) -{ - return NULL; -} - -static inline void memcg_free_slab_cgroups(struct slab *slab) -{ -} - static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp, @@ -2314,7 +2364,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) { if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT)) - memcg_alloc_slab_cgroups(slab, s, gfp, true); + alloc_slab_obj_exts(slab, s, gfp, true); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order); @@ -2323,8 +2373,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { - if (memcg_kmem_online()) - memcg_free_slab_cgroups(slab); + free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); From patchwork Wed Feb 21 19:40:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2EA0C48BEB for ; Wed, 21 Feb 2024 19:41:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B3F06B009A; Wed, 21 Feb 2024 14:41:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73C6C6B009B; Wed, 21 Feb 2024 14:41:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B3D86B009C; Wed, 21 Feb 2024 14:41:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 44EA76B009A for ; Wed, 21 Feb 2024 14:41:17 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EF77A160AC1 for ; Wed, 21 Feb 2024 19:41:16 +0000 (UTC) X-FDA: 81816829752.27.21B43B2 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 313F210001E for ; Wed, 21 Feb 2024 19:41:15 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Vvlw7H+n; spf=pass (imf14.hostedemail.com: domain of 32lHWZQYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=32lHWZQYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DcVDeB20sah+c6m+q7qbj/Bohzcv0RbxXXpsl4x4Og4=; b=DP3e1Xno16zVDGH8fGwQZuCN/hxjTGKBECrwpGyaig/D36fLQ41MF+a4y91S5YHpi26WoX vxK1vxMaIdrcbyhv0EMagZ4n/FsorEiejqkKvHjZ9gs5L2JdVSFvxnUDec+TAcuayd+OM8 lSljwny/6KIfL0aYhKH6QpIxQJdPrN8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Vvlw7H+n; spf=pass (imf14.hostedemail.com: domain of 32lHWZQYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=32lHWZQYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544475; a=rsa-sha256; cv=none; b=H2x77SO8OiFKNTAkHFfj1AqGV39wH0ZRvOiXe6i3w4rlaEbsdgxCqwH+m5nt2WSnF/Ty3h tUZeiGbHGfHYd9Px4Rhl2OAjjOKnr90DgmHzi873Nt78dajvyaeHCU40gWmbbmvT+f1m4J VeIFA9WXDtMtPM6UOLxkRHt9eufIfHc= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6088fa18619so9932637b3.2 for ; Wed, 21 Feb 2024 11:41:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544474; x=1709149274; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DcVDeB20sah+c6m+q7qbj/Bohzcv0RbxXXpsl4x4Og4=; b=Vvlw7H+nh6ruP/PkfuO+jWE/pGrsBhM4cpRsfvkNpvEasKOsPN7wR6LOr/X7NedzrG o2AAl4AMphoYp4c3rZc0TF8HhrKkHkKdr7+rhCvGpkStUMnYbQoVK8rFs/My9LLwY1tD yu57dggV9DeqO/i8FmvIPIy735VjWXIIeBiqrrefrYKZm6hJrqx67+WP/R/fXfedTIl/ O3ieazb8Jh1wbYvl3IXjIBGEdPyGvdVxkWA4ev+LHRDgQi3IjiL4IF9PHik9wAShDm6p aPopacDXoARL/NozCImJOjisSOycw5u1Xa9ibQosZADoZeQQcYP5f5RGjibXucpISPBJ wVDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544474; x=1709149274; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DcVDeB20sah+c6m+q7qbj/Bohzcv0RbxXXpsl4x4Og4=; b=cOpb0PUgM2hpuA4QgeSxcCAaskc4++w0yqNWMlatK7s6K8/kedEUPf6awjBMYekPvx Ojtk5IB57h3Ky2Cwe52GBjZA6a+I1LY5L9N9i7N5ImVi3XZcijlEr7yWUvJU1G6J94SC hgM39SONcD5GOcGO+AAGWSN49Ju2hGKbcQTSp/obJlfveaCSzws1XhtfLlFvxsK8PYnU aBq7dIT7BWNOp1Ejfk72O50lpk6ZSaQ+NGZH6s1gEgBWD0OYCVNhPEGJPUBN3JjjGDYD qBd+RnkCT5GXj2STkT16Ndg/R8tj4gba3CUDAtxSaoPr6T5ifJ/pcy2UU0lgVqP67z3v msAg== X-Forwarded-Encrypted: i=1; AJvYcCUybi6XQLcCSQSvCkVxWdHQsgI35S54Z9JsnaFuiC+uh+AxH+MPqGLGL0RLCgzffxGw+fDsZTnpSRZn5+gDCLJNkOY= X-Gm-Message-State: AOJu0YwJ+DxjLHft1kWaFlvKMPRJmMh+c3PTDXeGMvL4h1Dkc+FwL0iG jUClwav/0k4w/OeHi53OgXrwkSXzAQ9Lz3WWGZjK6xTRm9gKcNPqGFvuoqHFtBg6R5lP4iL1vjP ZIA== X-Google-Smtp-Source: AGHT+IH75M2lZrHB2S9wjqCstmCY896frmoFhg/GjACV0w0wTBslVAISl72HA0I1QmkMPlsLusmP2hmQVrQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a81:4ecd:0:b0:608:9561:fdbe with SMTP id c196-20020a814ecd000000b006089561fdbemr96126ywb.2.1708544474186; Wed, 21 Feb 2024 11:41:14 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:21 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-9-surenb@google.com> Subject: [PATCH v4 08/36] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 313F210001E X-Rspam-User: X-Stat-Signature: xcmdao7uwopdkfjdrudbai4rtdtysafc X-Rspamd-Server: rspam01 X-HE-Tag: 1708544475-209318 X-HE-Meta: U2FsdGVkX19ft1ZGXFfBMPj1C7x7gsGIrbiCMBbwHPGuBTP2svwZ+vNpI1WIqlLAGWvT43rFbomRteRIva4JQZLcOOPulHeQnJsIyKBP6MYVmV70ToSK+XaYhrDf7m2mji0GBonKF06xb959l95MlrwnOiDZxuejf9aPkifnkvzo7NfI/ypq82IU8iGJtPUAdfa4svtbYHAPYgKdV5O4HQhWL0oNE90WfX13XfRPup1jUoA8MC4ww4Is2lDbddyPHDqW3c49Fevv344WtqRr0tXJ6LjTiQzEXXHhSJOZ4w24PTS66Tbz2CLksv5X2AV5skjDhN3dn9+cXH/5rqFo9VueCi3odAZ/W9llwFxaLCPrWORBKYXBQA65KEm9zm/gxdmjEuCZuNfI29ttnx1njSUH/+bXaZtKeOtdxFmBWWxbRjKK9b4sLsJnugMAsPh1LYTUe8aNhZLoplDac04+07MyLXw4WxfgHJwvfhMH1084at54Lko8oeFeOarF4uxMbYNIlJ36p+FnbVFKhkAayUXIfW/xydOmhnCI6Fc43Oq14kZXjiV82ChhnNGo2laHKGKUCR9/g/HwYLN5jjj3YcFh/e3A3JwsVKAWmemBc/4giRyTfAuqhq4J+wxq6L2METTCEBzmfhQgyHSyOUR+/6qWvN5RPP872tzXEMpU3+iwy9Ql9WbxoLfM+oPk+Ibb0pGaB0OhcEWm9WGgNd+yZCgY2mBC69lG80HI1WFwVwFfjmFG35+z1Zw4oobmR5gM/RnzVm3qN444biQ/jpBDOnbqZaiuDY9Kz1GHlkNQrUQzsPVc7mW17xPkWguq50jAsHpmi6gORueX/MbyivkF+YAvJbAfDmT1S+ilqkYtfRH9eMP+wBuFPCLNnxpsb5tYh0x3HLgDGJvTNjhb5p3XBnJWLlF0FqWfCKznsBO/AlgFiDbGfEvDsUChevGsExcETtAKGVdUXRASw5CeeYa s9Z87wd2 bXUOOCllleaolj/rv4PF6ChUIfGjulbWYQJFxTK6SIuuyJ0pOWbUIcNCmlaJy4DbszuVTais3pMjPsh+vT4IlKf5M+MGYmFW/kLrveqVSMaj2RwLzZHuuZoRtWAhe4SdXX2iOYBX1Gf8Ao7NMah2FfsXNs+evgn16eM9v+AZjD8hoPot8gGxU23qrVQ1kgRN6r9uh93TqXQ3/RO3kufBKu+ClaicWclF0keN3GGRADwSl/z5PGf2rY/rl9FXFZqYtSjvvM3RvQBF9tPSwpCFj8AJwTyt7XFRGqdHNoIPH7YpkildqFkhsBOfx2By3FafIBjsZ6W9gltG+A/+QaMJNjSOHvcquS1yIpJqmnwLOO/9E2c7GFLW2EeC5GbJjVk15JxUfpD94JJ1sKq2hsOYBn87PSE4Z6Mht0gZrv2e7IDB3fdT0W42NJK8FXGKyh3Y2zY8t/0RrwE9coyzzUaUaUu52Be4kUxRE+pnRblofVNlDlBv8CwTjVDKH4gg233CrUKB7auDuhymhAubtcaa6kj8XkHin1gHRnsttiM+vIH0ZM/Z/hLbGoT+J7Ns9aHLttJ2nyED2qjYIQdKz5KC4p96NiIuv2Hsl0INPteemCRcTjG9e+emdwIS3qqeDVpwSz7pbqeOG9NRbmqPlHUBsOgTVDGz+ADq4+86XQnFSsQcge3Oz1H7l+g5ztqvodRwu8gRHT8mc7ADRAhIsie06kIzWddVwxwNM2MEMy/qegmLKCEulQvH/mby3yMxJuRIUkzor0IamkLVwsliVuzH0XYptZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/gfp_types.h | 11 +++++++++++ mm/slub.c | 2 ++ 2 files changed, 13 insertions(+) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 868c8fb1bbc1..e36e168d8cfd 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -52,6 +52,9 @@ enum { #endif #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, +#endif +#ifdef CONFIG_SLAB_OBJ_EXT + ___GFP_NO_OBJ_EXT_BIT, #endif ___GFP_LAST_BIT }; @@ -93,6 +96,11 @@ enum { #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -133,12 +141,15 @@ enum { * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT) /** * DOC: Watermark modifiers diff --git a/mm/slub.c b/mm/slub.c index 76fb600fbc80..ca803b2949fc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1899,6 +1899,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; + /* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) From patchwork Wed Feb 21 19:40:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84720C5478A for ; Wed, 21 Feb 2024 19:41:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AC4F6B009B; Wed, 21 Feb 2024 14:41:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 935436B009C; Wed, 21 Feb 2024 14:41:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7637E6B009D; Wed, 21 Feb 2024 14:41:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5846E6B009B for ; Wed, 21 Feb 2024 14:41:19 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3FC801C0776 for ; Wed, 21 Feb 2024 19:41:19 +0000 (UTC) X-FDA: 81816829878.10.E4CE025 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 70449C0011 for ; Wed, 21 Feb 2024 19:41:17 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kRyPKlmM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 33FHWZQYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=33FHWZQYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zEkUZGmdT8FEHHQk8lZ544d0po9Jm+ENmRP2MkpUgMM=; b=bpAGAyPgr1maj6f5Su89djNO/ze9z/iqZK7+4FoLchqqU5+XPwU1a7MDoGOV6iQZN0Qs9C GuXLRsj7gqiRW/J6LY3dTDKzkRkHkbQsw40PmCTAo/dJPEtaNvOVcWCs0u3sjueDSOCHI/ MgvN21CZn4MTjU9duwz/9QLxTUOh8W4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kRyPKlmM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 33FHWZQYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=33FHWZQYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544477; a=rsa-sha256; cv=none; b=oJpdk+juhUmfD0hNdnk1TGA8sIQ10INk4cWHWDB3x0tsJXNVnPEG2DAOaekF1ARuzkrODO U5gCWPNKaqefKtErm+j6sNNyLEGZoCFATw5e4b7kBTrl3a2SCWrCzeJxzneZxt1WkG+TbN OdllbumuNckJBdFJGMzhbeK0spM4FhY= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dced704f17cso3554141276.1 for ; Wed, 21 Feb 2024 11:41:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544476; x=1709149276; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zEkUZGmdT8FEHHQk8lZ544d0po9Jm+ENmRP2MkpUgMM=; b=kRyPKlmMApLzp4YAh8avbwulFmKTfA/EnixXKYn4IBQHTk5+cfrmrhcMXxl7ZIhLv6 2PiHKvp/esVZeNFeIBMYNzY6tdi0ynIi+pyf0Jty8I/WyLBpIeswTZzU2DTFvQ8ct8Wj GN2S5HAb+t+w4tUkbq7EfXHl71CPKKg/KbTpnVau/+nG8bmUJWWx4CrE1LA1CmHqCG9m Hnh95T2pOdi7rgD+QgGEvLienVfDz/toR3dDKbWMA1iGn5JEvVfkE5R+ZPgZWEaT0w+N /7LkHjxI3TFEly7bgsrGYRzlc4kpAJ0U68FrIanvjZJd59FP1+pFPiCoekbsBDXd0vzy wRYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544476; x=1709149276; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zEkUZGmdT8FEHHQk8lZ544d0po9Jm+ENmRP2MkpUgMM=; b=PiOTgOLPUIfmfL+TGLv5eF/1F9rOuRqRnvNXu46hgjCOEBZFhDg9oIXwc/KrVMdBry it2lRFg47pnFJEd8C9W0a5PryUFBWG60SRux7/P3vrHRUgnBEzZtc345aPvVpqvrTFqF EDbdyIPY/SlxtBu8u08KH34N7GpQQKMxqTsAcA+OfLTuQS9ScE5w5MB96eGG1orUpgAp KggesuVclBK1CEB1WN/g0K8UGYZw74dEdsyy/JZmA8imKbTGqsDIWeHo9lHZbGsKeU+G BzaqrT8cxGzF8j/35mDuKYIhzwQ0FwE3PmAFLyia+TXXbjh0OHsXTiQpFJojG0UxKqS1 Bo9w== X-Forwarded-Encrypted: i=1; AJvYcCV4ktwuDx35ZOFOIaYFe/h64lNnxIZYqhwWU/JWv0whVy47wxn/ufptgwSc3h7O8tq2pso7YuAZKHVgC8E2oDJDpoE= X-Gm-Message-State: AOJu0YwkYakVdQhWNiEfwUi2kq/gRtOkIdaTqmtFPAci/TRuBreZBjIu JeyJvuBwyKnaLwEXYa1nx4nlypcpbDE54xQg8qTnsgbBe3rnI44IewscAKKjEWiPqjCdXPOJ9Sv vqg== X-Google-Smtp-Source: AGHT+IGFqZQMKrIKQYeCMZ0dyCYxCh2jZRWMkTdBUYYdfqnWgxWQSfzS1xjoTphhPvwCFL+Ef45OR+v8sww= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:26cf:0:b0:dcc:41ad:fb3b with SMTP id m198-20020a2526cf000000b00dcc41adfb3bmr6923ybm.10.1708544476345; Wed, 21 Feb 2024 11:41:16 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:22 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-10-surenb@google.com> Subject: [PATCH v4 09/36] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 70449C0011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: f8a8ijsr3oqu1n8kgcbcnbnt9b3q87wy X-HE-Tag: 1708544477-998720 X-HE-Meta: U2FsdGVkX1/Y2eZJ2UfKO1dG5hihH0jfwfg/46HslHKCfJki375V4wwIpgNJguWjYQy43nlJOOhcT1RXrjYIRDBrF2BYsgnMMVuwV+jN/KUFxwyswMpOWyuqjrOoFRdmyjzfHuwvFOmhHExks5kYvjya2laj0uFaTURIyRkP8TCa4s+69E2g9CBuRo8ahDvwOza8E5Ai+A275RvEDCeeFZEdpoUU08nNYY0Fnda/egJlyLuwSLgOPbLCYGYWiTRJh+w/yFLXo1QQg4VQBrwye8KmbtaW8Z3mbokwmWS0IE++GnXonIgky/cXpnAsxQMLZFepkcfVxPdTlL/N7hV6r8aOG+0PeuKawUpJ+JA9I8zwAmoM8n7+VLWPl8o6frvCBwFwH0tuq2SDYStK1npr2vVbWp2YhIWGxKuIVhoZHIklldQ/F20OwaoR38Brxk20p8AmtbRPC4kQ46KC1Uysh5toS3aB+jJpwbr2j1ANyw02esYnEBuunKKoMhzeAQHBupaAfIStpTZXwImfZXJchn6PIpMEpQWMlvhwstnwEurAMIY/NSSwA0+ozz59ESl+3sQMgV0etszqbUJO1WwxIZJ4muHZ9y1BSpuHCOWhAC6x7XVlHxZOGE75XI0lfl7kdZ1ShlBb2yB5zBKcV2tsYs4JSVEIltHW0270AchbCgd5e3k8TqPCk0Cn+TtZIBZsjHQ2UXvI179Y+bkiN7nj3fTzE/9K3uYfftW00cajREVwF/4iQzPWBRLZOg/KNkgbc4TURU99AS6BqAFtSGXyJXHJedlPQAbWU5cETVw4nKRXUOQdXDWCV2j4JMYQHm+/54Jetyc4sfz0wlWURFrh55f0mf1hUlNEe1OafYPcvaVS7CNbRL2HMFYu3pToc5pMJ9l92vLutNdm3Qawi422FWizGnJCSBQPsZA9uDJ23c8xl2U08rXI1Feneu83Qjl0xfT2eEqoLHf/oaVCtWX f8e0EAr3 VhqnCTi/+x9wwTuQeMuFX6jZB//B3MJ42H1DoaubOF3PFHAYJQUTS1KtODNU1BdyedR4bZGsQvAPo4vtSXf+CwV1tdzwB1OktSY2aIA2lvxmaTVSZB+XUpJEsayscXS/rg2JzpkJ1u74Y/aX0Bodk9bmFlw9cKvruSvmEekvKldI3jUn4oS4T7jIuZ2oBSz6Tckq6NgxKxUMyzCEN6Gpl1uhNbwu/WdkHxP401ti1gRi1B2yMcwDGUywX8A4Aor72dBPsHNPggFYF1yDA1Dzlbmom+/sVRpqm3OVTia6XuO64CQhHfF/QtuGpXiXSzm98LuFdrI4ANjIF0GFnQBa8d+N1g7jtIbayZeqTXdk/2JCVQTAbcXEhv10LccCwFUVTI3ekaSrFkP4b5CRQzQvmQvfN9btmxhryOSu8SKDKNM4IZ0ZNjWJtu6C1s7DpCqrJqrVrxUW1lXSuoNaeH7MMhsSCXE6G0WsLNH1l+1KqeZzv+Suc6G1gcfgrpJZI/JjLuxzZ8TBEqIeaZLgr+mwXAmFFqnGsKoKy1I4KEVR5JhPiyVp12ESxDTzzkzjADWicvI6lNUsDcSm4geFl5h4XxNcYDle7Q29mDWt1i+gPzGs/88J9RtW/xnzxLRq1Xe0oCjAS66GD1Gqy5MvQKFObdx7+LluzO/012oUKmu7qsdC7XDczFmIeOvRDpotVAK2m8+TvbcqCv+o2SNs9DBnn4f86OTP9ZxG/hatlAn3aOdjs0raI2aupKRVwFAE4MsHbfXEpSmIoPa0G/19fmRj8ORHIdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 6 ++++++ mm/slub.c | 5 +++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..58794043ab5b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -28,6 +28,12 @@ */ /* DEBUG: Perform (expensive) checks on alloc/free */ #define SLAB_CONSISTENCY_CHECKS ((slab_flags_t __force)0x00000100U) +/* Slab created using create_boot_cache */ +#ifdef CONFIG_SLAB_OBJ_EXT +#define SLAB_NO_OBJ_EXT ((slab_flags_t __force)0x00000200U) +#else +#define SLAB_NO_OBJ_EXT 0 +#endif /* DEBUG: Red zone objs in a cache */ #define SLAB_RED_ZONE ((slab_flags_t __force)0x00000400U) /* DEBUG: Poison objects */ diff --git a/mm/slub.c b/mm/slub.c index ca803b2949fc..5dc7beda6c0d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5697,7 +5697,8 @@ void __init kmem_cache_init(void) node_set(node, slab_nodes); create_boot_cache(kmem_cache_node, "kmem_cache_node", - sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN, 0, 0); + sizeof(struct kmem_cache_node), + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI); @@ -5707,7 +5708,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); kmem_cache = bootstrap(&boot_kmem_cache); kmem_cache_node = bootstrap(&boot_kmem_cache_node); From patchwork Wed Feb 21 19:40:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1412FC48BEB for ; Wed, 21 Feb 2024 19:41:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FC4C6B009C; Wed, 21 Feb 2024 14:41:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AA876B009E; Wed, 21 Feb 2024 14:41:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D68A6B009F; Wed, 21 Feb 2024 14:41:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 662E26B009C for ; Wed, 21 Feb 2024 14:41:21 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1818014055E for ; Wed, 21 Feb 2024 19:41:21 +0000 (UTC) X-FDA: 81816829962.22.524005D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 7CF9CA0025 for ; Wed, 21 Feb 2024 19:41:19 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=R5pFpkTo; spf=pass (imf15.hostedemail.com: domain of 33lHWZQYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=33lHWZQYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544479; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RtUIa2LF9EHhDqysRu/yxXfpC6tMV2p9GldnGdUYEsQ=; b=qA6iYgwFNN49w2XjpaidtRGq8qk85eLklY3s/hwsp0P2ZMwU1xw53Lp8XMnr2btF0kq49Y PldQGo8sNVbdXJs01PI3Fy658QR14sal4YPSgHULdN+GsqbaqEBxG/pMPCocKXavAmNia4 vgx5P4bzfDB7s3V5yhDy+OaonqWDcOM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544479; a=rsa-sha256; cv=none; b=RDnMYCy8asZoVZ5/5jKEfi/QIItedWdLxwHymMxIREg9XMPEYSejqKb9xwGdO9EIEE35qB 9+B+kcGH5mmz8nn1nIo2f4vmfIefnjcu0TwyA/DdXcYGbJZ8TOTjNe2A6uASszGdKHhWuE jjaAjLP+XMrv7BQ1rW4vgQ2ziX+NNoA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=R5pFpkTo; spf=pass (imf15.hostedemail.com: domain of 33lHWZQYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=33lHWZQYKCBE9B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b5d1899eso213355276.0 for ; Wed, 21 Feb 2024 11:41:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544478; x=1709149278; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RtUIa2LF9EHhDqysRu/yxXfpC6tMV2p9GldnGdUYEsQ=; b=R5pFpkToUvDYKzCGWrWXng78mzmO1rCg4gGp6+3eRvmulaK00BG/zzeOuDyemhxnit 6I1Lz2Yt5f2D6E75z5NSul3ZTZTFDhazuPK+JyQCE9jHY2rePfpLRF1cT67vR3PKM/z5 hpJifbbqT8o6OlS85mQGKB2jZvoriWHCPamH0kq/tOwMY3WGmM8X/kr/01Bd5QlVApaE iA3o98qrQn4wfCIHTpIoBzLqQZLOO1R3fgs8CWG9vHSyv2ZiJURKz/PjfO49KF72P6Zz DZQmORZbYV7z8SJMLnbi2XxGQX7XVEKpJOu5cImYIDrH5L6f1Xsyp/ivNIcD4ltsu+mA iUIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544478; x=1709149278; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RtUIa2LF9EHhDqysRu/yxXfpC6tMV2p9GldnGdUYEsQ=; b=eu1od8zuHGWNqZjhgfG3E4L2oFmIyAmsBFNd/tejNVQt45dxu5mubF5WYlY//dkuV2 73gOIffvxIire4qN0QfGhgPY6aw3PIirW3aQw017rTMl1VLZMTQ3YoTukPOMsuShcOE5 s9FewWPrqm76XgHT1bdQE9GYx+q8r7T+cfrLOH1hkQ+qmTqeVylq0hxrwPcFmgOxQzpC fyEPeehJL7yJ7r6mT6gZwVwSuOuWtxjpHBX6c2ZJ1YWkLR2NUd5zMsLdRsuq/RfuDYEn gQNRdFOmhl0I95xXhxD3alli+dhJDAPLIFLYKEfXZn7VoEuL9QMt8Y+CeF291g0XfRrC D9vg== X-Forwarded-Encrypted: i=1; AJvYcCUfNNxxtXGxBCTCH2zJ+7iPnF/bcLkmocZ1svcERrye23sbKyzvsCJFfV+KdCaqRuzwqVea8o1T6OqKHb2d+OTGXOU= X-Gm-Message-State: AOJu0Yz2lLcD7R0SV1Ktb1H5eDGYk7Nf7q8o72T+JGpih3hxucfItGsy w1/OF7K0ynk0DXtjZKGkB4sDnB9m/wX9B7BNlQ8T+jUZf1r3SP3fcMCuM5GbXs5UIxjO0gmouZF bkQ== X-Google-Smtp-Source: AGHT+IEwwSNQL7Uk2yZ0VknWIStJpVKOxx3COiSHNCEPyEAxFFFoBLgf8pA7JP3oX0B0ao2tWkEzMtP+Sno= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:3f06:0:b0:dcc:2267:796e with SMTP id m6-20020a253f06000000b00dcc2267796emr31950yba.2.1708544478455; Wed, 21 Feb 2024 11:41:18 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:23 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-11-surenb@google.com> Subject: [PATCH v4 10/36] slab: objext: introduce objext_flags as extension to page_memcg_data_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 1pb3kuxj4sew3rry3fucxdzaw6eqsggx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7CF9CA0025 X-Rspam-User: X-HE-Tag: 1708544479-808199 X-HE-Meta: U2FsdGVkX184D82xervv/PtkUOlYHOEqD97otgj2LIAAthpB2fLSUiyiSFQB0LxIXUvyxA0ExbcXdxcozp423une3EehTTST1SIH9HBA1hPgp6JjW3pvmakW2Y5TdkMQdsusMkGs3sej5ttBHp9nFr+WndCjWoLegX+pY1pGgEagjYxE3iTc68l8CL5nOP1HPzfOp+p87P2kLUpa9suShCS3EGsiXsUoyO78w3d69cJ1F5+f7j2KrR9Dp9tPIYXkEQ9EZWEG/lincB4kXAJL2In9hM4JNSuWgt1hWN49rVHLwF6QnsaDQ0tbpe7zKd/CpzIqoTsTxBizyliZCTt14EB5uUHtDBJJkYDoaCugtcgNYpJ6d6eIQa2jDWvDqUMBpMRZXN4y1nt6rFTivjbpCgyn+XgLvIRAhb5G3Z6WC09PX8pcgmKU7Ikebsw65rnPReQWgNmb64bPtNjAsuaXkj+eUm17PCQ9B6IfFpEHEeZ0EpOmrvoQh+YaAMIjwoyFeVosWqeyshJfX8nFxt/wOYX2fBSQZuIBRE1axw7vL0evFZAQXRmBCySJKRbVqiLuQGGgpwAFZSeSuNJcl8FBou1QNuQO5B1vgqUxXS8JuROcxXXhBnT7hmbvCWNL/KEW/xe/E08Yq7HSK5ivZ4fx4JFZ5VAHACVL7j6Y6jD7/UXps/sYWf/uCniLoH6z4kfRUsBIusrVlz+Vuk9BMKPFwh62gVaG9MXcghTKuASVZSCJinUKyv9lS1Re+KGqMWmFOq8wC3iyedaXYiw01A5qpVt1MH1WAnfwSoHSccTgfiQ0jANxzlV5nljU7/qrOdIkqJ6d1sJ1IhbNq54SpR74FHwW/uYxxAjMVeIq8LSgsxyffSlwbtucBvvIyoA7uHsvFI2oQrIml4RTccW81RyShxWGm1mbn7HGgeurrv0ErHKHQlDcVwW2O5i7xQpFpsAcL7ePQCXaPyuRoA9iwCb 6/I0JUm7 35PRurbpACIF2N5RpKBF7oyj9KuJQMHOtSv3K5i5h0LKQ9DPAObcYmQ1zNkRBtj/DUVjawAJy5FuHR+culu1XIBzkgrEw+WNt7Ds/m3NjQG3m7bF1/a2s0wDvsT4/uCgCuw+l3HiF8xXhP+//rC5z+rlkseSwi/OxO8zVySMdmwIqLvhcMbhiOXe3IljB17YSKpphsmizp0UDZy4E0LVDWgAiircEDYzgq9P/HEDU81QHfg4B7V19FZcye4lc3dg1lptzqT4R9jm2ludfKGPw649FthTIYD8EyZkcfeOWrSZ9js2wRD4Dkk8fxztpb3pPO99P5QtmBSqbDzNXEbrWrnpC2KDKgG6BtoSKlRMxIGC1afuhzvoKU4pyJuSgtR99tLAokBTPrd7SSJirO6CededMvsyprBC4vqwur/cdjW3OAu0q/kxHSKVfRynsyGIJQvrmnuuAl17zZ3GjgaHJbKQRMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce objext_flags to store additional objext flags unrelated to memcg. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 4 +--- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index eb1dc181e412..f3584e98b640 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -356,7 +356,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { + /* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem(struct folio *folio); @@ -390,7 +405,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -411,7 +426,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); - return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -468,11 +483,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -511,11 +526,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/slab.h b/mm/slab.h index 7f19b0a2acd8..13b6ba2abd74 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -560,10 +560,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) slab_page(slab)); VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); -#else - return (struct slabobj_ext *)obj_exts; #endif + return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK); } int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, From patchwork Wed Feb 21 19:40:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAF4AC48BEB for ; Wed, 21 Feb 2024 19:41:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6F036B007B; Wed, 21 Feb 2024 14:41:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DF0476B009E; Wed, 21 Feb 2024 14:41:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1CBA6B009F; Wed, 21 Feb 2024 14:41:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A12196B007B for ; Wed, 21 Feb 2024 14:41:23 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 81611A0BAB for ; Wed, 21 Feb 2024 19:41:23 +0000 (UTC) X-FDA: 81816830046.06.75E8F5A Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 90F424001F for ; Wed, 21 Feb 2024 19:41:21 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Kc7h4rMu; spf=pass (imf11.hostedemail.com: domain of 34FHWZQYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=34FHWZQYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544481; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2W4P7NFQiFc4UeX84KC+r4Xo/VJXM+I42piR5xvvbG0=; b=p02oawz+fO/e0kr2i5Xpey1ROcz2I1+VXhpUfya0tDsJ3AcQbvijJVYyRZ8PkqO5Qa/tDv FHG3d7d4FIZPLSVnpXa3QgK0X1E+Nf/c7h1YcTK3f+OlXzF8IJkCMk5P45aTsxLNt9c/09 W8P6W/afZ9tORHcXlvQTw9sscDnkQQk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544481; a=rsa-sha256; cv=none; b=kBj3bA0uKIBW1hoooYAZwQbMulZOknbgL07DpILX6xe5X+cnL5cAzU7Ej8rehhj2U20UxT hun0H7j5magCnuLGt6j6UZDkoc6bOD1+KyuJVH40aafjnWjC4Ao/ODqXiB0Oo3DqMGScPF fB7phe1VNaHKVQqFwY0DmDtUNNvmwKw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Kc7h4rMu; spf=pass (imf11.hostedemail.com: domain of 34FHWZQYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=34FHWZQYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60804b369c7so74078067b3.2 for ; Wed, 21 Feb 2024 11:41:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544481; x=1709149281; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2W4P7NFQiFc4UeX84KC+r4Xo/VJXM+I42piR5xvvbG0=; b=Kc7h4rMuIyWipLsKbbBAdljwBzZOqZ0w4A8ZkJE3Gy3iZwknt+EI5D4Zi1UYNNB0Qa HAv+wBtj/UvDaHy8BrvW+kPOJYgZhOIealkYJwgKQa2JdCXDJPDoh3hNvKXEAYrVLeWb h6WxUvPYvCDWcCui5XKbWCpnlcmiSdGNXl0S0Ik21thtSRgAmC7lhqU2Ix3M725a1jSX 99ymun0s2XyVDvcNvIm0cet7dvmljpYPsgR/8+rQSPRJnLVe9jFfRCYpPYkSnBSYERHo wEZKHeNzm1xuIp7A6N5jHvc1/xaq5eif5nhI383hJKwkrIyWHg0qlnK6xCxT8bTEWYc9 jujg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544481; x=1709149281; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2W4P7NFQiFc4UeX84KC+r4Xo/VJXM+I42piR5xvvbG0=; b=wIWiLtwc2vlWFgbzhoB1OEtt3V+8hhlB9S0sQhsyAOe+h1Dln0tub8ar+xzI3fwwmg dFlfamlUon/VNwZBsIXk1RbWtOI67srqsaD8xW4p8sEmHHcSQKntvqFWY+Gbl0G1klsm HUioRt3hBHUc4r1hBnIVrX7biXdfqU5VXik7fdpYFI/Hv0YMsbrZLoGWHJgFODPCJTqE dVKLYULuNQbhlT0f8gbdS5IS+dsMp3EcPWxobfE8U3AvSwsMwf9T2eRiWNQT5jxcuOIR d6D90r5n7Oo6uSVC4Wc4KQu24Ng6OG+Ttu+8dlouXfp/rULPf8+7ZXUSnI/lMtJCuHH9 tl5A== X-Forwarded-Encrypted: i=1; AJvYcCX9AhDk1rHuatjHkpbngzN12KWUSol1JMy9/G6kroZfCmAbndqT5ysy0ceuq9J7Q5v0ZM7H5eb75PcipEM6p623+xg= X-Gm-Message-State: AOJu0YzvC8SecDZnC0QYDTt8ZF83qejGWAhinFESCMz7KiFK8BfCaLex XYzTlNvyc8lxs24sShagvYaTCbQrHg9qu6Zd206GisAFGPhO9+6f5qCv70V5rJMP2ttdZ2DNd3K 6eg== X-Google-Smtp-Source: AGHT+IG22NJCzfpd37plznN9P1eucQzR45NOEFz2H7svwKCMA16D/2m3MKR+Tv8mzzgFi1+pnCzIavLrQko= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:1209:b0:dcc:c57c:8873 with SMTP id s9-20020a056902120900b00dccc57c8873mr68426ybu.9.1708544480467; Wed, 21 Feb 2024 11:41:20 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:24 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-12-surenb@google.com> Subject: [PATCH v4 11/36] lib: code tagging framework From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: 6fkae35uq616na7dn5kagdxbk9x1nrmy X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 90F424001F X-Rspam-User: X-HE-Tag: 1708544481-51563 X-HE-Meta: U2FsdGVkX186F4x5Pi3y84uDMtI9V2SXzXI7LUk4z+jSsjIyS60xpHSDFsDufxRq0pKHEP3HqXlZ1NFw8AhykTssvde6P7mI5DFaDQWFt3aiRASwz+HJ130PgDQGEFsW52UCGslKPpU/DjxfguYmbZ+GciSUGCrwIkO5poxXsyxoU9oJgA1sXhuYkauUKLMUQkRkDl6F/9yNRpUvd/vjGxAfzUf9CY1o5b6sF6LMXKUGH9n2bF89HlhzwW3qvJMqKCcxzxu+7EaoZjEiOjgYzG9igF5XJoNclSg6W8x0ojwy5gVgndb0adMOGqzyPuPnsFzs3ajXGBtCYb3OJC6aqZHnoEEAVahkWavyabQnfYrNFRew2SxkyJxZ3Bo5Gf63zb6+xyStrrQRua0v9yxiYbyKefaMn8HuytZwtM0bHK0HjYcYSi6V5wN5Ux90clawPKUaWUdqrCX7ywY7N+aA/vW0MW4WYtLZOQwS3H/c6OKjVunIcA9buLg64hwC60wWgZXnrYVlllGFgHkcD68K/uAuXCJ1u3cdH+lLz3wK+io9/GRc3EJ9972Zst3l9unmKjBLc9eVq209zd8i751GccGVIC19fjh5Y0giAVN1/Fwnsxb0nKVWiCEKDzoOJY6GCdlaRvtOkhFjXLGcSv701Mkq2O/wJyYQpbaD4kVcTHXDfeOz9LfZ/Pw/77z5Gk+YSMezMi8rAL3Gkyr7wrDGYazqsFYYqslN6JRp8MC/kXtTxDvWwlt1YS9XOSUNiWUfVmwqTlqnvZY+CN3u0JTGD0Y+zcgsAbpKVpPbCTWsxD5Zf76i7v1GnboJxmatsqBozi9Q0Wwiu6+PbDhcKqb8stNofjJ1RXffEPLUagIy91cPukscqT8SW3VfZY/EU9zcB8uTYUld9a/BRXNhDXh6hFrw0FZwsVxDl5cQ9atomcm1a1b257UmCEiW/XfjBYNYmcQAw/IOlwE1bi6R5LX jEk32pRG xxxyOjLEwVdG0NTZdNQ81BJiZFiWGbikiqCQymArkO4B5XQ01nvU+cSOVMQF2NpCqD1SfFzqUmzQZ3/iO+c+hxwLGS/eSYgzeUkJOtHw9D5HBr7SsXnoAakTTxJHQ/pTsqdGggscP5yl4nPr4KX0vgLwBEIgJXT6PjFPLONfoOxNf4RmyGaoju8SRH72Dkew8K7EMfd53VFTHqTOW6J1BaLkOpgz/WZwiniI/PGd6hnIjwUCXYBNQtdR+TrYLqd41RrngTThSnz8lx05G0602xLREppmLXK/rtGuAJ07V2ouxe8TkmuzW7f7IQSS1eCOXb/qiVmPlLSAmwrb97HNTe8Ayyc99UKGTOQl903aYifr3PTv0Jy9y5VhBlVwYmNTSkVarUVwVPMOCnaralw6wzFAigLvj4dJ8EGAJ3dLSLEuXgLTn+dhnh8BgpDTcNkTt0HE7HaPjW5d302owQwiN0U54RqynxOsas7SlNvokRfCxALLw5IAJdg3mH3ly6Ww4J1uAN7GAzX44P1DHKYMnXztqTCfvjtSXRzrl1FshU10yxg3HjcUrt2Rbl7Pj59cu+7JtHb5fCeUsADYvebl8LFIQURyl/bOciUYSNukddEWPUiQcnWMjTCW2MlqHId6aEXClMuFQsoVTnpxLIadFOnCNEL33+vnS3NEYutmPbf2GNXI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 68 +++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 219 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c diff --git a/include/linux/codetag.h b/include/linux/codetag.h new file mode 100644 index 000000000000..7734269cdb63 --- /dev/null +++ b/include/linux/codetag.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tagging framework + */ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include + +struct codetag_iterator; +struct codetag_type; +struct codetag_module; +struct seq_buf; +struct module; + +/* + * An instance of this structure is created in a special ELF section at every + * code location being tagged. At runtime, the special section is treated as + * an array of these. + */ +struct codetag { + unsigned int flags; /* used in later patches */ + unsigned int lineno; + const char *modname; + const char *function; + const char *filename; +} __aligned(8); + +union codetag_ref { + struct codetag *ct; +}; + +struct codetag_type_desc { + const char *section; + size_t tag_size; +}; + +struct codetag_iterator { + struct codetag_type *cttype; + struct codetag_module *cmod; + unsigned long mod_id; + struct codetag *ct; +}; + +#ifdef MODULE +#define CT_MODULE_NAME KBUILD_MODNAME +#else +#define CT_MODULE_NAME NULL +#endif + +#define CODE_TAG_INIT { \ + .modname = CT_MODULE_NAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +struct codetag *codetag_next_ct(struct codetag_iterator *iter); + +void codetag_to_text(struct seq_buf *out, struct codetag *ct); + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 975a07f9f1cc..0be2d00c3696 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -968,6 +968,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 6b09731d8e61..6b48b22fdfac 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -235,6 +235,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/codetag.c b/lib/codetag.c new file mode 100644 index 000000000000..8b5b89ad508d --- /dev/null +++ b/lib/codetag.c @@ -0,0 +1,219 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include + +struct codetag_type { + struct list_head link; + unsigned int count; + struct idr mod_idr; + struct rw_semaphore mod_lock; /* protects mod_idr */ + struct codetag_type_desc desc; +}; + +struct codetag_range { + struct codetag *start; + struct codetag *stop; +}; + +struct codetag_module { + struct module *mod; + struct codetag_range range; +}; + +static DEFINE_MUTEX(codetag_lock); +static LIST_HEAD(codetag_types); + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock) +{ + if (lock) + down_read(&cttype->mod_lock); + else + up_read(&cttype->mod_lock); +} + +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +{ + struct codetag_iterator iter = { + .cttype = cttype, + .cmod = NULL, + .mod_id = 0, + .ct = NULL, + }; + + return iter; +} + +static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) +{ + return cmod->range.start < cmod->range.stop ? cmod->range.start : NULL; +} + +static inline +struct codetag *get_next_module_ct(struct codetag_iterator *iter) +{ + struct codetag *res = (struct codetag *) + ((char *)iter->ct + iter->cttype->desc.tag_size); + + return res < iter->cmod->range.stop ? res : NULL; +} + +struct codetag *codetag_next_ct(struct codetag_iterator *iter) +{ + struct codetag_type *cttype = iter->cttype; + struct codetag_module *cmod; + struct codetag *ct; + + lockdep_assert_held(&cttype->mod_lock); + + if (unlikely(idr_is_empty(&cttype->mod_idr))) + return NULL; + + ct = NULL; + while (true) { + cmod = idr_find(&cttype->mod_idr, iter->mod_id); + + /* If module was removed move to the next one */ + if (!cmod) + cmod = idr_get_next_ul(&cttype->mod_idr, + &iter->mod_id); + + /* Exit if no more modules */ + if (!cmod) + break; + + if (cmod != iter->cmod) { + iter->cmod = cmod; + ct = get_first_module_ct(cmod); + } else + ct = get_next_module_ct(iter); + + if (ct) + break; + + iter->mod_id++; + } + + iter->ct = ct; + return ct; +} + +void codetag_to_text(struct seq_buf *out, struct codetag *ct) +{ + if (ct->modname) + seq_buf_printf(out, "%s:%u [%s] func:%s", + ct->filename, ct->lineno, + ct->modname, ct->function); + else + seq_buf_printf(out, "%s:%u func:%s", + ct->filename, ct->lineno, ct->function); +} + +static inline size_t range_size(const struct codetag_type *cttype, + const struct codetag_range *range) +{ + return ((char *)range->stop - (char *)range->start) / + cttype->desc.tag_size; +} + +#ifdef CONFIG_MODULES +static void *get_symbol(struct module *mod, const char *prefix, const char *name) +{ + DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); + const char *buf; + + seq_buf_printf(&sb, "%s%s", prefix, name); + if (seq_buf_has_overflowed(&sb)) + return NULL; + + buf = seq_buf_str(&sb); + return mod ? + (void *)find_kallsyms_symbol_value(mod, buf) : + (void *)kallsyms_lookup_name(buf); +} + +static struct codetag_range get_section_range(struct module *mod, + const char *section) +{ + return (struct codetag_range) { + get_symbol(mod, "__start_", section), + get_symbol(mod, "__stop_", section), + }; +} + +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) +{ + struct codetag_range range; + struct codetag_module *cmod; + int err; + + range = get_section_range(mod, cttype->desc.section); + if (!range.start || !range.stop) { + pr_warn("Failed to load code tags of type %s from the module %s\n", + cttype->desc.section, + mod ? mod->name : "(built-in)"); + return -EINVAL; + } + + /* Ignore empty ranges */ + if (range.start == range.stop) + return 0; + + BUG_ON(range.start > range.stop); + + cmod = kmalloc(sizeof(*cmod), GFP_KERNEL); + if (unlikely(!cmod)) + return -ENOMEM; + + cmod->mod = mod; + cmod->range = range; + + down_write(&cttype->mod_lock); + err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); + if (err >= 0) + cttype->count += range_size(cttype, &range); + up_write(&cttype->mod_lock); + + if (err < 0) { + kfree(cmod); + return err; + } + + return 0; +} + +#else /* CONFIG_MODULES */ +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } +#endif /* CONFIG_MODULES */ + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc) +{ + struct codetag_type *cttype; + int err; + + BUG_ON(desc->tag_size <= 0); + + cttype = kzalloc(sizeof(*cttype), GFP_KERNEL); + if (unlikely(!cttype)) + return ERR_PTR(-ENOMEM); + + cttype->desc = *desc; + idr_init(&cttype->mod_idr); + init_rwsem(&cttype->mod_lock); + + err = codetag_module_init(cttype, NULL); + if (unlikely(err)) { + kfree(cttype); + return ERR_PTR(err); + } + + mutex_lock(&codetag_lock); + list_add_tail(&cttype->link, &codetag_types); + mutex_unlock(&codetag_lock); + + return cttype; +} From patchwork Wed Feb 21 19:40:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99B23C48BEB for ; Wed, 21 Feb 2024 19:41:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8EFBB6B00A1; Wed, 21 Feb 2024 14:41:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 82E7D6B00A3; Wed, 21 Feb 2024 14:41:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 650116B00A2; Wed, 21 Feb 2024 14:41:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4CAA06B00A0 for ; Wed, 21 Feb 2024 14:41:25 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 24C1940B48 for ; Wed, 21 Feb 2024 19:41:25 +0000 (UTC) X-FDA: 81816830130.23.B1119BB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 675774001E for ; Wed, 21 Feb 2024 19:41:23 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XuMV7Z2n; spf=pass (imf01.hostedemail.com: domain of 34lHWZQYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=34lHWZQYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544483; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XR/SP3Bk5ahA1k2JZ7LXmVl+aTAtC05e65Lb1fPVJe4=; b=RkWj5ZS3Gq3iqR+AyfoNbuTFpwEkFt1V1K4NHwZr74lL292YHcY4RlY+CmEiR9e2+dW6bx J66jy44WDJAvNfkGJhC26eJY0EHBuOtnAjQz3vZmYtEiyIN3UrDX99caIKlxmoPw9sB1b8 bqmXJUDjad2wxClsr2o/zgznqzap/Ug= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544483; a=rsa-sha256; cv=none; b=hZA6YXMwKkIK5PKa3pSdAC4bfTMUCAW7B/UAJlyJpqaxnWbYOa1ZHveRK92U6N7WVGd9Rc Kt+xcz4VQEbvLGm9q8atRnIo6GDlAQhAenFkjKKPN90O0VCXRIWjYkqJ1m7ERZPym26JKv t+WqhanEIKRKjnZkmfs0krMjhhhtmYE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XuMV7Z2n; spf=pass (imf01.hostedemail.com: domain of 34lHWZQYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=34lHWZQYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6088fa18619so9936067b3.2 for ; Wed, 21 Feb 2024 11:41:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544482; x=1709149282; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XR/SP3Bk5ahA1k2JZ7LXmVl+aTAtC05e65Lb1fPVJe4=; b=XuMV7Z2n6ZQDiNca7GDZqnDCTIsBmCpCKXGkTzIhhIy4JGWNmJ7/nq+CHLccjKdL2Z O8tNir2EI6F3atvmtX01sW4Ew1YX60fKhZAHKJzEITkjsao8aSOy1fguVjCNDEzfsw4X 132fv73go7COZqpznjuh7JoKSK/4KQQ6uSQTnzZLLCyvb54QLY7iP18zta0SnOGd+8mZ MhrPdlXoO+OI2VwUUbGFj4X4NqnQ7ZAhRNmcvq71/0OUfmJ9hUbBUEbgdWUZz6GwbgCB aXDSmv+q/Igm+C0l3lsqG2cjaSDjDsx1NnJ75Ok5YPQHfZ7wPov1tVwbnIIvC9arMXGH Dz6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544482; x=1709149282; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XR/SP3Bk5ahA1k2JZ7LXmVl+aTAtC05e65Lb1fPVJe4=; b=vxbBHCWSFt5waHmGGKaHiHEwYtfSMV5+yI7mgn79R5xTu2Sg5MamZvCof7sWYLrpuB kNpbAclL7OOh5LP1Xh9ToymknNXhOvJwC8X0lSTWdAxe/Lcw1rN9gvFqOwv2iL5/fY7u SfG797yMxkRV+iNQ24XKDoYZBjEjXMFnCZzDOfMVXghs4adKLqDmu9wSSG+d7qM+b9n6 7HpPhgjYcJdjhkRtA99wmI1gmaNqnloy/f0aeeuNBub3fX7ddUxXyQgRSqLYmYhLSOiG y+OWEzpdXKJnz1AAUCHDnrzg1RmsoMwf7hybVfctYs3AVYc88rvvOX0lc3TfhQQHwr2I +b5g== X-Forwarded-Encrypted: i=1; AJvYcCWNbNT0lvu3njBlmwox4Ood3WfwS9zCz1HA1v4bM5iRDzWzdtyfBDjygAI55porhoxYN3VY3mtPYlafbAq4oxUdM4o= X-Gm-Message-State: AOJu0YxF1y9RBqcdWNDwFZHiNdHkCl6ULbv1bJsPzGhm2GtHGYbQ5I8b qyxuF36KyCulreiABpJl7+KmCNUFJVWDGjK8ryPUrGeJ3/yxGBAkpGHZAZOcNfgwkE+T688wP33 X2A== X-Google-Smtp-Source: AGHT+IEyG+wpxFtVFmzBeD/y9m0tOzDM43NttM6Pb+slw2U8N4KR3QxBKfz+mbBGKekgVDUk4tSk0gcLUqs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:910:0:b0:dcc:53c6:1133 with SMTP id 16-20020a250910000000b00dcc53c61133mr14837ybj.13.1708544482428; Wed, 21 Feb 2024 11:41:22 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:25 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-13-surenb@google.com> Subject: [PATCH v4 12/36] lib: code tagging module support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 675774001E X-Rspam-User: X-Stat-Signature: m9b4japc3rsrwya3kdh4qycoz9qxx8o3 X-Rspamd-Server: rspam03 X-HE-Tag: 1708544483-772951 X-HE-Meta: U2FsdGVkX18tf47HcHdJzCQZvGYUtoahnEpdmwJQCIABpq9bCc74i2ia53HWoLRfSPksxZ90NLzDOMd4M1r67MJ6Qn16LyZjlq1g8jsOJOMeScL4dEz0WbEJei1XMIfUosUB5Zfh5Rnxg6T3uV2sfwXdeg6MMr926GP8moRteTx6FX01L99lKP1KD7yupyToww89yx/PT5yBtkfkzh+5yM793WCl+WyPDqY9ggNB4EtlH/7RniqJGWlztOA15ocs/mD3LF4K5x+fkYe/RHDR06vRF7rr2XOAhD4jrwm/GS7foy2uzmKDeBI4vOfb4D1flkR7FVEEiMiWk3t+7JCTlnn+B02vRpWd9B8igUP5X0md0tbbOR1/IQz50f8EoVvEMFwhbCLFtH0sn0Vxquhq4C6x9kw3g9L6aVuRnZ0dkB7DV1Sntzu5y23lMK0WbVrILYEIdYrzPqkQG/ekMRa4c/AVRzP4bOkl6BsB/l92W/B9DijOE6v3uge94GRhzvSHJkA9V+j2oYdGBFgV8nVMxdr1Q/J1GDTGPD2kvmW/Jk0C/dy0ykLUdnk28EcCAL1tSD0ZaU8JUe1yQ2jBsadk6Xt4r99DUNve9g0loOtw+Xo+AUvYHzUjjjwdVL0px4kPkgMrhY+7s+g7+tZqb1RXDRpYLmkfoqwzhKSheuHUsmoh/3j0be8NclHZ63a2Nj/T1EXAUSi4fx7iAtO2C6EtHR9AmjtdOREgpRTeK36rMQCz9f+a8aTi8aUaLJi+G6nXfwP9xhFkS74B342USrikqfLsEX8VRW8ATYUtFdzot9wXJVuPWP8uYd3cgpK903y3DlRlTZlVYnPZbwQgPlAwUa0ErxfaapwpTN5k0ik/r7LHlzi5tVhuYbSmzkE6kr6p4rHceYEFR0C+ArdI0B8LTMo4sEjjZhabghZq3wjOtY6VxNFuYzBmeUzfTHCMHcmgL56c4JD3MB+dIIHoLvK z2vJbzF9 gAbHZdDpzEsN0Pgd5Ej77zGJ5cwuWfYA2lU2CV3q/24pFbloOOIi61rUDnQ3Syc55mWx/cMoI2F6/XP0gqbEhPp/eh/cuZ/r5MkhoF3HMKz9p/nqCa6gstXRq1BGQ5Y3ILG6lq6x4kxFkRXVImLAyDpaRpzGIJMqs3C5itds9M1ASE0XlmZeF1nvi6xaTB0KG+lK4/FxvtP2Lxqb8yXh5DkPKUVPcRlx18/L0N+ZccRiuSXlxd3hej4noOCX63p3uEwLU2NhUewKReyWtaGssd4x0KuTculZXay3PuhgEd7ibS72A4UqmAzpuzZ4f9CZFt0kEzkmbez9AP7cO3gNt34jJqR9f2d+Yn0G9uAoRfqqP81bOcss0ULpL7GiqS6yInMmE9ilgxtmjeYfcV8xDpIwNBlsvqauqaU+3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for code tagging from dynamically loaded modules. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 7734269cdb63..c44f5b83f24d 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -33,6 +33,10 @@ union codetag_ref { struct codetag_type_desc { const char *section; size_t tag_size; + void (*module_load)(struct codetag_type *cttype, + struct codetag_module *cmod); + void (*module_unload)(struct codetag_type *cttype, + struct codetag_module *cmod); }; struct codetag_iterator { @@ -65,4 +69,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct); struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); +#if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) +void codetag_load_module(struct module *mod); +void codetag_unload_module(struct module *mod); +#else +static inline void codetag_load_module(struct module *mod) {} +static inline void codetag_unload_module(struct module *mod) {} +#endif + #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index 36681911c05a..f400ba076cc7 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1242,6 +1243,7 @@ static void free_module(struct module *mod) { trace_module_free(mod); + codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -2978,6 +2980,8 @@ static int load_module(struct load_info *info, const char __user *uargs, /* Get rid of temporary copy. */ free_copy(info, flags); + codetag_load_module(mod); + /* Done! */ trace_module_load(mod); diff --git a/lib/codetag.c b/lib/codetag.c index 8b5b89ad508d..9af22648dbfa 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -124,15 +124,20 @@ static void *get_symbol(struct module *mod, const char *prefix, const char *name { DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); const char *buf; + void *ret; seq_buf_printf(&sb, "%s%s", prefix, name); if (seq_buf_has_overflowed(&sb)) return NULL; buf = seq_buf_str(&sb); - return mod ? + preempt_disable(); + ret = mod ? (void *)find_kallsyms_symbol_value(mod, buf) : (void *)kallsyms_lookup_name(buf); + preempt_enable(); + + return ret; } static struct codetag_range get_section_range(struct module *mod, @@ -173,8 +178,11 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); - if (err >= 0) + if (err >= 0) { cttype->count += range_size(cttype, &range); + if (cttype->desc.module_load) + cttype->desc.module_load(cttype, cmod); + } up_write(&cttype->mod_lock); if (err < 0) { @@ -217,3 +225,49 @@ codetag_register_type(const struct codetag_type_desc *desc) return cttype; } + +void codetag_load_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) + codetag_module_init(cttype, mod); + mutex_unlock(&codetag_lock); +} + +void codetag_unload_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + struct codetag_module *found = NULL; + struct codetag_module *cmod; + unsigned long mod_id, tmp; + + down_write(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (cmod->mod && cmod->mod == mod) { + found = cmod; + break; + } + } + if (found) { + if (cttype->desc.module_unload) + cttype->desc.module_unload(cttype, cmod); + + cttype->count -= range_size(cttype, &cmod->range); + idr_remove(&cttype->mod_idr, mod_id); + kfree(cmod); + } + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} From patchwork Wed Feb 21 19:40:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F716C5478B for ; Wed, 21 Feb 2024 19:41:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010736B00A2; Wed, 21 Feb 2024 14:41:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EDB4F6B00A3; Wed, 21 Feb 2024 14:41:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C907E6B00A4; Wed, 21 Feb 2024 14:41:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AFD566B00A2 for ; Wed, 21 Feb 2024 14:41:27 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 61ED7120951 for ; Wed, 21 Feb 2024 19:41:27 +0000 (UTC) X-FDA: 81816830214.06.5A1EEA7 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf17.hostedemail.com (Postfix) with ESMTP id C8B2140014 for ; Wed, 21 Feb 2024 19:41:25 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=H+HADhDS; spf=pass (imf17.hostedemail.com: domain of 35FHWZQYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=35FHWZQYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BvZ08b/3oavcBvAkiQZyOMSmbVmpa+4ihUtpMl+Uc6w=; b=uTRxaZyTFO/UfUt+0q6f2z7RUtkb2CgCu+usi4k2tyZjrv3sBWRsKtR1vHbWKA2nWDK9tk 3+ozKFLM4SZjVlSCcyMOFwYxQJ7b3zX7wGl+rIOVgTAeND/7t7oTEW04UtJJLmymEiG9x1 rei8XN6F4Lm0Yhn+QxNEb2toGEDaH8k= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=H+HADhDS; spf=pass (imf17.hostedemail.com: domain of 35FHWZQYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=35FHWZQYKCBcFHE1Ay3BB381.zB985AHK-997Ixz7.BE3@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544485; a=rsa-sha256; cv=none; b=xl542zgSYC1q1qG/31Zg588VmfXWSDMjnabHNxXetpTdYM/OwJLJ/A6Fzgcck/SWjp1mSN MLJHLIFNi0vEc+AWbZqr+SDvmzxR2JZSfaivyyy6gONjo/lU0WZKj7QuzTYxSMSYZBn2hP 1DCxHW/g8zn8/womDpG8PBP6PXWVt2o= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-608886b1a3aso15399287b3.3 for ; Wed, 21 Feb 2024 11:41:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544485; x=1709149285; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BvZ08b/3oavcBvAkiQZyOMSmbVmpa+4ihUtpMl+Uc6w=; b=H+HADhDSKgp1qOWPdQCkYLBZsmEFJh8sYQvawy5Jv30/IiTo+fcQaRk8BEK/wH1Su/ Pnp2GmD5S9pTlnwZlKJeJYsyAhAZxw7K1ftQPIzaE5nMrBGC/O2nFZeoS/u5nO8c0nOB cTIKbCUX24Oty7gTCalYoBSRzz+SZB2Rgt8uVyJHB3ZhPV3q25RWWeWwj3oPgcpfdisg 2FvbESpDdxraUQ5GQEBw9zqd3r+uqGQiVEb1ftjpO86df2Nmj2UoyCCvwwVb4O2KpHGm IR46nbS+R83rSN7OLXeKuz+/6zZLA4Km52V/9AfDKmciB9yP3C7DX0TDzrgTWtCK2Vlq VKMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544485; x=1709149285; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BvZ08b/3oavcBvAkiQZyOMSmbVmpa+4ihUtpMl+Uc6w=; b=Bo7TGxl+4BoLU8XD4BQ1MjN7vegvMZBFlQ2Ox7cP9pjORBR4jOPuqmeyEsQCIbPa3C vWMksHwv15nfyr1+92FR5h5FRbYcLav5/tkpVl+Dsr7gGZnwxHFz8yhH4MBLvUn7ZKF0 z+Wt/JmdFjI2iQ/bzBj7BymwPXkGCBpMXYTc6ObVxP6iDIHUfIhSVUpr/114Efkr+wNp T4IpQMma0zyX+QNGS1+Rwv/XUu0iCArrL3+Mn3YkXTaDmmivlC35aNg2LxHpYU1+hvi4 GpICSCLfXLNWOnk7QuA4cnyvMC1V5lMMdLfBOH+wNmDneVULMARp8XQ6Gn0x6Q/SCiLT yylA== X-Forwarded-Encrypted: i=1; AJvYcCV7XdH7dJid0ToOyvDSmVc92LRIkAbB3Q5kKqnqls7h1VDvvkdnDNDEfuYUE2eLI5tOHnMRmC/qIXl7it55wWhUuks= X-Gm-Message-State: AOJu0Yx2xlfrPZkNtH11PRePmEw8eU3gvdPN0Wf3xfzAwQPhhcLKCJn8 Zg8bYmNZlS4JC5VUDiG7clB0sqwvN5/pSU2NyZmuyPftUs9UGVpZiUvakQ5k5nMvjK4oK3YTb9p B7g== X-Google-Smtp-Source: AGHT+IFD0XqCfLl4qYutYla1ZL6XgTrX5S7AO9uCahzWsTmMPZ4BnZi9haYMKld7cdjfFju71XrA0wuJzw0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a81:7956:0:b0:607:c633:2997 with SMTP id u83-20020a817956000000b00607c6332997mr4958335ywc.5.1708544484752; Wed, 21 Feb 2024 11:41:24 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:26 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-14-surenb@google.com> Subject: [PATCH v4 13/36] lib: prevent module unloading if memory is not freed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C8B2140014 X-Rspam-User: X-Stat-Signature: eynm9f7oadut9quapjy9iuyuy7n3fs44 X-Rspamd-Server: rspam01 X-HE-Tag: 1708544485-974454 X-HE-Meta: U2FsdGVkX18fJ+XqYNsNYX4o6YFpJCohZLv3So8XmTV15/tM9AO1IMomp4OltyxJ2gMMFNUntSu9g8gD0D9sKIwX2p2fzwXbeUBgfDODlrpAl8is52TQzT26hfY/XwD2e1HLB+Ud9IJw2awyjU4NWfPhTN4U9rZP5p/cQkmUZCqWdbo3uC+S9zucFgRERzxQznrKAv16+flwLDuo0hGUncj5mgWnz79QoiIVbzMNHuu69yFprclCQRvKlRbMk0UyF7ca5x5EXOMNJXURG2gOMx4G0FxVcpF0mdQ8dNT5sGz2TJ17iNrfgqeSWujU8y+S3MKo7cV0z/QYyFOqRXFIGfmszDCa7QpPZ2ryn3Dp0PKgMeuOfmhTiCLx2pz6xtGEdLAZyMJarDl6ftn+Ewii+qdUyPCX311SoX9duBsJ+zMv3wL/UfNOBYaemM4fQP8HaQuap6tuI8GOh4BD91bFT6RUPMJgkssNHBcy38SWIE8gYhX0ZO88gDv14+n9Bg1OG3PojuAO6RPLADOCoCY1zfjREnQMeebthSg88OrbIAGnAJP2D6Va+FkSCqUn5TWJ+1h8gHlIkzX+V1jLEQFE3PWoCQuVzgwUKBkluEtsfbW+7lnjllrlojZ06UfTPFoY2SBsf88GTnm5woE8ZdgI3umkPkyQbSfvr5DCkGIr90xcgaBvp2zt4jOywj/4WNefdl+eJQWPFToEUesTLI+nhna5Bz3s/JKwp/2FEce5oj5rvnFom4CE2pz0QVZxBRytZxtABKzt809uvEGHbUn/0KEortzGVi1JKWwfO9daUkiH8swmMef/N/2udeX4bYQCyw8lUgex7aH38JvGuDUjY7WeTdbiwLbP60336mzfmiAwaQL9FEEQsiwsSBRK6EJ/Hz2+aYBtlhv+1knau2RVlcaGYyBwAEoLPmo33wegBA0HsDNgAEwKc+x84RR7Z/kxJmiTgSWXNsQufxbhp/n XCtcNOs8 xgbDCvLoANEDpTKbI7f0Ixk7V7BAuzK2gwNpRTNHN1bzCRH8wjM9LXYbS+fa8gatyEyaiM8RF/etLvp4lm3aPz+ONKOB3ETsPWWk2Jqky6IvfBBmXclWo22QiTT2+L4gyOSpIi5W+J7SC8XE7kdtIS04hzwFgT1PQBlvgdSLqktI56P9UcGXmZUuvyeJpKvzLXBa4o/dxnS9aFbW0YmmXX01LCM8my92cJKuPbTVGAwauu/AjdJtrAA8UschJ96nzu1mHkKqZ8pt+gWj7i3Hzl7lmXlhP28P/OCPLstN8/08OSS7wfbnjDksDoeq+a2ZNHPGqTFtaAGvLGJY+rfxk4/nH/9+062n28jt3dMvtdniHGiDYAOth6hfbkKrN1s9Uwg8PgI3PVYMFC+C8oe6/TUYSd4DorGgdgs8C2RgM+o1gOPQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 23 +++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 26 insertions(+), 14 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c44f5b83f24d..bfd0ba5c4185 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,7 +35,7 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - void (*module_unload)(struct codetag_type *cttype, + bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); }; @@ -71,10 +71,10 @@ codetag_register_type(const struct codetag_type_desc *desc); #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) void codetag_load_module(struct module *mod); -void codetag_unload_module(struct module *mod); +bool codetag_unload_module(struct module *mod); #else static inline void codetag_load_module(struct module *mod) {} -static inline void codetag_unload_module(struct module *mod) {} +static inline bool codetag_unload_module(struct module *mod) { return true; } #endif #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index f400ba076cc7..658b631e76ad 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1211,15 +1211,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type) return module_alloc(size); } -static void module_memory_free(void *ptr, enum mod_mem_type type) +static void module_memory_free(void *ptr, enum mod_mem_type type, + bool unload_codetags) { + if (!unload_codetags && mod_mem_type_is_core_data(type)) + return; + if (mod_mem_use_vmalloc(type)) vfree(ptr); else module_memfree(ptr); } -static void free_mod_mem(struct module *mod) +static void free_mod_mem(struct module *mod, bool unload_codetags) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1230,20 +1234,23 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod_mem->base, type); + module_memory_free(mod_mem->base, type, + unload_codetags); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); + module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA, unload_codetags); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { + bool unload_codetags; + trace_module_free(mod); - codetag_unload_module(mod); + unload_codetags = codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -1285,7 +1292,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod); + free_mod_mem(mod, unload_codetags); } void *__symbol_get(const char *symbol) @@ -2298,7 +2305,7 @@ static int move_module(struct module *mod, struct load_info *info) return 0; out_enomem: for (t--; t >= 0; t--) - module_memory_free(mod->mem[t].base, t); + module_memory_free(mod->mem[t].base, t, true); return ret; } @@ -2428,7 +2435,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod); + free_mod_mem(mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/codetag.c b/lib/codetag.c index 9af22648dbfa..b13412ca57cc 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -5,6 +5,7 @@ #include #include #include +#include struct codetag_type { struct list_head link; @@ -239,12 +240,13 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -void codetag_unload_module(struct module *mod) +bool codetag_unload_module(struct module *mod) { struct codetag_type *cttype; + bool unload_ok = true; if (!mod) - return; + return true; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -261,7 +263,8 @@ void codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - cttype->desc.module_unload(cttype, cmod); + if (!cttype->desc.module_unload(cttype, cmod)) + unload_ok = false; cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); @@ -270,4 +273,6 @@ void codetag_unload_module(struct module *mod) up_write(&cttype->mod_lock); } mutex_unlock(&codetag_lock); + + return unload_ok; } From patchwork Wed Feb 21 19:40:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00F47C5478A for ; Wed, 21 Feb 2024 19:41:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 567CA6B00A4; Wed, 21 Feb 2024 14:41:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 516E06B00A5; Wed, 21 Feb 2024 14:41:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F5326B00A6; Wed, 21 Feb 2024 14:41:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0FBF26B00A4 for ; Wed, 21 Feb 2024 14:41:30 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DBC65120909 for ; Wed, 21 Feb 2024 19:41:29 +0000 (UTC) X-FDA: 81816830298.17.C8C77C5 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf01.hostedemail.com (Postfix) with ESMTP id 1483F40018 for ; Wed, 21 Feb 2024 19:41:27 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lt1IGJg2; spf=pass (imf01.hostedemail.com: domain of 351HWZQYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=351HWZQYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JrVRrtoAzVQVUCD8y1+sVdT1Uw826vby3bEDKcnM+kE=; b=D6VUygk3e5XQw2LTuRSPNFwPui6+acesuj8WaNn5n+Q7AIwPHIwG5pCewnOLCoVIkIubKk Xu52bC3u3PibWmdrNLXD0FZzXKfOvDtMkGBV3GXTWyytJVAV8ftUMJMxv6qzB1zY3mQfIh ZZotLnI6Qy1JhdovNaC7NgXV1CNauNg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544488; a=rsa-sha256; cv=none; b=rnUWlNDVlTPEuerTCbOkHeDRJhbw/FL6nwNpV9DkR1ygxKE5r7wmte86cpJV6uwvsZSaPs zNfJLuoK5Jwj+UHLAGAHXnc78uPQFTCbnQEvJSoErxpYkiV2OSVH2CK7gc/4QdjEHYiWAt OLYsLRk2Dp0B+hMh4T+UzsVVdrY+XO0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lt1IGJg2; spf=pass (imf01.hostedemail.com: domain of 351HWZQYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=351HWZQYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608852fc324so16607857b3.2 for ; Wed, 21 Feb 2024 11:41:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544487; x=1709149287; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JrVRrtoAzVQVUCD8y1+sVdT1Uw826vby3bEDKcnM+kE=; b=lt1IGJg2TMWDF1KrZtEvPmA6ejz2IoyfFdGarMUh83q59dV4rfCC6fbKIdKL/vlXjU 8Jc50PJauy1+tfO3ARrP4eTWVVAR5aXEjMq+fVsrGimgP1Ddj5AMZ+IzfstDiuPhVRzo Cpdm6EhxI/qENBKGB4rpP9/bDD4yH+qrbxwjY7LqvS8gQJ9pFlUi6SUQHx5v7NFVCXPi fLqzVzdcbX6XFJ+E7/TI6rghPimAqdBh7ACtqFe7zvMM2cY+11Vp6gGgsnf5yOcByVgJ b7ZGsjv5u2sknL8PGupMwEKl+YPN1bc9eCTQBciFgg7yQik3mzHh1JgUXHl1oK8f5vmW JVlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544487; x=1709149287; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JrVRrtoAzVQVUCD8y1+sVdT1Uw826vby3bEDKcnM+kE=; b=oeRo1mJHtOaDc/p9a+hTc26CZJ5RSftOAbsQHUN1jMGR7D76iC2sS4XVPcLXZljB7x FKo47EKFukmvJ6L4kRAyoGvX6qFDbRIb2HgVDBbuzs01Jjw1k3Cg4qqAFcwnbaVp4ypS +yirFW9wbghnrdOLXBub7OMuym2WNjKFMIV7zb1nyutWu0nySYMLmApbPHIpi2Rk8mCQ QXmExe3Q0CryH4z9k/pzJRCZuEAVtP6WI+qD4jWoPVSYJrg9da8iMu9VzOOrAfD3YLiy MP9CYU9RvWmRM9BjJ6laQXVBuTTgPcNhocpMUMa/UVop7N+yWzFqIC0ZBr9u4qORQPLS /A1w== X-Forwarded-Encrypted: i=1; AJvYcCU6r6hXDeCQRPXCMUsqceyYAZsb0MgA/QVdporVEIAQgol0+CyVUi/DVpraj/YLVoeV0V2/0r3D9zT7iHiEs/ov5AQ= X-Gm-Message-State: AOJu0Yxx3L5zHhTzdGRYmc6vcYrxDKboxL1qzkGMynvE3VMK9SopFoH3 s2MkGML+Vap6rn65dIljPCQe7YzJTSzG0t6Clh+B2kj1q9ZQ3fj/1ggVQ93NNbEJyRL2YemZKSj TGw== X-Google-Smtp-Source: AGHT+IFmNfinkkKIzGGaCYKbYCAXNv5GWf58HZ/9Pf/444E+5bezStghE1u2DK+BzAlsJ3VPEO8FS39skac= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:df4b:0:b0:608:5e12:ba68 with SMTP id i72-20020a0ddf4b000000b006085e12ba68mr1532689ywe.4.1708544487045; Wed, 21 Feb 2024 11:41:27 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:27 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-15-surenb@google.com> Subject: [PATCH v4 14/36] lib: add allocation tagging support for memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 1483F40018 X-Rspam-User: X-Stat-Signature: 8m91ajsr5k3jn65jafjhpgx1crd93t1q X-Rspamd-Server: rspam03 X-HE-Tag: 1708544487-807409 X-HE-Meta: U2FsdGVkX1/gIhO65VnbQKhlqlwSASOupabppL1YG2qbXukp8iidLsPjvYUwPJRXc957WknwYbk8wqgh5toDawNZq5gZT52w0F85Y0tSwJA1sKJK35Uh8Sz8Qc70WdYt+y0AoXr8fQ9cn/MCwcky2CyzK7M/ZwEIpLmbUG4yNciFgbFDPg7b3PlAa5B4uqEKm57n5xskJhyEWHvAxw3GmvEoNQHFtnYQh48GTB3mNhdLEMj93FeNRDfnHHm4U9W2ak4TV3ZDGnvD4jfTmHs/R1WiGZZyR30sjZ71YcOu26UmLQVI+LSth2Kxr9SazdxnSxQcnu0KS1Q7MJjTtrSSP40nOlgxiZlSwvbAWpSN3jaQYY9ZIkzq7X0aoN7y23hIl1Z7rhbAkULmUUgveRlzmpUhI2pyebU8sq/cS5VleM6zbZOnrFUTW6vZFX+OuQ20Rl/6DdFnOl6ea7WzS36AfPxG7FDaaTNl7rw8eh0N3UKEoynMODpW40zt8Kha9nZK5YyGFthcCFWrLbjtVOE892oJ58cBZV2nTvAczbKBzQM5y2lWWELFjxMW3e9upfJWEl7k0b3zYxC5XCQ599OpznQqQNoity9Hg5cLfCD76sdlzqVeO8jvSgU2oJlYgg2u/nYsO/H+1NX35pQxUaksB1l1bnlW6JV9BoavzRxEiwRbogvtEIiq8A02t/vZ1eYRWY13ysfgkAhupBqrDVrpkI3dA7kHGF0c6Ux3RLCgUTgQTuTYOsNM4dLm69HDyxJcsRv0sd0OUh+yoRU0K1Em+T4WhUrrhspfNtH0HgInZIUWjodtphDDGRQRO6GFw6b6jpyZgNr4o5uYGJc2K4yRQY/ylVxWTVTljl1lR37CSoKrwODkrjSZ09iWfr0CSPhv4N0Ty+AhBjJEedTg3pAtcCd0+BhSAXUrz8kGZk6YVgxWRkvdBDLkJ3vXu9B0/lpgX0xv4S1O8WRepHbXaL4 VrurP43k GuldvJkWO9Z0IcmSVQAFY9nLLOcEYupzPywGbgHIE73V+CuxkHlCbH8zbiHta34DPhwpghNzdmoZWMR6Hy+WY4m2jzsktHsme0ur69G23RCq3rb6Z76Q8jVIia9LQ8JyYqmD5/Hs3SvlTxxnB6ci/wevmbgXw7nNRIzKpOcJzVLqKtGj8SeoKY5FeOM87UcC1vpvvAsQym+VHwiOsbVdkVezgRnL7aU9AGfPuriQzDzvElr6GUCiFFfpvVEbLcQfTsTTlnkWb2McXcYVX23T3xkwvMhL9DgoJ5MkbaZc9sBzUTuJDyHqDBm3riBCkDh4dx177/cmT7XD5366BFJxScNErh/QXFk/EPQBhHRbdW8gFt8hBlOrzDFqmfWKv4Qp7Ostd9rh+QjX6RH3D5xUbsX+LTw/YRaAtw0+/o0WdqfiJbId04p5lzigbbRqdgxxiNHMf4V/8YRCsMgW5W1v64TpeiD4IgMYzuSRxXShIw6fsTmKKWpxkw/fX9WhO/gzG0IWZOsMAXWdBGox/bZJ6yG20PRCxYAiL5+OZfOXMEF3Q/oeqK/HDFc23kBqwtUt74Vz9iIBRxGsmYYGTuups33/7chsgre2Tn89k0FIy9wSAa2reXpxm3zsEkRH2loIYBXoErI2fl7fko3s/JaoJFdzpvRRhUgiXi6cIn6Yg0uoeygsbqTu473+xXw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It registers an "alloc_tags" codetag type with /proc/allocinfo interface to output allocation tag information when the feature is enabled. CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation. Memory allocation profiling can be enabled or disabled at runtime using /proc/sys/vm/mem_profiling sysctl when CONFIG_MEM_ALLOC_PROFILING_DEBUG=n. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT enables memory allocation profiling by default. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- Documentation/admin-guide/sysctl/vm.rst | 16 +++ Documentation/filesystems/proc.rst | 29 +++++ include/asm-generic/codetag.lds.h | 14 +++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 144 +++++++++++++++++++++++ include/linux/sched.h | 24 ++++ lib/Kconfig.debug | 25 ++++ lib/Makefile | 2 + lib/alloc_tag.c | 149 ++++++++++++++++++++++++ scripts/module.lds.S | 7 ++ 10 files changed, 413 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index c59889de122b..e86c968a7a0e 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -43,6 +43,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- mem_profiling (only if CONFIG_MEM_ALLOC_PROFILING=y) - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes @@ -425,6 +426,21 @@ e.g., up to one or two maps per allocation. The default value is 65530. +mem_profiling +============== + +Enable memory profiling (when CONFIG_MEM_ALLOC_PROFILING=y) + +1: Enable memory profiling. + +0: Disable memory profiling. + +Enabling memory profiling introduces a small performance overhead for all +memory allocations. + +The default value depends on CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. + + memory_failure_early_kill: ========================== diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 104c6d047d9b..8150dc3d689c 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -688,6 +688,7 @@ files are there, and which are missing. ============ =============================================================== File Content ============ =============================================================== + allocinfo Memory allocations profiling information apm Advanced power management info bootconfig Kernel command line obtained from boot config, and, if there were kernel parameters from the @@ -953,6 +954,34 @@ also be allocatable although a lot of filesystem metadata may have to be reclaimed to achieve this. +allocinfo +~~~~~~~ + +Provides information about memory allocations at all locations in the code +base. Each allocation in the code is identified by its source file, line +number, module (if originates from a loadable module) and the function calling +the allocation. The number of bytes allocated and number of calls at each +location are reported. + +Example output. + +:: + + > sort -rn /proc/allocinfo + 127664128 31168 mm/page_ext.c:270 func:alloc_page_ext + 56373248 4737 mm/slub.c:2259 func:alloc_slab_page + 14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded + 14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio + 9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node + 4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 3940352 962 mm/memory.c:4214 func:alloc_anon_folio + 2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node + ... + + meminfo ~~~~~~~ diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h new file mode 100644 index 000000000000..64f536b80380 --- /dev/null +++ b/include/asm-generic/codetag.lds.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES(_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS() \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5dd3a61d673d..c9997dc50c50 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -50,6 +50,8 @@ * [__nosave_begin, __nosave_end] for the nosave data */ +#include + #ifndef LOAD_OFFSET #define LOAD_OFFSET 0 #endif @@ -366,6 +368,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \ TRACE_PRINTKS() \ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h new file mode 100644 index 000000000000..be3ba955846c --- /dev/null +++ b/include/linux/alloc_tag.h @@ -0,0 +1,144 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * allocation tagging + */ +#ifndef _LINUX_ALLOC_TAG_H +#define _LINUX_ALLOC_TAG_H + +#include +#include +#include +#include +#include +#include +#include + +struct alloc_tag_counters { + u64 bytes; + u64 calls; +}; + +/* + * An instance of this structure is created in a special ELF section at every + * allocation callsite. At runtime, the special section is treated as + * an array of these. Embedded codetag utilizes codetag framework. + */ +struct alloc_tag { + struct codetag ct; + struct alloc_tag_counters __percpu *counters; +} __aligned(8); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) +{ + return container_of(ct, struct alloc_tag, ct); +} + +#ifdef ARCH_NEEDS_WEAK_PER_CPU +/* + * When percpu variables are required to be defined as weak, static percpu + * variables can't be used inside a function (see comments for DECLARE_PER_CPU_SECTION). + */ +#error "Memory allocation profiling is incompatible with ARCH_NEEDS_WEAK_PER_CPU" +#endif + +#define DEFINE_ALLOC_TAG(_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ + static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; + +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static inline bool mem_alloc_profiling_enabled(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &mem_alloc_profiling_key); +} + +static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) +{ + struct alloc_tag_counters v = { 0, 0 }; + struct alloc_tag_counters *counter; + int cpu; + + for_each_possible_cpu(cpu) { + counter = per_cpu_ptr(tag->counters, cpu); + v.bytes += counter->bytes; + v.calls += counter->calls; + } + + return v; +} + +static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + struct alloc_tag *tag; + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n"); +#endif + if (!ref || !ref->ct) + return; + + tag = ct_to_alloc_tag(ref->ct); + + this_cpu_sub(tag->counters->bytes, bytes); + this_cpu_dec(tag->counters->calls); + + ref->ct = NULL; +} + +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes); +} + +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) +{ + __alloc_tag_sub(ref, bytes); +} + +static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN_ONCE(ref && ref->ct, + "alloc_tag was not cleared (got tag for %s:%u)\n",\ + ref->ct->filename, ref->ct->lineno); + + WARN_ONCE(!tag, "current->alloc_tag not set"); +#endif + if (!ref || !tag) + return; + + ref->ct = &tag->ct; + /* + * We need in increment the call counter every time we have a new + * allocation or when we split a large allocation into smaller ones. + * Each new reference for every sub-allocation needs to increment call + * counter because when we free each part the counter will be decremented. + */ + this_cpu_inc(tag->counters->calls); +} + +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) +{ + alloc_tag_ref_set(ref, tag); + this_cpu_add(tag->counters->bytes, bytes); +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +#define DEFINE_ALLOC_TAG(_alloc_tag) +static inline bool mem_alloc_profiling_enabled(void) { return false; } +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, + size_t bytes) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index ffe8f618ab86..eede1f92bcc6 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -770,6 +770,10 @@ struct task_struct { unsigned int flags; unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif + #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; @@ -810,6 +814,7 @@ struct task_struct { struct task_group *sched_task_group; #endif + #ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity. @@ -2183,4 +2188,23 @@ static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); } extern void sched_set_stop_task(int cpu, struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) +{ + swap(current->alloc_tag, tag); + return tag; +} + +static inline void alloc_tag_restore(struct alloc_tag *tag, struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN(current->alloc_tag != tag, "current->alloc_tag was changed:\n"); +#endif + current->alloc_tag = old; +} +#else +#define alloc_tag_save(_tag) NULL +#define alloc_tag_restore(_tag, _old) do {} while (0) +#endif + #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 0be2d00c3696..78d258ca508f 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -972,6 +972,31 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on PROC_FS + depends on !DEBUG_FORCE_WEAK_PER_CPU + select CODE_TAGGING + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance and memory impact. + +config MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + bool "Enable memory allocation profiling by default" + default y + depends on MEM_ALLOC_PROFILING + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + select MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 6b48b22fdfac..859112f09bf5 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -236,6 +236,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c new file mode 100644 index 000000000000..f09c8a422bc2 --- /dev/null +++ b/lib/alloc_tag.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +static struct codetag_type *alloc_tag_cttype; + +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static void *allocinfo_start(struct seq_file *m, loff_t *pos) +{ + struct codetag_iterator *iter; + struct codetag *ct; + loff_t node = *pos; + + iter = kzalloc(sizeof(*iter), GFP_KERNEL); + m->private = iter; + if (!iter) + return NULL; + + codetag_lock_module_list(alloc_tag_cttype, true); + *iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(iter)) != NULL && node) + node--; + + return ct ? iter : NULL; +} + +static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + struct codetag *ct = codetag_next_ct(iter); + + (*pos)++; + if (!ct) + return NULL; + + return iter; +} + +static void allocinfo_stop(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)m->private; + + if (iter) { + codetag_lock_module_list(alloc_tag_cttype, false); + kfree(iter); + } +} + +static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + s64 bytes = counter.bytes; + + seq_buf_printf(out, "%12lli %8llu ", bytes, counter.calls); + codetag_to_text(out, ct); + seq_buf_putc(out, ' '); + seq_buf_putc(out, '\n'); +} + +static int allocinfo_show(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + char *bufp; + size_t n = seq_get_buf(m, &bufp); + struct seq_buf buf; + + seq_buf_init(&buf, bufp, n); + alloc_tag_to_text(&buf, iter->ct); + seq_commit(m, seq_buf_used(&buf)); + return 0; +} + +static const struct seq_operations allocinfo_seq_op = { + .start = allocinfo_start, + .next = allocinfo_next, + .stop = allocinfo_stop, + .show = allocinfo_show, +}; + +static void __init procfs_init(void) +{ + proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); +} + +static bool alloc_tag_module_unload(struct codetag_type *cttype, + struct codetag_module *cmod) +{ + struct codetag_iterator iter = codetag_get_ct_iter(cttype); + struct alloc_tag_counters counter; + bool module_unused = true; + struct alloc_tag *tag; + struct codetag *ct; + + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + tag = ct_to_alloc_tag(ct); + counter = alloc_tag_read(tag); + + if (WARN(counter.bytes, + "%s:%u module %s func:%s has %llu allocated at module unload", + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + module_unused = false; + } + + return module_unused; +} + +static struct ctl_table memory_allocation_profiling_sysctls[] = { + { + .procname = "mem_profiling", + .data = &mem_alloc_profiling_key, +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + .mode = 0444, +#else + .mode = 0644, +#endif + .proc_handler = proc_do_static_key, + }, + { } +}; + +static int __init alloc_tag_init(void) +{ + const struct codetag_type_desc desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_unload = alloc_tag_module_unload, + }; + + alloc_tag_cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(alloc_tag_cttype)) + return PTR_ERR(alloc_tag_cttype); + + register_sysctl_init("vm", memory_allocation_profiling_sysctls); + procfs_init(); + + return 0; +} +module_init(alloc_tag_init); diff --git a/scripts/module.lds.S b/scripts/module.lds.S index bf5bcf2836d8..45c67a0994f3 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include + SECTIONS { /DISCARD/ : { *(.discard) @@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif } From patchwork Wed Feb 21 19:40:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0EB2C5478A for ; Wed, 21 Feb 2024 19:41:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 650DB6B00A6; Wed, 21 Feb 2024 14:41:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DC8E6B00A7; Wed, 21 Feb 2024 14:41:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DEE16B00A8; Wed, 21 Feb 2024 14:41:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 23EF16B00A6 for ; Wed, 21 Feb 2024 14:41:32 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id EC0FB40B82 for ; Wed, 21 Feb 2024 19:41:31 +0000 (UTC) X-FDA: 81816830382.02.2F9EBE1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 1252AC0023 for ; Wed, 21 Feb 2024 19:41:29 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RQvyhvv9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 36VHWZQYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=36VHWZQYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NsY6yp7iUYgjdAZV5EeYqd6Hb5QC4jbV+S+/t/cRTHY=; b=n8ee2f1XUMneSUN3tkPd+OqT187BX5tDeU0VVylHExbTQwdUA7GkwJ1gJhF29T6/f5d8jI ZkMrmToVRZQH99VKZ2DwJ7jyegNqdH2VmWoc0Wwkuv0EfZUgCdaG2yVMVZ7vD3c7/9qZjE CdbaNho83ppE63h98lJOqim4c5IAbZg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RQvyhvv9; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 36VHWZQYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=36VHWZQYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544490; a=rsa-sha256; cv=none; b=F+Y3TljaOLB2Czl6kqQh1xUme/us+X97Z91DryoTIfKpLPBs1mtz8h2eeM19kbjYnxCmWq n8kJ3zDiOExpQJz9eDQIViLd1ZNQB94C2drGAhLjhdL04NQS7JQEitTxXQeXP232UV43WN J3gXhF7mrOj6gMWfWFn8kCsxJNsLung= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so9722267276.2 for ; Wed, 21 Feb 2024 11:41:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544489; x=1709149289; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NsY6yp7iUYgjdAZV5EeYqd6Hb5QC4jbV+S+/t/cRTHY=; b=RQvyhvv9kcGLkdPK+RJQjpGqs85JldVFyyOPkCTWw9o4+Rs6nPielefKHmQ8S3NCte MqrZtY02FFVe9iDw8D0pJlU/goCwT4vJhWVSs9PCi02deTcoYqsaD+MM0zxgph2k9JTY VFhCV4+Rm0fxcaup5HsF/FQ237Y+3xgegDUAqA2kZ00a42x58jN5L8GMaZR8XQWZdiSz K2dDrw4L+oYM5irIqZ3OKRT0Xf9a7hci4qvwvVtqfZi/hl5Uee46jMY3fn2aZHfEbv+o SpgG0TrqgSICT3jeMmfYFoeRfLmoMZk6kB9ojHcFjy8zyXgFUxyKvZjZIhPoVXXgvm5o 1D1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544489; x=1709149289; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NsY6yp7iUYgjdAZV5EeYqd6Hb5QC4jbV+S+/t/cRTHY=; b=l30Fxt1xs0FoUpqnJv5HEVKXWar/qTCcyswjkYUWyLK9khbgsThHX8oxWAIBWKPcfA kOMzLtsDpAvlbZJ9/1pZgA01YJ56zhc3NjbWVRwgCsovLUd17kkotqWPFZmrNvXsce4d dXpGzxU8rIu8rpOU24n+p/pTtw2IL3Rk3CepvkAQzh+R2jn24XF+bsmvViqvoShLN0RO OABhxQ4LelNUlgzD5bdDVlE6tDGQM+veweYP2SLgqdtqARE9P8xSnRINjUfboE1NRMfS Sd7V/KifVCghZ1hTCU660ebAnDVS+fwkW66k9MiG3oQstEfZFSLxNGT48Khskv5u6opz 7YyA== X-Forwarded-Encrypted: i=1; AJvYcCXWIOCFGtArctcioSQqJqaDqydEWGUs1WE66giyzj45eyd1lnj0vmww8gzfBChulBwLKQ5jMgc0ybDqaAR1QNwv974= X-Gm-Message-State: AOJu0Yz42B94z2/ks1tL2LTd/0MtWltS36w40W6fzUzoVHEAondUs8y0 dpRPb+2IHD7la1ULtRd2hG6PAzmMymAu21jvDy1rs2BZeLs/CIujXf5VAUrMDuM4gXZHzIHFQhI fUg== X-Google-Smtp-Source: AGHT+IFZ5XVcmon2QJStstDyBIA3gzHqbB5IX2wfVdmT9vHWe4Ppvi0XQTFXMMtNZdjP18VCozhnu+gMIw4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:26cf:0:b0:dcc:41ad:fb3b with SMTP id m198-20020a2526cf000000b00dcc41adfb3bmr6925ybm.10.1708544489081; Wed, 21 Feb 2024 11:41:29 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:28 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-16-surenb@google.com> Subject: [PATCH v4 15/36] lib: introduce support for page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 1252AC0023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: thu17pmzr9zwschuf8zwuxxojw6rskhi X-HE-Tag: 1708544489-77731 X-HE-Meta: U2FsdGVkX18kgxyK9s9/UTTgL18YEDu4hrcq1irbhi3Y8H65Pz1gb96uLTj4rYLKXwr1zh47G/8f0ux20s3/BmZbsjV+XDP6+01wQeEC1hOEGnWxatMBi6vkDxq5dRA/qz1potGEN/HrjxS81wl5LNyEA8hsXj4ktaoCc70CaFZ5bIQPoeQRRbP2uFhUpbIgPB1hNK1H/wy+ht8oQYjb5gBSS7sMcvezEsxW20qJDwUbqexmfhLEp3wDxexJv/MeFphySJGruI69nNJqcJnWz3GrKSpGzhKaCCAz1KHO8njvjv+pibyW5+d4/JFcUTo39z/0pCNEfNjgzvp4aMFpLUYtHocJIs5q73FMyWFA5w9Ep7RfWarTlLimvHsh+0MeiZcIImcjQkdd5ldacef4BZdUGs6BS/wQsd1w2Mjs7l8Bwq9nZnXPKyGiY26SVx4qyjUImR7KSZo8plXKjkdtWmdOiQj7y1FEICsGMToHWXgZDM33LFoLxy8arqV55a3D9Zxz5ayogcDmmOapG3z/mNC789X/k2FcnA1Xu9HfHHoBtIdk+XvW44dF4yceRXz4w/RNMsxV0UD9OWNwPpGGTUEwQus1PznwMR0hhH+TDUPrEiYsYeh8wXwaNm349YGTaZSY0JUpdAtVAauqX/TehNGQS8yrRvlaZGaMJ1kaLDfaYk54Qapmg/khD7U9NXdbDQLsyECcxPZBuxOW/BUMrvc1glJ05ly4ATRZNP7mxYL7Ni5s2uBncQ4nnEe+S/N6LVJQ/V+dgzTDCYYDVmME5wkPhpU2KtJZMiy5x+K4fbiI9sd4hlTgsai+imdIzOJb4rOECdSjIDnxmFXL98Tz7ThDCc/Jipov0+ogEgwCtaTUKDRDNN9AFILf4Ahb672BPEzw1jxxalretcg2MMIDw2/ZNAqa6XlPiue6Rm+icrzt8z76hzk/oYE7eoCDXxf3sGEpemfNg8KQWj5GbUn qi+9LjU5 ZTErAvf7wzZLhCBb7o0C8zyFmOM00wakbEeUAsg1ktsvKcurUFtVO9/QzGfu6+tzwm7p6CXSCseCE5fLUHXvPQo3+MzEVVlTqzlYCusQ54lMl9llJGpiA5PkvsSqScK6zSh5U5ja+DJ1G9LAfVGINQJiuy5wtiT9P/lPnsiiVXQQbLe+We9LJsjwJMay1VJs4/F3SJwf8zVhaMA3hQRAXd9LoAbCG0StKiZCiv1QNn0U4IFvfA2+XJ/+9EFNZqjtU2DylxznL2fy08m2cZOVrwY+0zgA2DsRybdtpn+AvkchA0clYLWo1qVRjy9DpeamtwkMtwYMEfnzLWmUizB3kUEhcy0qHLlNHvY5IC8B9/PW0S0gtdxcQIwuuRWLUy4bTIOh/3vADlrP2pa0gz5rZrkBet3j708sdYtHWGxLW+ju8VBkc1BYsixpnPFm95Fu8jcWZcS4mukEqGYf+4mpoA+KJbumyyYByGWguY/po4YYbU61SnHGRMvLEa7C/TO3iZeCN58da+uGB0WBhU9sDLjnZsGM7qcAVDr4Cvqd5jR92xyUeA5OJo40mMyP4lR57lZke5pJhiVU4+Hi+VD2SlvkQvJxx1nl42nsGnWC48Xz59YpARnmXg7DUodFgznlDHul/waxtISDvpJnyCvLNUXWrPp29p83lOr2k X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Vlastimil Babka --- include/linux/page_ext.h | 1 - include/linux/pgalloc_tag.h | 78 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 ++++++++ mm/mm_init.c | 1 + mm/page_alloc.c | 4 ++ mm/page_ext.c | 4 ++ 7 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 include/linux/pgalloc_tag.h diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index be98564191e6..07e0656898f9 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -4,7 +4,6 @@ #include #include -#include struct pglist_data; diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h new file mode 100644 index 000000000000..b49ab955300f --- /dev/null +++ b/include/linux/pgalloc_tag.h @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * page allocation tagging + */ +#ifndef _LINUX_PGALLOC_TAG_H +#define _LINUX_PGALLOC_TAG_H + +#include + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +#include + +extern struct page_ext_operations page_alloc_tagging_ops; +extern struct page_ext *page_ext_get(struct page *page); +extern void page_ext_put(struct page_ext *page_ext); + +static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +{ + return (void *)page_ext + page_alloc_tagging_ops.offset; +} + +static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +{ + return (void *)ref - page_alloc_tagging_ops.offset; +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) +{ + if (page) { + struct page_ext *page_ext = page_ext_get(page); + + if (page_ext) + return codetag_ref_from_page_ext(page_ext); + } + return NULL; +} + +static inline void put_page_tag_ref(union codetag_ref *ref) +{ + page_ext_put(page_ext_from_codetag_ref(ref)); +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) +{ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE << order); + put_page_tag_ref(ref); + } + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) +{ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_sub(ref, PAGE_SIZE << order); + put_page_tag_ref(ref); + } + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 78d258ca508f..7bbdb0ddb011 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -978,6 +978,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index f09c8a422bc2..cb5adec4b2e2 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -115,6 +116,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +static __init bool need_page_alloc_tagging(void) +{ + return true; +} + +static __init void init_page_alloc_tagging(void) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size = sizeof(union codetag_ref), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL(page_alloc_tagging_ops); + static struct ctl_table memory_allocation_profiling_sysctls[] = { { .procname = "mem_profiling", diff --git a/mm/mm_init.c b/mm/mm_init.c index 2c19f5515e36..e9ea2919d02d 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 150d4f23b010..edb79a55a252 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -1100,6 +1101,7 @@ static __always_inline bool free_pages_prepare(struct page *page, /* Do not let hwpoison pages hit pcplists/buddy */ reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); return false; } @@ -1139,6 +1141,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -1532,6 +1535,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); + pgalloc_tag_add(page, current, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, diff --git a/mm/page_ext.c b/mm/page_ext.c index 4548fcc66d74..3c58fe8a24df 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include /* * struct page extension @@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif From patchwork Wed Feb 21 19:40:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C693C48BEB for ; Wed, 21 Feb 2024 19:41:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3214D6B00A8; Wed, 21 Feb 2024 14:41:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CC9F6B00A9; Wed, 21 Feb 2024 14:41:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1473A6B00AA; Wed, 21 Feb 2024 14:41:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id ECC266B00A8 for ; Wed, 21 Feb 2024 14:41:34 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BA35514083F for ; Wed, 21 Feb 2024 19:41:34 +0000 (UTC) X-FDA: 81816830508.08.A845B84 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 434B840020 for ; Wed, 21 Feb 2024 19:41:32 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tY6MEn6W; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 361HWZQYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=361HWZQYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GoE6bGSaoBD7i3pA8lR/Run1pysn6CP5hI9CoxzKmn4=; b=goDDHQQ+D+h8NxUXUn8GqqN5zrN4lnbY0mdG7AX5SXeVnCPljYp4ExIwwdvhHUlgzk2NGu xn7lm2YwYUUaEkjmwoPDiIjuehFPg6yQ7PdjdxcETi42Dq6s1D9gqt7QI3yOPgWzm4/iQN mM5O7E3jnFo9Vwr3U9y2HaJGl/VCO9M= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tY6MEn6W; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 361HWZQYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=361HWZQYKCB4MOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544492; a=rsa-sha256; cv=none; b=cl70BaBQVlyit3MKk52+qqKqb77T+gFMjpYvTtb2Q3cEyR47z2u0ffCjDYwUnpm+lY05Oc tUBL5C1/Q1/8qM7NXLnT+cRjIwNP+GSgtRNijDGlT61u5DgtuD5DQ2Bqqm64swtHSqI4si JN5v/QoWy8zSWeblZHpM6vkJhX4+soQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60802b0afd2so1238167b3.1 for ; Wed, 21 Feb 2024 11:41:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544491; x=1709149291; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GoE6bGSaoBD7i3pA8lR/Run1pysn6CP5hI9CoxzKmn4=; b=tY6MEn6W5Di4gFKovneyI9sLQcnFvyjO6kj/4tIPYoMOyO77fN+PvdSTwc1ftH+qWJ QfdcmeFZTBKOQffk96NsdmpN4jOCahvolnc1cApyDEip0E0WRUKw9vWdCjGhos38qCkI tThXEM8S/G7Iwnja/779Gb6wbkGo6ffwFPPMM3i3g4In6G5Ew7yjL0ZE9XootaAhiLML Awg02ZCbzdpNwR+a9akPvBnxPxtp/L5UD+2En3YVrwAQmgpGGajEPvI27y9f/8vUV7bc +wX436Lu06iB3ejah0gt3+B/rXvWsDs2qF7MDkxXrlL4btWze466a+RPvVqoQpxQ77et W5FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544491; x=1709149291; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GoE6bGSaoBD7i3pA8lR/Run1pysn6CP5hI9CoxzKmn4=; b=v9bF0Exb00bpp3NTYyFpeSVCrvdqyMn6Y3p+Ty9u2Fu/wc8yTiRwg8t0oFskmGPDxW kpLVVklE+9l5a89b7yMEa6YRajOhpVawbYmh7p82UTRFdgFoqorevgSv8vyvtBkfUEx3 zxxrbfExlscZVFMDpYBbCArieeeMuD8ljw3LvSstKVv+wWpSA+w82gEeEoDfG8cpbEsR gHsYiO74mXUXv6UKAzjZNlXo13v+3FbZD/0tl7eXJ4J0SX+EJmBa22aIQvYyHrmohdzX dCOmNMtc2pQ3R8en8XvumiaoStULLRmKLQWjfX3mejpejGiFX/ok6gIl+LPPaAzevpM5 f8og== X-Forwarded-Encrypted: i=1; AJvYcCWkiD0POuE9qbIpI2U/ff6CODb0ONYFGoxjXaxeP7BARtYfnYvTvhD+GzbLfsZ9mzApSbDwkEi56+zO3SIjRgfWZvg= X-Gm-Message-State: AOJu0YwHTkMWH11qo3w3T4l5HVMdPFd8W6iuziIWjrg4gqyXj6pYevbD PKR6yRXRg2KbyYmwHw2LMXPNVKvv9aqCeY2//hJk97c0rUHU+LsJonw/zBzbqfC7AlI+kjI7QWS j+g== X-Google-Smtp-Source: AGHT+IG2dizM99WzNXjXCm4NRVCQCOVTKSh+Pv+zsrsy5Io1Fqept1ZOsQ0OERnmDAPF2nVohKm6aRgWd5Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:690c:fce:b0:608:7c19:c009 with SMTP id dg14-20020a05690c0fce00b006087c19c009mr105006ywb.0.1708544491160; Wed, 21 Feb 2024 11:41:31 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:29 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-17-surenb@google.com> Subject: [PATCH v4 16/36] mm: percpu: increase PERCPU_MODULE_RESERVE to accommodate allocation tags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 434B840020 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: g53pawoz4wpd9kqkfd5cfb89sbupycja X-HE-Tag: 1708544492-63358 X-HE-Meta: U2FsdGVkX19ne49ousyurG5TMJU/h2G1Dwm/tE0P0ZqZClu8U2H5HLzLQAks+9XsNcLJfiM9QQsWNsQ+XtymwMjkJ0mi7DWc0UpBFhh6H+GgZtEo3kwuczG9u1AbRVzDAFIEJFX5Io/P8Bi7gp0B3LGTHDLgtSKk80veUCGXZWcLbCLyWP19lxOCflcBHvuQfnC5ZFOst0Nxz3OapG4+JnwfQJFCzGULJxrISwe5sGOiGAjxPNULYk02FdxMdLSgrAoEVyAWYP4OSIO7dDUo/GfQdcWLA6CxfLDk5/E1x3CMCCMdc6AZFdJ9JGSKDa44JnsYHluLUXSqqQPSZRVGcOqT13qP4tE+Y8KY1eqMkRN4xONLPzQ/sXx9F+2mU5ydx9Dcu6+Cejiddy3NLuxO57iAPUs8yQhtLSgzipIOIb1SuTmw8ilOHef6uEKuU6ZdRfuXcNj0faksp1bb8FyxT+0UoYLomaEJOZLpxNbAiUA/atxCoLQkewmJ5Is0D8HtTSzrJrkwdM8GqxXgX6pwAziBP5tghJ3h2PDv3yA8Hw2ykW3Ief148hPmV+WBK5wgxiODOhD5WAunNBcMV469yepnQ5HFNgqKy9eAykbL3TXQ/Y/YPqVYQGe9WB6qRzBLdlQBK4AdtIxMaq2IPAGpAWhY9dw3mNPqOV4dYGUhhKCVINohf1T4KnPdneYiAcJ82u0jUAMw2VysA8ywmUSQmrzn6An8/7jlKlYsGSyEVz34k2aTDvvr2/E0O4CblBliQdW5Nag5SDk667kSZ8tit1HrUFLSih03hrlS2gvjVdHRBZVCuk9q732IcYhgceCRx8heUhaoDI8VlEXMVKCR9i/w+BYxnLEKmfl20Kv9r31hLk+5/Y2ieDgccl9033XRP4oVbR2QLXqwTjuVmYMGwvdrBLaY+Qi3/HjYyztqeyzFaMSt996DxS9bbnQw9OE4Ty4ydkD9ouum982PfmR qhkX93E7 LCebLhMBJWw2C/QUo2xu8nY/Px7CGxLGHaxi51y5xa6Fp8dKuk2zJnjSVIi6JnHx15oxvZ6nMgm1Q/4yoiwFwrULpsiFlnr0o532t5p0cg8HdPju4t/2wl4JHxGkgcyt0c5UDqaxH4LS6cPSe++TtTFjQXG9qBdVvNKqNzeBSfMCQB7GhJvCEBlXTus2+S2scgUlwcA80Dw01QDDKRbe5WWICzL2dOVkBGH4vh7uCTVbFhtkr+ckNfcHSb4IRH/dP2yr/ivAW0Oe9J6t1Y7WkduZYVYudq24XsDh4T4bMWklo9hPWOJTsdHORYi1fbTD7x9aQIszqsDWNngjUJyxtyI7N7kAqDb7iXPSkkGxCCkOBYcECnwcUrioSBIJbpu7M2iZXZA5bsFq+MsGwjIRhUvLe5rxA1nyVdvgpI4zeG7sj0bvOq+ajsnXqvfK+7DkR3afUFvtgqMzTjUTFu/rVkaDY/NyVOyJy26KgqXAanwiPbLXcGCJv1L+yg2bq8qQYA48Jm1pn0r6kbB98g/M/OoYcsKid2lTwWXApWK6pPDqyz8H7HqsLMfLrVo7iVl/T/ko/44Irop9+JZ+oP1uZ/RpiIRAd2o/0nN0cMy+Po8FffY4YdV9+gL5tJFg8ALTpQwQcRhBTyBA5Uw/0N5Bxa9CQLbC6U7vo3CtdsnPxwBniTnQhO1SLviftE/TyvUm+v0dR6sDgq+vQDQvPrX4NkztdWd3tUIbnGDm0wgtn60Th5gHNUAa8UffdCKK/ixpt6RtT/n4JHw+y2+AubPizyk2NUSrzk4G9GhGEKQ+vbzKDC6oIuKr/gDDDriiJy/8nq6iEAY+QWqSJLkA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As each allocation tag generates a per-cpu variable, more space is required to store them. Increase PERCPU_MODULE_RESERVE to provide enough area. A better long-term solution would be to allocate this memory dynamically. Signed-off-by: Suren Baghdasaryan Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Tejun Heo --- include/linux/percpu.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 8c677f185901..62b5eb45bd89 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -14,7 +14,11 @@ /* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES +#ifdef CONFIG_MEM_ALLOC_PROFILING +#define PERCPU_MODULE_RESERVE (8 << 12) +#else #define PERCPU_MODULE_RESERVE (8 << 10) +#endif #else #define PERCPU_MODULE_RESERVE 0 #endif From patchwork Wed Feb 21 19:40:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43343C5478A for ; Wed, 21 Feb 2024 19:41:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E15C6B00A9; Wed, 21 Feb 2024 14:41:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EFA56B00AA; Wed, 21 Feb 2024 14:41:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F04C6B00AB; Wed, 21 Feb 2024 14:41:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 31D3A6B00A9 for ; Wed, 21 Feb 2024 14:41:36 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 11D09A0E1E for ; Wed, 21 Feb 2024 19:41:36 +0000 (UTC) X-FDA: 81816830592.28.D9CD414 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 75C0240006 for ; Wed, 21 Feb 2024 19:41:34 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TtLMBXbr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 37VHWZQYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=37VHWZQYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544494; a=rsa-sha256; cv=none; b=gofJvR+atsMMpCsfS6qDiRxKSQETaeJQViU9Jx0wp8xHZg1xZgu6NaIbt0xg9YLqOPenpt WtZZWArxxiCUmESsemtxekcZwG0IEVJ1Sm0uqUGPCmbpK3iKcJxbvcAg8jva4fgSaVa1dR nl3l4OGaPBzXICevHy/NKdf6lMCapKc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TtLMBXbr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 37VHWZQYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=37VHWZQYKCCAOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/ICmwOELDDa5gDiXJ0QfSPYNwNSjuLTkQBKISvcVpvE=; b=SDz38ZASuL++DXwfYSmBy8IrBYkL2g4gMnYsoDXnwKxqvOdxAsycFlhRAxJzbyORn+QVUz drPEebgsPyT05QJf5Mzk7J6L/BS/Yajc4kGK4ds9nFFp7WwjRcfgo5ctXeMrVkvWKUQjw8 OrZVxKj7CmF8dgr6E69qZHN2kPyeKh8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6088fa18619so9940217b3.2 for ; Wed, 21 Feb 2024 11:41:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544493; x=1709149293; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/ICmwOELDDa5gDiXJ0QfSPYNwNSjuLTkQBKISvcVpvE=; b=TtLMBXbrd7N8l7eIMduJcAMJjRTAunAoYwtKB6wzK9PK5vKzxEIZ7md9xi9yDW+AZE H1hJES4gv0DVhI8O8UAOmQABucsqQ1FzY6128rwyV5k1U/IOU3zkkJE5vwP8yuGpIGPo nAYfCF2LiSzu4LQF39QPP/1QIVUvdGYBTjxpwMsC6RW6NtZ8rrwnLyTc6vJkLL5QwGzw FhRh9n2MC2yC9d0/LrJqi7U5Eca6tEZiQ3Zapmc+jfGxXWbSYaRDSUHnipvF7B9M/Q+6 8UGswJW/FIv6Jv9Dl9S5rR1EKPixG2J3SrSniyUH2QkVFSx2OWdKyH/e13ixhvlNcLvU saRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544493; x=1709149293; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/ICmwOELDDa5gDiXJ0QfSPYNwNSjuLTkQBKISvcVpvE=; b=i1ZUCXfatNR+YwNOkPwRSXTtPUE3LZdG4p7NT6CkIV/90bbOlZj5lE+ocyuM/pmyyp cByPdWHKm6JpfRQ3/4oDn77zitTbJm8b+Xa0KnlmPJ7r4GH6bqBy1u9aCTAUdpG98WFL hxsc4rAsIImAn0C1YWRi6sYaRTpaWij3uMD3i0+DR+HptTh7qFv4BqppyN/3ENMQwyL/ wxLs945KEFGhsRBE2nW+mZl94eY8Y59r0u2P+Vnj/xXNWNr6w5NaL6Z5RwNYjwAxrGye Fq0tAZmggd/CSAd07fMOAs/kYPORCQ3COwnh5KjVtE/CBpKA3pFE4jKjStkqW1vM+4aa WsSQ== X-Forwarded-Encrypted: i=1; AJvYcCWKWHQCCOkr9vSpvDe7rS103DPpTN1IvExRXGg2+qDv5AOp99YRBnreKT2VB2XMCzF13YmLYKU40QqqPLjvhuP9eqs= X-Gm-Message-State: AOJu0YwRXj0IbcGMcdfIeExyUbplc4Gbk/A7cvj960Y5X2mHrHApSaxO tpgKsYeylNo73WdLzraTbUfaLSTZbIMHyaqgUZ3yDyBP3Uag4Fd7qGKZ2XjdAFm7IXwUGKmlAJW 0WQ== X-Google-Smtp-Source: AGHT+IFRKkAzXFrqX8FMuNNkdeozK6lGgluj6VzXaaQV6asduSfbKwLn9SW2CaQIMM08vePlCmcgUKW9cG4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a81:ab4c:0:b0:608:2137:27f6 with SMTP id d12-20020a81ab4c000000b00608213727f6mr2122409ywk.5.1708544493446; Wed, 21 Feb 2024 11:41:33 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:30 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-18-surenb@google.com> Subject: [PATCH v4 17/36] change alloc_pages name in dma_map_ops to avoid name conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 75C0240006 X-Stat-Signature: r7htdycx6n4ezffygha9yb7p6kqhxij4 X-HE-Tag: 1708544494-540020 X-HE-Meta: U2FsdGVkX18oWemGw1EmYipay/MYPbCYxkdz0xFFK2g1Lbh4I1rGyp89aoYjL6zJL0pGuPib6XLpBdnDtYPMMbGJF/Kmwyncl/4McK10I7gvgrgXAwmfCFeWNBsiKaYTUgex4bq0d7WDrevHkpOslZBgxveceFX8XZPWq+IOMwdG+17a+dQnugk5PaVSnpT5JJJHmRDpgt0RIQF4isYoCNqMWdUnoixWgyaK8PU6DCpn3JmszUufDhltAqaMNq2ZTdkQFt7Bwejt3Q+ZVdD0vOtwCUlk2nNn7snILiv16A67rZu//nHz5SLymXFIHzCJNfibbxm5iK9k+/hYfR5J2cCOU/LfkNkhq2jfoeG3OAZD03NusWr8bEQBqdZGxOclyQrkEAieAxUc84HCRUNhdVFa98b4kmmoRR8tdx7CvBkAfL/lxC2iOtqTkFom/D/Y0cl7FzuYjF/wMCtni0TPP5m0OpmXmJkxnbwhvXbBJWGjclZJTvaKHRsh2NtQKFOVtKDq9VtCOVisG8F9xRFX4w7FI9+YegCLlzoqTIxiC+I0tYvnXngxEWJiRAIf3zAAKoJ4slU/f6p/++u+FU9O7LCgQEt5a7xWdsRQwDBJmu8V0143TRnP1aIEB+fLtgVH2ClWppt6DKSTLw/ArdE8rvA2n5uqJSV86GeT4dh+BJ2UCglPF2ihUg9UwpdMu1j4VCbgLkO3GOM1VwnUcJRR5iL6kK1yqOcA2j+eLeF5m4GdL9sgEvsqZy1Yd28E3vGuU4ew418jwSQnOtRlf9QmExeg4MZWltelBu59gmUJ7pZbU2MIEJ7EUB53gSbtsANF/QvJsM4bTOhdUfcCVIzp4JaOtEyWTpz+cUbhiDThLFxUUzE1kSQk6plhMOmig1507PUGs4YNrHGCO849U/Oej5wZ2ToFu8i4A7NuP97Jivd6Hlib4s1039HxGFLRmdg0IlafAsYgXvolBW0DN9N hHEj8mB6 0svlAtDzvDULn9965uVg0I0vkyy7zXu0rd9vb/QjKPMCDisvnN4HWyR4A8KnDYz9bRLOKlLF3Qf0MQjQlIbFkdQlcbvHFR5stH69Q00UvaqhrKNVWxEL58vIXWUnO7o9j3fXYAeq9omZvOpK0VCwwW9yWzKp7ijFXxvPjc2vAhzUrpXxMmbISV8V2Op0dPE/8psE5NRRIZShi3YelOcFR46dV4MV7ozxEOTOW2f9PZ6Hvphv6PbCoytayi82uQbPivM4F+y8fUIFJX9CWlcejXtsxYp+FZ5S50RhFy5w4lYhTldsua/lcyL3J7OOlNgGMsmFlYe6mzGzo7xMTXAOSuqt9RnFYrvl/3gFiWnAqPwQoyhL14cezT0U/FWEkf1S/si4gd1RkWQFB7/StgokNjuveYFa9qwhOD8Eq7e+Q7prmp3W/OcYZCN2AmHXNLIDQBqCDDJ8aneZLfvvbOqxnTCkLKA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Suren Baghdasaryan --- arch/alpha/kernel/pci_iommu.c | 2 +- arch/mips/jazz/jazzdma.c | 2 +- arch/powerpc/kernel/dma-iommu.c | 2 +- arch/powerpc/platforms/ps3/system-bus.c | 4 ++-- arch/powerpc/platforms/pseries/vio.c | 2 +- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/parisc/ccio-dma.c | 2 +- drivers/parisc/sba_iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 13 files changed, 15 insertions(+), 15 deletions(-) diff --git a/arch/alpha/kernel/pci_iommu.c b/arch/alpha/kernel/pci_iommu.c index c81183935e97..7fcf3e9b7103 100644 --- a/arch/alpha/kernel/pci_iommu.c +++ b/arch/alpha/kernel/pci_iommu.c @@ -929,7 +929,7 @@ const struct dma_map_ops alpha_pci_ops = { .dma_supported = alpha_pci_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(alpha_pci_ops); diff --git a/arch/mips/jazz/jazzdma.c b/arch/mips/jazz/jazzdma.c index eabddb89d221..c97b089b9902 100644 --- a/arch/mips/jazz/jazzdma.c +++ b/arch/mips/jazz/jazzdma.c @@ -617,7 +617,7 @@ const struct dma_map_ops jazz_dma_ops = { .sync_sg_for_device = jazz_dma_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(jazz_dma_ops); diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index 8920862ffd79..f0ae39e77e37 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -216,6 +216,6 @@ const struct dma_map_ops dma_iommu_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/ps3/system-bus.c b/arch/powerpc/platforms/ps3/system-bus.c index d6b5f5ecd515..56dc6b29a3e7 100644 --- a/arch/powerpc/platforms/ps3/system-bus.c +++ b/arch/powerpc/platforms/ps3/system-bus.c @@ -695,7 +695,7 @@ static const struct dma_map_ops ps3_sb_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; @@ -709,7 +709,7 @@ static const struct dma_map_ops ps3_ioc0_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/pseries/vio.c b/arch/powerpc/platforms/pseries/vio.c index 2dc9cbc4bcd8..0c90fc4c3796 100644 --- a/arch/powerpc/platforms/pseries/vio.c +++ b/arch/powerpc/platforms/pseries/vio.c @@ -611,7 +611,7 @@ static const struct dma_map_ops vio_dma_mapping_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 2ae98f754e59..c884deca839b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, }; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 50ccc4f1ef81..8a1f7f5d1bca 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1710,7 +1710,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous, diff --git a/drivers/parisc/ccio-dma.c b/drivers/parisc/ccio-dma.c index 9ce0d20a6c58..feef537257d0 100644 --- a/drivers/parisc/ccio-dma.c +++ b/drivers/parisc/ccio-dma.c @@ -1022,7 +1022,7 @@ static const struct dma_map_ops ccio_ops = { .map_sg = ccio_map_sg, .unmap_sg = ccio_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/parisc/sba_iommu.c b/drivers/parisc/sba_iommu.c index 784037837f65..fc3863c09f83 100644 --- a/drivers/parisc/sba_iommu.c +++ b/drivers/parisc/sba_iommu.c @@ -1090,7 +1090,7 @@ static const struct dma_map_ops sba_ops = { .map_sg = sba_map_sg, .unmap_sg = sba_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/xen/grant-dma-ops.c b/drivers/xen/grant-dma-ops.c index 76f6f26265a3..29257d2639db 100644 --- a/drivers/xen/grant-dma-ops.c +++ b/drivers/xen/grant-dma-ops.c @@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask) static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 0e6c6c25d154..1c4ef5111651 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -403,7 +403,7 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, }; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4abc60f04209..9ee319851b5f 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -29,7 +29,7 @@ struct dma_map_ops { unsigned long attrs); void (*free)(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs); - struct page *(*alloc_pages)(struct device *dev, size_t size, + struct page *(*alloc_pages_op)(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 58db8fd70471..5e2d51e1cdf6 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size = PAGE_ALIGN(size); if (dma_alloc_direct(dev, ops)) return dma_direct_alloc_pages(dev, size, dma_handle, dir, gfp); - if (!ops->alloc_pages) + if (!ops->alloc_pages_op) return NULL; - return ops->alloc_pages(dev, size, dma_handle, dir, gfp); + return ops->alloc_pages_op(dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size, From patchwork Wed Feb 21 19:40:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 835F4C48BEB for ; Wed, 21 Feb 2024 19:41:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 353F16B00AA; Wed, 21 Feb 2024 14:41:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 258636B00AC; Wed, 21 Feb 2024 14:41:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0112D6B00AD; Wed, 21 Feb 2024 14:41:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DB2886B00AA for ; Wed, 21 Feb 2024 14:41:38 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A88F2A0BA5 for ; Wed, 21 Feb 2024 19:41:38 +0000 (UTC) X-FDA: 81816830676.04.B9378F5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 014C810000A for ; Wed, 21 Feb 2024 19:41:36 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FVrCYH5c; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 371HWZQYKCCIQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=371HWZQYKCCIQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544497; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uPmfz5BtrFVGTzdrgLXfZ9Hlsd+huw3QS0R8m2XwZkA=; b=5hhLdthG4xE7swEbClw7LCMLUPGS0YcR+XiOJBvw4CHdEAxeYBDYK4VAKVJGFkldPetjF3 Xnf1ewB/HgSnkYwYeb6tZFnYD9sGKgJP9uMm5UierEUdSqPe/iHPeUq/Z1r2pbvo63jiMV KKqYnCl+7avgXCcPX084aRsbZCeoZ3U= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FVrCYH5c; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 371HWZQYKCCIQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=371HWZQYKCCIQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544497; a=rsa-sha256; cv=none; b=0QS+vkJDd8l0+t4SuUnwGFblyayWGQW1frVFi74BxXq0uOWXEJvLdsCuJyzE4fbDhFB4g1 nS1r8OfG9ufIYcbCRXBAop3fp8DVOG67RbTv5YeDEKqtKBUGOP4AcizSUZHBh5kJ+t4cX2 NIRI6OtxA3vViEObA4T2G1yi7fXa2nM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b267bf11so6914771276.2 for ; Wed, 21 Feb 2024 11:41:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544496; x=1709149296; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uPmfz5BtrFVGTzdrgLXfZ9Hlsd+huw3QS0R8m2XwZkA=; b=FVrCYH5ckGPzueTH95AMhrRPVNrTrT8jUcp4zhgl12KwBhaFIufdkIh9LaboV6piwj 7ltwK1xJNxNzCroExiqYALl7Ut4H0ZqeufJy1Hhe+4moa/WD9UCz/e1OZ/WP2nMRTaCe 5u+4GYl6qFgiwGudpfNQo37lT1vwd9u8jbX5uAelA0x1nxMwVWfrGVCDnpHwlGmI9NLp /eA5P3oub5k6Yem9I6KTFrIGo3ryBLoIj+MJ2kaZPX1heqkwplp0Jnuw6884MfQzOCWz EIWOR/r+DWH0psJTYJSuxZJ6sx9GInWaBhup9hW0S8Nd6CAez2L07BayesBaYyniYsno 8FnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544496; x=1709149296; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uPmfz5BtrFVGTzdrgLXfZ9Hlsd+huw3QS0R8m2XwZkA=; b=vJDW1xfdHKWo9HlYEFLqaUal5bjDJHaxeUSPtPUfD8CI5DmK9L8jrarCauG6MxvUTu 8imGRA1eRby/NTiY49CLmmPuPChBoYGoHkzbvGe6EMGzbOYGGh5hgtXv314CflbnXUip Y6HMcf45/FGakN+EqYPoDHkKbI2Rj+Coe4pbcJ8EyDsMShMYEltneIQ55iV6cnYb9GeU MJueLqCwY/BejE4KFvqeBTQ5y8Zx/+wdrqP7z4RbpuvSIyjWHuO6A3LFpNULuw7MGwFo LOWKjvqBz4zRAQ9550RSzVX3HE2F2OocZ9xm09Qlccbu6+j3OIwr1NHilKxrG+apyx8L XAsg== X-Forwarded-Encrypted: i=1; AJvYcCWR1d6M5qma6kOkeiliNf499+31crqHm37yJqLHF5N3GwA4dt+BS6HmgJxdCgx/80vDbKXVNNiRmY/y6KMSkgKO5Ns= X-Gm-Message-State: AOJu0YztT2As+dO22jVAWRPUl/bQwTwyPAZki9EYPFPIRyKGCoRszV7X Md+MPbeVOeImfm5nc7uzK4VTxHH3ayG+66d8TTEfcUx3yjad5unr6kajhQnSk8R5TEK66ECcfiT ldw== X-Google-Smtp-Source: AGHT+IEGiZxEZLQu4Fx/810KCe5SfUo6pvZp4KIqXBDa3HNz9YA+geChHzYsqxNzexrQU/zYLXIZZIJbRIM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:1209:b0:dbe:30cd:8fcb with SMTP id s9-20020a056902120900b00dbe30cd8fcbmr15521ybu.0.1708544495831; Wed, 21 Feb 2024 11:41:35 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:31 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-19-surenb@google.com> Subject: [PATCH v4 18/36] mm: enable page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 014C810000A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ax5cpar5pi4ospnm1r53spotkpjdgk7k X-HE-Tag: 1708544496-137873 X-HE-Meta: U2FsdGVkX1+OsyZgmIEXwphQeIxDy34eCHye/vDj0yt6DE24Fscd6HJep+iJLxiaZbzeH6TSdLSjSZGpKeC4WTTddYpvqlrCaDvPjJEsdXgUOm5tLKZNr7rRVIQGkoJz7IktTtL/pVxmMGi+Ry68EFFLurXdwNyjb6lB3Uu93R7d3BcF87xWoISgurtZhM/hiUL08yT3z8F5bux0W7LELhRTOEwidzqLxX9jgDBVZxr76irmLhj7Bb5NAUDC6JwcQ7yZo1FTYaBZHB0LGu3nZ3kLU4NUCx7ICNKNfxKlkI7nCXrlwWIANtfCk49BustaOe7nyodk7GE6flqy4FgxfP43LHpg8STQqR2bo+pSOCvf7YE//C1lOuCQnRitWIPVH0JZHKRCCXSzjLwJ2U6z4xjYOfXLQGQVH7o1lBRUILz6wgQQTxfk1UNFkPRjb2fQ3Qmu82jNjHFwZv5da2scqdXdfPfVh0xkyR6IstcOALsXq382Hs5YqK80hGxYJaXDs3Pf9Bpo2jI3JpYRKGomRCiENzO3sn0+pNw4Kp8ys7YSur3evzDgq6yPdJNQlPtuSQVQ5HSlRde9B/KwwA+bqvD21INdhPp+NiosaCfD34n054zPdDau7r6rWOGxeTm6M8fTjWi10wpTwzrS0rl+ZB+CpPpdDW1pNkXJ/+S5uJ13TM/BRh2iOpk5wOALrfQ9Bqym0XiqdfDs+H+KOPivpp3/cHvEW2cJPAZO9amr+ADG5TP6nKtzVnsWHojzQJ03xcf/dR6R3F63a7s/JBYVieSiU5BzRfL/pmvnUIwqRn0xZki/GCSrI+OOnDYzxX6Oq+1l0pgLfawqkvVPOh0+5dRKTLxHgsUwYEjh2DLyh2CqNnvr/2AGwrtphoVaHX/ZgCRjJtidARU6pHDILHlpG/BgeT0lVQGOo2s5fx+3hgPudhxl5+PbTF3NxT8orlY3mPxkeUkHltNOzC3T4Ax m8ZYnl2H NGj7NlsSqmnjZvqhXWyAvHA3EaH/KhqjtYfd0CVxlGYvrvCufKmaMF0UbFcroHsB8JDfp5EkeC8kI0AytVgf2SVGMuTY9tEWJPH7hliCdAcSpJ/1h8I+KFx7qPvoWUscO0gKpLltc/Ba20g0wBSZAN/JaBUWBAn0ElFu20+o2C3K7HnBTn9hSjC6rTTaqR9t+Xkir5JNNLNinuaTNza1ZF8xSied5IiV4U5eLJtgph5xpKX8MhlTnSbdkshjkRdLLpBEHxM7NFldpH7k3FNjArd7UEZIFhUnhC4AcU/XG1jxbTfMHl8mBDyVgPcQBj0jvaNkFBvkwtfMeK+JVokjHqu2Cl8BOJfhDg9b0rOVg89QuQ0+FOyZLm6s2MxzvJKVkiY17ox0ela3nGwclcPXm08al7G/sbaKAAv+M X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 14 +++++ include/linux/gfp.h | 126 ++++++++++++++++++++++++-------------- include/linux/pagemap.h | 9 ++- mm/compaction.c | 7 ++- mm/filemap.c | 6 +- mm/mempolicy.c | 52 ++++++++-------- mm/page_alloc.c | 60 +++++++++--------- 7 files changed, 164 insertions(+), 110 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index be3ba955846c..86ed5d24a030 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -141,4 +141,18 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, #endif /* CONFIG_MEM_ALLOC_PROFILING */ +#define alloc_hooks_tag(_tag, _do_alloc) \ +({ \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(_tag); \ + typeof(_do_alloc) _res = _do_alloc; \ + alloc_tag_restore(_tag, _old); \ + _res; \ +}) + +#define alloc_hooks(_do_alloc) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc); \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/gfp.h b/include/linux/gfp.h index de292a007138..bc0fd5259b0b 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -6,6 +6,8 @@ #include #include +#include +#include struct vm_area_struct; struct mempolicy; @@ -175,42 +177,46 @@ static inline void arch_free_page(struct page *page, int order) { } static inline void arch_alloc_page(struct page *page, int order) { } #endif -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) + +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); +#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array); +#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__)) -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array); +#define alloc_pages_bulk_array_mempolicy(...) \ + alloc_hooks(alloc_pages_bulk_array_mempolicy_noprof(__VA_ARGS__)) /* Bulk allocate order-0 pages */ -static inline unsigned long -alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, list, NULL); -} +#define alloc_pages_bulk_list(_gfp, _nr_pages, _list) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, _list, NULL) -static inline unsigned long -alloc_pages_bulk_array(gfp_t gfp, unsigned long nr_pages, struct page **page_array) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, NULL, page_array); -} +#define alloc_pages_bulk_array(_gfp, _nr_pages, _page_array) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, NULL, _page_array) static inline unsigned long -alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) +alloc_pages_bulk_array_node_noprof(gfp_t gfp, int nid, unsigned long nr_pages, + struct page **page_array) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, NULL, nr_pages, NULL, page_array); } +#define alloc_pages_bulk_array_node(...) \ + alloc_hooks(alloc_pages_bulk_array_node_noprof(__VA_ARGS__)) + static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) { gfp_t warn_gfp = gfp_mask & (__GFP_THISNODE|__GFP_NOWARN); @@ -230,82 +236,104 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) * online. For more general interface, see alloc_pages_node(). */ static inline struct page * -__alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) +__alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp_mask); - return __alloc_pages(gfp_mask, order, nid, NULL); + return __alloc_pages_noprof(gfp_mask, order, nid, NULL); } +#define __alloc_pages_node(...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) + static inline -struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) +struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp); - return __folio_alloc(gfp, order, nid, NULL); + return __folio_alloc_noprof(gfp, order, nid, NULL); } +#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__)) + /* * Allocate pages, preferring the node given as nid. When nid == NUMA_NO_NODE, * prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, - unsigned int order) +static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, + unsigned int order) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_node(nid, gfp_mask, order); + return __alloc_pages_node_noprof(nid, gfp_mask, order); } +#define alloc_pages_node(...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) + #ifdef CONFIG_NUMA -struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid); -struct folio *folio_alloc(gfp_t gfp, unsigned int order); -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else -static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) +static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order) { - return alloc_pages_node(numa_node_id(), gfp_mask, order); + return alloc_pages_node_noprof(numa_node_id(), gfp_mask, order); } -static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +static inline struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid) { - return alloc_pages(gfp, order); + return alloc_pages_noprof(gfp, order); } -static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) +static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); } -#define vma_alloc_folio(gfp, order, vma, addr, hugepage) \ - folio_alloc(gfp, order) +#define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage) \ + folio_alloc_noprof(gfp, order) #endif + +#define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) +#define alloc_pages_mpol(...) alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__)) +#define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) +#define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) + #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) -static inline struct page *alloc_page_vma(gfp_t gfp, + +static inline struct page *alloc_page_vma_noprof(gfp_t gfp, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = vma_alloc_folio(gfp, 0, vma, addr, false); + struct folio *folio = vma_alloc_folio_noprof(gfp, 0, vma, addr, false); return &folio->page; } +#define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) + +extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); +#define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) -extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); -extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long get_zeroed_page_noprof(gfp_t gfp_mask); +#define get_zeroed_page(...) alloc_hooks(get_zeroed_page_noprof(__VA_ARGS__)) + +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) __alloc_size(1); +#define alloc_pages_exact(...) alloc_hooks(alloc_pages_exact_noprof(__VA_ARGS__)) -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); void free_pages_exact(void *virt, size_t size); -__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); -#define __get_free_page(gfp_mask) \ - __get_free_pages((gfp_mask), 0) +__meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); +#define alloc_pages_exact_nid(...) \ + alloc_hooks(alloc_pages_exact_nid_noprof(__VA_ARGS__)) + +#define __get_free_page(gfp_mask) \ + __get_free_pages((gfp_mask), 0) -#define __get_dma_pages(gfp_mask, order) \ - __get_free_pages((gfp_mask) | GFP_DMA, (order)) +#define __get_dma_pages(gfp_mask, order) \ + __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); @@ -357,10 +385,14 @@ extern gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma); #ifdef CONFIG_CONTIG_ALLOC /* The below functions must be run on a range from a single zone. */ -extern int alloc_contig_range(unsigned long start, unsigned long end, +extern int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask); -extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask); +#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) + +extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) + #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2df35e65557d..35636e67e2e1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -542,14 +542,17 @@ static inline void *detach_page_private(struct page *page) #endif #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order); #else -static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } #endif +#define filemap_alloc_folio(...) \ + alloc_hooks(filemap_alloc_folio_noprof(__VA_ARGS__)) + static inline struct page *__page_cache_alloc(gfp_t gfp) { return &filemap_alloc_folio(gfp, 0)->page; diff --git a/mm/compaction.c b/mm/compaction.c index 4add68d40e8d..f4c0e682c979 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1781,7 +1781,7 @@ static void isolate_freepages(struct compact_control *cc) * This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct folio *compaction_alloc(struct folio *src, unsigned long data) +static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; struct folio *dst; @@ -1800,6 +1800,11 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) return dst; } +static struct folio *compaction_alloc(struct folio *src, unsigned long data) +{ + return alloc_hooks(compaction_alloc_noprof(src, data)); +} + /* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no diff --git a/mm/filemap.c b/mm/filemap.c index 750e779c23db..e51e474545ad 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -957,7 +957,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, EXPORT_SYMBOL_GPL(filemap_add_folio); #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { int n; struct folio *folio; @@ -972,9 +972,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } -EXPORT_SYMBOL(filemap_alloc_folio); +EXPORT_SYMBOL(filemap_alloc_folio_noprof); #endif /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 10a590ee1c89..c329d00b975f 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2070,15 +2070,15 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, nodemask); + page = __alloc_pages_noprof(preferred_gfp, order, nid, nodemask); if (!page) - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_pages_noprof(gfp, order, nid, NULL); return page; } /** - * alloc_pages_mpol - Allocate pages according to NUMA mempolicy. + * alloc_pages_mpol_noprof - Allocate pages according to NUMA mempolicy. * @gfp: GFP flags. * @order: Order of the page allocation. * @pol: Pointer to the NUMA mempolicy. @@ -2087,7 +2087,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { nodemask_t *nodemask; @@ -2117,7 +2117,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - page = __alloc_pages_node(nid, + page = __alloc_pages_node_noprof(nid, gfp | __GFP_THISNODE | __GFP_NORETRY, order); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; @@ -2130,7 +2130,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page = __alloc_pages(gfp, order, nid, nodemask); + page = __alloc_pages_noprof(gfp, order, nid, nodemask); if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ @@ -2146,7 +2146,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * vma_alloc_folio_noprof - Allocate a folio for a VMA. * @gfp: GFP flags. * @order: Order of the folio. * @vma: Pointer to VMA. @@ -2161,7 +2161,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * * Return: The folio on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage) { struct mempolicy *pol; @@ -2169,15 +2169,15 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, struct page *page; pol = get_vma_policy(vma, addr, order, &ilx); - page = alloc_pages_mpol(gfp | __GFP_COMP, order, - pol, ilx, numa_node_id()); + page = alloc_pages_mpol_noprof(gfp | __GFP_COMP, order, + pol, ilx, numa_node_id()); mpol_cond_put(pol); return page_rmappable_folio(page); } -EXPORT_SYMBOL(vma_alloc_folio); +EXPORT_SYMBOL(vma_alloc_folio_noprof); /** - * alloc_pages - Allocate pages. + * alloc_pages_noprof - Allocate pages. * @gfp: GFP flags. * @order: Power of two of number of pages to allocate. * @@ -2190,7 +2190,7 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned int order) +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; @@ -2201,16 +2201,16 @@ struct page *alloc_pages(gfp_t gfp, unsigned int order) if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); - return alloc_pages_mpol(gfp, order, - pol, NO_INTERLEAVE_INDEX, numa_node_id()); + return alloc_pages_mpol_noprof(gfp, order, pol, NO_INTERLEAVE_INDEX, + numa_node_id()); } -EXPORT_SYMBOL(alloc_pages); +EXPORT_SYMBOL(alloc_pages_noprof); -struct folio *folio_alloc(gfp_t gfp, unsigned int order) +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { - return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); + return page_rmappable_folio(alloc_pages_noprof(gfp | __GFP_COMP, order)); } -EXPORT_SYMBOL(folio_alloc); +EXPORT_SYMBOL(folio_alloc_noprof); static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, struct mempolicy *pol, unsigned long nr_pages, @@ -2229,13 +2229,13 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, for (i = 0; i < nodes; i++) { if (delta) { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node + 1, NULL, page_array); delta--; } else { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node, NULL, page_array); } @@ -2257,11 +2257,11 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, + nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes, nr_pages, NULL, page_array); if (nr_allocated < nr_pages) - nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL, + nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, nr_pages - nr_allocated, NULL, page_array + nr_allocated); return nr_allocated; @@ -2273,7 +2273,7 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, * It can accelerate memory allocation especially interleaving * allocate memory. */ -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; @@ -2293,8 +2293,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, nid = numa_node_id(); nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid); - return __alloc_pages_bulk(gfp, nid, nodemask, - nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, nodemask, + nr_pages, NULL, page_array); } int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index edb79a55a252..58c0e8b948a4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4380,7 +4380,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, * * Returns the number of pages on the list or array. */ -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array) @@ -4516,7 +4516,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_trylock_finish(UP_flags); failed: - page = __alloc_pages(gfp, 0, preferred_nid, nodemask); + page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask); if (page) { if (page_list) list_add(&page->lru, page_list); @@ -4527,13 +4527,13 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; } -EXPORT_SYMBOL_GPL(__alloc_pages_bulk); +EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, - nodemask_t *nodemask) +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -4595,38 +4595,38 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } -EXPORT_SYMBOL(__alloc_pages); +EXPORT_SYMBOL(__alloc_pages_noprof); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page = __alloc_pages(gfp | __GFP_COMP, order, + struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order, preferred_nid, nodemask); return page_rmappable_folio(page); } -EXPORT_SYMBOL(__folio_alloc); +EXPORT_SYMBOL(__folio_alloc_noprof); /* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if * you need to access high mem. */ -unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) +unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order) { struct page *page; - page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); + page = alloc_pages_noprof(gfp_mask & ~__GFP_HIGHMEM, order); if (!page) return 0; return (unsigned long) page_address(page); } -EXPORT_SYMBOL(__get_free_pages); +EXPORT_SYMBOL(get_free_pages_noprof); -unsigned long get_zeroed_page(gfp_t gfp_mask) +unsigned long get_zeroed_page_noprof(gfp_t gfp_mask) { - return __get_free_page(gfp_mask | __GFP_ZERO); + return get_free_pages_noprof(gfp_mask | __GFP_ZERO, 0); } -EXPORT_SYMBOL(get_zeroed_page); +EXPORT_SYMBOL(get_zeroed_page_noprof); /** * __free_pages - Free pages allocated with alloc_pages(). @@ -4818,7 +4818,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, } /** - * alloc_pages_exact - allocate an exact number physically-contiguous pages. + * alloc_pages_exact_noprof - allocate an exact number physically-contiguous pages. * @size: the number of bytes to allocate * @gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP * @@ -4832,7 +4832,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, * * Return: pointer to the allocated area or %NULL in case of error. */ -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); unsigned long addr; @@ -4840,13 +4840,13 @@ void *alloc_pages_exact(size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - addr = __get_free_pages(gfp_mask, order); + addr = get_free_pages_noprof(gfp_mask, order); return make_alloc_exact(addr, order, size); } -EXPORT_SYMBOL(alloc_pages_exact); +EXPORT_SYMBOL(alloc_pages_exact_noprof); /** - * alloc_pages_exact_nid - allocate an exact number of physically-contiguous + * alloc_pages_exact_nid_noprof - allocate an exact number of physically-contiguous * pages on a node. * @nid: the preferred node ID where memory should be allocated * @size: the number of bytes to allocate @@ -4857,7 +4857,7 @@ EXPORT_SYMBOL(alloc_pages_exact); * * Return: pointer to the allocated area or %NULL in case of error. */ -void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) +void * __meminit alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); struct page *p; @@ -4865,7 +4865,7 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - p = alloc_pages_node(nid, gfp_mask, order); + p = alloc_pages_node_noprof(nid, gfp_mask, order); if (!p) return NULL; return make_alloc_exact((unsigned long)page_address(p), order, size); @@ -6283,7 +6283,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, } /** - * alloc_contig_range() -- tries to allocate given range of pages + * alloc_contig_range_noprof() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either @@ -6303,7 +6303,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * pages which PFN is in [start, end) are allocated for the caller and * need to be freed with free_contig_range(). */ -int alloc_contig_range(unsigned long start, unsigned long end, +int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; @@ -6427,15 +6427,15 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } -EXPORT_SYMBOL(alloc_contig_range); +EXPORT_SYMBOL(alloc_contig_range_noprof); static int __alloc_contig_pages(unsigned long start_pfn, unsigned long nr_pages, gfp_t gfp_mask) { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return alloc_contig_range_noprof(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, @@ -6470,7 +6470,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, } /** - * alloc_contig_pages() -- tries to find and allocate contiguous range of pages + * alloc_contig_pages_noprof() -- tries to find and allocate contiguous range of pages * @nr_pages: Number of contiguous pages to allocate * @gfp_mask: GFP mask to limit search and used during compaction * @nid: Target node @@ -6490,8 +6490,8 @@ static bool zone_spans_last_pfn(const struct zone *zone, * * Return: pointer to contiguous pages on success, or NULL if not successful. */ -struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask) +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) { unsigned long ret, pfn, flags; struct zonelist *zonelist; From patchwork Wed Feb 21 19:40:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E96FEC5478B for ; Wed, 21 Feb 2024 19:41:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F1786B00AC; Wed, 21 Feb 2024 14:41:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 486156B00AD; Wed, 21 Feb 2024 14:41:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F4B96B00AE; Wed, 21 Feb 2024 14:41:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0ECE56B00AC for ; Wed, 21 Feb 2024 14:41:41 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EAB24A0A75 for ; Wed, 21 Feb 2024 19:41:40 +0000 (UTC) X-FDA: 81816830760.15.A9E60C8 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 53C8D16000D for ; Wed, 21 Feb 2024 19:41:39 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YV0S4Y3C; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 38lHWZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=38lHWZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544499; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1hWNqJFIgMmzuIRMCt7lzLBdngxXaEaZ9g59MUlptRo=; b=l1x3zBBRRIr3sdjN7enzVa2Bu7ehLuYHIRDBuaiY7AtrHRlNbNxy+dBoYAdy1dtVJSZGfy dy5TKBh9FRNmHHLEvOsEyKo/15s5RqFa/Sc3MX2URMxH+yDxJcBr2kjA2iJ6/CPUk833v5 mP5VCtXsgYVHHVE3HEOPjwYtFJY4v9I= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YV0S4Y3C; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of 38lHWZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=38lHWZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544499; a=rsa-sha256; cv=none; b=EF8uga2sZwmxCH79J3u/AHvRBaHj7b5Qw4k1lWq5hfFWJsFYeAL6revZHMGWsO6sD3GcKj HWPm0IWNlOEjqEPVas6FFWo5oUo+23s/4LEWmt1964jipHy0WVvdbrl+x39of7+ozpeMeC R6TGZtJdaYBsKeI8E58lH/5ntxQ+A8c= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcbfe1a42a4so2066395276.2 for ; Wed, 21 Feb 2024 11:41:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544498; x=1709149298; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1hWNqJFIgMmzuIRMCt7lzLBdngxXaEaZ9g59MUlptRo=; b=YV0S4Y3ChLDi6BCj/aUsvVGctFuKt2+jJYqGdSvPtE6xhZwyqvyG5Z8djNdMprnTrm UH8apNV+oiGQl9LlUd73potiTDtuFHGUl3f4UT/mBUjHd9CVChUuo0sNy7oOlPjRX5q8 pe/gJsHVOKV38ApcHCQ3+JWR2fjZBMhLx7OI9GmoMJsuauuX2PfFSipMQvdwkI3Yd4qj Oi38skxo1CwXI3VF5Qsua2CBBDeC0KJVropZJ1xao4gH3awSg8pmpUzrbla38v0eh5it p4P23JpalBdQTTqkJ3y4seOxHpN9eQtKCxvwjE6PeAqk5r3Gp/QLD5+HgmVU/lpWNkrw QkEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544498; x=1709149298; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1hWNqJFIgMmzuIRMCt7lzLBdngxXaEaZ9g59MUlptRo=; b=quWcQE+48tMMjoAMAflJ1qKYeV+46goqhkwAWY9lnTYOuSAXTN2XsfJJhnD14y79q8 fq8QNcqwrJnCaNo/2RsycuwJA6cb8mH3KMXuPnplahfTgv48lBkdkPZi9c80F7Q+khlE zkrbzTGEwergNQNlk8f0YGZGpljszpnlMd0m6uUp/1bF/AGFIFoz29nuo5VVNDoygbDp wefDPtYqgas8wL6OiTsUXod56qCuXI57OpqYQg2OMDK8/CpPxLg6T83mt3ZFs46Jvu09 3Uw5qGvVG/fKn695cQMkAExqeJpqztuc/YPpi7OutObn+9vNbqkR7R5zi38j1wmyKxt1 S/nQ== X-Forwarded-Encrypted: i=1; AJvYcCUD48LvnZjvne6BmmQ1Um5oYp1OKweLHWsWFWBPycAIDV9qJIg+f45c6fMt2bPncxyFqWct7bVxgfLjNBqBYp/Gvyw= X-Gm-Message-State: AOJu0Yzj4uYzMu8k4us9dblzPnoGSBmwWvfNjiiTPgQA1dG7e4qDIpkh k4muPMdLy0X/IbvQnYA7u5EuSLxij20aU8JTs+YPf5Nk3ZjVrGiTibjNyKJ49/XnB65W0Dju7MX NOw== X-Google-Smtp-Source: AGHT+IF+lk/FYOD/Yxg3sPkq6eArvA9PdzVkhKYXiOVnRPx3H40+bQfIfbP0HZr0UMzCJhoN0p6KojopSmw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:18d6:b0:dc6:dfc6:4207 with SMTP id ck22-20020a05690218d600b00dc6dfc64207mr68537ybb.10.1708544498016; Wed, 21 Feb 2024 11:41:38 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:32 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-20-surenb@google.com> Subject: [PATCH v4 19/36] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 53C8D16000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: fr8g6o1fqr1bq615e37t468uxqeke7ja X-HE-Tag: 1708544499-584360 X-HE-Meta: U2FsdGVkX1+wmtk98O9/oqOLivJGOKEUeX+GeSkCmhcHxUzR6QnegUttreg0A9g2CJoO9CvR/5iiyNotg9tgwAkM/N0mMHgsTndEUIbeVVtmZP8KOzkjDhgZkNDsVgGUDnf7E51o8CtMm298KI1+vHFt5y74dT88I+cqxJlj6g+YyNQ12q+qGZHFKOolnHwFq27tF8Goe/qYR9l07AULbpqBBQUaehnO4mZYo4jJgQfTppi8IYq3wa74zHOapWhobotWZA3YsFU2LRDSrpuHYUxHMG9wG04xcovuzH2ckGtLDtXjrOyFThCTnjk5ar3+/hAqsR+grZhSTI/Ekc1XtOZGk6g6cOI24m1SIfWxLQrGaGajwLlztAv+JWHSWtsqcg9+Pcr8RLUu7GAgc5R6oUasvodvJLNmDSIF5VW7RsKCxFR2Q39keYyVB5vzvn4vk+nqdUZZYTvZc3KAwkc0BcX0YNI17Hnu8f7qZB7GstZSK6XZUzNItpb9a34DO9SMDbt4NrdHep3Z4snaPUNPXmD0eY96fHkq1ifPOglE0+ZW1nMhchX0RIAi2eVjelxXmxNoZF3i9ok1VNQ1EZ5ZRxlani3RFctPNcdi3fMzpk4bxedxKqWqEoM1rtNQReLkfg8vhUlruDqOv5+LhJu3E/ojQXqu/xc/act4jZdJ7ps2KwUrE6ufxHlanbdR3sLV6EZGa8GBn7OOR3WydITLsuj8xK8/uW64FECrEp5J/3riq0MmI1VlXYJDTOrI0UKvxID8YghPsJZNcjYx7lKWRpM6HAUlhZ11LoknbaNmKUDuNZLwNNif0sKPfK+J4fHTacQ3k2gONf2jZHWg/CGlFUoXT+f1/Hj7mlKn2oXiqAig4vtmHNOIR+jMY1qpVVSmJdLw+VWGlyUlXzsRrvXaOdpw6BABL+4t5n2ttUGx4KWaDVT1YMzT/hmX18008ZQAQKASidyYuJbQhASKUOG JVkB1lwf goIcD385rzoMNpY6ZjXM1E/j4CgqyxIpPF/9DHVj/yvzYs6svSFte/uaLoFMzZNlgYcQeCxzI2aOCPRSY2pXaVM/m+PatLYBEyEAFY4Hl4kOjNQtSJOX30yZpuIzA/0kPyPbeqyXre1eZA8/KjabQzxP3mj7wJELnpIp2JbO73LvNIhlRJFCJG0lnMU9aO2UJ5megY2qY1AQ9TiKEVIowQObNkZ/wM4MFkQsPeD4lj+QO13gmWS4tWJaX/OsVlQBnxapifDHlZMzoobZJgKtmZ3lcIO46g5JkV5NKbJufwtXbwT0WGAzvTWSm+T1tlFwKc7HjicQBW2N9ELA3mvKPpq4qs2Fig0HMbWXfZYIN3iA2VZnToM6PIKhgX1/fUp2rp38TcpuerDG4pHl6msdLs3Uj+qcbUwFiMUiQNFSo8FKM/ns= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index b49ab955300f..9e6ad8e0e4aa 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* Set new reference to point to the original tag */ + alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), tag); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 94c958f7ebb5..86daae671319 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -2899,6 +2900,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, nr); + pgalloc_tag_split(head, nr); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 58c0e8b948a4..4bc5b4720fee 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2621,6 +2621,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); } EXPORT_SYMBOL_GPL(split_page); @@ -4806,6 +4807,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); while (page < --last) set_page_refcounted(last); From patchwork Wed Feb 21 19:40:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9A7DC5478A for ; Wed, 21 Feb 2024 19:41:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7160C6B00AD; Wed, 21 Feb 2024 14:41:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69D2E6B00AE; Wed, 21 Feb 2024 14:41:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AAF36B00AF; Wed, 21 Feb 2024 14:41:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2698F6B00AD for ; Wed, 21 Feb 2024 14:41:43 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 00960160B58 for ; Wed, 21 Feb 2024 19:41:42 +0000 (UTC) X-FDA: 81816830886.02.A8C73DF Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 6448A140006 for ; Wed, 21 Feb 2024 19:41:41 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XUfTpFmJ; spf=pass (imf23.hostedemail.com: domain of 39FHWZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39FHWZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tPPJBer4xtwbKjpAS2Ohhk9ejLfkpDLeASf186TPNEc=; b=zA2JfVA+d08fwfwMqaqjTJO804Xoa4WQEwgzWQjv6MWe1N0AnlscLkagUGNm/kxirmwhHu SAyJuzng9EPywYa3/DFA8lG/mpQGS74QcblNZzfDcdQ8ogd95Dos0SkofDTGa7Ndffk9a1 7yOvAgxGM7iH3ES8ODvUc44m9L8bZDs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544501; a=rsa-sha256; cv=none; b=NCtwfmaWLSXTMklhBWsNr9PZcCvwn0ZClD1VT/DYXTqCVVNIN4VdRUJE30JUDtRjlNMMk4 NKsOxsd59NhVST9w3r/mqwtyr1eI4gzL3U5SbHg4bbM5ijOWVouajMbTCKEehsphWZFW0q D0YVtzJVveBGdUjMAeemfLD3XfTcPiw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XUfTpFmJ; spf=pass (imf23.hostedemail.com: domain of 39FHWZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=39FHWZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26783b4so6791556276.0 for ; Wed, 21 Feb 2024 11:41:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544500; x=1709149300; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tPPJBer4xtwbKjpAS2Ohhk9ejLfkpDLeASf186TPNEc=; b=XUfTpFmJgUpzhcnYnCbU4a/N7sA/A5JX0ylTJMpeGVq3JL4eb7oU6JfM4INUYdERkQ CHZkdQbC6sK+VnHAjq2P3xgSlEE1FnORosiWhxhJV3F/3ppkNS+69uqfME+RzE9n9U0z 0RREWqy5viXnpVtoZtuNsijrGgLCZ5HJJoHM2giO8v/nzOisq64S70ibiD+T1g7x06gh +dDHN1NCceIJYxI8EL15Qj8aTyF8tVyS4Wuu+QjpfWhzCzqGszM/nstUD2jM8PkxWMmO 2MNWe9wwxEGKQQ7TW1sOKD4J6zzuer9FX2GJnwiGWh2V8VksEbNQRXZiQy73NS+8j+Og wgHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544500; x=1709149300; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tPPJBer4xtwbKjpAS2Ohhk9ejLfkpDLeASf186TPNEc=; b=Yh9pkCkCr98kxBoPfJSL2vgZ1PR8eMfRTsXzsZT46S03XZf3aIqMRvsJjeBYbSudKK HcvhZIUMP0aFLl5Vhic7TFQyD1D/ZRL6O576ycg4pRZdWdiM9Z2bK78j2qH02FPjI29Z 0N4GaxmaYIpcc8bYeN3sswSfsNPkTXDH047DmFq3KfUWYSuK1dDyGgIW9guiPpmIBQzz qX8QDN7CR6u6gdGQ1MfqgUXNtnzpxPeNdlZqI8IcuitF2sieQMM0Bg5XlPyDfCBMSSkj XTaxYyxB5aCFjFnVD816quyRBFLkDZzrgmODTbOhkk2iE5oO8Q4fAGhObxrLpBqDmrGo EKnQ== X-Forwarded-Encrypted: i=1; AJvYcCWqRxtP9i/6GpAJ1RJS7VLbS6EZOX4Ayj+KXHcEZ6lNGF41Zf4E8arevTTI8MeW1XCQhj/Ydk93xsGb35+7CWARK4o= X-Gm-Message-State: AOJu0Yz2UGQcdVbhy6eQMe4NhWiHwysrsjKahAiOxXpolT7j6AngUtBo 33sBUk9YaBQGFKXp2HjicSnH4Bjdl1t4PXYEALMRk4O2j2MuXrkb4nzsNy8bmN6Ze6kFBVSx5xr uhw== X-Google-Smtp-Source: AGHT+IFeU9fBllBbqeIlv/z5gUqfuWqisiGNNlq6CXUqUD/DUi+PbsZNeU43mHL5nXzTrIUoTAFCk30SjNk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:1008:b0:dbe:387d:a8ef with SMTP id w8-20020a056902100800b00dbe387da8efmr14985ybt.1.1708544500266; Wed, 21 Feb 2024 11:41:40 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:33 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-21-surenb@google.com> Subject: [PATCH v4 20/36] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 6448A140006 X-Rspam-User: X-Stat-Signature: 7fy3ay35otqxa59p8g4qagojnz7njxho X-Rspamd-Server: rspam03 X-HE-Tag: 1708544501-613161 X-HE-Meta: U2FsdGVkX1/uo6Psgv+4T6ni1sc4lrpdecGqyOywTe96i6pj/joLOrnYzJbnm0mSjevlct1/9tPW27fQmxm1G9YVUacYnmheQIvnaoRiYxmGr2yn0qHQTja7zg9A2oBzxHaXyc2b/0p2hQ2Xnx3TnQIEdpPw3m9mQWzTDrDDfSSdlJmmr2M8Vl03+xKckoTV9MsANODjaUH7BFX7BmJj7NuVwys2+yMMrBM3VXvijQ8q6N64gjMyfTDb97/KWS4ND+1QdFpL0G5xoHqEDU7dCWRtE3bfXA0QdvrLKGfGWOu85u6ohsnRORtnvbhd4YCdfBosgvKr4ngL6xzjFKvSDB2whMJPnwL30yBFhVvS47V7h3/Z24OXvKlPytEEpPr1/pW9i5Gq6TFycErjIjB8TItvQRLdaZA0VtYx8Ld8aMarqIvK4Sqnn0+a9qL0DYAMRCW4T9kRyzBtXrP7zn9djY8ItGQzoD1jkW3nQKIrHnl1Z+G/trXOGO6pRk6EEHGlDytSaknH2gfVObEkewabKH96qp+0LPmqMoKYLF5aqWrFrZlmPCPC3ckshDD05ilx/II5xuTAjX3TehwinJPuwzvbApHKoGh0GKh2K23YRpEFaKtd5UcTB7zVgWwX3nPVx71tFq0zExt0bAdavHna/0F+VweBMCO4GFEa5LkY+KNybOuWOkQVgXA9SQM+dozoc/PwFoG37SdTD5/CagPYWZNsTKzFm+y4ino4r7gUTAuyuqbC1EnLJ2LAcmc+v8hk/FD8jkXKPQY5J8WS18ct1vAKcuKAUUEPdSXzUXvaFfWYS0xZUEBJHPiO97kEEyttxdQSFD/j3cg1k0l3a9nAOaV2ut3WmEh6GbeOlNy8Vp9OGE1MgeCIpGga68XkqS2f/3zRgLPsvqWXp2wYu+DWcLGKHs+BL/A3ZlwuCqiX7fZAQH7C2xWPTH7kVk9DW7G11WhI/r239E19Efo8ZH6 7BW4vezE Q6cpF3f651QdjUgBMnBhEWqSpB+e2tvmLoLdxsJhZXvFe/nZzaufAGMKtFw8wIHXTIPlqIOQVmqcq91ek6+8x7/gmXU85mbyK/vnEbV1njhsSl0ezcYyL0IHZg3oCjiJbz5bWUR9KXOI0V1JYbleeaYH6AzSZiTOHz6cp3aj9Vn+OzwylmGV/JWWyZGiLwwbBvjG+6pGu1qh6v1cdIMkK9XtUBz+coBCXOJJL5YYlnL8trhkMEZlLOa4+KymVElHlXfsSpjV/bzImEDmIU+lNPAKUsmPpuHruGJauB9VKhJXFhyKw8GnhFeDxZHmqfovmBNMP7meO6HxcCHUvlUYWoePG5qGyi9u+hACm0gGH1+xICwCjq+u0z39+8WiO/5mTYU43vVVZt/ApeDpUBviZWNZZEyjI9mdHqoN9Bo+A1cbvB6YWVQ2iTeHnYCXBT63SAUTlubqwCRtRjUFpFpYKHc0VfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/page_ext.c b/mm/page_ext.c index 3c58fe8a24df..e7d8f1a5589e 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -95,7 +95,16 @@ unsigned long page_ext_size; static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* + * To ensure correct allocation tagging for pages, page_ext should be available + * before the first page allocation. Otherwise early task stacks will be + * allocated before page_ext initialization and missing tags will be flagged. + */ +bool early_page_ext __meminitdata = true; +#else bool early_page_ext __meminitdata; +#endif static int __init setup_early_page_ext(char *str) { early_page_ext = true; From patchwork Wed Feb 21 19:40:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45CEBC5478B for ; Wed, 21 Feb 2024 19:41:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCC906B00AF; Wed, 21 Feb 2024 14:41:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D050E6B00B0; Wed, 21 Feb 2024 14:41:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B80436B00B1; Wed, 21 Feb 2024 14:41:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9E8E16B00AF for ; Wed, 21 Feb 2024 14:41:45 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 76B73140B18 for ; Wed, 21 Feb 2024 19:41:45 +0000 (UTC) X-FDA: 81816830970.13.42EBCAB Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf03.hostedemail.com (Postfix) with ESMTP id B3F6820019 for ; Wed, 21 Feb 2024 19:41:43 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ei3R4t+5; spf=pass (imf03.hostedemail.com: domain of 39lHWZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=39lHWZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544503; a=rsa-sha256; cv=none; b=isXL1NZUCzdQNM4d9s3NJv2Cbf8vpyXoqmDyI0cOK3BFe8wKqlzW6cfcwPgpv7xo1iqcsK ppfd44l6r47dp8RRB5wlE/66A7fHdVGfUrCwqblIB1AxFU64bvI41/ydSvFvYCwFEkMsPw 9/s0lEggBWN8iOhyR2mTYh+yjDbOwrE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ei3R4t+5; spf=pass (imf03.hostedemail.com: domain of 39lHWZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=39lHWZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4D2ym4tmaIuNf1w08quK73QR9EK/cyrgSXHS1sTLYhg=; b=D68lpSiShXpOvI82HJXHnTG6X7Smh4KpWiCfMWyzMsWwbrJHQXHV/SH7yDBqD/JU3sVYpb b0ICnkeN+Qck/PtmQfTdi3HtjDpTR1sfhin71rnvuOa9hUQIk6Jqo23X2mNC2q8LT2o2oS M1SY1rsvxb8FUpygYwn8aOL4TC5IcE0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-607a628209eso127389077b3.3 for ; Wed, 21 Feb 2024 11:41:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544503; x=1709149303; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4D2ym4tmaIuNf1w08quK73QR9EK/cyrgSXHS1sTLYhg=; b=ei3R4t+5iGeDpXZ39V1uZ8ROiJFCguZJ2wLjQCetGyUmFRQkqplZcKtInUFoOiPywA wHa4jEOMNG2EF0JslVKvWAYYJWZPYvsna1FeWqeeWUFPVs1om61vVpAsmaTxheSaVEbD Ff1qZ1pTdzspqxlUu2eoeYDab6UqpHs34eUqHeBKAZb+GoejjnJAhxthCktz6HWE9t5v QQtaOT8Xy7NzlNhjUaOuj7G4cLDfHcwQSpFD2ylUxdJBWVLLXgKTaaEgoPPJcSkcXPQT 1+KDMJwzETYAdp5l/KeIxqFGJ2SNK69e59Jf8spDh/heWjJ+QEVF4hp5uWAVeSoRAfkb TMZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544503; x=1709149303; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4D2ym4tmaIuNf1w08quK73QR9EK/cyrgSXHS1sTLYhg=; b=XZ1T/iXab20Tzhi4ofiZuPsmx7KFFIaQHq4OiQb3sYvPApRwAyyHw2APK9U+08uhgT mLWVXGMv5V0ekKuXbWLwgWO4xvnLbCpo6Bzn0TMOymgVUB/t0hBJO09mJK8QzHnztuzM qs48ekKHu2cS9VilX1cv4YqRNqmTl1iCcoy/ABj5Wf0K/CzpHozEFlDD8VA03YmnoKoR F/40nsLIvSvWQcdNCRi9E3cqAGN6cTj5PEYpRRjkaVxmGQYOxyHS2uOmVFkRcTClJq8+ rIAjtuXw/MKlrWWYXrlQ7kWflV414XLsHJIU0DuKgv+Dl5qsxIPxuYkkD0iuE3G7a+1I UQog== X-Forwarded-Encrypted: i=1; AJvYcCWX2VUuIBu00ihrB/PRLgB8abk6WyhCqf9MgP2vSY1SGw3w9jtxyefdNySkGTNwkCw5TYzBUin75SiMPCqazywVJwY= X-Gm-Message-State: AOJu0Yx5yzMPP0GZTwuaizbGGRS5SH6+ZDgrMebJ2dXILTknpiEG2uYe /7oGspK7O+NDkkZ1t1rFEfvM/vSfjpkuDAIGClzwwQXZbyNiTVLb0PYAql0ZYuMiNwl/GpxYnoW 39g== X-Google-Smtp-Source: AGHT+IHUFqe0R6lIIi9NA0qHx8ZRtyarym315AvQcyj6s2JvP2F2gSCMg1vFWmN120oFVZKD/Z/VCxMwQvI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:e611:0:b0:607:9268:6665 with SMTP id p17-20020a0de611000000b0060792686665mr4677003ywe.10.1708544502298; Wed, 21 Feb 2024 11:41:42 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:34 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-22-surenb@google.com> Subject: [PATCH v4 21/36] lib: add codetag reference into slabobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B3F6820019 X-Stat-Signature: qzky1zfhioap1cj1jcxo3sy4tyxspqmz X-Rspam-User: X-HE-Tag: 1708544503-502128 X-HE-Meta: U2FsdGVkX1/fF5ThiopAK9ZHPvxzz3NAkmAzQahQ4fdNCh0WoYU5lcG4ZF6PeCtxliw8zh/fh4EXXTk6Uu4ulV3bnoMnalVS0yQOGzjp66CRk01nCLa8KRtOr2co8nWVO11XaEd66G9sqW4dlmJboVmi5Bz3gkU9SCMnIXWXqamcTy8X3DnFlrW3/0W8XTsB4BpqUbwNcIMaVOMwoG5sJg4N6ysTkwt7sPqb6H6RSBkZdJ5k5UJ3+xVKonJz/zWH788jBfqQi6yh3VujOCLFFSSwgqCOYHCkTw7sY6DizMbZCWqi/6HkAbGlfl1/6xhU2HKm3wtz5+cUN7kzFME9+cnggjOFHRzSsuqi62YQPjag8kMXGchUtOIGZuWYdKmnZ+88H+hTNehjxxkXVDCPOweAARywLqmIW4BtCEM0PPbpr5Zf0mYmDJ9D8e53dAGDdCQNam/jC7Duj/dcZRNw4jtIlMc4uTfhH5Aygk0UFWUv1NaQMxSd+prmOzI9vGJkKbbZp4T5MzD9ZmgSFFL8+RBS5GWr+67qNkEQnTG0Kf1C0aNgMNq1Ls5xfcFXkiRhekAk8TDPtE1yvDmh6+A7zarlpizDqP5sE6ElvCXz5jcksBPOcZI2xaVo9DtLDC267ar4DuZIcYCKFhp7CQDQeizSxmTH96odl8yYBvyILiStjxVdskclWpzkLwiu2YmpbQ1SHdqxMWwHaRCfwwrmvwH0Htn3+1pFWIcGP4n/xSQtH940sJDadJCxNd5ddUIYOXMiEMVXh//0Rzf8Ct0UQ8sbQJuVFORe0oyW3vUVpjrjAQ75DxBd462pw0evh3E1Q7LKP8sp32BZTjba/Wwd/9hD2rRYqQL+cOFLh84VzKjtP+V7skfZQ/c8f94CjNRcv4x7r1KBqxh9h76M1Cbsf4hEAOHqyr6s7qiDSrSMt5oqsrO5kb6zkVGBeqVrjprd28zU6UIpt8E5dzVcZV0 BzyrgD6L 5TP87yxll55Mk0FIIfFDIT2FwbKpGuOLW29yD2QdZXGWU/1am0ZEAyiO++nvEbkok52Y1LPvpset+Cy6ARY1tnlBx0b1FS8JZMoFJepjvbipHqGD0JEe05DhJkHiHSSi7NtExIDqq5SnNkQPkNNRlhkJFvO9MDvgqIslUcOHONGbwStaSWIWFl4wye4Uw0idDRFV5pkq4/FvGX2z4bqQ48WW5Ox5HY8QVC9ByLw1b7OhQBSf/8sTRmVd1FIbbL1xt3mFp8RoGIP0t/ofxB3sHnJ7CgL9P5sSTvyvzkWKUw041aR4Lm8OBSYwSiM85nNPqnJsWJzvpibj1CV9X27pHMVwx2iyfXmjgR8LbHP/yVyJQbHZrhsoREkmHgT2GTio65AkQY0yEpeJSP+r6jrPlVOy8+UuPoTiniM6r2FBA41d4rfbf12kS6vcG9bPcONi28YvNPA5lE+178Rdi2vpP7xbyAxulyD9iByT8tiYWpuSe6gZQPWJs99rpags/wDqBlcNAJtbtQy4ad0uJALyj79Vu3Ma/ai+k+9dnJB361DMchZQOOU8S/m2LV0qqikhbg4CyqkeNHr7qtp+qoKbUjdYxrkvdNgdRH+jgDuCusb+JMPUAeTEgQy6uSixKHWsAUzf7Le72Q0qftmSm9uqqEx4/sg6Wc4TQ6Hgc1CrzYwPl7VlX0COOfUAMuaRhqKr+J/xzqWWoDo4osi4cR5APr8tiMR2eNVb9Z0qH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + 2 files changed, 6 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index f3584e98b640..2b010316016c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1653,7 +1653,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif } __aligned(8); static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 7bbdb0ddb011..9ecfcdb54417 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -979,6 +979,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track From patchwork Wed Feb 21 19:40:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A810AC5478A for ; Wed, 21 Feb 2024 19:42:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A70646B00B1; Wed, 21 Feb 2024 14:41:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FA236B00B2; Wed, 21 Feb 2024 14:41:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84B7D6B00B3; Wed, 21 Feb 2024 14:41:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6D5396B00B1 for ; Wed, 21 Feb 2024 14:41:47 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4363840B60 for ; Wed, 21 Feb 2024 19:41:47 +0000 (UTC) X-FDA: 81816831054.22.1E44AF9 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf13.hostedemail.com (Postfix) with ESMTP id A56DA2001A for ; Wed, 21 Feb 2024 19:41:45 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Bds0vIaU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3-FHWZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3-FHWZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SFqjsc0WslhSBLDjO+3ojiIHoGv8W+Erk8fAgLgypEk=; b=B12v/0xg/+BroZ9UNgydbSpHxOuzbBqom1hNU6z/vMowm6OP0vRYTumZwjEaDSYouYore+ vy8qKihhLG67jEgNDfBpvY0nCMtKXFldrNgbrlw+foKdk5MLMsfZp+ysQXbLSLjLm86rez gXZ3zXRL60XALaIvGei28H3+pZD/ryw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Bds0vIaU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3-FHWZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3-FHWZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544505; a=rsa-sha256; cv=none; b=hxoDPX3ptjdw0hBpNb0Z+jNoDQfzDbWu5vFPQ8go5eoCMjuoGerCE+szZNl4isUe31OtnM Lrwk1ZeGXGNtSVa6BImt+kwi5CUYUed97oL2HSFJ7Zd/tzLQs/kLxw5r6AW8jJq05QCCMe hpCk6FQtH1lM0nQXhLWlk+5z5u6Ck9k= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-607c9677a91so17929847b3.2 for ; Wed, 21 Feb 2024 11:41:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544505; x=1709149305; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SFqjsc0WslhSBLDjO+3ojiIHoGv8W+Erk8fAgLgypEk=; b=Bds0vIaUC1JWTe1BxpC6ST/ywEXBP8Uiy0BEMxZwECWSgTR5jbcWqgUQYanXCICNvp nD5DBu1mIVYz3v6kiZPVFo5x8qAgN6Ux8XR/ypRujEAOXtZyzI+tEvX6p84IACjMmMP7 h5Jft9S76WoSIwEIrzjvdThSlabH9060NAgCxefn4/9rjhbBuK64HK1Kij4ijY6zN+Um cEDyj6NQFOSApG3SArE092TrzLEuaTfMSrElaf95VBOpoEZnUfwgacdHHIxTZk9AAjRO leOtgRa22vjobe9+DojeEDp+frVP15hZYjvxYbfHAQVESyQp3goBrhVsFPuawHnGo4u3 vU1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544505; x=1709149305; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SFqjsc0WslhSBLDjO+3ojiIHoGv8W+Erk8fAgLgypEk=; b=CgkYbAS6m0PJu/v3BIiRYVZWQH4eqI+ugjOz4q5qV3TZZ69RqoO8Yx15j8kU9JmULx dsxTcXILsyVAgq61OGU5FsS01SYMcbQlG2ksbeJhuE90SsDnxSCk1OHDUATB1ELO1pLh 9CKC6uDMRQc66/0TxknFJVH1FG4muchRqBOGPBooXJCfK4grr1DKNKIVMazhYabvJ3pB ssDiMTmCsoBrDbyQAUGFYO5l6+4oC9o12SflmWbm6HS51qsTMXwHU5lOo8YjKoM5yuS+ 5raQ/qi1INBDQq65yGhkLh0EIhLP9q2XKIxxL2WIgLiooyKGowRrsiuP5oaLBh1qcc9E TQ/A== X-Forwarded-Encrypted: i=1; AJvYcCWOiXFA0F7KNYtMHZc1wZHe9leWIex2ABSL5GEvxpRLdtcGLFVZdP71Knk1bmcsVEGb79CMLeC/jVP9vS9XfC+olGU= X-Gm-Message-State: AOJu0YyrVMzltU/pqY7FzsHU6ihvW1FFM9coEDmAW304qM/BTMi3kjxu uK4dB68vl9je1VAXW+n5cISHmqce+tAT7ARlVH0M4ZFqiXK24P/U/Nldgsw1aqwx3gP5xoBzGw+ y0g== X-Google-Smtp-Source: AGHT+IFEtEW0GI6YMD3iPDE2fE3Szl49cxeT7/Q/vgrbPAzl796e5Nz4XS4xSM5+xHy4B42fIJc5lCjZ/j8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:e692:0:b0:608:6894:120 with SMTP id p140-20020a0de692000000b0060868940120mr1109228ywe.4.1708544504629; Wed, 21 Feb 2024 11:41:44 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:35 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-23-surenb@google.com> Subject: [PATCH v4 22/36] mm/slab: add allocation accounting into slab allocation and free paths From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A56DA2001A X-Stat-Signature: 914u1wxtmc5s8fwscy8sctha1o3ah7c7 X-Rspam-User: X-HE-Tag: 1708544505-128283 X-HE-Meta: U2FsdGVkX1/bQ7YPEdNHZ5eGS3hjJg3XLrUESa13QpLTcsHF5JXCSl5QDxcXdC9+3mWKUmKyhYgvVcT15Fh/qR4y1ckTZRnLCAbxcwsDFLgrWsZbqPdyXEn7f9qC2VCAyVXWCErcqOu4/M9BZsA0U1M9jH8g8hsg7E1k7RXuaIOHJul1iLCJR18bTkAaHObpst2kcFgkfvRDZeXk++01EUQ9uySSrgS1nr7lTUw8VEIzkDH8rL5Y6a8OvfAH/45LyOYPLupznKkqQp1oN8XWhZrcyYdpPui9AX4JHqbWyUPjOKT4HYs6k1UoWIrp4oajro7L5t/fBNZvWDtZPGlLQCU0Crf937QEVIzPssE5xLkJENTXonf8yJE8zS0ijSSPQSev934uR1Ly0OHQ7zplZKFZZY2MmItgCjhEiYqI2RbJXIVG8wHRNtjFYLJITb2zlWGpvpKAmJ3SjOl/1C0LE2S7yIWMdanK2L0pGFC/RqrFuIhuZh6yOpeMu0LovTzy4H54akxL+yVMFksUPzD/vOgPbJT4J0oRh4CH0kxLgz+nfqmVJzmfX4k1pX3BJcpZeyuRWe1LciT/G+yEcnD78JtCIsoo1zC5Vp2pnFZZOwLeuB1DagIOdW3fQKtGCNQe+k9EeCPTzmpjKmJ3IHXGAtS12Ko6+PzXYHCrZTdr+uuFfEOd7QPas2fSu597M4yXCgvTHQyWK7wL52EBOurTchQa+xyZTwa+TNUvZbPWcSTqHFMyyr8NP09lMQHGO8asKSnwQTkVYuMDwb2t8F6xU/Wqd3TZP8mOvbsXcMI9KZ6psq10YoSwV5n/JJLWJ0i3E6jAq56v22rFJ3HsWnJ8cXwjeObSpyDnzraWK5tZnt6fgW6RK9MhBvcFlUfwlKzIVHIh7rXWJFFT9YEY9M+0ZxtQaCf3ueAJ2rt0ekKYZn8dbhSjZW9M5p8vF72RqVUG6o5XVlhmQ8KtkT55JEi CweLWuAo /pWHLe5RS5OmEHxVLRvQGokjsdAvfqb6SNMbeXG5KGDHC8lHw2vpJzWIVfPtRaumdEvhLqvqVE0WXaapSBYaknVmiR86FgST6ccICV9dMgjDKntYVAe0gfc44n2Jz1mqMwh+/m9zeI0ZdzeLe07df9YoQTO8IpGOzzjGLnTMlcSqrMk1D0u/P7LnVj7WkC10Yi56O9J9izeSwRAB/zRAzygx33W9U81fKg5JykAEc8yJy4zxoU8FM8KvlVfjSu7ooPdHB/tkW2yGe3cHROQvjLbMtTSUPWF9Yja4Lx8GzfwCquHH+Zda6MwZDuF42HtYtRGqFWtZrqYf0FKkquCFp7YiKZ0rHxXhYcoeOnSO9JHwMRBwRQbPkEpn4K0amTKITTb55PCQCCpRwgtdUHo4HWRBCryegBfMy4xf8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Account slab allocations using codetag reference embedded into slabobj_ext. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- mm/slab.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 9 ++++++++ 2 files changed, 75 insertions(+) diff --git a/mm/slab.h b/mm/slab.h index 13b6ba2abd74..c4bd0d5348cb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -567,6 +567,46 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); +static inline bool need_slab_obj_ext(void) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING + if (mem_alloc_profiling_enabled()) + return true; +#endif + /* + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally + * inside memcg_slab_post_alloc_hook. No other users for now. + */ + return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + struct slab *slab; + + if (!p) + return NULL; + + if (!need_slab_obj_ext()) + return NULL; + + if (s->flags & SLAB_NO_OBJ_EXT) + return NULL; + + if (flags & __GFP_NO_OBJ_EXT) + return NULL; + + slab = virt_to_slab(p); + if (!slab_obj_exts(slab) && + WARN(alloc_slab_obj_exts(slab, s, flags, false), + "%s, %s: Failed to create slab extension vector!\n", + __func__, s->name)) + return NULL; + + return slab_obj_exts(slab) + obj_to_index(s, slab, p); +} + #else /* CONFIG_SLAB_OBJ_EXT */ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) @@ -589,6 +629,32 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) #endif /* CONFIG_SLAB_OBJ_EXT */ +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) +{ + struct slabobj_ext *obj_exts; + int i; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + for (i = 0; i < objects; i++) { + unsigned int off = obj_to_index(s, slab, p[i]); + + alloc_tag_sub(&obj_exts[off].ref, s->size); + } +} + +#else + +static inline void alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + #ifdef CONFIG_MEMCG_KMEM void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); diff --git a/mm/slub.c b/mm/slub.c index 5dc7beda6c0d..a69b6b4c8df6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3826,6 +3826,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) { unsigned int zero_size = s->object_size; + struct slabobj_ext *obj_exts; bool kasan_init = init; size_t i; gfp_t init_flags = flags & gfp_allowed_mask; @@ -3868,6 +3869,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, init_flags); kmsan_slab_alloc(s, p[i], init_flags); + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); +#ifdef CONFIG_MEM_ALLOC_PROFILING + /* obj_exts can be allocated for other reasons */ + if (likely(obj_exts) && mem_alloc_profiling_enabled()) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); +#endif } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); @@ -4346,6 +4353,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) { memcg_slab_free_hook(s, slab, &object, 1); + alloc_tagging_slab_free_hook(s, slab, &object, 1); if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) do_slab_free(s, slab, object, object, 1, addr); @@ -4356,6 +4364,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + alloc_tagging_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. From patchwork Wed Feb 21 19:40:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 327FDC48BEB for ; Wed, 21 Feb 2024 19:42:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53F8E6B00B2; Wed, 21 Feb 2024 14:41:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A0E16B00B3; Wed, 21 Feb 2024 14:41:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A6E36B00B4; Wed, 21 Feb 2024 14:41:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0F78D6B00B2 for ; Wed, 21 Feb 2024 14:41:50 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E2E62A0FAC for ; Wed, 21 Feb 2024 19:41:49 +0000 (UTC) X-FDA: 81816831138.03.E06FBB5 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 22D50140004 for ; Wed, 21 Feb 2024 19:41:47 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QDbTprAI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3-1HWZQYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3-1HWZQYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544508; a=rsa-sha256; cv=none; b=JlkY1gmZQjCrLCH/E1xrHO3aRCa7t7wSasy49X9Aa8uwSFYpgVbD63TL7rDGxSN4NaIYIS Dq1evYA5Wbl+m5PpJX2f+l/G8jLigqUEUPTo2xeU9wFwoR8nf9VGuXnlcmlBp5TZzHcNZZ 0rkkYMPoEytdYYczppc9zBWDGb4K7rs= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QDbTprAI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3-1HWZQYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3-1HWZQYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B9bBzslVXKxus/Eql90kYDF/kN1EZHGoql92bbefNg4=; b=XH11Gc6rH3Lt+3o/XgN/xTKHNPMn94frfY/g0/W+ve1lzcs7PP+OfeLtuUjOWS48ujy5fv P5zLPRRBaci9dV3zldbFDDOuxURyTftLW/V9nIJ3aJVaVgktDADINBSSN3MJRDu1zJUfQP qRrpn9vgJaaOR7eA1wYok3VFDRd2Sds= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64b659a9cso12524819276.3 for ; Wed, 21 Feb 2024 11:41:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544507; x=1709149307; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=B9bBzslVXKxus/Eql90kYDF/kN1EZHGoql92bbefNg4=; b=QDbTprAIQXkiCedjxK7/c7dOAG7idrgsd5KP/3hzc/mmlc16S1vtNTpnsfaEQ6+yGe UmGIrY6vFd2NNCSt5rE9BoT1L/LFRaQ5SAJDCCNXhNVePMhdI65F8V4QwhmJ8mIXi+/+ i7C2EVlLyKVjAt7TY3dyiki2e7GIf/Tin79Whj9nh25heeQYoNX2l7vrPf+PkBCOM8Za ba8y+kbMqTWDBxaNEWVfzaDs5Mn4qiNCMkXUI4NrLKIxYc0uBsTBEzb8tv6lVPiwl1q3 E31KVyQlVuLZYA1kpEBWEcC0rnXhdtq6Ehi7nkv4JtzyoURewiHRnZVjoMXnt0x+A7mz jIig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544507; x=1709149307; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=B9bBzslVXKxus/Eql90kYDF/kN1EZHGoql92bbefNg4=; b=KTGlF+B67LYHopdLhk5E9PgoiU4NzWXjRlAUwWfUTCdSgIC5PUXixvQ75DRSTqle5N k86PyEN1fPo7LPbjxWC1TMlhcm/gYdKSwHvII18Jr86VcLoJbThOcsJJfSgyE+Z6LZ2B 0FaR7JAqHj094T0yevaZJc36NuK6p7DCnHi1Okfw6x7eGr9VFEOZPOZAzGU2XYlaFz9s Mrou7BOqQXhbALCr+RVjNGIK+b5orhn8OGAAv2MewX+/9UtARDHJylYY0B02Oe58E1si Py9GyAU3TFfCcbnVWEhZWP8mjQOONxBV6giu0Z163/JaVTAk/ZhxEO1qifwaaLVPqvVK b6Rw== X-Forwarded-Encrypted: i=1; AJvYcCUB0DYLOQKSlzP+tkWssjQtld9y7lewQthc/hEqT5JHmWh6Pw3fr4sqS7Cc6w7hT/hp3xtAGNvyn/sJqM8QB48XbiI= X-Gm-Message-State: AOJu0Yz047pc/MRzXiUfW2x74dlZLcPmyNC5QaSw2SJgf8tpYXxPrzda CU2GIaHC6U3aApqQLlXplbQFYLuC/yCF92GV2DKm+tV3uuN5N6PtseWPXYwmI+ZWYZH18XN/aUt 4CA== X-Google-Smtp-Source: AGHT+IEGw5nl2ExuOLala9gTYXJk9n6bTpGvYj0ETG5ZayMmW5LwulYLaI6C/wEhMG33SuMmfk11BFRCzc0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:ade6:0:b0:dc7:42:ecd with SMTP id d38-20020a25ade6000000b00dc700420ecdmr69488ybe.6.1708544507041; Wed, 21 Feb 2024 11:41:47 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:36 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-24-surenb@google.com> Subject: [PATCH v4 23/36] mm/slab: enable slab allocation tagging for kmalloc and friends From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 22D50140004 X-Stat-Signature: tfbw6khxwofhkiwdkba5g4o3q4wdwzac X-HE-Tag: 1708544507-802078 X-HE-Meta: U2FsdGVkX1+O7QJcBSEK0NI5t3S7ijL8vmBQ+5u7zRn7SJAUNV30YK3ERnacFn5gOklNzlqmxgaVdzYQHZa0x0tm+McyDf2NK32rgwHtArcYB4jEx6LefS4u1/pTa6/HZSASESTWeO463ZTGonAk46LUcLmldGAsCa7Q/vl6246tuw5KqxDrF5bEmPKQvsDj72ynDdRpZc1ZzlUwo8e4ozEPSIyG7NXKaMTE/Ij3RC/bbC8K8CIWzQlQdGdzOazltrjDslPo9HG6Kb7YKNaZVF1wnPdzmkpHRhihcIKqHjz7wsAv/VFAVHMsqWtt1sewrLhaRlOry63JHcicYAyHWG+adjDRP+Vkjy9b++Dj8Z+8CpILR0l+mN/O/x3N/fNV+a7miZit59KTV9EvBpzzIOOHRtOhCNkIQq9MJ1wbr1WDi5P0OJo8ACcaZ6LqbIrClxGLu7Eg/kHUVectiIK/ZKN0gMJg4CFKyyVn3NbsjiYlU5lh3615oplKuSeHj3E+Ty8biFYmLO6vSlesN24hC+6xDZupNmhF1mHrXbQlEI0dddFUWckhbIDZuhMRMUpg2/sBGxA999He+gqRTmCmsljCxENek/ikUu+8aWvQgKJurSnQMBaj27sIkoZk4hcA6lq0WM88+ZItJFpOrfnqtZVVJeF7F5hdLHj2rZnwGx6EtF9MWxnIE7Sa7bJSWbwSq3OEZX5v9PPKyOyDiC29ntGY6gen5I3+tNd840muEPFsSI4sG5uTB1UHCDLNF5llhvZ3IfFv0VrNgDFgTEzwic9V0lLokmMtiGlob2sFVuzko1OFrMFlPaNc7uniG68ySsKEKV+v5EUkHaj1UsrA80LDNUFgbw1TbgKOBL67Nxv2LEZj4OMJK9vJaIGvqRiNUVKMOyYHC8MoKqCkWOqMlHuTa+TWppXvNmYjeXhYHjfuJzKxpft/9l08vuiBFgLkYIQeDvHMeqbeASVp+9i 3spye7Jn /9c/hnYa/wHxKEgBcjlpFnDWVvpAW1XwcNzIInVZ8oSZRF9OvhJwMx1J1ZUWcHqVqdWC0rY7txXqkfhEf2I0W/cxJUV1rG60URy70ZyZwVzOyjxVF2bhoCybq0FgPRX20ZM0no/+iUejY8VqvVSAY7t2KOFweo7cttxiAPVScEMtFtZQpxhCiZyqh1Qfnuv2Yo+Cy7/H1RmF7k2fd0yuwOlkEg0joGMb6gQAOcIembdZyvVhLL1MImq7pAdEl7S00p+rm/MHa2Vy4iA2xZy/7oZf7SwrL+KbLF1zfzQCHKnH1saK9D31lmHbG1BcEMrlOV30DzU9vqzl/iB43njFc0EAt6O+cWS0rLesIuSeEYBqBbvOYN7YN0nm4yL0h4d+sSpN8XdCep+Y9XLbskS7ujuvqIAr4pR7FBOpNRjClyn/OHmUE3JJv2xYsu47hY9LHAP7jWuh69b+t9SojVXBjhhGA8KhwNPBrHerVbRHisWJncWDGNCfsfRHDPiwcBktnf8YuCNkBP8XEAdALJaVVathIq+y9suOqlWxlaZRuy2NqEakuFdojDwXCf/XuNtJdDlCMpUTH7OsT2y5+RAyFhJSYLvVUpCqh1bbxYXpd563AAk4KyxZr+TGZ5wSrcXm6FSYgLDwFDBwR/IvB+vhdzRLA0ZGRZIP1jHtWblFrGohPrXiKWsbVtx7J2ry95CL5sLPEdGXd4BilqhzxCLnv0ZjmfUe2tWAMOPOU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/fortify-string.h | 5 +- include/linux/slab.h | 169 +++++++++++++++++---------------- include/linux/string.h | 4 +- mm/slab_common.c | 6 +- mm/slub.c | 52 +++++----- mm/util.c | 20 ++-- 6 files changed, 130 insertions(+), 126 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 89a6888f2f9e..55f66bd8a366 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -697,9 +697,9 @@ __FORTIFY_INLINE void *memchr_inv(const void * const POS0 p, int c, size_t size) return __real_memchr_inv(p, c, size); } -extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup) +extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup_noprof) __realloc_size(2); -__FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp) +__FORTIFY_INLINE void *kmemdup_noprof(const void * const POS0 p, size_t size, gfp_t gfp) { const size_t p_size = __struct_size(p); @@ -709,6 +709,7 @@ __FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp fortify_panic(__func__); return __real_kmemdup(p, size, gfp); } +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) /** * strcpy - Copy a string into another string buffer diff --git a/include/linux/slab.h b/include/linux/slab.h index 58794043ab5b..61e2a486d529 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -229,7 +229,10 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +void * __must_check krealloc_noprof(const void *objp, size_t new_size, + gfp_t flags) __realloc_size(2); +#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) + void kfree(const void *objp); void kfree_sensitive(const void *objp); size_t __ksize(const void *objp); @@ -481,7 +484,10 @@ static __always_inline unsigned int __kmalloc_index(size_t size, static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) -void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#include + +void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#define __kmalloc(...) alloc_hooks(__kmalloc_noprof(__VA_ARGS__)) /** * kmem_cache_alloc - Allocate an object @@ -493,9 +499,14 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_siz * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, - gfp_t gfpflags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, + gfp_t flags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_lru(...) alloc_hooks(kmem_cache_alloc_lru_noprof(__VA_ARGS__)) + void kmem_cache_free(struct kmem_cache *s, void *objp); /* @@ -506,29 +517,40 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); * Note that interrupts must be enabled when calling these functions. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); + +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +#define kmem_cache_alloc_bulk(...) alloc_hooks(kmem_cache_alloc_bulk_noprof(__VA_ARGS__)) static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment - __malloc; +#define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, + int node) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) -void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t flags, size_t size) __assume_kmalloc_alignment __alloc_size(3); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) __assume_kmalloc_alignment +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) __assume_kmalloc_alignment __alloc_size(4); -void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment +#define kmalloc_trace(...) alloc_hooks(kmalloc_trace_noprof(__VA_ARGS__)) + +#define kmalloc_node_trace(...) alloc_hooks(kmalloc_node_trace_noprof(__VA_ARGS__)) + +void *kmalloc_large_noprof(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); +#define kmalloc_large(...) alloc_hooks(kmalloc_large_noprof(__VA_ARGS__)) -void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) __assume_page_alignment __alloc_size(1); +#define kmalloc_large_node(...) alloc_hooks(kmalloc_large_node_noprof(__VA_ARGS__)) /** * kmalloc - allocate kernel memory @@ -584,37 +606,39 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_align * Try really hard to succeed the allocation but fail * eventually. */ -static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); + return kmalloc_large_noprof(size, flags); index = kmalloc_index(size); - return kmalloc_trace( + return kmalloc_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, size); } - return __kmalloc(size, flags); + return __kmalloc_noprof(size, flags); } +#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large_node(size, flags, node); + return kmalloc_large_node_noprof(size, flags, node); index = kmalloc_index(size); - return kmalloc_node_trace( + return kmalloc_node_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node_noprof(size, flags, node); } +#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) /** * kmalloc_array - allocate memory for an array. @@ -622,16 +646,17 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc(bytes, flags); - return __kmalloc(bytes, flags); + return kmalloc_noprof(bytes, flags); + return kmalloc_noprof(bytes, flags); } +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -640,18 +665,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_ * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) */ -static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, - size_t new_n, - size_t new_size, - gfp_t flags) +static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, + size_t new_n, + size_t new_size, + gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(new_n, new_size, &bytes))) return NULL; - return krealloc(p, bytes, flags); + return krealloc_noprof(p, bytes, flags); } +#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__)) /** * kcalloc - allocate memory for an array. The memory is set to zero. @@ -659,16 +685,12 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags) -{ - return kmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kcalloc(_n, _size, _flags) kmalloc_array(_n, _size, (_flags) | __GFP_ZERO) -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); -#define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) +#define kmalloc_node_track_caller(...) \ + alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -678,11 +700,9 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * allocator where we care about the real place the memory allocation * request comes from. */ -#define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ - NUMA_NO_NODE, _RET_IP_) +#define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) -static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, +static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) { size_t bytes; @@ -690,75 +710,56 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(bytes, flags, node); } +#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) -static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) -{ - return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); -} +#define kcalloc_node(_n, _size, _flags, _node) \ + kmalloc_array_node(_n, _size, (_flags) | __GFP_ZERO, _node) /* * Shortcuts */ -static inline void *kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags) -{ - return kmem_cache_alloc(k, flags | __GFP_ZERO); -} +#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO) /** * kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags) +static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { - return kmalloc(size, flags | __GFP_ZERO); + return kmalloc_noprof(size, flags | __GFP_ZERO); } +#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -/** - * kzalloc_node - allocate zeroed memory from a particular memory node. - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @node: memory node from which to allocate - */ -static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node) -{ - return kmalloc_node(size, flags | __GFP_ZERO, node); -} +extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); -static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) -{ - return kvmalloc_node(size, flags, NUMA_NO_NODE); -} -static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node) -{ - return kvmalloc_node(size, flags | __GFP_ZERO, node); -} -static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags) -{ - return kvmalloc(size, flags | __GFP_ZERO); -} +#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) +#define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) + +#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) -static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kvmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; - return kvmalloc(bytes, flags); + return kvmalloc_node_noprof(bytes, flags, NUMA_NO_NODE); } -static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags) -{ - return kvmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kvmalloc_array(...) alloc_hooks(kvmalloc_array_noprof(__VA_ARGS__)) +#define kvcalloc(_n, _size, _flags) kvmalloc_array(_n, _size, _flags|__GFP_ZERO) -extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); +#define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) + extern void kvfree(const void *addr); DEFINE_FREE(kvfree, void *, if (_T) kvfree(_T)) diff --git a/include/linux/string.h b/include/linux/string.h index ab148d8dbfc1..14e4fb4340f4 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -214,7 +214,9 @@ extern void kfree_const(const void *x); extern char *kstrdup(const char *s, gfp_t gfp) __malloc; extern const char *kstrdup_const(const char *s, gfp_t gfp); extern char *kstrndup(const char *s, size_t len, gfp_t gfp); -extern void *kmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +extern void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) + extern void *kvmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern char *kmemdup_nul(const char *s, size_t len, gfp_t gfp); diff --git a/mm/slab_common.c b/mm/slab_common.c index 238293b1dbe1..5f9e25626dc7 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1184,7 +1184,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_track_caller(new_size, flags); + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -1208,7 +1208,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc(const void *p, size_t new_size, gfp_t flags) +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) { void *ret; @@ -1223,7 +1223,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) return ret; } -EXPORT_SYMBOL(krealloc); +EXPORT_SYMBOL(krealloc_noprof); /** * kfree_sensitive - Clear sensitive information in memory before freeing diff --git a/mm/slub.c b/mm/slub.c index a69b6b4c8df6..920b24b4140e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3920,7 +3920,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list return object; } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, s->object_size); @@ -3929,9 +3929,9 @@ void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(kmem_cache_alloc_noprof); -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) { void *ret = slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _RET_IP_, @@ -3941,10 +3941,10 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(kmem_cache_alloc_lru_noprof); /** - * kmem_cache_alloc_node - Allocate an object on the specified node + * kmem_cache_alloc_node_noprof - Allocate an object on the specified node * @s: The cache to allocate from. * @gfpflags: See kmalloc(). * @node: node number of the target node. @@ -3956,7 +3956,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_lru); * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -3964,7 +3964,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); /* * To avoid unnecessary overhead, we pass through large allocation requests @@ -3981,7 +3981,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) flags = kmalloc_fix_flags(flags); flags |= __GFP_COMP; - folio = (struct folio *)alloc_pages_node(node, flags, order); + folio = (struct folio *)alloc_pages_node_noprof(node, flags, order); if (folio) { ptr = folio_address(folio); lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, @@ -3996,7 +3996,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } -void *kmalloc_large(size_t size, gfp_t flags) +void *kmalloc_large_noprof(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); @@ -4004,9 +4004,9 @@ void *kmalloc_large(size_t size, gfp_t flags) flags, NUMA_NO_NODE); return ret; } -EXPORT_SYMBOL(kmalloc_large); +EXPORT_SYMBOL(kmalloc_large_noprof); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); @@ -4014,7 +4014,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) flags, node); return ret; } -EXPORT_SYMBOL(kmalloc_large_node); +EXPORT_SYMBOL(kmalloc_large_node_noprof); static __always_inline void *__do_kmalloc_node(size_t size, gfp_t flags, int node, @@ -4041,26 +4041,26 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) { return __do_kmalloc_node(size, flags, node, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc_node); +EXPORT_SYMBOL(__kmalloc_node_noprof); -void *__kmalloc(size_t size, gfp_t flags) +void *__kmalloc_noprof(size_t size, gfp_t flags) { return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc); +EXPORT_SYMBOL(__kmalloc_noprof); -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, - int node, unsigned long caller) +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, + int node, unsigned long caller) { return __do_kmalloc_node(size, flags, node, caller); } -EXPORT_SYMBOL(__kmalloc_node_track_caller); +EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); -void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, size); @@ -4070,9 +4070,9 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_trace); +EXPORT_SYMBOL(kmalloc_trace_noprof); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); @@ -4082,7 +4082,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_node_trace); +EXPORT_SYMBOL(kmalloc_node_trace_noprof); static noinline void free_to_partial_list( struct kmem_cache *s, struct slab *slab, @@ -4691,8 +4691,8 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, #endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, - void **p) +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, + void **p) { int i; struct obj_cgroup *objcg = NULL; @@ -4720,7 +4720,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, return i; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof); /* diff --git a/mm/util.c b/mm/util.c index 5a6a9802583b..291f7945190f 100644 --- a/mm/util.c +++ b/mm/util.c @@ -115,7 +115,7 @@ char *kstrndup(const char *s, size_t max, gfp_t gfp) EXPORT_SYMBOL(kstrndup); /** - * kmemdup - duplicate region of memory + * kmemdup_noprof - duplicate region of memory * * @src: memory region to duplicate * @len: memory region length @@ -124,16 +124,16 @@ EXPORT_SYMBOL(kstrndup); * Return: newly allocated copy of @src or %NULL in case of error, * result is physically contiguous. Use kfree() to free. */ -void *kmemdup(const void *src, size_t len, gfp_t gfp) +void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_track_caller(len, gfp); + p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; } -EXPORT_SYMBOL(kmemdup); +EXPORT_SYMBOL(kmemdup_noprof); /** * kvmemdup - duplicate region of memory @@ -577,7 +577,7 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. @@ -592,7 +592,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -614,7 +614,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = kmalloc_node_noprof(size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -643,7 +643,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(kvmalloc_node_noprof); /** * kvfree() - Free memory. @@ -682,7 +682,7 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) { void *newp; @@ -695,7 +695,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) kvfree(p); return newp; } -EXPORT_SYMBOL(kvrealloc); +EXPORT_SYMBOL(kvrealloc_noprof); /** * __vmalloc_array - allocate memory for a virtually contiguous array. From patchwork Wed Feb 21 19:40:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FF10C5478B for ; Wed, 21 Feb 2024 19:42:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F044F6B00B5; Wed, 21 Feb 2024 14:41:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E8DAA6B00B6; Wed, 21 Feb 2024 14:41:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C43586B00B7; Wed, 21 Feb 2024 14:41:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A7F896B00B5 for ; Wed, 21 Feb 2024 14:41:53 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 74D36A0FAC for ; Wed, 21 Feb 2024 19:41:53 +0000 (UTC) X-FDA: 81816831306.30.7F5ABC4 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 951F880012 for ; Wed, 21 Feb 2024 19:41:50 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TBtrfZ8h; spf=pass (imf30.hostedemail.com: domain of 3_VHWZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3_VHWZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X6Mc1467Tr+n3B+nCTUL6nWniTp2NFe2UCsWnkAH0JA=; b=rbWzXXLHfe8qJnj3KnwOFwhXgCP2TUJNOtSVCf6+CLPAJmIMGE82stq07T8ZFkt9BD0bIQ xopK/9H8ckPVV7WrvrmcWM/LryjpbfmWOx66Jx4TsvqjWcIPSx05uEP9T0/CBF8ophCuCJ ggh8XTEKyuJcF4kVYXnlTdsnUIcTORM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TBtrfZ8h; spf=pass (imf30.hostedemail.com: domain of 3_VHWZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3_VHWZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544510; a=rsa-sha256; cv=none; b=0BbYMx2yRgCKKzKdzxybFWXq80IdheSNpx85JTmPLW1W6BEaXUDl+U+Mr3LPk1LJHruZW1 0meK1FtZ1LBxXRalDzpiyDr2T1+xNzBHGh6JzZs7VCC9ypgMnyYKK+tFBgnUR9FW4Q2SdC KLRP/vv9DfoPsONcXEL016KaogzeXOw= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6087ffdac8cso18221647b3.2 for ; Wed, 21 Feb 2024 11:41:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544510; x=1709149310; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=X6Mc1467Tr+n3B+nCTUL6nWniTp2NFe2UCsWnkAH0JA=; b=TBtrfZ8hJUBjDgYVUyGmFxkWxOdGF7GtVsXLynGfDR8SsFdN2LUxZMQoncHuvlaSPF YI5ynWjgqGJOT1eNOld/U6CSGqY95Cy1a+LG8qWTt5qxenf3LQMGGX8MgV2MN3vdy2n2 R/BMSgXlTRHiTrnu9HCM1CjUCwcG9YWD3pAYqUG3IX4BKLmgThR/Vryxn3D35KDV4Jdn SmZvkz4UNjEQ6LiuDP4dvOP6znFBYT4aTEld629tZ66jIZ7ZIrx9jRNgMOn2kCVLNbXk GmdP6moCBQ6Uf3lQeh3BZilp6+GO+t2J8boRJxP9ggpncQtbJgZvNbEOVmte0GoOyuDm USSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544510; x=1709149310; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=X6Mc1467Tr+n3B+nCTUL6nWniTp2NFe2UCsWnkAH0JA=; b=dQt9ZVwio8OVXKE2SCDkEkiI+35ha33bYqFWZuR8c6/EDflOtH21rHcDpAKhNntCOA xrq3z02/t/oK6U7JE6rVHGnFiUlIthcnUo1/s3X+ysPkMylYZIruxHrSbow4v6CwrEZW I6bdANaRzXiSdz1SYbL1gzl+7PT88rbjIojW9+52Uu3Zi7YV0d7RJir/aUhfdR1CK1eB gdxuQ39+XcDoEj9eA/yWrlqcFi6XEBkVesTbSS0Bxk8/zJ5eqdu6zMsTGKH3OHAdQudP mFYT8lZ0DmXXgYZaxQxX8la6lX2Al2/1wKdKx0pwqFnPIr76MBFXpKt62oiO+y3vz7Af SMjQ== X-Forwarded-Encrypted: i=1; AJvYcCV/tIkxmYeL758K+snCXQhAqDwaiO/rp6eJy5VHStqi0Kd39RBvffFJHMLl5avI32120mrt6tL7OvznqGzRbZBMo1Q= X-Gm-Message-State: AOJu0YwdsBx7hVxYCJSQuigN2IDbHAXFwQMK1G9rFg9Et7waVNNURzxE 9rvEUDT/z1CrIu9YthociKPmhdWdEX8dNJTD5SY8OBgWwRrNa2BCKM1Iw89CijsYNlyjuEi6csq FNw== X-Google-Smtp-Source: AGHT+IFo4X8oIPfRZXy8Mvx3jkM76a/Fo26RcQOQzbZJGg3aZKwzpwtV53XrK+fKFAFr0Qz0jo83p6b+B/Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:690c:368a:b0:608:55be:5e3d with SMTP id fu10-20020a05690c368a00b0060855be5e3dmr1661247ywb.0.1708544509616; Wed, 21 Feb 2024 11:41:49 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:37 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-25-surenb@google.com> Subject: [PATCH v4 24/36] rust: Add a rust helper for krealloc() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Miguel Ojeda , Alex Gaynor , Wedson Almeida Filho , Boqun Feng , Gary Guo , " =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= " , Benno Lossin , Andreas Hindborg , Alice Ryhl , rust-for-linux@vger.kernel.org X-Rspamd-Queue-Id: 951F880012 X-Rspam-User: X-Stat-Signature: i9jtzuxppog6xqxo6ehaxtinh7bti5kp X-Rspamd-Server: rspam01 X-HE-Tag: 1708544510-90937 X-HE-Meta: U2FsdGVkX1+pyVMNBkdTG7+IhX84HeM21GjAiWEawGf5XwwhimRvW2n75QrLZzI64ix4YqjFO230sClGfswfkOykm9OM4OgRAtYSO6gGhVhd4R+JzogTFIg33BKO8lMVb02wUnwrHA7VOYJwzR5mlzAfCUPpPczdq4RIlHpT6stmSJm3FSYrawmH293aw0zs6TJdIcS7dw5wGBT4YojGtxog4n2HprpS3jfgz4MaxqGFOY4wudYDBmmmYhZmTRc/m7CIgoNO1Ghp5nbvsZiopN7eb0YOjOf+/bMaBZXUucesxsHrqUVFlrFWZ+Oplcxo3zmZ74yiSY/aErLw9BAjkgXsS9hXxFdYvO98Jj88cAiDQsPYtvArdFqxjbicv4/6xSinnvJrsAgIa/3itbvCxSm9PENbTrFVzzH+fIjlWnwWw58CLGDwRBF+PY9IFBbh6qpmz5FYxI55W4ngdbS4ODW/+8uyRTGOG9T+y/R9uF5skdZo2bi//q0pqflI/+/9Ro26kJmkdGedS4FO83NADt43RlSvk73+HhI1iO7ldDvisEH+l44vFg7Yth0E/mgWVT881H8dH3DuxOjZ93PkKc+fJ8Gv5yDUjwLCYx8BQYV7irZ7dcDhDrBL8HBzJsRbUhQ/N15rddSfG4oT7zPFnAQ8MDZeTAVZsj+hm1VZsGwC0ewzW5SKS0XuwSWR1ExRJ9B4Se8df56mWRWsy8B5W5idKHa8Oyq6iW3dKVCde1U1iopGVAK/2s2WpzQyg0me1L8lFnJGyXPZDAWxRtKlLv5LClCcGRd80P0/PqVUvEHxPEUZff2eF9Hef0dnGgA+Hdr5psxpG4VEGB/5l8/tcYXP/9oNCk85koJ8cEdSrMLFirfDLSGMmRSf0DnfiacNvDh6m/nc0tPCh/rf4FLCHVb0gGCuzL2mRaITWrYL4AR/MyUcrCY+W4WUnXyZcQJFs3xdBJMtNsLVmSJWksP ZHl8TAbK UL84A3Hro73JJHDCPnl63sqAT0wNelVSuG5YrL3s+MDm5IrKivWjZlJr/9PZ1bg702ii2LekE24nAikb3Qf3xcbeTRYAQof7OvJHYkBVqLwAabHxMLv9BQe3kdD7hkWPvTNuv63ATAeWIStNN7BaAGugH1GhUT9mN7+P2AP6dW2SlPQs+h7Yo+2DaK2S2hNoML5nf3gB5iZ9O1GYsGBRDQVlcq11n3SbAGW+bJy0ZdHqJqJk7fN0KQk6Zs0j2yLHRA4wvC2g1XBZuDC8NeS0w4ALUNJg5+VEWB7Sg8ZuHtTbkGiHlGrZNg3jEkg0zs39DSp2ewsz2FXOQ7StqBlhuS7fe0hqnsDnvCxCDjqKbULz0gkmbW7QBYL1w2vZLzPj7y/tMafHEGCbXZAbB6EsiNK8wBcQ8g5jJZlbSvdhwSzhTPwqLNRINnaR5PBYncgDLq5UBkrd6w3zPJja/PLtxLAbi1uh/azZ8qrERpDU6Z09jxS1EYc5icl2aaMysBaPBS91XZvq1HWpm+r5CzX/9MO18HFl95QERPgLb4F2oW8R0YIryUR5tOUzKlQQPkgb9ectettpOPEpqLdvsCLLtC+00oNJ0I0I6EicqgmRcVcjmx5YV9VrKqJIS1x3m71I+rD0YOy/HqLLsqMvbzT3zvOjy/hfVfyKxBRcfjlWjkaS5auInOA3CGcAxWHZXeuv/AV1e+3qGgDrMJA7LbDpFXdLqQoIPzE31Z38Ci0HdbPQSBtuMKTt2lxt+rtgohyaZ704AH1TIaHrpF4YRRkFNXSdB41xeT7QYlm05FqSTh2e47HL/fMj1urlB+uo0LlTF4BXHAy7aXz/TAyx/c+DlPs0txR0Cbc2FCEXCpF8eAqeMDejJhpTPExjxIS6Mv38aBaEsCMYxZGeDytiJpyu4sP7VrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Memory allocation profiling is turning krealloc() into a nontrivial macro - so for now, we need a helper for it. Until we have proper support on the rust side for memory allocation profiling this does mean that all Rust allocations will be accounted to the helper. Signed-off-by: Kent Overstreet Cc: Miguel Ojeda Cc: Alex Gaynor Cc: Wedson Almeida Filho Cc: Boqun Feng Cc: Gary Guo Cc: "Björn Roy Baron" Cc: Benno Lossin Cc: Andreas Hindborg Cc: Alice Ryhl Cc: rust-for-linux@vger.kernel.org Signed-off-by: Suren Baghdasaryan Reviewed-by: Alice Ryhl --- rust/helpers.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rust/helpers.c b/rust/helpers.c index 70e59efd92bc..ad62eaf604b3 100644 --- a/rust/helpers.c +++ b/rust/helpers.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -157,6 +158,13 @@ void rust_helper_init_work_with_key(struct work_struct *work, work_func_t func, } EXPORT_SYMBOL_GPL(rust_helper_init_work_with_key); +void * __must_check rust_helper_krealloc(const void *objp, size_t new_size, + gfp_t flags) __realloc_size(2) +{ + return krealloc(objp, new_size, flags); +} +EXPORT_SYMBOL_GPL(rust_helper_krealloc); + /* * `bindgen` binds the C `size_t` type as the Rust `usize` type, so we can * use it in contexts where Rust expects a `usize` like slice (array) indices. From patchwork Wed Feb 21 19:40:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14AFEC5478A for ; Wed, 21 Feb 2024 19:42:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BC666B00B6; Wed, 21 Feb 2024 14:41:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 31C5D6B00B8; Wed, 21 Feb 2024 14:41:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0AB556B00B9; Wed, 21 Feb 2024 14:41:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E48466B00B6 for ; Wed, 21 Feb 2024 14:41:54 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BF46612077A for ; Wed, 21 Feb 2024 19:41:54 +0000 (UTC) X-FDA: 81816831348.23.A796B81 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id DD88A4000C for ; Wed, 21 Feb 2024 19:41:52 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=koHPh63D; spf=pass (imf17.hostedemail.com: domain of 3_1HWZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_1HWZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544512; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YoA/f6AawOcdYOB/BdjtQvWuhk/43MsQdrDrIZmlE2o=; b=1J4B+8LC7quNVIYGmKgEAota1I/5mxmD1ypA+nf+omJvaeVQtHPh08g5p817qJAknSl+aT XDM2Pq9YS7y1f84H5RvSsmcl5LyveH+DO/vHuIKl8GmhfZMCWbUk5xbfiKrGnb7k5iZnkc Yi2QVQ3pGmAhFG3KEuzRcEUrqQYTDfE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544512; a=rsa-sha256; cv=none; b=55im1bf3ZGrTvv8Y5uUqRkoUEl1joF8XGwbyug3m4czk2FwSu376YYNFJ29KyGZ4iFBext x9sX1jYsy9oajHZCjnICvA/YIEMGYe2OwdrovJxayb8qd/Z061iDLfAhWNaD3RD805/XGR PVFv3vCc7O3+Rc+JNKxPrSMaCf1hB1s= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=koHPh63D; spf=pass (imf17.hostedemail.com: domain of 3_1HWZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_1HWZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b5d1899eso213911276.0 for ; Wed, 21 Feb 2024 11:41:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544512; x=1709149312; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YoA/f6AawOcdYOB/BdjtQvWuhk/43MsQdrDrIZmlE2o=; b=koHPh63D53uB2csa9wlQ5IJHJRvc5wF1D2qDWZ4q2j301V1RJq8hmmhLvj9alRgrHe n0NaPNdwPilM+mmjIU6tM+2L388d6ONe0afKPGb41hZrrGaKbsLrZSYLLVIFpoHZxDZh zzCfjoXFNghH5Ry5y054js958MFMlaw51WjO+oFtYXF0AT93GfXwo/WgGXKItRZ9DUWe kfPgL+WJOlertACYIjMDQgIm7E3fOQyjK+EF3KXaKOccYg6FLXcSFnTIRZ1iXzmzSo8S 9ucocL9hL/cRRtgrNlOKj8EDRAfBUyq9xh02AJhDkJWptb6zfWk302U1SA958ZRWe3ng IunA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544512; x=1709149312; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YoA/f6AawOcdYOB/BdjtQvWuhk/43MsQdrDrIZmlE2o=; b=f+IWqsgzM0Ii4NF6wGWkpDSdgQ+yZMYFBnzAqHds2BU2bms1qzne/Dlenxn2XY/R/2 IYKUfIs1s4gHgB2LKjuSUgIHscJ3V2hC9F4U5+1C7uRqzMmo7rquppPl+U4VlsybN09H MrIfaI6GExoEwIAH50m5UWWfa3eIFcytDk74XIya3MUUzdU/VxCuz1tr3Oo0hv3a+G4q GhSlbJPpRakIj5KhVg6XEmkWo2NaE4XOwYljcJu0QXgFYRJAgDWre9DD8gysB9xUx9t1 7p1JYG5zi53HNySUWllwrgztYo3KtICs/igp4XsR6uiMANV1Uxk8ijI60IOprHvwp0Bj HrKQ== X-Forwarded-Encrypted: i=1; AJvYcCUPqk2Vcrkw1MTDpBCXHcOXamXJLXBBBWvHtqBJfrnclYCwbEe4gpdZlmnJCSoVNUGMuhO+eZltFVo9eMC7fZw9I2o= X-Gm-Message-State: AOJu0YwZBPGrPdcYQEktDQFezpYX5Os2fuJhV6obs1GHBgza7UkuoMHJ KBtmdANZUhLt7jbuvvfhZKdpqj5zFWk9a3N7p+GvLnj5W61SMk9AiB2uDi4txfw7P+V4GZYbESe Tmw== X-Google-Smtp-Source: AGHT+IGZfB101wk4OhxM7/wqk5VOpnI9XqfFl6eMKnLGtgB+oOIFBy+VqLgKH5b+IHr8remNjBUIBw3H4Zo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:c5cd:0:b0:dc7:463a:46d2 with SMTP id v196-20020a25c5cd000000b00dc7463a46d2mr46740ybe.0.1708544511751; Wed, 21 Feb 2024 11:41:51 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:38 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-26-surenb@google.com> Subject: [PATCH v4 25/36] mempool: Hook up to memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: DD88A4000C X-Rspam-User: X-Stat-Signature: nxinsumxnb9s6pfjyyf8ni787rk5b6yh X-Rspamd-Server: rspam03 X-HE-Tag: 1708544512-454035 X-HE-Meta: U2FsdGVkX195pmTq6f4+I+fZY22OIEMWIS2q28iWeGcDbxpHf4r5PZlLFBKqwzYcKgkny3Xb0MjDFq4BsPocKKcqHYayu0scvA5ttVVFxPr6k1qEypq7CPhOkVklt0dl1KqRGqqxZaqAZqqCAbQOETCYhtGxMgzmynoAU5fu4t1dis0+MYm772D7EkneYeWKOJi8TV6aLOuOpZtnyQqjUHI+gUSHjysNWy+8fR/Sp/cBF5qlFlfWso2IflfDeJw/Olu3fA1a5RL8ng8pAfzFuK8hJLq1cvRf1RToGgYGWDM8KZUl0QxTwko/LruxKT16mgQpoJRn5vCjCmRIiCidDXL1uZKtDnw3u3Wr8sXEt+WmEr+8pvNU7/ITHWI4WBMgrg0mg5Hjy8TgmVfBBzVV2Dg5o16lIaWwouBH3CzRN7GVoF0iVFapaUi6DOSCB6ktqS1iXobZtpLSrzvplRU2cTVbSQuygAeM6HhUkn59CRMQIdlUffu0+uZJwrOXc/+x9Q0RDUBfuHE1oRHYUrDYs8m75c6hyUN/MUrr45/20wCHG5w9w0+sM2twUiy3Rn8QRLesmTogab6dwkQSVZxTyc/AIeNxgGqrpvYKtwly8u1BqpfPHX6vgUB/At/LQ8P3Bp4LEYFNDGpmFf8w1T+F9Yjangk/la87++NkFuUIhm5lwIO3DkDkLUMPCrygN0PrKFuY/ujkFomB5KPzGOt+VrDRib2RLANON1wNujXYwtFDNfDcfirks/fkokKMjJi4cwplfeGgpvgAdXvoP54GSSxSGRyh90dlhB/DRrahVIcEXgdK9YJbvk1f7V8FkV7UF16r7kNHkn2j3bYI+0Dlzw0ZJbmuYAXiJglbtUYHoEs6T8Hd/v2tjmkSoky/6SRHC0sRhT6SuIGwiE9zEX1+pqF3AtRFROUG2fNJJB4VMr3E5sj4kvIPndN4kppQg4Ev6s7i9yljfunjAw4dTBw fEbkSLkn 3YTghXOzuw8G7Lo+WYcBiv0+XQzxbeOvcZM1w/CSRcKJ2lRFK+L1UJCCRepvVsy24f+PHT0wde7AsZU2Jn21SYMW+fGdzSkGlMUkqa+my4gxIQP3E0mAY5pQYvb+hcEoDbfixk+QiuvBY1eUWJ1rmVH0CPNXdb+YF3ISfl4q5tqq4ZMYjzmyKdpSE2JPTa2R6nFxtmaa1IHyzE79j7KqN3ymRD5j4qMzSZoO5TLSo2uVU2L97my72c1+X+xPRdmJIQ+CsM0RCrsNOUoFHQjIxBcHZ64cfV/i8AfDP9wuhCa+FZ/kR0jyM1uWKrEee8YGBppP+dVNmhSkRilJ1RPU3KwBmJnsPqY/QVIQeR+KGcnkrX7vlskbnkN0yTU+YoQ/XDDr6wSgAfQ6SvlVbVfFwoPwV5Jdt3teFbUMJbVddRXgoA+7rROnShd6DJMMJP3LFvXHMUpxR7tvGiMfjwIziTfUCD6tfdY+9krvW6xqiflQK4n6WGms+YUSGCBv5PXaNJPD+9Fq5+ZXC0J1lXZkbtIQSfhd5sESgD9Hb+nUDYZR+9vxkvdQWR6Zl/QGnrks/VZZDLvDlSj71CUYzEObng8JZwsfgzME7xBPixixAHHkhWYzKCnaVEITCfUsND5qDLi+fFvtXF54jtFHaPRiU8wOqQxqg0Nk7SgDMSoyCM21MeQwSQz4jjC/Htw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations. Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 36 ++++++++------------ 2 files changed, 49 insertions(+), 60 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 7be1e32e6d42..69e65ca515ee 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -5,6 +5,8 @@ #ifndef _LINUX_MEMPOOL_H #define _LINUX_MEMPOOL_H +#include +#include #include #include @@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id); -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +#define mempool_init(...) \ + alloc_hooks(mempool_init_noprof(__VA_ARGS__)) extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); -extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, + +extern mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +#define mempool_create_node(...) \ + alloc_hooks(mempool_create_node_noprof(__VA_ARGS__)) + +#define mempool_create(_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE) extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); -extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; + +extern void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) __malloc; +#define mempool_alloc(...) \ + alloc_hooks(mempool_alloc_noprof(__VA_ARGS__)) + extern void *mempool_alloc_preallocated(mempool_t *pool) __malloc; extern void mempool_free(void *element, mempool_t *pool); @@ -62,19 +78,10 @@ extern void mempool_free(void *element, mempool_t *pool); void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); -static inline int -mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) -{ - return mempool_init(pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) -{ - return mempool_create(min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool(_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool(_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) /* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the @@ -83,17 +90,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); -static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) -{ - return mempool_init(pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) -{ - return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool(_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool(_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) /* * A mempool_alloc_t and mempool_free_t for a simple page allocator that @@ -102,16 +104,11 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); -static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) -{ - return mempool_init(pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long)order); -} - -static inline mempool_t *mempool_create_page_pool(int min_nr, int order) -{ - return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long)order); -} +#define mempool_init_page_pool(_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool(_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) #endif /* _LINUX_MEMPOOL_H */ diff --git a/mm/mempool.c b/mm/mempool.c index dbbf0e9fb424..c47ff883cf36 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -240,17 +240,17 @@ EXPORT_SYMBOL(mempool_init_node); * * Return: %0 on success, negative error code otherwise. */ -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) { return mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } -EXPORT_SYMBOL(mempool_init); +EXPORT_SYMBOL(mempool_init_noprof); /** - * mempool_create - create a memory pool + * mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function. @@ -265,17 +265,9 @@ EXPORT_SYMBOL(mempool_init); * * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) -{ - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); -} -EXPORT_SYMBOL(mempool_create); - -mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { mempool_t *pool; @@ -291,7 +283,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, return pool; } -EXPORT_SYMBOL(mempool_create_node); +EXPORT_SYMBOL(mempool_create_node_noprof); /** * mempool_resize - resize an existing memory pool @@ -374,7 +366,7 @@ int mempool_resize(mempool_t *pool, int new_min_nr) EXPORT_SYMBOL(mempool_resize); /** - * mempool_alloc - allocate an element from a specific memory pool + * mempool_alloc_noprof - allocate an element from a specific memory pool * @pool: pointer to the memory pool which was allocated via * mempool_create(). * @gfp_mask: the usual allocation bitmask. @@ -387,7 +379,7 @@ EXPORT_SYMBOL(mempool_resize); * * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) +void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) { void *element; unsigned long flags; @@ -454,7 +446,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) finish_wait(&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL(mempool_alloc); +EXPORT_SYMBOL(mempool_alloc_noprof); /** * mempool_alloc_preallocated - allocate an element from preallocated elements @@ -562,7 +554,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) { struct kmem_cache *mem = pool_data; VM_BUG_ON(mem->ctor); - return kmem_cache_alloc(mem, gfp_mask); + return kmem_cache_alloc_noprof(mem, gfp_mask); } EXPORT_SYMBOL(mempool_alloc_slab); @@ -580,7 +572,7 @@ EXPORT_SYMBOL(mempool_free_slab); void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data) { size_t size = (size_t)pool_data; - return kmalloc(size, gfp_mask); + return kmalloc_noprof(size, gfp_mask); } EXPORT_SYMBOL(mempool_kmalloc); @@ -597,7 +589,7 @@ EXPORT_SYMBOL(mempool_kfree); void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data) { int order = (int)(long)pool_data; - return alloc_pages(gfp_mask, order); + return alloc_pages_noprof(gfp_mask, order); } EXPORT_SYMBOL(mempool_alloc_pages); From patchwork Wed Feb 21 19:40:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 012A2C5478B for ; Wed, 21 Feb 2024 19:42:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58E756B00B8; Wed, 21 Feb 2024 14:41:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5197F6B00B9; Wed, 21 Feb 2024 14:41:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 258BA6B00BA; Wed, 21 Feb 2024 14:41:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0095B6B00B8 for ; Wed, 21 Feb 2024 14:41:56 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 922C6140B43 for ; Wed, 21 Feb 2024 19:41:56 +0000 (UTC) X-FDA: 81816831432.03.8AE49D5 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id C8558180007 for ; Wed, 21 Feb 2024 19:41:54 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zMRI4Ae5; spf=pass (imf24.hostedemail.com: domain of 3AVLWZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3AVLWZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SBIp7l6FYh66kLRcBRL71yRbl3V2EKRiwuHeAcwsTzw=; b=Qoalu7H+wZiVrt7PNA0qYo0KIq9ijtAhET9+HRWyi8vFv6+tE4BFwouPeT3ullAG3Ni6NL fIwlHyRxbO4APfx3Ishhbz3uPC4fYnwUrErZgY6C4gk4256mgEI/ZEFMTUTfp/SJKz2xkj 86WqV6Ajg5k1fE+L0OVjZKaVeKLYaNw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544514; a=rsa-sha256; cv=none; b=bO7n0+6chJ1MDlmNzBkj+dHMgamwWLsCAYNnWWtAQC8wLUdsktBwExqcqd7Oj4ndkGaFwY 4u75hPfW1cZH0TqXIvorcfYN1XkhMNb/reDUYG9KNKfVZvJ8KokULaGwQ570kbYxU9VlPT limHxXNgQ92glBD1la0eV2t3tbhdAuI= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zMRI4Ae5; spf=pass (imf24.hostedemail.com: domain of 3AVLWZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3AVLWZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6087e575573so22039637b3.1 for ; Wed, 21 Feb 2024 11:41:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544514; x=1709149314; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SBIp7l6FYh66kLRcBRL71yRbl3V2EKRiwuHeAcwsTzw=; b=zMRI4Ae5/o1RViZDLr1InOQ/OzdjWN5eXGOMhvqV8vLq+ETan1yW+HmbRCurn3eMLg 47WBpc76+cxniSfqlGI3wmmSJpJOaOuIiarpTGwQKlJ4AK/ScXmZW57uVyX1wHI27w04 QxlNpT9xLZNW7PbpYkNjR2100FezHFd5WTo89zsV0WiVdrSl6RLI0H59aH3j89B+s+YL nUCMpC9eNvHZXjvy/x2p9WVxxLMd0f6hydFtlvT56dPWzq+G1i6ZyjLlHa0fZvzWuI7b Lecu/EuHrQSxcBZlwK4WYvzEre0vXzjGB6OZiVK0T9f5FsHlF11tTdoG2U9SRthhcAjN 8UKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544514; x=1709149314; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SBIp7l6FYh66kLRcBRL71yRbl3V2EKRiwuHeAcwsTzw=; b=SeRTHTL51G0mYzIKVOBMea/6QsrT1PUrlXcIbkaB5WJwLCsri95WsayWOAOnoWQU25 9Bh0zO3FhPbMB1Luf2nWeAZ0DaOVu6BmiwtimaE73S2bb1x1qXW3kL26EfJdu+et8GsY 8vQ1YZjZmKQ1ZEY68a5k7c+udXYtpvpqUhAA60VXdSY/aw3Xz7bSmQdX7FwPkP3vlCCk Mt/ahA0gRaOET/oWA4HRXziPNwgk4cssoTGgAM9QApZqzJHH+4XCV5mUgU4M2Lt5xbD/ C/3p3yTymWpQh1jfJXHN0ebmomuevaxo5SaqFiB4qykGvyJHbGagbZWmp4+xC+ShVfFx x7Rw== X-Forwarded-Encrypted: i=1; AJvYcCVRzGzuX3xyMBNG0bluF9l/cQzQ7KDMkIkdyFFOgzVW8Bgv6R+Bmoq6Gwcwc508t7f66tY58BNDb9PbjIWWzGIcZjU= X-Gm-Message-State: AOJu0Yx4RKwEADA19YIqTZAKUEkOfA4r8NTplHIUUfm2w3EZe+fnIxIL 6NXm9g2Cnk8jQP1OBEd0OObOKtDxJL/PnSm9sl8IwrOltPocczS/9BF+j7jxSCHXstVnTrUxJ4k upg== X-Google-Smtp-Source: AGHT+IHLKaXp7N9l8d01o/qA3LkmgVnAKUksSqDfIHvWiz4IkTdB0wq71PyEfTle+YY/8d+F4iJoyhChRR4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:690c:3388:b0:608:40e4:d05e with SMTP id fl8-20020a05690c338800b0060840e4d05emr2214356ywb.7.1708544513792; Wed, 21 Feb 2024 11:41:53 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:39 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-27-surenb@google.com> Subject: [PATCH v4 26/36] mm: percpu: Introduce pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C8558180007 X-Rspam-User: X-Stat-Signature: 4gtb4zini1swz99kzqniem1fm91zzxez X-Rspamd-Server: rspam03 X-HE-Tag: 1708544514-974090 X-HE-Meta: U2FsdGVkX18RfGIqgo6uqqzCSM6o/k3wBSFUixQ5GoVBWqzMmp5mmrSZVhg2P1JUZV2XoP9lsGnxOkUbrpjZGldN/3mgbZ/lr2I049dF5CoJUhi+Vo5UyX71hM1KyBwoHxxFJZLtamtGw6iMhN2PV5tCZCUUdNsk1TAwhUtoSAd20HAdjf37ZV/iqRGIbfPKHm6emndWxTZ+YGVey5Fgr0vbrwOKl8t5TziF6Rn4tWTiidKyjDRn3tcsoMEIRIy0FHeUjskYiGa0iit4UiUm/s0zyRBA3Ejc4NM6v4v1j89pBCHEaRlFsflOO+ZFbG2jYQ0r2FH1Gx7zSX7E6hIFkHB6vDKeZ1RDZgUZ6wVX9HQ7Q3tMsbwcLBcUnFfbJeaprjEV8sKcPvKVCin8flrMxxGKWQ1+q2RX5cMiZownJbFIRrGAdWhxkZ3M31c9NyvqaMlPyiUtO37JrE4/TnEkh/SI+qerJ2HD8mrCaswMNycZvcUU85LG+JFGubSdQaFMF4F2UMq4vzjC+RobCauowVDZDbsb8GTR6KRZ9rHaZ+7dDQ3TqwCKMBNmK5xjrwiIkn1WUtQ2KsvU8EjhIsIdEOzxg1kaoBAPPgTj4nfJyhtNmcG5hs9hpH1zJAnd+CZbMQ+FXhuYrRVHFP6eQPJI9i1fQQzOAFn+WtriIOYHo4fXASkN60l7FPlCvp4qqZuiPndya92O3MvPP7Op81LbAIUXEICgJvaAaSCleOjeCA2LJRoq5yqovrMLbbZnnFCzaCoWZghIaks/N8PXaV+qFU0kaKKBWAdsBOCoRYeOvR3UEtSKyevVdDfSuD5Wfnhu96mwIpXmscTMxP0Q0uWIAUsTu+sO3R2qnxEYxsZXWQrHNhp5rkYQrVZ0Feo6pFg4xgLT4b8VFJ8aCtkq49I+sFLNuo/fgQLjIesEmwNKJk5GAMD17qgYPpnVxy79a4e+4crLb0kqanUniKd2cNx nR2dNJOK OnnpzTTJhsouCqGgHAO6IHd73lE57J3CDm+meocfJ18aRAiE7lmvG4AxwDFkCevG4GW6iaxI/yhdbcRqmuIzWbFoeC+26tO5duPWFpSqEoNQCGTKXeMna/f+1GXPH4fHozW0OKrQreFrWQh1ekqgrk15nhieihZNkV7fyQXhJ9DdiEXALv46AdpFO3NsEvPrXFvlsI6LRya3sauHbNWRL7n/hxuZrnBFuXDrwwX42QHYiBb+h1wvxBpAbAe7hNa8E3I1K1Hh/O95+HPOQ5iBenZh7TKUns3jcWhMbADCsfo2J41E9Hnf0zkW+XpKSY5Th9akwLjxbsTmQ50nrDhgextjmDhN1ruoVzsX/MSLr59KZ0G+SPgwSovXF0kTIEAP0jcBL3Zs0uppsw4yy9tnV/8zORQgmHgSIMuMv5XR+yzZX/1KdldVynJjE+cjqGN1bxsu4DbB6LgP3JvvbpofUB4GXIGpKXda8uCQV0UKX/00Arrm9JDzQBP8Q6JrpZwV7PkgEo9WoDBQxa3l/l4dp1bSRhuui40hGclD/CYBp9hWUgXjbmSDY6LCivFm16vxHYN7mojinoW4Mo3IwNrxRsa6uHPmRpBmMv6R65TDpk89ZlIpNSBAZHt1wCqUMbcuWTTMMbde2YY/tve7TSzrrhX3OLdBnos4O5CU7Oxs9ZtBoabzVPB4aRDJTHw2932tcoeG55azquUxqP5A/xCRcF/R5YL+L5xhkjfkH2CLR1THYFsksEvn7l9QZ+XaKeH7JtewDiVe3EN5cAl9WtSFUpiENR3zkObqLhJduG9HSswl+jgAAeH4D4QaAzcOvXoM7qIv7zli4454xCia6q4W5Y6eItGQNiT0cp4Vs0u6/SE9B1xPMGD4k8lwa5fzWySYmCjZj9wUY4ImhdVkrL+DlQ1sk49WvaMrNElVrRuZCRfgVZW38KLOalLDuYB4d59d1chMif320uZFN6qY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Upcoming alloc tagging patches require a place to stash per-allocation metadata. We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: linux-mm@kvack.org --- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index cdd0aa597a81..e62d582f4bf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -32,6 +32,16 @@ struct pcpu_block_md { int nr_bits; /* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif + struct pcpu_chunk { #ifdef CONFIG_PERCPU_STATS int nr_alloc; /* # of allocations */ @@ -64,8 +74,8 @@ struct pcpu_chunk { int end_offset; /* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM - struct obj_cgroup **obj_cgroups; /* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT + struct pcpuobj_ext *obj_exts; /* vector of object cgroups */ #endif int nr_pages; /* # of pages served by this chunk */ @@ -74,6 +84,11 @@ struct pcpu_chunk { unsigned long populated[]; /* populated bitmap */ }; +static inline bool need_pcpuobj_ext(void) +{ + return !mem_cgroup_kmem_disabled(); +} + extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; diff --git a/mm/percpu.c b/mm/percpu.c index 4e11fc1e6def..2e5edaad9cc3 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, panic("%s: Failed to allocate %zu bytes\n", __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups = NULL; + chunk->obj_exts = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT + if (need_pcpuobj_ext()) { + chunk->obj_exts = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * - sizeof(struct obj_cgroup *), gfp); - if (!chunk->obj_cgroups) + sizeof(struct pcpuobj_ext), gfp); + if (!chunk->obj_exts) goto objcg_fail; } #endif @@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail: pcpu_mem_free(chunk->md_blocks); #endif @@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; -#ifdef CONFIG_MEMCG_KMEM - pcpu_mem_free(chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT + pcpu_mem_free(chunk->obj_exts); #endif pcpu_mem_free(chunk->md_blocks); pcpu_mem_free(chunk->bound_map); @@ -1646,9 +1646,9 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, if (!objcg) return; - if (likely(chunk && chunk->obj_cgroups)) { + if (likely(chunk && chunk->obj_exts)) { obj_cgroup_get(objcg); - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = objcg; rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, @@ -1663,13 +1663,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { struct obj_cgroup *objcg; - if (unlikely(!chunk->obj_cgroups)) + if (unlikely(!chunk->obj_exts)) return; - objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + objcg = chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup; if (!objcg) return; - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = NULL; obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); From patchwork Wed Feb 21 19:40:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5F6CC5478A for ; Wed, 21 Feb 2024 19:42:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E69D26B00BA; Wed, 21 Feb 2024 14:41:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA2D76B00BB; Wed, 21 Feb 2024 14:41:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCDEF6B00BC; Wed, 21 Feb 2024 14:41:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9F46F6B00BA for ; Wed, 21 Feb 2024 14:41:58 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8315B1A0B79 for ; Wed, 21 Feb 2024 19:41:58 +0000 (UTC) X-FDA: 81816831516.21.C35AE76 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id D622D4000D for ; Wed, 21 Feb 2024 19:41:56 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p1kvq0xl; spf=pass (imf12.hostedemail.com: domain of 3A1LWZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3A1LWZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544516; a=rsa-sha256; cv=none; b=kV+Be4r6A5RmW2M9MuXZo+BPO1X+dfOeUr8EIqbcHjmUeZRZv9Gqfn1q8KIAta2OZMgJ2l d/L/Gs8EktlkxyMrWstWkrH9fv7NqkhrS/WDbPavq7MXMn39ulHwMTla6t028pffrWfy0e yYWaZBidr/nkTzCOvt5djxY9k5Q+tKk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p1kvq0xl; spf=pass (imf12.hostedemail.com: domain of 3A1LWZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3A1LWZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544516; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uHb4H3aBf7zVkxjSohmSP0JQMXLGrYpxjhF/b1cUoKA=; b=bjH+kXx+LD2Olk7GKatB6Ux1x/TNdLpZq9eqN6N8FMy693YE0/VT9OAbfxd1i2GNdBY2Gc D22bu1ajf1s7F5MCLpHwwtV0GbyyTdEWPPwGart+t24h59eMG1dcsq/5RnBFMwa/a6IifG DyFmCMFGABgljSYYj3Vs8QixhgsG6xM= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5ee22efe5eeso102886307b3.3 for ; Wed, 21 Feb 2024 11:41:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544516; x=1709149316; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uHb4H3aBf7zVkxjSohmSP0JQMXLGrYpxjhF/b1cUoKA=; b=p1kvq0xlQQNEpL9+f0lsqoXRPemupZ5P/Icp7q5XILoaHeO9CUknNMfrRTIRAwVooM a401kEO8zt5pMd4UQoUQkCZKAGjRgOBXLCSHk00tSfr+30SiqBA5+gZFeAXk046zxLL1 UrZW7T0PXzdLH9w2jDJDOBDbd3c9v9wWpHJWdUfdb1cHCAIW1uxPA9GFulqxtorQ8at1 nhy4cAYhhkDZ5A+zA92qGh9U/URYh15bk2Lagr4IpSdyxlFVlk90W3zfON+JiZiyIqe4 2EcjAViuir3RgqgCPVLcZcwyejt2xK+R5B/4y/Duyee/uLAL2NGlPoLfe7nmOA47EbKa uIqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544516; x=1709149316; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uHb4H3aBf7zVkxjSohmSP0JQMXLGrYpxjhF/b1cUoKA=; b=ul0GvhZdcHlNn3iy1nHqi7CGhkQQlY0cdKGsxlyMyJXImhxXF8lh6S191e4VuqR8dB Nm8tGiwv9yJ90K6r7pc88Cu895C7fNrVOOg1CmbjB1SP7J+L8A+ejx0oWZHcxYofuTXn yq3uskRT291QuXa0S0wCq1ipLuRhykaVOKUw15tftGctVAyI+tpzuIxRtCUdJ3sZxU4l ZtokTSyEHxVbFw5HrVJA97qdhy1lEV97V+fDa020ZxAfG3brGpthbVXapyk+qFjZc1zC FdxaFWrj0oTXtCOrvV26V+JZphNVc7TlRh2uwZpd4nbjhEZse15k3X9C/OUrgpf/rQZP ifcg== X-Forwarded-Encrypted: i=1; AJvYcCW9RRTAGYSWXCZrOD0xanRqAWKkcsHsix3Z1PahBUPxJBfJi420xzwPeD8+Jady7ScuoXyUmxNhsV+75uUsTs1WSr0= X-Gm-Message-State: AOJu0YzcOoHFu0ESyR+MVW+y9AuNWPGtxJ8n/CW6HF2H6IFXDRC35eRE 7ea90a+u0kSEsa4u7BqIfnpQfzwgdNsbdzVT7cLyqrWZYHTB2bmPJaF8XHHSYkW6xXMTTNWvLAy zgA== X-Google-Smtp-Source: AGHT+IFiyKMub1eGZLzZF9NObSWfIJfU8Z6wUZOBIvD/PODcgJqUEzoNtVZWO2NPbtCUhIy8HmGJP9rxfPQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:d5d7:0:b0:607:9bfd:d0bc with SMTP id x206-20020a0dd5d7000000b006079bfdd0bcmr3270706ywd.7.1708544515839; Wed, 21 Feb 2024 11:41:55 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:40 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-28-surenb@google.com> Subject: [PATCH v4 27/36] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D622D4000D X-Stat-Signature: kfc5u56zgptsqidmbcrmpiks1zfkzqap X-Rspam-User: X-HE-Tag: 1708544516-9248 X-HE-Meta: U2FsdGVkX1/IiG+CW6AvG7OeXoDf8ey6uO8Fr0EYs7opEA1ptTgHDJlyhZBolFvXFXE+V19tZfx/IpIChNA+pCG5riF+FQtZAJe7y8v8i8neGBE1xBqhHpJ67/5oacE3uL+OMF81ZtmgIVjgP6fwzu5pJjZLyZlF/rVSrlK2xPJfEIi9Zvx8tgaFdVfhGYjZ7Q8ao0o83jB+BZm7+4eshGewSMnyJ5SHmRW+1BqFY9fSQmu/2evLqYoYk+M4OzfqLoPKVetdJMDBF63JOo+POmE49F7ZvvtUQ8rl7MbOzz9xd7g1ePuhf0+Yyp51m9pcA278IJ3SnTSYcs7wXs4W0v6XcxXy9ba+WRrOAfN041w0TH00Higi6LrfHtiGaYUsafIrNNGGakNAUiMBxokKRcJt63yfarfqITY8ykKV7m46/1bovayQ6c9/42zxA6d4Mdm9AsdJa3CaqyrP3V4D8BtpC3vk2w0vsErSfBSDa2pTcVOv1T1eAXOtyjoeuZYmplyeJiLG2UPbYmE7zc3PAr6Y3SOL3WXlNy8nsDSKLWzO3nGziO84EiJuqav+G+CnL7hGOeXuX+lFH4A3h499p1Z70kNV0KOuhfurHpDDmV0zaUrv0oetTrgvMFb+CQvZjcwMkR76e6NUa6h7TaNygIDyCsEZXqfil+tn/toiS6HVcl/YJ3clxSAF3jJ8TkZ3SZxAhEszIxiF4e3Fy7W4GXJaHQ5eH467TtJJcWpgDboJCXWc2wFME+eaUbZAFRRbNHRuaMjP2ZgcrUlbyS2GEbzJlDF4oxMQ64mWmYcYKKdSzOfKy0jF9R3vK167CQlcbBvZaxVPmjrCHOmawuUjmYR+n8hxwIi9Uvh5UX9HZ7p9/9uX2mFI1LJtc2ryCLVDyh+0iN3pxTgr/okHDyCj/9tDlyiK/4yELJyGbxA9/60NJwETgm2sOIuyjWFOekaYxPhoGX74l0oaj5VOL5u rQ6eDGHd TwA7MlwxKnxAJpPuw8mLdqDjG+23C2r4JJV453LzleZy3k+Yvb1dJ8iWgRx+IM/Uv76diQkr7jRPLKmvjXLSiAGS0NHBR/Vwsg3cCIHU9EtdGu3N7R1s3QnN6kSIlY0WKq+xWY144vVY78iVonbHPL8aWJAyOHmHGprewEpaSjBVwylv5TuwHz9Kvh17JbxN730fVP1UWEN8Rps1Ky7SKLj1GEhLvjzK6CI6jnrjUp8Uf0ZozLeb7vE7xsaSYELINpz7ZUbaYBmeXGyPeMK5n1E6mCuWN31YakFcjDw9rNkfZXTdT0MmJlan/2eCzJ/Qwc3ctJBo2xTpv6ebfgVbVTD6E2vAKIs+Gr1SoFKDWfMxsHS+1s3JWpwbfAo5qHkogaNqMD/Z2RoVjoGAefMHpBv7eks3ZHaC8+YcKT/9n5Xt8+5KUddU3ah6gO2kbeMXTbLxL+006YwbIcCA/+t0Kq3KdSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index e62d582f4bf3..7e42f0ca3b7b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -86,7 +89,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 2e5edaad9cc3..578531ea1f43 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1699,6 +1699,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub_noalloc(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes From patchwork Wed Feb 21 19:40:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 916D9C5478A for ; Wed, 21 Feb 2024 19:42:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93A0A6B00BC; Wed, 21 Feb 2024 14:42:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C0C86B00BD; Wed, 21 Feb 2024 14:42:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 712A76B00BE; Wed, 21 Feb 2024 14:42:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 49EB26B00BC for ; Wed, 21 Feb 2024 14:42:01 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 19CC912077A for ; Wed, 21 Feb 2024 19:42:01 +0000 (UTC) X-FDA: 81816831642.26.63AFA36 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 1A1F720022 for ; Wed, 21 Feb 2024 19:41:58 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dDWjvhoY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3BlLWZQYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BlLWZQYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YEtQtrsg9A+6qi4G17OgAejNr2caNgy1uwZYQh9NpiY=; b=0mLjJxwp85ppzs+svTFgT7Wwhi7qe5qIrxdlXSGJWqMGWAel0vNEsAn0+YivncVXdU9RkX VGWxXVjlnEcSDNCJcJo2pcVzTH86mvKeHaGssgWiT4hGUlsFNyUyv6fcewr8A4zQxdhXiW WNY/WYCTIR8qJv7KVyOEPcz7lT75NoU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dDWjvhoY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3BlLWZQYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BlLWZQYKCDknpmZiWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544519; a=rsa-sha256; cv=none; b=Mb/CzG8q98wUMUE38CTyBAbgStPArol5+CWqdyWSvp/GahfEydY7fn3WpyVthNVtbXswzK r3Q3I8EJg4uT+aRXGiz7edn2K2LK7yiyGsVu9TMfB8L22UK5p4/6wJwp0FxXGCDzRaav2u figr1fSWXtTTFmGfchBvggRbHXDIOaQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-608084ce3c3so17028587b3.0 for ; Wed, 21 Feb 2024 11:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544518; x=1709149318; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YEtQtrsg9A+6qi4G17OgAejNr2caNgy1uwZYQh9NpiY=; b=dDWjvhoYrmcv4CexN9/PskwfX/JaO26JdSgb60bgNV3dp1GDNfJ49Xcpe6nhJ70FlK t6H+PjEIAcRqy59tPXJgexJ8yQjXjWpPJlWwrHvX80fNxGij+vyNh6rWTRTWvpTsYKUT F2+mZRPh/UxV/j5kcYp74m8VVRvO0m6CjxRzHSMiR+ZjcPWMc7ry7Nm9E4J532zvI3/A RxrDmVe+Plk7w2B2u8jvdrxgtnNojuFxYtmn2AM0T5BrvMmX5+xDv+kvBYWyhqQnbw3l jDNUMP2BSp4sDZJqUz37stD+L6g3sOOx19nocFeX6jaUS78vF6F5OkwyQQB/8EHeyoIE LoBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544518; x=1709149318; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YEtQtrsg9A+6qi4G17OgAejNr2caNgy1uwZYQh9NpiY=; b=YANY1OA/N8NwtZkcYzrkkfxYH3ZTkS+vl4YQImwCSfSOUUQCsFicpH/eeLr3x0JmGj tbRwstg4XOy0Tty+bZ2yBJ3RzLTOkJkb11B+vKkqyJLDmaJ/TInHwOY/uIWFMwLLYu6K IgDoAC76U2g+mbQyaHzlBJ20tAo++GQ8Ik8PHHdVKmV34syAVPEjqlq8s1DWTyzC+Auv e4TUj2Sn3PGpJprnm1ySEL4A/xqKWh4Ftb5j9t/IewBIwEhRCLI4IRQz9lBjFEnyAdzP tix4lbDe6D5AKliR7X6lfa1UNRTF28yuA3bNVsjOiOh/UiRm/0sLk6E/vntLx4xI+Q5r E2dQ== X-Forwarded-Encrypted: i=1; AJvYcCVgnrH4RdHSUCw9YwNAPIEizgmP/yAgS2tExL66i/X7vnw4k9AteD+LvCTDMbFo9ftMvY8ubTLdnbiAhKz7xGGoay0= X-Gm-Message-State: AOJu0YxrnTsP8Pns0PT7iJSh9H8QKPgDxjXLLlu9pGLjcjb2+1koLXnB ZUIaHujVJUX16yOcylN/zs20xySPT9S1AI+DCqvE+FhU3b1oiN7Fo+6XC7t4+DKX4J6ElENUFms bRQ== X-Google-Smtp-Source: AGHT+IEnhHBEks36/xCu/yIm+JxPq4OifsOn0+nP+Rvhl9lD+QhdtqS+LgLqlaoHkRcwKrzKU7ev2mWoy/w= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:690c:3388:b0:608:406d:6973 with SMTP id fl8-20020a05690c338800b00608406d6973mr2030736ywb.5.1708544518169; Wed, 21 Feb 2024 11:41:58 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:41 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-29-surenb@google.com> Subject: [PATCH v4 28/36] mm: percpu: enable per-cpu allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1A1F720022 X-Stat-Signature: bws3t338u6z51m7bwg6x5s8g43siwhtc X-Rspam-User: X-HE-Tag: 1708544518-485429 X-HE-Meta: U2FsdGVkX1/pa2BrCtFl2Zscw7L/w2TaYWTtbkzhaHGAroe5oxJz66mquGjDyPJAZDqgSJypAZhdREgxwlQrarD+Vo7eXY/0984tIjdSBXMVCbZHQ0WuChVv7FtCTbZ30aI1WMyH0VXfR4Xvtg+e+SXYSjgf5HsDdwdNdEtnd8iVGsPzXKyY2+VBwBxrkAzaquP3iFPWMlNoAdEtKUdmDBM053jhPUCjS0cnuQBmuDSBPlEAcb7+A8UptiquDiMXLvM6brZ7uG0Q2YPl+3BQEW+ayscaYhaSQDr1cnUAjBpItOH94PtHrf3ElojOVbb+xZ0jFUtnr6q/Cg6Tunx/Zw6/fuW+4NoWXtVUQ7mu4EA+211Ub9OmDg15j1MQmcXky8aeVPlj36FBY7VpnxfiDAOs6DTYHK8cRcR94vM9UMmz2SOQpEwC7iHo3BFJP4/PWcR1oLCPZql3lgN8HFYa3GnEIFo3IABT2u2qP/veouKSetFzpO3n6oYf+aQaxqnEl+H/2WhmTinqQMsdOkYgD5oJxUOLuu6ex1g0T8iJ6rmSAskfUz/uYzfF8OgSpsNkaGcsOO7AkQnccj5t5fnxWcDevN8gJ8ri4bjYlbzU0K3qL8jI+sh70WlVG0SXk54zvCM6mW8XhN37mtN/YlHf19H/5UJPy9jeePq+URvmofS+5QXlnq+Qoq+kOABR33w270zM2s5ksHWSrJkDtieTNKLC3TugtfsBM8XRJuNV2+3St2qpL/q08XvUonoNwIEH1pnnXZrvWSopcRMCHh4/v9i+s5oJiPg73/yPre6i/u/i4+53P4lXsUhsemXNwWvs5xwPriybEqJj3mw/rK7l2PEcIt5/TBMI/QICkOO23fuZ4lLgm1dz/SdOPdCnq939fDmwPJ9W9f53zymiEUAmx9KVIn9aXWBAALZgeE8eAumL9EqImafuwrKCZRushEaA6u7VMv82lwIqNpD8Esy nhmwYF9R Qvzv6dAcWIy2pmeAzCQmDKBbY4/oD9f2+nRnebInDqPg1EaTznZEpHFBBmwlkt4leQlN4gBGAGa0PUmxWIIm33R5HVviPv9NHbQk34Yvy9Ok8ehDHjEyQu7Gx202k+QVu17q+UMn2gxq04XIEbHaC+s6XKvCWkWhDo0llR5c/jjI0tsIqa/K80XEOVdDTRpn8aB3h/loyBhb7KBFRTuZ26QZS9ISdodSX6/hlw9t0BXHnZsw/AQ+DevDshOSad08D8eSMpUQiODhyr2gm/4uA6lmBooS4M8wuAqIG1tbbxXUbmlXf9RWPZuUv7iMzdJIy0+wB0eg39Jb4sF2MyRuTSlNRJszfQ6oRhhI0TOUGUOy09XwoOeq4+RcMF+Ifzly9clBZpzkBOsKrTNOQ9wfdx6XbQdK3EU7IQZt0k3cWZVoQdyWPlaKbDg78rovONhyyFaNtOqf8s/cCgVIZpNA689wxnd736oKhN1Yls3f65RCsXHbsRbYTYxeKMpsqEXInQTpCvAl0+jj8nq1GWQnR2JfG0MRGWa72chn51SoX18Myri7O5tmxR6JXTa9tVFExV7snvjkLXNXyfkR76Gpzn2is5MbTUl8xdgkjKwtE95g7Ewq3A4Nb3yJIWVsi8tdd8bdRopKdiK34bIC0r6Iq0qTvPuWiOymjZHICGFA+1AFPJLI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/percpu.h | 23 ++++++++++----- mm/percpu.c | 64 +++++------------------------------------- 2 files changed, 23 insertions(+), 64 deletions(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 62b5eb45bd89..e54921c79c9a 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -2,6 +2,7 @@ #ifndef __LINUX_PERCPU_H #define __LINUX_PERCPU_H +#include #include #include #include @@ -9,6 +10,7 @@ #include #include #include +#include #include @@ -125,7 +127,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); #endif -extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr); extern bool is_kernel_percpu_address(unsigned long addr); @@ -133,14 +134,16 @@ extern bool is_kernel_percpu_address(unsigned long addr); extern void __init setup_per_cpu_areas(void); #endif -extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); -extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); -extern void free_percpu(void __percpu *__pdata); +extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, + gfp_t gfp) __alloc_size(1); extern size_t pcpu_alloc_size(void __percpu *__pdata); -DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) - -extern phys_addr_t per_cpu_ptr_to_phys(void *addr); +#define __alloc_percpu_gfp(_size, _align, _gfp) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, _gfp)) +#define __alloc_percpu(_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, GFP_KERNEL)) +#define __alloc_reserved_percpu(_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, true, GFP_KERNEL)) #define alloc_percpu_gfp(type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \ @@ -149,6 +152,12 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu(void __percpu *__pdata); + +DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) + +extern phys_addr_t per_cpu_ptr_to_phys(void *addr); + extern unsigned long pcpu_nr_pages(void); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index 578531ea1f43..2badcc5e0e71 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1726,7 +1726,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s #endif /** - * pcpu_alloc - the percpu allocator + * pcpu_alloc_noprof - the percpu allocator * @size: size of area to allocate in bytes * @align: alignment of area (max PAGE_SIZE) * @reserved: allocate from the reserved chunk if available @@ -1740,7 +1740,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure. */ -static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, +void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp_t gfp) { gfp_t pcpu_gfp; @@ -1907,6 +1907,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + pcpu_alloc_tag_alloc_hook(chunk, off, size); + return ptr; fail_unlock: @@ -1935,61 +1937,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, return NULL; } - -/** - * __alloc_percpu_gfp - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * @gfp: allocation flags - * - * Allocate zero-filled percpu area of @size bytes aligned at @align. If - * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can - * be called from any context but is a lot more likely to fail. If @gfp - * has __GFP_NOWARN then no warning will be triggered on invalid or failed - * allocation requests. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) -{ - return pcpu_alloc(size, align, false, gfp); -} -EXPORT_SYMBOL_GPL(__alloc_percpu_gfp); - -/** - * __alloc_percpu - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Equivalent to __alloc_percpu_gfp(size, align, %GFP_KERNEL). - */ -void __percpu *__alloc_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, false, GFP_KERNEL); -} -EXPORT_SYMBOL_GPL(__alloc_percpu); - -/** - * __alloc_reserved_percpu - allocate reserved percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Allocate zero-filled percpu area of @size bytes aligned at @align - * from reserved percpu area if arch has set it up; otherwise, - * allocation is served from the same dynamic area. Might sleep. - * Might trigger writeouts. - * - * CONTEXT: - * Does GFP_KERNEL allocation. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_reserved_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, true, GFP_KERNEL); -} +EXPORT_SYMBOL_GPL(pcpu_alloc_noprof); /** * pcpu_balance_free - manage the amount of free chunks @@ -2328,6 +2276,8 @@ void free_percpu(void __percpu *ptr) spin_lock_irqsave(&pcpu_lock, flags); size = pcpu_free_area(chunk, off); + pcpu_alloc_tag_free_hook(chunk, off, size); + pcpu_memcg_free_hook(chunk, off, size); /* From patchwork Wed Feb 21 19:40:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01AF1C48BEB for ; Wed, 21 Feb 2024 19:42:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C71866B00BE; Wed, 21 Feb 2024 14:42:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF6AB6B00BF; Wed, 21 Feb 2024 14:42:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D3B16B00C0; Wed, 21 Feb 2024 14:42:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 859D66B00BE for ; Wed, 21 Feb 2024 14:42:03 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 61B2912077A for ; Wed, 21 Feb 2024 19:42:03 +0000 (UTC) X-FDA: 81816831726.08.D7FA9B7 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 7D9E7A0021 for ; Wed, 21 Feb 2024 19:42:01 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eh1mfdTu; spf=pass (imf25.hostedemail.com: domain of 3CFLWZQYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CFLWZQYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544521; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NoRXBaM3T4KBY/6WQ4BLlRy2g0mzIwYzZZ7lc9Gsip0=; b=loqSkMODAAnIOtrRjWwnF/Gxoe7nIi9Kiy5fOzaPqIaCU8VtR+3C2XI6LjJonZH14tBFJ3 hAbWHOCG3vHqXeqdtbpma7DkmaFeHcpNKGQLrkFcxgILLZrgiNU5DwDkYJAgrbKYdpPqCF K0pwptxJl0HPuc3sfYpP9dtBDt9BPsE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544521; a=rsa-sha256; cv=none; b=wS4P6WJyZXwFt6TVYtm9lAvNLvaCrB8DmiX5mD3VDr/65xVVk8Uml/u/3Li1ypl/ZREoEt vw6VyVPi8uzhAA5eHeiqONIF0rswlHhhSPU/McxTxZo1Jn9rVb63KCcPJRyMEsWzsa7r+L bKPFgQXEgKDWgO/bbnLhnBUjtIYKNDw= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eh1mfdTu; spf=pass (imf25.hostedemail.com: domain of 3CFLWZQYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3CFLWZQYKCDsprobkYdlldib.Zljifkru-jjhsXZh.lod@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608852fc324so16617077b3.2 for ; Wed, 21 Feb 2024 11:42:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544520; x=1709149320; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NoRXBaM3T4KBY/6WQ4BLlRy2g0mzIwYzZZ7lc9Gsip0=; b=eh1mfdTuUl12KrYcVPrCdJtRg4xNNvgZJ2ylwRoG8YJHQce8Jdr15cBfRHijZtNE1E 3UKNScLwldTJMuGHsVuc8PXJRWmBHJjPBcLn3yQXDXiw/HbAXh8J8PEXMkTRLjNMAKtG /67P3c5lPnzWbBeyfcSdl7MBhEmlYrPadH5vGDxDVc3HKKm9sOWdgaCagFaEo9ZJJB1k q8oQbi7JuSQ1CKrwmR2+QxgdiIEJZhEcGmgLYbC5FAhnSBRw5Ekq2OhzWpUr6B/GBYeB uenkz3G8poRBVjj+h4Ze4tD9VJRb+CpO1TZfAEPuO5W0Ef1OKaN5xH1h9xWHJHSPWCQo BHBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544520; x=1709149320; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NoRXBaM3T4KBY/6WQ4BLlRy2g0mzIwYzZZ7lc9Gsip0=; b=SqnifbC0WuhJSIZwEnB0Cp4MX/O9UmY4lpYtI7Ph61rEEidft2PxKMeQa+AB1n9Ibv yVkRJLoiX6JkhpBxB7+98+KurjCj15EjXIsttSWDxffW91yqmTZ92h2/WlElnCYLjqDn czMpKeUj4OSRUvEjIYCgcYd6lpdQkYuhPv8yprsasC3RKoez5rUawsD9JUjtetEhzDNp OPYq4xyePr/3zCgQefnMzQWNByEOUIf9ca/88Dppt58ICKHR+MNxlLV7/ABgh3t9S3vC 9T2kUaS9czwS4lhaH28YfQyvtoRe2qRGGwHouwrya4BejtYg4COUg9fcR6OUVceOhuip F/Xw== X-Forwarded-Encrypted: i=1; AJvYcCW0ZZ8jWjzFI3oguVcEO/+onBWKXxLtbVVY2K04pnvHFSs89nPwQtiBlJHELh9zuWr7IZL/2Juqi5ojwby+mm0+cBI= X-Gm-Message-State: AOJu0Yz8ZCRieDlJD7uZA/XT2c4+PP3XVe5t4cODx00yBS4ep9Bfgu4V eQfeL8RJya531IeSbsr44JGQAPBnAL7fsFIqpxgnBMf34xPV7c8xVWVOk6THb1fKh6zNck2GecM SWQ== X-Google-Smtp-Source: AGHT+IG1cmJLMXAWhkOV/br+PjMHR2GKUnB/GnIEF9I88UH1pGaMUb9WVwYhhuTb4WS0cyEWnc/zHgxybpQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:df4b:0:b0:608:5e12:ba68 with SMTP id i72-20020a0ddf4b000000b006085e12ba68mr1532961ywe.4.1708544520523; Wed, 21 Feb 2024 11:42:00 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:42 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-30-surenb@google.com> Subject: [PATCH v4 29/36] mm: vmalloc: Enable memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 7D9E7A0021 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: roieeg3uu8td3zk5szmq3pfeed3s9aab X-HE-Tag: 1708544521-699199 X-HE-Meta: U2FsdGVkX1+I3+jJZguzMXHjxYI9LO4xJop3rYTY+AAvgdY0KUkBmFQrK5rtDHi/7Vr8CMJI8HHYH1YEkMWH/An6QdKgm4o7EJnbTOZRlopWeZ0QcMFvqyTs9VTAchulXTK1+b7SJJcKkp17cUJtJLrNWB0L8XIojKAOLQYag060H6L4HyXpGfRk/URypxVbmeT1VAzv291XPiGcRKZRYic2hzqvvg5vf79nYHnV1agrXxa/lnGWxJnfHlrh/d+CAKYerbnjKjfZ++dSFeotuXs25efHfB6R/EH2EuPUZjZL9C4Vct0ZamNITs+G5Hyl3c2jIfLZWSMT6YbcRR9UZj5oDMI41YlD/xi8RxxbPwrncCXpCBisB6KAB18ArT7LFtd0L/8vOnUTzc7r/cUyPGN4ms9eylUYXfrMK/X9ez/mZuce9CyIOTD+yHatKtR9x0LarC8wAVlAyXIHphwSsD3dCjV66nVXHIOoSpRt9MQ0JrpW8RKUG6pe3u4/hBy68BFwPHsx0QqaVRSCKtBWnUOKRoi5ypMTmdPoxii+xnGAfbN4RvFOhYAtZNQ3M6824Zhj39p0khkTQn1a2gNEc7w943gp4TXY/Hc+fnzN/2lSal2fn8VPMbyovV7qQS+3kqdx6oi/GsxKV95DzrAYW0H0v8IHOOG4h5qrhdL9IbvWItpcci12VAkgURDHno36LLIajr+IvSYYa2Ihy5/lVjmHdjR2LfUvYH/0+BcgHNKGif4T11xuMVVsu3mpDwyVRgk632TMSAa2edxbWZ0fNfwSlWyYxme+BTpvYrWZQP2I9TymulRKb+eAfVH7X/HaGkj1aihbQisvPCkahVpgS8DJOG9sbJQTmzdeUxvBh5xt5J4+g9ggA+VAOLKNxqq/7aB41o3MkthSTF1v+fDtL8oHg3a6L55+xOQF6y+qx7z6lQE+svjI2bHmKcF5Af9jjzotp1Uk9rmfXx5XoxX f59wxltb KGw1h3ZHILuO2Zr4+hVDzIe/fk6jGHtDZQUDG0Drt4s4K1wA1Srmp+EFJ5SyO9+cShXJ+9FCsmt4fefBEnBEIDpm3uNWQZ43AqW4Aoo5RhuKy07Bc5uHOzooHq0ZeOcp6II7AlEvIC1JDRd/pONJXqY4cgJ46ePySoq8OTnLQPxrnkVFYAmg7c54QdTe/YiLt7mPyUGB9eJ2VdouD+9k5TDyvverVqh4rExzcdafUQs9aCbVWuNfleXfLqUdoL5XqBZOuVoZWxuP1aappR/srleotG3lNiSeX3dirG6e2PTjWa0MohvS570rv0M0iPcqraXlZ46qlZEKJXeNR3zZmYjMAvcndxPsfNhvFy6Xa1J2knHcNspTwXuReTfRNjE6lAmULI2MaZWXLqQyavPi3X0AEmMzUyW+LeEz/FgptWyogNx1l3DRlhk3mgEnjB9yHOwH65ZPpRON2O2LIHri4jSacN9qblkpI7yD5HLdFIsy1uqIvI8zD4fe9UAA4ebAObo3jTPLhV/G8yytLFpQvMKUk3rMxIHOFGy1Iwt5fBbLJrNXvCVCIJo0HXqV/qdnKwhJQ00lc7VlTQuvwxTd+RHOqFvERt2fe7IZYvhKYvC3WAmraw2E8CYtNVUWTuK4KPKwfWUvEur+kLnRdYTDRxqi5myzZkLZKUR6t84EIm/nVwps= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This wrapps all external vmalloc allocation functions with the alloc_hooks() wrapper, and switches internal allocations to _noprof variants where appropriate, for the new memory allocation profiling feature. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- include/linux/vmalloc.h | 60 ++++++++++---- kernel/kallsyms_selftest.c | 2 +- mm/nommu.c | 64 +++++++-------- mm/util.c | 24 +++--- mm/vmalloc.c | 88 ++++++++++----------- 6 files changed, 135 insertions(+), 105 deletions(-) diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm.c b/drivers/staging/media/atomisp/pci/hmm/hmm.c index bb12644fd033..3e2899ad8517 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm.c @@ -205,7 +205,7 @@ static ia_css_ptr __hmm_alloc(size_t bytes, enum hmm_bo_type type, } dev_dbg(atomisp_dev, "pages: 0x%08x (%zu bytes), type: %d, vmalloc %p\n", - bo->start, bytes, type, vmalloc); + bo->start, bytes, type, vmalloc_noprof); return bo->start; diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..106d78e75606 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -2,6 +2,8 @@ #ifndef _LINUX_VMALLOC_H #define _LINUX_VMALLOC_H +#include +#include #include #include #include @@ -137,26 +139,54 @@ extern unsigned long vmalloc_nr_pages(void); static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif -extern void *vmalloc(unsigned long size) __alloc_size(1); -extern void *vzalloc(unsigned long size) __alloc_size(1); -extern void *vmalloc_user(unsigned long size) __alloc_size(1); -extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vmalloc_32(unsigned long size) __alloc_size(1); -extern void *vmalloc_32_user(unsigned long size) __alloc_size(1); -extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1); -extern void *__vmalloc_node_range(unsigned long size, unsigned long align, +extern void *vmalloc_noprof(unsigned long size) __alloc_size(1); +#define vmalloc(...) alloc_hooks(vmalloc_noprof(__VA_ARGS__)) + +extern void *vzalloc_noprof(unsigned long size) __alloc_size(1); +#define vzalloc(...) alloc_hooks(vzalloc_noprof(__VA_ARGS__)) + +extern void *vmalloc_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_user(...) alloc_hooks(vmalloc_user_noprof(__VA_ARGS__)) + +extern void *vmalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vmalloc_node(...) alloc_hooks(vmalloc_node_noprof(__VA_ARGS__)) + +extern void *vzalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vzalloc_node(...) alloc_hooks(vzalloc_node_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32(...) alloc_hooks(vmalloc_32_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32_user(...) alloc_hooks(vmalloc_32_user_noprof(__VA_ARGS__)) + +extern void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc(...) alloc_hooks(__vmalloc_noprof(__VA_ARGS__)) + +extern void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) __alloc_size(1); -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +#define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(__VA_ARGS__)) + +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) __alloc_size(1); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__)) + +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define vmalloc_huge(...) alloc_hooks(vmalloc_huge_noprof(__VA_ARGS__)) + +extern void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vmalloc_array(...) alloc_hooks(__vmalloc_array_noprof(__VA_ARGS__)) + +extern void *vmalloc_array_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vmalloc_array(...) alloc_hooks(vmalloc_array_noprof(__VA_ARGS__)) + +extern void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vcalloc(...) alloc_hooks(__vcalloc_noprof(__VA_ARGS__)) -extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2); -extern void *__vcalloc(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); +extern void *vcalloc_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vcalloc(...) alloc_hooks(vcalloc_noprof(__VA_ARGS__)) extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/kernel/kallsyms_selftest.c b/kernel/kallsyms_selftest.c index b4cac76ea5e9..3ea9be364e32 100644 --- a/kernel/kallsyms_selftest.c +++ b/kernel/kallsyms_selftest.c @@ -82,7 +82,7 @@ static struct test_item test_items[] = { ITEM_FUNC(kallsyms_test_func_static), ITEM_FUNC(kallsyms_test_func), ITEM_FUNC(kallsyms_test_func_weak), - ITEM_FUNC(vmalloc), + ITEM_FUNC(vmalloc_noprof), ITEM_FUNC(vfree), #ifdef CONFIG_KALLSYMS_ALL ITEM_DATA(kallsyms_test_var_bss_static), diff --git a/mm/nommu.c b/mm/nommu.c index b6dc558d3144..face0938e9e3 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -139,28 +139,28 @@ void vfree(const void *addr) } EXPORT_SYMBOL(vfree); -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { /* * You can't specify __GFP_HIGHMEM with kmalloc() since kmalloc() * returns only a logical address. */ - return kmalloc(size, (gfp_mask | __GFP_COMP) & ~__GFP_HIGHMEM); + return kmalloc_noprof(size, (gfp_mask | __GFP_COMP) & ~__GFP_HIGHMEM); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) { - return __vmalloc(size, gfp_mask); + return __vmalloc_noprof(size, gfp_mask); } -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc(size, gfp_mask); + return __vmalloc_noprof(size, gfp_mask); } static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) @@ -181,11 +181,11 @@ static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) return ret; } -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { return __vmalloc_user_flags(size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); struct page *vmalloc_to_page(const void *addr) { @@ -219,13 +219,13 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL); + return __vmalloc_noprof(size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc); +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc_noprof); /* * vzalloc - allocate virtually contiguous memory with zero fill @@ -239,14 +239,14 @@ void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc) * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL | __GFP_ZERO); + return __vmalloc_noprof(size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -256,14 +256,14 @@ EXPORT_SYMBOL(vzalloc); * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return vmalloc(size); + return vmalloc_noprof(size); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -274,27 +274,27 @@ EXPORT_SYMBOL(vmalloc_node); * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return vzalloc(size); + return vzalloc_noprof(size); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the * page level allocator and map them into contiguous kernel virtual space. */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL); + return __vmalloc_noprof(size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -303,15 +303,15 @@ EXPORT_SYMBOL(vmalloc_32); * VM_USERMAP is set on the corresponding VMA so that subsequent calls to * remap_vmalloc_range() are permissible. */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { /* * We'll have to sort out the ZONE_DMA bits for 64-bit, * but for now this can simply use vmalloc_user() directly. */ - return vmalloc_user(size); + return vmalloc_user_noprof(size); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); void *vmap(struct page **pages, unsigned int count, unsigned long flags, pgprot_t prot) { diff --git a/mm/util.c b/mm/util.c index 291f7945190f..19c90036d3cc 100644 --- a/mm/util.c +++ b/mm/util.c @@ -639,7 +639,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) * about the resulting pointer, and cannot play * protection games. */ - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } @@ -698,12 +698,12 @@ void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flag EXPORT_SYMBOL(kvrealloc_noprof); /** - * __vmalloc_array - allocate memory for a virtually contiguous array. + * __vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vmalloc_array(size_t n, size_t size, gfp_t flags) +void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; @@ -711,18 +711,18 @@ void *__vmalloc_array(size_t n, size_t size, gfp_t flags) return NULL; return __vmalloc(bytes, flags); } -EXPORT_SYMBOL(__vmalloc_array); +EXPORT_SYMBOL(__vmalloc_array_noprof); /** - * vmalloc_array - allocate memory for a virtually contiguous array. + * vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vmalloc_array(size_t n, size_t size) +void *vmalloc_array_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_array); +EXPORT_SYMBOL(vmalloc_array_noprof); /** * __vcalloc - allocate and zero memory for a virtually contiguous array. @@ -730,22 +730,22 @@ EXPORT_SYMBOL(vmalloc_array); * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vcalloc(size_t n, size_t size, gfp_t flags) +void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) { return __vmalloc_array(n, size, flags | __GFP_ZERO); } -EXPORT_SYMBOL(__vcalloc); +EXPORT_SYMBOL(__vcalloc_noprof); /** - * vcalloc - allocate and zero memory for a virtually contiguous array. + * vcalloc_noprof - allocate and zero memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vcalloc(size_t n, size_t size) +void *vcalloc_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vcalloc); +EXPORT_SYMBOL(vcalloc_noprof); struct anon_vma *folio_anon_vma(struct folio *folio) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..5239f2c9ecae 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3025,12 +3025,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * but mempolicy wants to alloc memory by interleaving. */ if (IS_ENABLED(CONFIG_NUMA) && nid == NUMA_NO_NODE) - nr = alloc_pages_bulk_array_mempolicy(bulk_gfp, + nr = alloc_pages_bulk_array_mempolicy_noprof(bulk_gfp, nr_pages_request, pages + nr_allocated); else - nr = alloc_pages_bulk_array_node(bulk_gfp, nid, + nr = alloc_pages_bulk_array_node_noprof(bulk_gfp, nid, nr_pages_request, pages + nr_allocated); @@ -3060,9 +3060,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, break; if (nid == NUMA_NO_NODE) - page = alloc_pages(alloc_gfp, order); + page = alloc_pages_noprof(alloc_gfp, order); else - page = alloc_pages_node(nid, alloc_gfp, order); + page = alloc_pages_node_noprof(nid, alloc_gfp, order); if (unlikely(!page)) { if (!nofail) break; @@ -3119,10 +3119,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node_noprof(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node_noprof(array_size, nested_gfp, node); } if (!area->pages) { @@ -3205,7 +3205,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } /** - * __vmalloc_node_range - allocate virtually contiguous memory + * __vmalloc_node_range_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @start: vm area range start @@ -3232,7 +3232,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, * * Return: the address of the area or %NULL on failure */ -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) @@ -3361,7 +3361,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, } /** - * __vmalloc_node - allocate virtually contiguous memory + * __vmalloc_node_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @gfp_mask: flags for the page level allocator @@ -3379,10 +3379,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * * Return: pointer to the allocated memory or %NULL on error */ -void *__vmalloc_node(unsigned long size, unsigned long align, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, 0, node, caller); } /* @@ -3391,15 +3391,15 @@ void *__vmalloc_node(unsigned long size, unsigned long align, * than that. */ #ifdef CONFIG_TEST_VMALLOC_MODULE -EXPORT_SYMBOL_GPL(__vmalloc_node); +EXPORT_SYMBOL_GPL(__vmalloc_node_noprof); #endif -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node(size, 1, gfp_mask, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, gfp_mask, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); /** * vmalloc - allocate virtually contiguous memory @@ -3413,12 +3413,12 @@ EXPORT_SYMBOL(__vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); /** * vmalloc_huge - allocate virtually contiguous memory, allow huge pages @@ -3432,16 +3432,16 @@ EXPORT_SYMBOL(vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL_GPL(vmalloc_huge); +EXPORT_SYMBOL_GPL(vmalloc_huge_noprof); /** - * vzalloc - allocate virtually contiguous memory with zero fill + * vzalloc_noprof - allocate virtually contiguous memory with zero fill * @size: allocation size * * Allocate enough pages to cover @size from the page level @@ -3453,12 +3453,12 @@ EXPORT_SYMBOL_GPL(vmalloc_huge); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** * vmalloc_user - allocate zeroed virtually contiguous memory for userspace @@ -3469,17 +3469,17 @@ EXPORT_SYMBOL(vzalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -3491,15 +3491,15 @@ EXPORT_SYMBOL(vmalloc_user); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -3509,12 +3509,12 @@ EXPORT_SYMBOL(vmalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) @@ -3529,7 +3529,7 @@ EXPORT_SYMBOL(vzalloc_node); #endif /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the @@ -3537,15 +3537,15 @@ EXPORT_SYMBOL(vzalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -3553,14 +3553,14 @@ EXPORT_SYMBOL(vmalloc_32); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_VMALLOC32 | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); /* * Atomically zero bytes in the iterator. From patchwork Wed Feb 21 19:40:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEF84C5478B for ; Wed, 21 Feb 2024 19:42:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2FD76B00BF; Wed, 21 Feb 2024 14:42:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE0EA6B00C1; Wed, 21 Feb 2024 14:42:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABEAE6B00C2; Wed, 21 Feb 2024 14:42:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 911B46B00BF for ; Wed, 21 Feb 2024 14:42:05 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 74A78A0BA7 for ; Wed, 21 Feb 2024 19:42:05 +0000 (UTC) X-FDA: 81816831810.23.D388B0C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf02.hostedemail.com (Postfix) with ESMTP id BCC0580020 for ; Wed, 21 Feb 2024 19:42:03 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=o6bfTOA+; spf=pass (imf02.hostedemail.com: domain of 3ClLWZQYKCD0rtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ClLWZQYKCD0rtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544523; a=rsa-sha256; cv=none; b=TmNV6AGsfzfS05LIKwFmFw+0UL2tlAG3HDmaOroC+xGGSphXzDKv2Yxm1zEDLrd9WFz5lG JWV277CyNNG0dWdvvUzSKv1jqhamxhf2JVPq952t8sWL+YbCnmOotCTJg6ThlCZIYN1R2Z WqFk6c7gP2cDc/uC51DIasZULANcPbI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=o6bfTOA+; spf=pass (imf02.hostedemail.com: domain of 3ClLWZQYKCD0rtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ClLWZQYKCD0rtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544523; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4ca2CRMzgUqo30S53DRYHeCmPf8DAT9MnFsYZ/BDy7I=; b=z/5kTLtFG0Awb1kNLZIaXWx2xk2ywuJEEqgwkCHAea+CjLY+IUkJlr5siZ2pBio6IBTz7Y JKC74jOK9+Drqfw7ura7buWj4L2knbPPfuCMEMiATRux71kTjwl9avQp6BgJxDxG5YwOZr HyowkzLUtHF3sWHou3qXT/3zFLBC7SI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60770007e52so82279587b3.1 for ; Wed, 21 Feb 2024 11:42:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544523; x=1709149323; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4ca2CRMzgUqo30S53DRYHeCmPf8DAT9MnFsYZ/BDy7I=; b=o6bfTOA+AwRb+B3M97lcMw18noBRECxdvL/PK0jAlkpKrG2LYW1iluc/sHd/UYIO6O qWWdYcxiXOrQap3chEXJIO+/0svAaIBsfBuLQfjr2dzS0otGCY3zz/N01wUBTFCiiXqr OyyHQUba38AtSjqPNBUz8kiVV5d3NLce2b+HbN+wnmjZJsGUL9Msrrva25nkQ24coXLY 7wbsqTy9EjsqwDtB6cx6KoZi5TKExx6BlD757oTAe2IAdPLkNQ84MkA+AI48/2wueVd/ 16TC2T951nfAezeMPLL3tb859pbE5HquKOVGDUd/Q0nfsIDUDRFumpTD8NLEPRmGHqVS HArg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544523; x=1709149323; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4ca2CRMzgUqo30S53DRYHeCmPf8DAT9MnFsYZ/BDy7I=; b=OhGYoEXGztENTD/18jMlc9eqTy0Uzt5vzg6myhV0VSEvNCmw4bMOhkqqGzCDsLFitJ rdYfKy43yw/cs8RSPziuOidP5oMm2Jphw4SA5S8ZM5J8bTGUwxj3UxSmk3W6PXFhjICw FOr6Tnkk2TzKaA+L7/BdziLmNgAEnOprIkIBrpbcgPH66bIqqinC+k6thsaDuu20SrTz KM1rmrmivB9JSDBOYfyxrtOE6vm8UhHeSG6fcDKT/hK/d6ujtIKQLhdTFz3lhS+7yUTL +d7fJdOYWRM7szETDahZX6FbvOO/4C8dGiDt6UjIb4pvatkQELOVScO1N1NMyeQNNSSi hIog== X-Forwarded-Encrypted: i=1; AJvYcCVLeCY7TKhFHtbxUYzr0chD0CiXFk00or0WkBhgtnfdHQEF4VsjgDVP2PHCrKwo5UbbF59eNipbGZN7S5NPqFLM7VA= X-Gm-Message-State: AOJu0YxfGzv7ZRuyXI3b/cUR81iGOLy6q2RoeXI7iInDT8+C7KAfCccl OQGBwGl7Mmb7iDMIL0Y+No5y6PKiO48L1B3EsH75np+HXpbB3uuro+jplmDkk6iy+iK9OxweV4V huw== X-Google-Smtp-Source: AGHT+IFffFYSdKTwRIoTuKanuxnAO1YVHDWWhEPlchVKFceg9jZBcLvA/YiXfOHsmmNJVcGKnn75V3mm8JM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a81:fe07:0:b0:608:22c7:1269 with SMTP id j7-20020a81fe07000000b0060822c71269mr2041757ywn.0.1708544522852; Wed, 21 Feb 2024 11:42:02 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:43 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-31-surenb@google.com> Subject: [PATCH v4 30/36] rhashtable: Plumb through alloc tag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BCC0580020 X-Stat-Signature: u7yp953y8j59nf8q1n5x39ho59nqmfbe X-Rspam-User: X-HE-Tag: 1708544523-579411 X-HE-Meta: U2FsdGVkX1+EVrP3LzIKnl0Y6T6Jmxq3lMsYfG2zPLHocVRo8qGzs4uEz34coepZHs7XoQsiQoaoyWs0PkEyYAAphKnKj8UgK22FU1sUh9R9o/ZdW34/qlQz0MthAWCPQdRLaXn9riyD5geaq4DvgdjxNGg3CeMLHcdbPeFYacdaVji+zKmK0+w1gMAa3gCTeh1lKBlX+i/WcXXG7nVBNgUttNbnsW9+vClbdl12bdOmlhtaGZQHDZNe4Kvr/pT+RhivF5l5GbFZi5askIsKUUHM6LoScPDkN89QMeaJZ6NJowB/DI0AL6zxyPsZaUicP04Pi4/AIXKKCbb6pm5w1FcfZWdyuQe+3r4dAFY/29V2Ev5639nEzc52zn4xn+vjKrwipYq64wSLSbzw1VmYpr9iG223qA1WxOgqDM7HJdx8eXNs4IH3B3A78Cv8z2Bv7GKp8M/S72up7j1Bak1zEILOvxNsuOiMT9Gei8dwnLyyIwq4ex/DK5MPp2WsF7yOdm8QfDaojRJSpOpT9TWBY9PO4KTLJkd6qqGR4Rad0WmQEcrs7DIVc26i7jH4AGIWMew1htWJSIhR60jCRVwfvtTERmmr0PXzAlJYwKBFjTT6sG5eVwbQxdknBK6iGDF246OlncN3cBl2AM4UG7na8gCOpGRIMunMb7wlazoqCXpVWNx1n1MOpBma4udbZSrT8V/fyC4ENQJ7X3yyQvzJTx5Iuuzi2gIBSBw2WucqJZ4I4tu+0e0OTzSILkcfaZB2tgXCzM2BBmnYOCyEfBCZAaPd5vuSUMczEkS4wzAHXmBIy3kwdFAdTCx/jAvzO1UvnFsmL2es1koYmGVnNgjekctyO9NqiX15F+rxwtVfgrzwQbXSAb28BV3B1qpErIWyT0G/MuteGGRp6xPr9B8CVKohZryugqOix6vibS9Cykj4vJdVA0Cg2GL7xRC+GLO0SUHCofdO83v7/ooA/Jd Msge1fn1 OXo6FqL84PEYuN5Wc0yNUgIGTM4glEGFV3FfBDz9NgY3J1uhcUrkfqPdgaacc7OB8z2vOTyrvBsgJfm3XuMITs+KYogjKVGkxjN5Zb+eCSt5dzo1BTyYNxJ5PoU+Xy5BEwXG499OjZjw8BztYrxRii84jKLrLeuOt6cZqwyeG+mO8YMexcLUA0FnHnBXo58Ai3WP7N/Eefxhy6BBaNLsnqdvOAx9de7+YTcsYaKdqxdUC53HkwhhlGt9eycG/wOEUQ+42y1zrjzjtk/VPfElCW8PEejLPeg3hztGtZhn9AETvNSN3T1+tiQHiVkVZZO6jiNgRphainlXw/kY85AOdn4hoPTNpVANGSyMFEgLUA6qMj+zYirDLojz7CrVYq72hqaApvyyU02lDEKJYyDZyGZF5KXqQ7bQkGOqS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This gives better memory allocation profiling results; rhashtable allocations will be accounted to the code that initialized the rhashtable. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 3 +++ include/linux/rhashtable-types.h | 11 +++++++++-- lib/rhashtable.c | 28 +++++++++++++++++----------- 3 files changed, 29 insertions(+), 13 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 86ed5d24a030..29636719b276 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -130,6 +130,8 @@ static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, this_cpu_add(tag->counters->bytes, bytes); } +#define alloc_tag_record(p) ((p) = current->alloc_tag) + #else /* CONFIG_MEM_ALLOC_PROFILING */ #define DEFINE_ALLOC_TAG(_alloc_tag) @@ -138,6 +140,7 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_sub_noalloc(union codetag_ref *ref, size_t bytes) {} static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} +#define alloc_tag_record(p) do {} while (0) #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h index b6f3797277ff..015c8298bebc 100644 --- a/include/linux/rhashtable-types.h +++ b/include/linux/rhashtable-types.h @@ -9,6 +9,7 @@ #ifndef _LINUX_RHASHTABLE_TYPES_H #define _LINUX_RHASHTABLE_TYPES_H +#include #include #include #include @@ -88,6 +89,9 @@ struct rhashtable { struct mutex mutex; spinlock_t lock; atomic_t nelems; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif }; /** @@ -127,9 +131,12 @@ struct rhashtable_iter { bool end_of_table; }; -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params); -int rhltable_init(struct rhltable *hlt, +#define rhashtable_init(...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__)) + +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params); +#define rhltable_init(...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__)) #endif /* _LINUX_RHASHTABLE_TYPES_H */ diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 6ae2ba8e06a2..35d841cf2b43 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -130,7 +130,8 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht, if (ntbl) return ntbl; - ntbl = kzalloc(PAGE_SIZE, GFP_ATOMIC); + ntbl = alloc_hooks_tag(ht->alloc_tag, + kmalloc_noprof(PAGE_SIZE, GFP_ATOMIC|__GFP_ZERO)); if (ntbl && leaf) { for (i = 0; i < PAGE_SIZE / sizeof(ntbl[0]); i++) @@ -157,7 +158,8 @@ static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, size = sizeof(*tbl) + sizeof(tbl->buckets[0]); - tbl = kzalloc(size, gfp); + tbl = alloc_hooks_tag(ht->alloc_tag, + kmalloc_noprof(size, gfp|__GFP_ZERO)); if (!tbl) return NULL; @@ -181,7 +183,9 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, int i; static struct lock_class_key __key; - tbl = kvzalloc(struct_size(tbl, buckets, nbuckets), gfp); + tbl = alloc_hooks_tag(ht->alloc_tag, + kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + gfp|__GFP_ZERO, NUMA_NO_NODE)); size = nbuckets; @@ -975,7 +979,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) } /** - * rhashtable_init - initialize a new hash table + * rhashtable_init_noprof - initialize a new hash table * @ht: hash table to be initialized * @params: configuration parameters * @@ -1016,7 +1020,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) * .obj_hashfn = my_hash_fn, * }; */ -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params) { struct bucket_table *tbl; @@ -1031,6 +1035,8 @@ int rhashtable_init(struct rhashtable *ht, spin_lock_init(&ht->lock); memcpy(&ht->p, params, sizeof(*params)); + alloc_tag_record(ht->alloc_tag); + if (params->min_size) ht->p.min_size = roundup_pow_of_two(params->min_size); @@ -1076,26 +1082,26 @@ int rhashtable_init(struct rhashtable *ht, return 0; } -EXPORT_SYMBOL_GPL(rhashtable_init); +EXPORT_SYMBOL_GPL(rhashtable_init_noprof); /** - * rhltable_init - initialize a new hash list table + * rhltable_init_noprof - initialize a new hash list table * @hlt: hash list table to be initialized * @params: configuration parameters * * Initializes a new hash list table. * - * See documentation for rhashtable_init. + * See documentation for rhashtable_init_noprof. */ -int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params) +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params) { int err; - err = rhashtable_init(&hlt->ht, params); + err = rhashtable_init_noprof(&hlt->ht, params); hlt->ht.rhlist = true; return err; } -EXPORT_SYMBOL_GPL(rhltable_init); +EXPORT_SYMBOL_GPL(rhltable_init_noprof); static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj, void (*free_fn)(void *ptr, void *arg), From patchwork Wed Feb 21 19:40:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D173C5478A for ; Wed, 21 Feb 2024 19:42:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7647B6B00C1; Wed, 21 Feb 2024 14:42:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6ED116B00C2; Wed, 21 Feb 2024 14:42:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A4326B00C3; Wed, 21 Feb 2024 14:42:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2FA466B00C1 for ; Wed, 21 Feb 2024 14:42:08 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 10806C0B6C for ; Wed, 21 Feb 2024 19:42:08 +0000 (UTC) X-FDA: 81816831936.02.5DE8DED Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 71F9812000F for ; Wed, 21 Feb 2024 19:42:06 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=10ImSj37; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3DVLWZQYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DVLWZQYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544526; a=rsa-sha256; cv=none; b=a/WG41+FTNf0R6U0H+YJtshGlswnnyWa9mmZetfK7b4QXd89cw6eRrwx65PjAcpbSb2ScE 8SrxGRbvdngTNPzk/aInksGGXk0yFzpiZDMM3OZQgtZqjX3e278/JDw6FvBNJrd6PHi+02 At9PYZtsDmCmPOGAyOHBI+x8epe44oc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=10ImSj37; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3DVLWZQYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3DVLWZQYKCEAuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544526; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LC7kagHHnwmHwKr3BMy6MtJI28X2XJIX7s9nejqSpcw=; b=J7ke8+h/6jHpyM0qsyAEd429YgpubSYLdnHlGi8IHb/fRuqKDXE0AbLrefPSP3+TpaOP5w rPkTSHWkeB43fGTPgXaxrQU5YAd35kDHfyq4H9FwVx5oMVkBQ6ifKf1MauA5vw9GCi9nH8 iMexG12K2SdshV8VkbPMDYVx0nxabUc= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-608835a1febso18674287b3.2 for ; Wed, 21 Feb 2024 11:42:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544525; x=1709149325; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LC7kagHHnwmHwKr3BMy6MtJI28X2XJIX7s9nejqSpcw=; b=10ImSj37I2MK6te/OE1EpB/mKek/jq3YFdU3s2NBxdv3mlKe70f+lbGW7caeSIOX+p jD9yoZEP8TIQsIhihMExGekuzc3lynjPaBmcKG8gD3Z+g7QOp9uQrNBaD3cLyPA8iSYF eib2XPbH7W83puyxm3dhUB3gDw9HUOp+FPZL3k4eK3ecwqUcJPnBP6wJij+Povi7l0ol Kq+0YyQXJpoUceYk0PeAku5FTdQN/XgrpmtFdyabpdAyoFuLLhAxOwaqxhKSvcRpEDwY gzYrV8LVdYYrx4qPPDpNy1B0U2o293u9tfUKg/SvRg54S35EQnxdStgdBNugVqNLErKM QD1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544525; x=1709149325; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LC7kagHHnwmHwKr3BMy6MtJI28X2XJIX7s9nejqSpcw=; b=vASzNkGzQDnGv4wYYPMZ+pbYJ2HU9hnKoayB+CLgjwaF9oNIvbkajaxgFTH6iiD6Sl JGqFaiNDlld68D18mKg1NLmJIMdb86fs6G+JCtfitxmlml/FukG2RXyiRNWGfZgxwce/ FqVTQz/bPeoQF3dJW7CGetyadCxHPVj2OBB0RsgAeDSkiDQPQWrzuYiYBZZPezK2HNMi KRQrU2jG6LNRgzO/l2MyxvHEWa36+S8irvKcrIz8Nlf4Z9Bm9Nriv64tZcd1uunZ8V1+ IuDCGyUzFOzRFZ4tI9+5oXqTt/X3n+tqXp6FWkjPPKTich5Tbi42IZpnInQBVrroP53F SsCg== X-Forwarded-Encrypted: i=1; AJvYcCWr6xvAa8A8nXDTUZ2AL7ymiXFIOinaFKLfnpBPwFUhtJo42nhK7ELbK2H9N5vXQdjHqSyE0Asj4xv2rFAamZeGsf0= X-Gm-Message-State: AOJu0YzbO76n65TlJ2BUL0NYYSPDDBom4Vs1ZI6dll2BhIeV/2WqEOJI slF24RI5we1xqXmkD2YmOHZU3e2Scv944FdvzCF2Mbsj7r2ym+y51vYnnjDVA2aOgGTSsTGEJCf Rig== X-Google-Smtp-Source: AGHT+IF/fw4+k0W9YQaAhd8X5D2pflUjenW5K2EydUoGfLHK5w3QTNH4iYX84WllsVXa/U8r+txZWwizBxc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a0d:e611:0:b0:607:9268:6665 with SMTP id p17-20020a0de611000000b0060792686665mr4677189ywe.10.1708544525062; Wed, 21 Feb 2024 11:42:05 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:44 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-32-surenb@google.com> Subject: [PATCH v4 31/36] lib: add memory allocations report in show_mem() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 71F9812000F X-Stat-Signature: 3iey99sqg4fi3drud5sr4gw8pscnzy8s X-HE-Tag: 1708544526-347484 X-HE-Meta: U2FsdGVkX1/628OknskFKTU9IK5U4jNTihRYz4RgM2/vKyyjKUZ+JC8rrq8B/5i7bY2a/BZYWFFQ0QPw9ohXN3Abz5H+x0cGEMSlTfDM+fHSMaoOgqO5MZNWDJRKFBTJVK3QNDNuLu3tZDEOhs3zeJZbFfJ/QpNLWbXv5EPvcwvMgAPKGGcmbLvbiHmo/vA11NM65rPaE3s2xEIXAp9DhegIUgC0fhV8nQrYWYu2+c0TopWXEau8kS+lgM1U7+CfBKzScfdkS6/jNDNGddfh9wmGOMCYSBwXIRf1wgplW79/z7hVw7WkugKutavAW4mVDm8fvh4jud3+hgAiPoEVj5kJ3uU5Pgiu1gSzkd1BfZ9Os0OcvfgOMHdpfcOhYFVfyx1iZwCOMxdOWLTGMz2ExwLjE7MTJJEJusSTNDkJ1iSRFhkTMwle07dOxd8DSHETqIjNuu8Q1NJiJQr+98VlMRk+UFx9GzTsf62E0rywBt9Acd9D/Y3rnmxRo+WlsMaLlUyoxMH3WprccD3N7ZFdGjfnPxuR7d8AUPODnLu7dwNib6e4RurnqLycydzkYafmzppM3knv7wjkZ5fFsj1mwe4kafwcCgd3iosOvSfpTH12/EM6Uq+z3fqYcGTnkEL7jQ29dVi23tJMYpsAy2zPI++owSJLyBti96XvH7c2qCy9nzLNO730OEaxiBBYaQ4bseGxIgDwVDCZIjuJNnn42nytStwjh1ptkSPpOONcr+lhAhd8d1wNgaOtjL2LWQecq9Tg32yQgGWfu0gj1HLtINsI4x/n5EmZWdshp4Sh+e2ivzoOEcsFEEMFjzw6myM0EaA6umYLfrJ9pSGD7HJGT8IdlapCyf7o/oWoXwWjJrQZebbtXfPqwKD/chQfS8BAYxzksMQres4w8VINeXfjincXVMVHkN+fT5RUwNPbrxaDHCojUUxhsmQMluU43le+nJ68XoPb2PT1QodmM/u gAPPdeF4 jigjXpt0qKgS6aKXntvL8Xq5wjioEZUnCZOvQTg2bY9DQcCa3Ct6ci9bTZ3CavTnoQLa38kNA2CaR7d7LlM7WotRV07nBFsDEYIKTmroMRQ0GtyjUdVv7kt1E799xW06QSB9s7xSub/0u8pZ31+yNVejq4vdmpxntglSvjmg/O80JI4khyXobKN4VZKDVZANPiVU+8XDbQ+sPr4CMR4wrRKe9uJmc4LzXI9/OP7vSodAXUStdqSZ/BVkyuqViw9mySxdtzIrJhT6jDOvIZWfpgwLdTCUvnUqjmJ/yKF0nime3Rn+CZvF6w787+4GuGTV4XrUphNRXQskLv1Gh8sLQI6L0U8LiFcAeA8f550sq5I1g1iWqvX6SAqtcGnqSlVzFqnCHGtctr8ioaTSU59Q1j0HkDXZN+PxokSKMI3kcs8qzPm5nwTLLmgxSEdT5JxXzK8/9WFcb8Ic4fxq1fpAlATBXvA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Include allocations in show_mem reports. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 7 +++++++ include/linux/codetag.h | 1 + lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ lib/codetag.c | 5 +++++ mm/show_mem.c | 26 ++++++++++++++++++++++++++ 5 files changed, 77 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 29636719b276..85a24a027403 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { #ifdef CONFIG_MEM_ALLOC_PROFILING +struct codetag_bytes { + struct codetag *ct; + s64 bytes; +}; + +size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep); + static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) { return container_of(ct, struct alloc_tag, ct); diff --git a/include/linux/codetag.h b/include/linux/codetag.h index bfd0ba5c4185..c2a579ccd455 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -61,6 +61,7 @@ struct codetag_iterator { } void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +bool codetag_trylock_module_list(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index cb5adec4b2e2..ec54f29482dc 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -86,6 +86,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep) +{ + struct codetag_iterator iter; + struct codetag *ct; + struct codetag_bytes n; + unsigned int i, nr = 0; + + if (can_sleep) + codetag_lock_module_list(alloc_tag_cttype, true); + else if (!codetag_trylock_module_list(alloc_tag_cttype)) + return 0; + + iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(&iter))) { + struct alloc_tag_counters counter = alloc_tag_read(ct_to_alloc_tag(ct)); + + n.ct = ct; + n.bytes = counter.bytes; + + for (i = 0; i < nr; i++) + if (n.bytes > tags[i].bytes) + break; + + if (i < count) { + nr -= nr == count; + memmove(&tags[i + 1], + &tags[i], + sizeof(tags[0]) * (nr - i)); + nr++; + tags[i] = n; + } + } + + codetag_lock_module_list(alloc_tag_cttype, false); + + return nr; +} + static void __init procfs_init(void) { proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); diff --git a/lib/codetag.c b/lib/codetag.c index b13412ca57cc..7b39cec9648a 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -36,6 +36,11 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock) up_read(&cttype->mod_lock); } +bool codetag_trylock_module_list(struct codetag_type *cttype) +{ + return down_read_trylock(&cttype->mod_lock) != 0; +} + struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) { struct codetag_iterator iter = { diff --git a/mm/show_mem.c b/mm/show_mem.c index 8dcfafbd283c..1e41f8d6e297 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -423,4 +423,30 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { + struct codetag_bytes tags[10]; + size_t i, nr; + + nr = alloc_tag_top_users(tags, ARRAY_SIZE(tags), false); + if (nr) { + printk(KERN_NOTICE "Memory allocations:\n"); + for (i = 0; i < nr; i++) { + struct codetag *ct = tags[i].ct; + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + + /* Same as alloc_tag_to_text() but w/o intermediate buffer */ + if (ct->modname) + printk(KERN_NOTICE "%12lli %8llu %s:%u [%s] func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->modname, ct->function); + else + printk(KERN_NOTICE "%12lli %8llu %s:%u func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->function); + } + } + } +#endif } From patchwork Wed Feb 21 19:40:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A789C5478A for ; Wed, 21 Feb 2024 19:42:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0BBA6B00C2; Wed, 21 Feb 2024 14:42:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C91FA6B00C4; Wed, 21 Feb 2024 14:42:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE4356B00C5; Wed, 21 Feb 2024 14:42:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9602F6B00C2 for ; Wed, 21 Feb 2024 14:42:10 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 782C8A0F68 for ; Wed, 21 Feb 2024 19:42:10 +0000 (UTC) X-FDA: 81816832020.18.F98C40F Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id A08D810000A for ; Wed, 21 Feb 2024 19:42:08 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3Wdy1QkO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3D1LWZQYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3D1LWZQYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/f5EOcne7K32iqTM6gsyOYn4NQeb5Q9RrrPaWIBtzrU=; b=EKyl4k6iyux/Fo6JHK3cNahLOzM6kobJ1T5zRMxuCQ/mBzw/Fs3kPWkwVIUU6qZffLuou8 JHSlFbsaB71xZi9fAKjruQIJOcy/6xAKUfQitG7LX9nf1M+rtAcaAgV580hbjJpNzvcb2b f3IINBj9vj+HlA0RehiY/WsENqWiBTA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3Wdy1QkO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3D1LWZQYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3D1LWZQYKCEIwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544528; a=rsa-sha256; cv=none; b=dCSyj4jcO/L8X4Zsva4N5Ki1BYHQ++9x+uM1EVZhD5NIXlh2UtNQ2B2YvHd31s9kDbMJDB JYyjgbUXY+dDRTWjYbtzGDi2uI396V0K5UgcANU69NsruQkUnpeWBsgr0yjrwio31w2nhF 7CBQlEC3x9V6MzGFwctmpje/xdmwAqI= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dbe9e13775aso10654736276.1 for ; Wed, 21 Feb 2024 11:42:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544528; x=1709149328; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/f5EOcne7K32iqTM6gsyOYn4NQeb5Q9RrrPaWIBtzrU=; b=3Wdy1QkOVUd/oh34rEqRGMvvqlkx63jyX02nquSgXvvTm0v0xCEeL8NkJYbzNC4fV8 iIBQcjB8Mw6/q6R8dj2swfiaR7ljnr40UAdFsYw4Z2ieITczNkSTgjVnVo8kAy6ggEdw AowzM75Ig7r6naEQrvqWmd/Nsa/Dg89g84eqwxMNaaRwGT+R8nYhNklJRVhHqp+rFTM/ POBUcJr6hyYW8uLz4ZcaYDVLpzTzJfiL4THDretG3yG7oc7fTwgsmeMSLQV3WTy/NMnL Swsl7/TiwJP7A5LPS0MQVOOdwJrCvkUFDXe7nrtfGsnyw1m4RObCjxfQssNYy3dNECCh jLdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544528; x=1709149328; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/f5EOcne7K32iqTM6gsyOYn4NQeb5Q9RrrPaWIBtzrU=; b=Teu82paG6z04zJbFCtCa7HaugCxnWYVkLU6DMZpWjfhXEhbeZoloRd8Vdx+H4RhJ9p dmf50UxieEzVJS0LzFCVk6WuPA7Zz1aBnU/4CVNE02WGzj5VuAyn6VQPfFt8a9l8o+wf mtTun/s1MgpBBe3eI3jVLZWBCIawNwbIviLIk8kqLosmeFORx++ML2LK422MwdZYgdIj mtoxhn545hEE9ynYjbHccRl7yYeonJ1k/nD6APU5Gw31d0usoyM5sb7niIwgu8B83LnI dS1EVNMSrluwzBZ915pW11L4e9Ua1vmG1BDQwTJbx1ntFRcQS/ypkuwXSwSNUQySZs6K 346w== X-Forwarded-Encrypted: i=1; AJvYcCUQG/0deM2tYQ1f6p1tR7a38cCfRIaKEwjkAOVbQrH41+fOj5t7aAuoTo/ruORJzGTBP1Pp1PgvkowIQ9onij2BJLw= X-Gm-Message-State: AOJu0YyBPhnOMR7WJ48eFS5qwzVfJJ0ki6c4O1ShXYHAwvQbj7zEfsDe EySKUTaz42oqVWX6n9Ej2DYGZlaW6/5MmmMWO/ark+EqA3O7Q6FtqluVpklkQqE2JJkddYcZpbu F1w== X-Google-Smtp-Source: AGHT+IGFiWXsCIvnXCn7DCHC4E3aqtfNjiBbLG3wTDrJrg+DapwG+olr3URRZ84R+eZCxmTFvxp+ectvo4c= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:134d:b0:dcb:e4a2:1ab1 with SMTP id g13-20020a056902134d00b00dcbe4a21ab1mr67096ybu.11.1708544527496; Wed, 21 Feb 2024 11:42:07 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:45 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-33-surenb@google.com> Subject: [PATCH v4 32/36] codetag: debug: skip objext checking when it's for objext itself From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A08D810000A X-Stat-Signature: kuizwbkaopk44ncjahpyz9qipfr5h5qi X-Rspam-User: X-HE-Tag: 1708544528-928643 X-HE-Meta: U2FsdGVkX1/guxKEKb49h2RGCkJall2Vj32e1C6PyDVj+TTPcO2PlpkwDvUqW1YphRsCDw2icnGviDOsnX76byuQnHurxOeybzc3O4ITjyNFJCFxNVreQOAUNOxNsxqpQKKehzdqxvycu7p5vZGS9+IAUVQwCM2EOogf8EDQ1jALjhT1RVAyJ5lahSRBS/tUm4A5dpsCiyGV9FpP81VpM6Sek3/CycO4uZmQrfkXTcz7Bz+OZNltrxWluCD55DnTjgjkINep66aW/HRFluuDCIvFiHhydSukRdcl59mjIuksKZ1QUK07k1ObCrLG6DmK7c4zmIyo2D1KTheSvm+xYrXj4445/f0I+yNvUJKZue40yIBScqAYPoFFuL4WHWpN3h0hL41t9FeaRUh9L8JcwYyMkuFwmC8EBYudysragdOEAMHnayXa4mFOXB3Iu7+OjgPwRWlvs1S25V/21JO8s+ClwyeaBPwnNbMQ+/pgxIXqyETHRBj4oFeG2PQjgTh1M2ZuS48vTWPoIWKMXRY6j42/S4qtB2slxFMLwaI//CrEpISIkUwgJd7a5cG3bM7BWDfS5BwkM9bSuaRNuQxsyBPzmbXro4CyWVuqlkeHt2zZZvg+O7HN6o+gAMnzRZLNhSEduprID7jJ8FMFVr/TCjVQI4G8juHvQJ2q2V4PjxxMpHRu13KPd+D0CLgi/wMb9Dday1o6aE7e3O/BWF0+eMIbcPwI/oAqWeJMvMLUyw811q9eOlh4fgrpsxWG+nHKu3F8DGfrjl/9OMl50JR6pC5fRPNVlhe6RaMxRcp5va3Z93WhcnXJgGZbqdrZ7Fonb4yCEklALHzqQ3z9mN5+Ey3QJHWa8QkoFZ4+B5zMIM97/TAtRzQ1/p4tjudhu53FIOlUEw6T0W7zQHFgNeSwMMJWSwPTgonGNu4X9HmtjCMhlv0NhLQpDodZP7Rjgd1G3mgOAVFMyRY0JC3IZYE F7VTHUDe MsfUrR2a999GbJW6Hr3uTZi/yUHZ0Qd+ngvs8D+i8i59sKjXsO3btoJRtMZLqisGfOE8lrXZYeAOAsVZzutNmWm8Vc9CcrDGEaR73lY2Ip4z48imlXkD0a0/d74Qs8BMf2shf5agWL289nEf9FTxiXjmhm8+pMjnllKazgcmYYp8sJqw7Wl4MgM8Sr1jQjkhQe8TOOqfXORZTTXHnhZ9U5//375LnyNLIbghVFDxfn1nS0ivRa+bjYherDL3N10LrJ93zqBCkD6cXD5dWeX2b4vDNtSC37slr6i1jJU5yS/SjG5xTtFUR77RJKSfpHyeMv12xgCSWzbrFzizES+35bmzBLkF/I2xswiLgTarR9evdi/2RWQGwVbDqcykNO6zZrjYC4S/F3uqB20ODYrRUGvpbdwvWp8i4mVQGqsK+GN/LUdsdmjBoSMKOzTARnsWZrKtSu7Wy5qjKVMCIrhIS+mINFeQ3B0JFKDBrcbaGLNAPQA7aXV7EdrJAjRPsaXthRY8J+yyCTCPZ8c5l3QY40CzqeqTyIN/lD3L9QaCxyCLG5YeMlZhZmx+MM23LPOW3h6593/7aKw+vxPk7g/xlXchu/fEROSCsgQli9qs3y33J5Y+H4beoRnH+J4OIvgrGHGnEqEBhszMHZh0jivWs8DeSknCVAddj6R4weLS9mNaEgoI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: objext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 26 ++++++++++++++++++++++++++ mm/slub.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 85a24a027403..4a3fc865d878 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -28,6 +28,27 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY ((void *)1) + +static inline bool is_codetag_empty(union codetag_ref *ref) +{ + return ref->ct == CODETAG_EMPTY; +} + +static inline void set_codetag_empty(union codetag_ref *ref) +{ + if (ref) + ref->ct = CODETAG_EMPTY; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + #ifdef CONFIG_MEM_ALLOC_PROFILING struct codetag_bytes { @@ -91,6 +112,11 @@ static inline void __alloc_tag_sub(union codetag_ref *ref, size_t bytes) if (!ref || !ref->ct) return; + if (is_codetag_empty(ref)) { + ref->ct = NULL; + return; + } + tag = ct_to_alloc_tag(ref->ct); this_cpu_sub(tag->counters->bytes, bytes); diff --git a/mm/slub.c b/mm/slub.c index 920b24b4140e..3e41d45f9fa4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1883,6 +1883,30 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) #ifdef CONFIG_SLAB_OBJ_EXT +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) +{ + struct slabobj_ext *slab_exts; + struct slab *obj_exts_slab; + + obj_exts_slab = virt_to_slab(obj_exts); + slab_exts = slab_obj_exts(obj_exts_slab); + if (slab_exts) { + unsigned int offs = obj_to_index(obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); + /* codetag should be NULL */ + WARN_ON(slab_exts[offs].ref.ct); + set_codetag_empty(&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + /* * The allocated objcg pointers array is not accounted directly. * Moreover, it should not come from DMA buffer and is not readily @@ -1923,6 +1947,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, * assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ + mark_objexts_empty(vec); kfree(vec); return 0; } @@ -1939,6 +1964,14 @@ static inline void free_slab_obj_exts(struct slab *slab) if (!obj_exts) return; + /* + * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its + * corresponding extension will be NULL. alloc_tag_sub() will throw a + * warning if slab has extensions but the extension of an object is + * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that + * the extension for obj_exts is expected to be NULL. + */ + mark_objexts_empty(obj_exts); kfree(obj_exts); slab->obj_exts = 0; } From patchwork Wed Feb 21 19:40:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3A71C48BEB for ; Wed, 21 Feb 2024 19:42:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1A076B00C5; Wed, 21 Feb 2024 14:42:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA3BB6B00C6; Wed, 21 Feb 2024 14:42:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCE1F6B00C7; Wed, 21 Feb 2024 14:42:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A3E696B00C5 for ; Wed, 21 Feb 2024 14:42:12 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 81C8E80BB8 for ; Wed, 21 Feb 2024 19:42:12 +0000 (UTC) X-FDA: 81816832104.05.E71109C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id BB7D340004 for ; Wed, 21 Feb 2024 19:42:10 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Gv2P6COm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3EVLWZQYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3EVLWZQYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544530; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CzDIuQ0nYms/65bHZ69fKMNRRXjNNeWyOZfzFtq6j+Q=; b=wMJ+Kl2+2kZGnMYsgdujyEymE41WmXiAin++Vm6TSCmr6K6U5oX4pIRyWEu53k8bGUlBd9 LGC0DvO+IJZk/2GkXbpsSS4sTciTotSulyjlmngMgPjCRG1OITxIaEOkzEq/n4Ffsm6Hd+ HptgurRxIbdzuqcFGyRpRHB9nfF7FcU= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Gv2P6COm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3EVLWZQYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3EVLWZQYKCEQy0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544530; a=rsa-sha256; cv=none; b=lp/cmfoaIKdh8nQUr10QJW2bCxtEFT3ScA8zjb0LnQm65vBiiogOa8GPOUTIAoZ+u5wCZQ rHQ9ui5EwE9IuxHEGkRUzj1HFwbRtdXUsZ6qIjoyRcjSxijZscJa5+LT8aiNLB9AN1lZH/ vRBt1i53dFMWDRjksg4nZp2WQo68G4k= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60802b0afd2so1243527b3.1 for ; Wed, 21 Feb 2024 11:42:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544530; x=1709149330; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CzDIuQ0nYms/65bHZ69fKMNRRXjNNeWyOZfzFtq6j+Q=; b=Gv2P6COmYBgKMLUowpG/6Q9LeAYT9QTkFwjQ0M6SUU4H9i9c2zcJ2w3bHgwuldNfre L04SJvmI+SoSer7gsPGNvWX1MEqV/wKZW2tJliukAnjHFAQjTD0CgBLEoUsa2QG77FMY aOFSmqcEjD6vfV8Qq4+eK9VSp/N3koDTxh3DO4+CSG9fYFKTkvLKEH//t0bmac7EAU1d aCXYh5tDtA/NLFupdAs7J19N5vVWHiny/jz6pXXRPfg2drhBr/VT0HBNF7sF6A5JotEZ IswDYhxY0XC3ECcGUe1wtsiLMgZcJwK7R/Gm0IX5hp9YuemDWZQU0i8kE4+c8igU8Dvl OsUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544530; x=1709149330; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CzDIuQ0nYms/65bHZ69fKMNRRXjNNeWyOZfzFtq6j+Q=; b=H4n/V3f8jGazaOoT2AKGDBZ3CiH1z25eucX7MVaoCB9IEqT+MZIoXRrkgiaeTqvUcT WaWzyw3fU1ZvwugkULpJcdw2V0kypS+svWQ1xeny0TjEpFattWuqTnNofNThAohE2X0W u9QoxaImJpR3ok1yHL1T8M2A2zvqFs2veERxm5r2QVL0HXYyNt3ehwnNTlV7DLkn4YKb 11Hg80E0zB85bHIHK5r7HnK/JQ1k/yCe6cIUjhf54+lZOKbsrxApGi9P1hPjbJf0Ex8x nuin5BbnuZhyvykv9KFFSXQic5VKJLg3eSu5jW5zsinxfpv/qj2O/4MIgrCCEOo38SiD oy7Q== X-Forwarded-Encrypted: i=1; AJvYcCXegAZ8MlS5OqmquGBF1pDSc960Z7mIY9nyRp3OxoaYoVaTC5Osl1OcudgTbv7RIGcY7w1YZQzWM0WkAjDpVIEFeZQ= X-Gm-Message-State: AOJu0YzyuNtdq4nDw8Vp8qbt/yhI2GQJSUvpSjH7qqpjQPn+Qx+469gH Z7BirYjgtgC0t0oWlMQVVV8VqrTwcb4AqT+YMRv832QtSvmGG0Yoy3HykNT33qziFwVNvH6CC8q zbA== X-Google-Smtp-Source: AGHT+IFe7qNCg82jNJD+RarPvS2YBMTSZ7x4HEvuAvHP+lTF13VKJQxl18VpWOGqr99d4PU9keICw1gSItE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a81:b287:0:b0:608:6b9:bf09 with SMTP id q129-20020a81b287000000b0060806b9bf09mr97625ywh.1.1708544529717; Wed, 21 Feb 2024 11:42:09 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:46 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-34-surenb@google.com> Subject: [PATCH v4 33/36] codetag: debug: mark codetags for reserved pages as empty From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BB7D340004 X-Stat-Signature: 8wq791znqqmksfnkyhgfzx7jgk6m8iy9 X-Rspam-User: X-HE-Tag: 1708544530-267487 X-HE-Meta: U2FsdGVkX19j6igEOvD/cYjTZKW0nPJzvaHsDT973mA8hbxkgs7it3/9is3pLPw+Y0F2AV0T5TaH42PiTvzha6DasyDz7K6q0z3YFWskkQMHyOMCTQinEzeu6Mt4gRfDZ7tk+Khyow1KWnZ9YPYHhp10hUHDD7q2PMa99GmvcbB34BQeaCnzmKcOlGJsXBnhHSBeBegRuOeVlSYD5Lnrcb0QY8mbITtyn0UDHwQrOlJaECueOl66Vl+FxqtV30tZb8iZfLSp2A90utvG8EEigLqaJQJ0XbTEuoYjOtqOOj/A/aEbPDH+d00AprR6sgUdNCbXyTD/yVYrC5AeFh3b/ZlYQ7Yqoc/U5pDWf3/rOwcdo3RI98NEoU0hmZ6ZkyZExOYrqsillVdDY8ePQTk/vL1UXc53XKaRJ2YfSzzVn0EljI4fRCr5Ni8oD8SLXDbdLCzZ+xk8hh0GTRq73xDDnV8jbbaNuWGPhID7XbF+AP+c1o5RApYAhqYPhm3gdsDgdvKIy6lqY0chwnq8Jy8o/taPTMdOXNjqCn6AIGXj1TWoXAE+17/6P4Uusbizr0XUs1QAmhC/X+JIMkzg4XihtbWuF+sfsNVSA+PVYNtH0sPUlMlcjeF+mKGIvjHuLsKYKZ3JFCRomIv2Q7FUPceByxOdyrSFGoFQTH3oC30PdTkZs5JH622eXpaNxyRM5iEzp0j7DYOelTYULYXC9vSAxhYiLK1VTlUxCKszjTy0aeRUEeGkqKFDNxK9AMjJkxgH7R9mIGiGaN9TZ8baBhhSAcwK5fxJ74XPTOHkgXmBFV4JmFrorOzNkVFeWQ0Uh6oBfE9a3xcKGwPfeMe/Gl2jXjyIh8MSowuWjZyFIo5KT31cUg2HUDGErtvKEDq7l6a00QYZtbtP5j27HhPqxXMll7/ftghRE+63W6Tul+Xc+asvP2BugGKRM3d0V5bA5FGHWuf6pKXXS7Kx/b5bRUc cQUIdF+q vZD6ededOU/X+5vyk/5+lKpUW2g1ai1NKoB32pu4XwDvQfShTTL1LCjEvb/eXwymJHBAq08e26f7340Cj7LayjAo/hiAlwt/KwxUfzln4alAiuQPdYRemBhGUwr7FrPlyxBVqyU1pEgglGzit0Voz1leynTrMzJSFt4vHhfYlSjKnjmVBlH2K2n+tYm3sZs3RXkIBdoCKXDVYKgG1wOrNTF/iewxtCK8Pc5e0RQqlmjwFClS4FPEYKzVJe6Y/1fzTqdyc9ly9M/rGoKdzUDwC9Q4pJNzqbHxjLRllRaeXL2uykCnSmWapS4By6QrWFJhkSjIS/Vf/NGRsdjUOk+Y4fUuQBnOtMw//9+PqjexAyOqtp+CuRFKcmr1xnCLtT4AxLH7xnZz0gKBaGtAXVkOXptKHTuX14kxhzyFEHY897tsjb/uUzBdIl/OtmYQSNOMieBj48Ks+snXgHdLCmVv/kncdDcyEcFYpgbZW7U+x6iUA6gHrpPLrQ7yoV05UunlAy+NkBScl8OW03Y0mll3pdkUsaXtNRs1F55KefXDt+e/bgKUZmiExevfp+RVSn9jzrD0V5Q/+iYCu5HZlDwN/8Abj31egFU9JP9Awmkb/5g2qJ3PqdJPzJlkgC6205s+18Z8xGmZ07b7n9XOzarZCVcFa8uQLUtna+GVSeZ+7OxMlxu6jL6nW0YM7qHplW+9xwGR6j6kNv4Ehi+6S5oqRC+Jb3C6/625hwyltDnXkYwRw2/c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 1 + include/linux/mm.h | 9 +++++++++ include/linux/pgalloc_tag.h | 2 ++ mm/mm_init.c | 12 +++++++++++- 4 files changed, 23 insertions(+), 1 deletion(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 4a3fc865d878..64aa9557341e 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -46,6 +46,7 @@ static inline void set_codetag_empty(union codetag_ref *ref) #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ diff --git a/include/linux/mm.h b/include/linux/mm.h index f5a97dec5169..b9a4e2cb3ac1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -3112,6 +3113,14 @@ extern void reserve_bootmem_region(phys_addr_t start, /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) { + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + } ClearPageReserved(page); init_page_count(page); __free_page(page); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 9e6ad8e0e4aa..7a41ed612423 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -98,6 +98,8 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} diff --git a/mm/mm_init.c b/mm/mm_init.c index e9ea2919d02d..6b5410a5112c 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2566,7 +2566,6 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) { int nid = early_pfn_to_nid(pfn); @@ -2578,6 +2577,17 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, /* KMSAN will take care of these pages. */ return; } + + /* pages were reserved and not allocated */ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + } + __free_pages_core(page, order); } From patchwork Wed Feb 21 19:40:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F048C48BEB for ; Wed, 21 Feb 2024 19:42:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 568836B00C7; Wed, 21 Feb 2024 14:42:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F1B16B00C8; Wed, 21 Feb 2024 14:42:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31BD06B00C9; Wed, 21 Feb 2024 14:42:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1661D6B00C7 for ; Wed, 21 Feb 2024 14:42:15 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E5BB312077A for ; Wed, 21 Feb 2024 19:42:14 +0000 (UTC) X-FDA: 81816832188.15.0C6490F Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 1BDEA180012 for ; Wed, 21 Feb 2024 19:42:12 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cZisyCrP; spf=pass (imf06.hostedemail.com: domain of 3E1LWZQYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3E1LWZQYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544533; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BN4n4r+pGIOuiPq9+HuwMZ/LVswbQsC+381oXCoHOQk=; b=g0R8Ii5IREPS/ylu3D58XFvd8lNIvPAX+yTt7c9mlKmhtg3hTkOXLgIxCtYGyoQXn1QtOW 6f84zOZt9WdIULIh3CfzyzPJoyIQSl/a3tmGIRg2ESJUoG6vt9ROWYL0tgffvsvZX832xx +4K6D7WIyvCv4fsa5lovnddCHnIA1Gw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544533; a=rsa-sha256; cv=none; b=u/Xq1eTB8HwVy40hekUJx0DSo8jY2QUc2EwC5NplcVP7MgRm9Gf47+th0CqTVk2WRexKd2 6du2/x0k5pKhGF/WMAB3BhS+b1b3g1X8PuA1jzQXnvinwuREZDu/bUC9dT5Br7VHYZPr1u gyjV8M9uNwFT0h/8z/wtuMTjv7hqslE= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cZisyCrP; spf=pass (imf06.hostedemail.com: domain of 3E1LWZQYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3E1LWZQYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26ce0bbso14756151276.1 for ; Wed, 21 Feb 2024 11:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544532; x=1709149332; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BN4n4r+pGIOuiPq9+HuwMZ/LVswbQsC+381oXCoHOQk=; b=cZisyCrPeonXae56aL6675SC9EaYlHm9Vzx1RTE2YRmx94ly19kUmryrTZWOkbdCQi uRXWHNyYe3F6SDlD1fHVi/rAnMlg8VkvJtThTsGPThac8Sb2MmfF4yuHWlAUoK1Uqy5I yVlzg/A7cb/sb3qk8QM5Tpd7WEJNwBVEOS7LVuTf7U4AEsETP3CzYtMHQd9j8ffPPKXD 2Ne+9KWxr4+kpH+0pM+uYs/7molWuPPTrd4/sLMqfdOMmwKCgud3gjWULEgQOvDK9OP8 2ro1mIwz1fB1FFkIZktj6pNrbCdF/+ZdtQBrExfUrFkAQJAddPsF/mXycRcpD5qxzAAt caaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544532; x=1709149332; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BN4n4r+pGIOuiPq9+HuwMZ/LVswbQsC+381oXCoHOQk=; b=L4qstj8ftM7OiQhWrpnnCN4OUy9g2ofXnBX08xncCUozA8PRBY8zUyWdU/u7AEZZat ViUJv3q7EwLu9U5YDZ/60lLVbu7l7mpii6+l4Ou8bTna3m6XIMTs8RiZvWAzXviEWlQS GY6SBsW9pX51SXRIZLYlbk6D0BmX/oHwDSaC4sTywW30o9MSOhzkCxt52+I9BA2V038y GpXqxzKiWg5f1n8ryuLQBWPo1VqvzvKAQC39A68V+g8XxfxZiS+fZMwKc7dzUt0wzVr6 ak2jszEgzPwtf1/7hD8HYzGm1Rtg4pDNq9LWiK87QX5I3ESTvj72CkiyIWpPwY+ITzja O8oA== X-Forwarded-Encrypted: i=1; AJvYcCV/DK5XAE6PpCifg31Swd89qk/QtlKkx0OHVfFthylGIdF/p4N+YHYh1MGJCaP4hcCJymWyKyVU1EiCeM6WAQQ+67c= X-Gm-Message-State: AOJu0YwZ5Cj64eAgEtGrCOXRsYWB5VSzCcWvdelhdt883o9bRAAzURza tYA3MWL7BVP4x85e4f481cC90UqYOaPmSmiD2IEX06D4hrHX/ZAIZTZuyc3239CuTn7ljhJ1fx7 Krg== X-Google-Smtp-Source: AGHT+IGccLdK1+8RZqTfFKQGLX6PM7i11Typ+L1RnqZIOYRT2MopGhT4QGJWEbhgFrEtPg85XhcRKYt7Lzg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:ab6f:0:b0:dcb:b9d7:2760 with SMTP id u102-20020a25ab6f000000b00dcbb9d72760mr67613ybi.13.1708544531841; Wed, 21 Feb 2024 11:42:11 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:47 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-35-surenb@google.com> Subject: [PATCH v4 34/36] codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 1BDEA180012 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: yur6fbamxzdd4gcptfmoo1c9cq67f8y1 X-HE-Tag: 1708544532-791600 X-HE-Meta: U2FsdGVkX19kbpqJbvgZidA0NFVRkGkbTolcmSxX8S0vAatYQuk0xWN+s+vFncbstet3n9dziVhl5823yibRxK6om5uhEYWl9QPJEOLT94MrMDSv8uHEfmB1l+t/uUjdGj02B5nJr5p/MF1ndT/k4zSq5FTyIgUft7V1lMn+A5UKHzBqBa3+Jkz9XA7ErtZSTNew/9XI9EaY7gpetOncmfX8oXuTeqHjitkNsUJa2bXqeg+ee1ewnnrg0705hiveiWhMD2mNBxxoqZtzNIogN8SdZ1Li8r1DmHyBlrWOHWDMy15o7FmQCqtB17nOfUbPNQscWMX2djL/HmpF4P0p/MgW1uuSm/MYPWbimOIph5wVfsuFT42agkLK2IihymMThrRW2yKMH3vJpCnQiFY5HzuDeSzCsrrMQhOgmH+1Sc2cHWshfpfKtjU8M/mraODZhpI5jxlikOqkCflRJ8+3Yef51FM4Sx4jbhY6ekIeiaVMDO7xtuEgEp7+/ebowVKKDmSUOu1r0snKdxVEEtXYd2y/zWYgXyDvViYlqFw5H3xf0IhCXLHMFPy9vf/wqWxHCRq0KGZEoJPjrvFRJy/agLsCyfXpNK/cQDLEVRr1QYrssp5Dm3xsqT87dnOSx9lyXXoGVmIrnXSvYXeGHwKcb3vbMFRPVKMaRSUW13Wm/9pICzc8yNa3dXrege8gw7+xqXFbUDDUUH02CN8kzEaIjnEIvrgQIT391lrRYd6rpT5CyLJ2YpY4icCTcVEQo3/6kIT1X7wfeteW7HwJrzYUbrZE2BQFroI+8T2GDHekk7EArwBIJWtSNBdoXPmpgCmACXGoCyw/6QEQTQVThCnCONhRKUAgJlJGYtO3QN67FW7L4cm9tlKHT/uGBga2kzXHxynqL4mjKG4z0JUX1fiLOaaRgxpXQq+zmHtoXHOOhUZd8RRHT6ePc447xln7sJ9g2+YSyGfYYdi3O3IZsiA E7/RhyOi q94T98yGSvZ7u1l5Le2NukjVo6xwJwGab+LbJOPgYZveWjnvjAVMqkB93VDqZnf6m7ZCWLCa3BU1VyzVKWqMcAzaBUo3zdMF5zq8NiFMnuw+ijxXcpBmE6E3O86wQSV1jEVcq6l5PgmTj5FfPOnffB1ESRVFk4Smgdg5uG0wubeR48DJRUagQQu+pxqpstj5tDY3NfuQQ368lQXmqDyGLgZZq04BnEqiRm29JRVwQhEq/jawSMwm0KzHNVmi1bwa/IZ7bCIN8gfI9hvuCd8x5o4ofz1r7IFCSEjF8QDeLyWtjbzxxIXhF8rNUfUIUGCtXZBySwYcfRtfz8O0nxDgAWvHB0puGv816Y+pP3gD/+U4FVceeZCyZEJztREVgrZT3M39M0L/lzyCQq2T48G3+W8XKthkPUd7LuUeNyVgxFNbqZBPxU3h5jN610EAgYMXLByNFxA2uwfouhjczV4suzu2fp0gJVvfq+AGSRdnYafNg7twXEIpP+wnpy0wcyKWj6MqjLQ8tPeykXwAJiIlAhbiIKLr23EHFcp/jugc2jlKevHp/Uu3/ewN5qFSlYFlrCXcoP5yWAMPnzIc1HHMqovNs/Nvv5MHrY8n05iWqazUfRyx05fgq3m24l0+75XW84u/WM0//R9czjaj/ZFC3g2szagjKOH84cnet2JCzZAS5hw9FwSbKpIha7S2shdHa5MToAjCRbqzEcATrZ5N1oDYMGA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 4 +++- mm/slub.c | 46 ++++++++++++++++++++++++++++++++------ 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 2b010316016c..f95241ca9052 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -365,8 +365,10 @@ enum page_memcg_data_flags { #endif /* CONFIG_MEMCG */ enum objext_flags { + /* slabobj_ext vector failed to allocate */ + OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG, /* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG << 1), }; #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) diff --git a/mm/slub.c b/mm/slub.c index 3e41d45f9fa4..43d63747cad2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1901,9 +1901,33 @@ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) } } +static inline void mark_failed_objexts_alloc(struct slab *slab) +{ + slab->obj_exts = OBJEXTS_ALLOC_FAIL; +} + +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) +{ + /* + * If vector previously failed to allocate then we have live + * objects with no tag reference. Mark all references in this + * vector as empty to avoid warnings later on. + */ + if (obj_exts & OBJEXTS_ALLOC_FAIL) { + unsigned int i; + + for (i = 0; i < objects; i++) + set_codetag_empty(&vec[i].ref); + } +} + #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} +static inline void mark_failed_objexts_alloc(struct slab *slab) {} +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ @@ -1919,29 +1943,37 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { unsigned int objects = objs_per_slab(s, slab); - unsigned long obj_exts; - void *vec; + unsigned long new_exts; + unsigned long old_exts; + struct slabobj_ext *vec; gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); - if (!vec) + if (!vec) { + /* Mark vectors which failed to allocate */ + if (new_slab) + mark_failed_objexts_alloc(slab); + return -ENOMEM; + } - obj_exts = (unsigned long)vec; + new_exts = (unsigned long)vec; #ifdef CONFIG_MEMCG - obj_exts |= MEMCG_DATA_OBJEXTS; + new_exts |= MEMCG_DATA_OBJEXTS; #endif + old_exts = slab->obj_exts; + handle_failed_objexts_alloc(old_exts, vec, objects); if (new_slab) { /* * If the slab is brand new and nobody can yet access its * obj_exts, no synchronization is required and obj_exts can * be simply assigned. */ - slab->obj_exts = obj_exts; - } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + slab->obj_exts = new_exts; + } else if (cmpxchg(&slab->obj_exts, old_exts, new_exts) != old_exts) { /* * If the slab is already in use, somebody can allocate and * assign slabobj_exts in parallel. In this case the existing From patchwork Wed Feb 21 19:40:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 306FBC5478A for ; Wed, 21 Feb 2024 19:42:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5A146B00CB; Wed, 21 Feb 2024 14:42:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF06C6B00C9; Wed, 21 Feb 2024 14:42:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90F006B00D5; Wed, 21 Feb 2024 14:42:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CE3F46B00C9 for ; Wed, 21 Feb 2024 14:42:16 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A287180BC5 for ; Wed, 21 Feb 2024 19:42:16 +0000 (UTC) X-FDA: 81816832272.04.CC891FC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id EDA8D100014 for ; Wed, 21 Feb 2024 19:42:14 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3NfQUIL/"; spf=pass (imf05.hostedemail.com: domain of 3FVLWZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3FVLWZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8ETkntKf5Z8TKkoIJeAz1krTkKQJvX5Zd0TKA8iBrrk=; b=LGDAaIAfgM5k22IAj5AbrKR9R34sTjCCkDoTcjs+LoDZ5tof9ekNwfyUCVoqDZtMKSu9GB EyckDBKOYmvftfI5TqKCiOoF6MZcec9AeC00ED1XZw+Gy+t5pWShqAolZ3WPP5MHzwKHwh 8frNZzBx5vKq3blDqNlL9/zMBWoykF0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544535; a=rsa-sha256; cv=none; b=eBDubAGbF/uFNprcsD7M/Lu2HBefgoe27Zywov38dRFK9mgRsrJA3YkhHgKvoLbQRKlPvd ZdImhy6NtHTR/BLDj8RjdN9b1UNoYhHh42lV+kJYlESWCO2ruigTr9R0pfkBZWxdMyhM4F YjTHgSgapyBV8HKyEmDj5p4/6256HK0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3NfQUIL/"; spf=pass (imf05.hostedemail.com: domain of 3FVLWZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3FVLWZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26783b4so6791956276.0 for ; Wed, 21 Feb 2024 11:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544534; x=1709149334; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8ETkntKf5Z8TKkoIJeAz1krTkKQJvX5Zd0TKA8iBrrk=; b=3NfQUIL/V8Fqfj7UFjootvkZF58cuCEypPYSDLY5rdZS7ytMOUgmbJKbNVHSD46yoZ yK+lwg4GpA1YujleHU0EBUnHwkad5crit3oM2ZQy0NeBFTRrtSlsovamSYtC04ENOfw5 8jdHQrjXr3RcG1tb+DFYIQUtXo34dB8a6rmzNC+ZsUuTZlrv591yzpdIbk+9qWOYZvU3 sB1HbmgERb0s3Bs4tP7R+ThvC/blMGb8FQFJR1yx/YMuAe1d2wogKr39w7W98qK4FIWY cPqqhcu5Cy+zLgNWuoqOxcrDi2bGwqD0sWFV/T+V3flxrGV/Ph815Ibbu4VRiQqAZVXl PNOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544534; x=1709149334; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8ETkntKf5Z8TKkoIJeAz1krTkKQJvX5Zd0TKA8iBrrk=; b=SoIRSluqZOIHThJwDjYUw8K+kH4apBE9RfnRYYlEp9xBiplvK2za+KRLfax65QY6Ou 6jJO8bcKDgVLkKALPDoWZ6mZj6qvtFCtBzQwJtgXHfXHGwOsXyYItpD/vOCNVDAgRYfN 8n5HYALIutBrUTQhfTNsmGEnvzl0BQYAEQwFNkqWUhCPYbSS9D0UXRSyNo30mORc+Kxd g1ifmctjcRyjmnXTktXUi1CCfwROhf34EMmPNjTh+70FnnQZpcxs+Y4M4V4sVqAVE29Y Nssm687eAxDYqO2dz6XWRYrPeMXAh06jzppSpJdx7/zA62FWXYFs0ul8yN5O1HP+8fZW m+aw== X-Forwarded-Encrypted: i=1; AJvYcCWV+YlTAwfFO3Y2SV1oy/nS0uzoPHa2AMOUuY4QtEgRJenQuHVS+7NabERqkRYgS5qIMJ/rzWKggESNqTSWM+vsN3w= X-Gm-Message-State: AOJu0YzBF2xn8fecHqA3+tfIT0aFuJyhUIr9diIga2/JN9r8Gv/zaUbW B5nDj6MJvnlbLhr0SV3fW8v2+BUF79lfBpN4nA2LdS3xYiRA5KTdar8FLulFLizbpyxreY1U0Py 5Fg== X-Google-Smtp-Source: AGHT+IGcBk16mzh5/xxbKr6yggh2Yyj7g5lycw9QHRAaELldFsCkqrypZfX0yKxpepdpGv4MQmcqVIlNfo0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a25:844b:0:b0:dc6:ebd4:cca2 with SMTP id r11-20020a25844b000000b00dc6ebd4cca2mr14813ybm.11.1708544533985; Wed, 21 Feb 2024 11:42:13 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:48 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-36-surenb@google.com> Subject: [PATCH v4 35/36] MAINTAINERS: Add entries for code tagging and memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: e4tjsxg8q1i94dfb84orj1d6tpdwgobt X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EDA8D100014 X-Rspam-User: X-HE-Tag: 1708544534-123652 X-HE-Meta: U2FsdGVkX19r8GjEhJGUz9fyhcpxVQypS6fWXO1unAg0wQeN7kIor5i3NL4hr+n1Q8wBR0v6otTjBNV1Zx7kf7bDmq5coCNIAYvsXItAk47pVYx/xlJxqMi7F54nGlCjW7IXa8BS3rGAxREhBfi0LxvJwWNx6frFnDm+2X5aTBBWsZTngAiUm5H53tOVN5l/MqMQWX81BxWww9I7gABiEymlOMdhTXZRb0AQ2qNAM3vhm3v8mZRJBoLmGLhBCOLUAUHSCoSSm/nuLjlpwupYrJDen90w7S4fYo71rqBFAhHQK/yT5XkwAjgAvdGy6A2AZgH5S5Xj41kRp9bV8TAYmqzsaKf0Wdn9TUqfhcdI6g6SyM3k3AwSe1LhibE5z1zmE+PSyP+i9Pdtc1Sk/XvtfJMDOwN0yyD4BSzxN1Ppp4bYfiKuEEyyn+3BUQHDEbNBPlu7pezYWkUvzKQI2dCFEZS4aEE2Dw1RRRxrm3iSc78bmdw5Y8KaDoxtpTk4X7kO3SlcmLETzvinpOeGLko5VBXN4Wwh+hfE4nnk+9PweRTkalqnYA3v40So4+hrXU9Ny6FeDboE6cBIBhypU8fyQ5eCwc4vqWoY2+E6pkA9mxAzocU7KBIU9uQ0rT0DBLcAryg+nyKXfNX5hsmhFEjjbGvHaJH63KB+0VcXx9r81a37aM1yV2eaF8PrLt4x361fT8Pw26POgEXh5qvdATKX/iMXkm9lXra3KUdo9C/o/Z6LCf5wibalisxx/1bU63Q4gu3+qmixeoYYJOFCBqlGbCYONfMI/RXhHIUABeyGqfhbqpS7bO1YVLsLs2yAhR9umecSGrRmIGrZ5cfX3GxLBu9N4tK9MPYegfBIyyIg6OVzWeVYt15vI2lCP0p5Kjccu+Y1CEWPOFiPiFsYWctnk58c4RdABUolkbbn7HPJwXh4EGtWhyUqpI5Cs8hFtYvTp25Mv5Jrq2ZDXx7LSZS DZipuj9G 8mzyDajTlyO8qfLbqtH8CWp4j3XtVdcmHg0PkePamAgWHC+fciKOcCi0UuAqJOyU8e7o0JKnrDEiVQMiHJBTEhC0h2KL9MvMDd8D4pFhTh4gm7vRF8/RTCVj5+hjYLRF5Qnftw1OOP2uGPH1OPBH2cYgh3FyVG+rqzP/EdQRE0mLmZphWh5cNfGU7+KgL5ICIwpg+A8q6yuTiKSlt26QEzmzO2j3rrLeNrfJRW/g3vNA34MUxi2CB5x5VOvUZXd7FrcbCOrilHc+VgGH18y2guvNVciLVWpZH7aPCmENbqfrPsz2Hcm7vuUyIgX6bJlnVwTo1hpd17n2TbdDrahLzMWFn0RZa2x7Z1gSmi8eRCOldxMAMkP6ww52kRnqBlwFbit9dqNrVSiaokeWcU7AVMk4rkzrzws1Y9mFF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new code & libraries added are being maintained - mark them as such. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- MAINTAINERS | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 9ed4d3868539..4f131872da27 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5210,6 +5210,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott M: H Hartley Sweeten @@ -14061,6 +14068,16 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan +M: Kent Overstreet +L: linux-mm@kvack.org +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski L: linux-kernel@vger.kernel.org From patchwork Wed Feb 21 19:40:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13566216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA81BC48BEB for ; Wed, 21 Feb 2024 19:42:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9C736B00C9; Wed, 21 Feb 2024 14:42:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D5C786B00D6; Wed, 21 Feb 2024 14:42:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F8936B00D4; Wed, 21 Feb 2024 14:42:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9DE116B00CB for ; Wed, 21 Feb 2024 14:42:19 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6C8391C0CD5 for ; Wed, 21 Feb 2024 19:42:19 +0000 (UTC) X-FDA: 81816832398.08.4768DBF Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id AF705A001C for ; Wed, 21 Feb 2024 19:42:17 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Mpb8R9p6; spf=pass (imf25.hostedemail.com: domain of 3GFLWZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3GFLWZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708544537; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iHJNBSQgY470Nm1vo9kQv1HWTlhW4/9LqFry/+3J5f0=; b=1H1gViQ4yYf4iHawnUV58aRbZYJ1GIWBlGDmFyWbTeJWLIax+wE1WBtjRCY2iKKNCbBSVp EbNPZeKqx423U9fdgzFEHdllxBOFDwFWJ0cVKxkKbJm8KHHylaixFvbBRp0oB7wWFaYotR 1oDsLZZkV7wEhOsBTnHWW2I9rV8sFjk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708544537; a=rsa-sha256; cv=none; b=xhNR9y8BO0WzC6jufnMojqckX4t8Vy6Yxn7+B+Sjq+Z6rTgW25vLUGRrFKc5F3LD76oLRI cPCHT9rqIqXHsRDxNU9/YgIqsbi8Xq/DdUKJjFGMaPflKJd2mlQfca1B0TG3N94AZJkRqE BxNmhtaCbJw2hB931Ec3d1gS7FwJlek= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Mpb8R9p6; spf=pass (imf25.hostedemail.com: domain of 3GFLWZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3GFLWZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc691f1f83aso76390276.1 for ; Wed, 21 Feb 2024 11:42:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708544537; x=1709149337; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iHJNBSQgY470Nm1vo9kQv1HWTlhW4/9LqFry/+3J5f0=; b=Mpb8R9p6XkftoELXGCt5DWf1FnZmbIkAHIXrJ7XPGLkE6ZpywodK5ZaXADbPhCDAaw MBM1b2R6B2mFGZiAY++HLi4f2t0/Zy6QORBFxEdpahIonFEpGOu5AiaXIBrQHOFIlGxU hLcDkpRxzIN1z0UtXSDjLzYiT87+vHeKMrFF3JqYHwiH6C7iux6JJSp316xIB0RH5bPW B6ezIRTuVb6TACrWLsMd3iUuvEzqntuVeSdWc2RzmD3mgRiiVbpbPIWdKIhbzTcQME/T vYi2DshMVKHwsgyQ8LJ13WZSvy1i3+Z8/ygEp8UnhujY8nN4C1An5di3QwpN7TPWmZ2C goTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708544537; x=1709149337; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iHJNBSQgY470Nm1vo9kQv1HWTlhW4/9LqFry/+3J5f0=; b=OyE1BGGOSvYQI+ySNtsgVwOdbq69xOW6WDCEePGo0rvghA1Fbc3uaUGPUCW4SjzOkG bhfa+ts2gVkJIyB1g7PvFmxryrWcfK17bOIUkUQxSaF+oinycWC+7tkWT42HwXo/k0pX wvyPNjWmknAFbwhqD454/AWb8IF2vjyEtxYFcpscalRBbrnvp3kYjUFrg1YXSNX5SdL0 vYjGNlfTB6WMiUFq1AYXVcLTzbLLHZ96mHUiX112f2FxJSxfTXh/65q/G1NoYBYKOhtH X0f9noGemTDyhUm6Kpi/AVrhzcJtXQQOceeWt6WOkEmQ95wpC5KHUCzAMwHRbvOS/Wvb 0APA== X-Forwarded-Encrypted: i=1; AJvYcCWovk6lV9imLEAkLSmo2WFr6kAOr6Zo0yoONpF6tdTIMqLUskH7azawBWdT4MMBsF8vx+YgD/N/9yGA8AKatQp/8fk= X-Gm-Message-State: AOJu0Yxw3vIBuKClsFTSN4yntdxZIkxPAOiLKhcePcx0se44qkiVr8Cy NgJkqtL4BAKqMakkzHBBrs5bQC3nYDFz1f2GI4d/DfiAO4gdZ6m5AR/pFaAEab7Kwb9YrQc3uGs JqQ== X-Google-Smtp-Source: AGHT+IHkeXMTMo11jEJZVLK3x/LUPbuFY0r5OfLvnPbsUo0+TD2woe9SeAhjFOGiRZt6wf97J57l2VuRNJw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:953b:9a4e:1e10:3f07]) (user=surenb job=sendgmr) by 2002:a05:6902:1008:b0:dc6:e884:2342 with SMTP id w8-20020a056902100800b00dc6e8842342mr25280ybt.5.1708544536561; Wed, 21 Feb 2024 11:42:16 -0800 (PST) Date: Wed, 21 Feb 2024 11:40:49 -0800 In-Reply-To: <20240221194052.927623-1-surenb@google.com> Mime-Version: 1.0 References: <20240221194052.927623-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221194052.927623-37-surenb@google.com> Subject: [PATCH v4 36/36] memprofiling: Documentation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: AF705A001C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 9k9h5gdpue3rytsgiogk46cg4t7r8mao X-HE-Tag: 1708544537-79461 X-HE-Meta: U2FsdGVkX1/LauYGd6TcQ8OqQWMUPj1/xK3IHxZiMIZDXTCqywKQ/vggs0A0QTcauI4lHzzpwj5QQMglEibOsk/sEOkINTrDttzcdqxBhvFCJOh9fua/P77GZ1WXYUOwqWLhT00H4JuoU76GSNsbM/KGPh32LEhjm6URsXSKFIqV1whFEK8oDrPkP3LF8584g+m3WO9hyUozNteHs8osQKFQBhb6a+xuP7Y+PPWc8D2R9Uw5UL8LfylwF9FFKMR015s+dem5u5+U56Ko8r77oUMwcUIT0EqEFePIegkMoPwjeitVvWd+r9vgJyKBo1laPezLx3Uac8X9BJn9sNpcEPV7IzjoQzTZvxAs/fj6SBsqrsP4AxovRxxBrQRi4yCqDQnUwcS3tBANRbRMS5n4o8xOnkIvyAozYZjB3T9psIRQnp8mdzlGYrYPygMaKKW4Kccrk6rj/2LrDGBllhGqpIekGdycnBahGFSe1dsVdoGM57YZHuqGTej3hZJF9BQVYk0h8PEqboUConqvt05W34b1w/NLkzqoOdN7Yn7+gMcAwovgxX7ntz6tjZRzpYbL11LYScoJ7zN50JnK6xNHFfUdvZj9KtvpTEWcZXsR8n1hhZkl5MidDhGkvKYNNsiihYgM0g8ftbYXO21rvRXRz2UVokGT8+03VsD1S+LxE20TSFxHh3jr3f9vlFZOuJfl42dt6OeCYZvNImSDGSrPqKEbOcQZD7Nle7w5uYiyAh6UUMCem/VOUY1m5SqlFFDjUQ6B1/6318cv6NyV3EJ1YRLgzp0WShaq+sPMq6YmwdAKexoXYS81j5ECKKDXkZKJjwUKLrN9066uydqKxdHZI6ZRT6j69dA7xlxX0kiWkj0kTqPOm3ENnt19VaXvVr0FNmIGay6lk0Tx3z6v82H0TBnP9VnhwV4ACZcCkpTUu5KY//KvXSqID4XE0TKO3dwOeN0m3i9UbjAz9s5LwJd l5JD1TnL 96qrpRQ5+EZxtMcszOog9Eh3sQRJ4ROesnJB5hZK2AvJYh1N/Uu3qjxdbSG+gxrdozbrU0fQ1KiExb1TkRdIg4Dtz46rGVpvbL7JhPSktRV25wVSxZO3gJPqM8kVHmDjGUzNBDfFdTu49b6p9PdqMG94vfDDNGTno0q+ItJD6yxy31Gq1LWFjtWPk0x0jllV75OTxlqK0hx7bwzed+SVH9ZTmhf+/Vfkz8GrEtdpQZE7oo85+qsUXnTwtYRtrJRM7RxRhhDenMlQxozJMtofzvvDHD7N0JA4YJnQyrnourfWGD6Wo1VATqfySkerpqXZsct0G5D3LzngU3DEgJ5bKsnj706wW2KhCtnmXiFGa+qAnSyP76viYHNVdMldZmANUCb1R43KIE3o7gKw3+2sv42FlhP0+sXEUsc1v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Provide documentation for memory allocation profiling. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- Documentation/mm/allocation-profiling.rst | 86 +++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 Documentation/mm/allocation-profiling.rst diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst new file mode 100644 index 000000000000..2bcbd9e51fe4 --- /dev/null +++ b/Documentation/mm/allocation-profiling.rst @@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========================== +MEMORY ALLOCATION PROFILING +=========================== + +Low overhead (suitable for production) accounting of all memory allocations, +tracked by file and line number. + +Usage: +kconfig options: + - CONFIG_MEM_ALLOC_PROFILING + - CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + - CONFIG_MEM_ALLOC_PROFILING_DEBUG + adds warnings for allocations that weren't accounted because of a + missing annotation + +Boot parameter: + sysctl.vm.mem_profiling=1 + +sysctl: + /proc/sys/vm/mem_profiling + +Runtime info: + /proc/allocinfo + +Example output: + root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec + 2.8M 22648 fs/kernfs/dir.c:615 func:__kernfs_new_node + 3.8M 953 mm/memory.c:4214 func:alloc_anon_folio + 4.0M 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 4.1M 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 6.0M 1532 mm/filemap.c:1919 func:__filemap_get_folio + 8.8M 2785 kernel/fork.c:307 func:alloc_thread_stack_node + 13M 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 14M 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 15M 3656 mm/readahead.c:247 func:page_cache_ra_unbounded + 55M 4887 mm/slub.c:2259 func:alloc_slab_page + 122M 31168 mm/page_ext.c:270 func:alloc_page_ext +=================== +Theory of operation +=================== + +Memory allocation profiling builds off of code tagging, which is a library for +declaring static structs (that typcially describe a file and line number in +some way, hence code tagging) and then finding and operating on them at runtime +- i.e. iterating over them to print them in debugfs/procfs. + +To add accounting for an allocation call, we replace it with a macro +invocation, alloc_hooks(), that + - declares a code tag + - stashes a pointer to it in task_struct + - calls the real allocation function + - and finally, restores the task_struct alloc tag pointer to its previous value. + +This allows for alloc_hooks() calls to be nested, with the most recent one +taking effect. This is important for allocations internal to the mm/ code that +do not properly belong to the outer allocation context and should be counted +separately: for example, slab object extension vectors, or when the slab +allocates pages from the page allocator. + +Thus, proper usage requires determining which function in an allocation call +stack should be tagged. There are many helper functions that essentially wrap +e.g. kmalloc() and do a little more work, then are called in multiple places; +we'll generally want the accounting to happen in the callers of these helpers, +not in the helpers themselves. + +To fix up a given helper, for example foo(), do the following: + - switch its allocation call to the _noprof() version, e.g. kmalloc_noprof() + - rename it to foo_noprof() + - define a macro version of foo() like so: + #define foo(...) alloc_hooks(foo_noprof(__VA_ARGS__)) + +It's also possible to stash a pointer to an alloc tag in your own data structures. + +Do this when you're implementing a generic data structure that does allocations +"on behalf of" some other code - for example, the rhashtable code. This way, +instead of seeing a large line in /proc/allocinfo for rhashtable.c, we can +break it out by rhashtable type. + +To do so: + - Hook your data structure's init function, like any other allocation function + - Within your init function, use the convenience macro alloc_tag_record() to + record alloc tag in your data structure. + - Then, use the following form for your allocations: + alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...))