From patchwork Tue Mar 8 13:10:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12773792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 777ABC433F5 for ; Tue, 8 Mar 2022 13:11:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1F3C8D0009; Tue, 8 Mar 2022 08:11:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCF978D0001; Tue, 8 Mar 2022 08:11:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C23748D0009; Tue, 8 Mar 2022 08:11:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id B593B8D0001 for ; Tue, 8 Mar 2022 08:11:22 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 73FA8972E6 for ; Tue, 8 Mar 2022 13:11:22 +0000 (UTC) X-FDA: 79221255204.23.0BE169B Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf30.hostedemail.com (Postfix) with ESMTP id E057C80003 for ; Tue, 8 Mar 2022 13:11:21 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id c16-20020a17090aa61000b001befad2bfaaso2370579pjq.1 for ; Tue, 08 Mar 2022 05:11:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6X23AJzOcszEUYA+Oc8WYhr7w/7aeKWJdCFlxFdfbxc=; b=QzDOs9yQq9R1CZFP5kH/i37OmN2DriXgh+Uc29/28MK52/n2BNL5T+LpOEtCkDH1wT SArng9TL/tXBv+Z80bt9GxJnPhLPtlJphCtIvRp0gdbabeClqLN5OJJWkW3w2ZTfWwQa Rnjfw7mAFLLj9wTWH+Eh6SOwOz0vOv1IZ2Ez9Hdj3Lib06/AoNebKC9gEtTDUdNrv4NQ 9DcZx2G8SWEzn6bao5aEaUSzUNDhlZZ1d3NO0Rldf6bomrFxx1CFs1Xa9YnCHp8zH9iB t5155kTpOmXdPuCwsVILsy5c1hsx6nSVXhy04JPx1GbVRQYFwe+KylhvR1CA514F2xZd 3C/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6X23AJzOcszEUYA+Oc8WYhr7w/7aeKWJdCFlxFdfbxc=; b=8QsrUQwmFpCp8JQeVgtzHi3xI4cAIn+5lOs/KQNzcLolcuUusb23jW85OlfN95rPwZ ReWQYtlMi9s4i/375OznuP2s99JFNBi1m2fv0XvSPtjRpwiO9I7A6f5w+fNCbc5qvugR ZDzzMz1uEvyfYoJgieAM+qjG5DUAKV1VGZiahXSG4zrANAI/+8dIf/funK/9cuW9bGmG yl/pMCY/nK82I8Yo0R6WT0Lur/XBks3mX7vsjQDW4s3m2Y1gkNefR4gcdNTxsKjwd2FO 45PETF3U5OUTv40bjjxNoo1qeW1zM7PQDkE5CGbKYsWgtphz4irpg+fN6GW9jjcajELl oBrg== X-Gm-Message-State: AOAM530N59GRVDnWk41nMIudkPEzSz4R+gLj9HLAwfHnzDe1HPALmsaa 7DOGLf7F91A2b26Hvo+DLW4= X-Google-Smtp-Source: ABdhPJxKOq3e10aQ1RmNw8ArFeArmumS9bRFdOU8RkbwmtWaG0AV0PF5C3HqteolvWr6aJIng6XTbQ== X-Received: by 2002:a17:90a:4fa2:b0:1bd:383a:2b6b with SMTP id q31-20020a17090a4fa200b001bd383a2b6bmr4727275pjh.108.1646745080990; Tue, 08 Mar 2022 05:11:20 -0800 (PST) Received: from vultr.guest ([149.248.19.67]) by smtp.gmail.com with ESMTPSA id s20-20020a056a00179400b004f709998d13sm7378598pfg.10.2022.03.08.05.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 05:11:20 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, guro@fb.com Cc: linux-mm@kvack.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Yafang Shao Subject: [PATCH RFC 5/9] mm: add methord to charge percpu address Date: Tue, 8 Mar 2022 13:10:52 +0000 Message-Id: <20220308131056.6732-6-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220308131056.6732-1-laoar.shao@gmail.com> References: <20220308131056.6732-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E057C80003 X-Stat-Signature: itf56r139asywpec6gkkhkxr4t5qwrcg X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QzDOs9yQ; spf=pass (imf30.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam07 X-HE-Tag: 1646745081-398803 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch adds a methord to charge or uncharge a percpu address. It is similar to free_percpu except that it doesn't touch the related pages while does account only. Signed-off-by: Yafang Shao --- include/linux/percpu.h | 1 + mm/percpu.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index f1ec5ad..1a65221 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -128,6 +128,7 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); extern void free_percpu(void __percpu *__pdata); +void charge_percpu(void __percpu *__pdata, bool charge); extern phys_addr_t per_cpu_ptr_to_phys(void *addr); #define alloc_percpu_gfp(type, gfp) \ diff --git a/mm/percpu.c b/mm/percpu.c index ea28db2..22fc0ff 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -2309,6 +2309,56 @@ void free_percpu(void __percpu *ptr) } EXPORT_SYMBOL_GPL(free_percpu); +void charge_percpu(void __percpu *ptr, bool charge) +{ + int bit_off, off, bits, size, end; + struct obj_cgroup *objcg; + struct pcpu_chunk *chunk; + unsigned long flags; + void *addr; + + WARN_ON(!in_task()); + + if (!ptr) + return; + + addr = __pcpu_ptr_to_addr(ptr); + spin_lock_irqsave(&pcpu_lock, flags); + chunk = pcpu_chunk_addr_search(addr); + off = addr - chunk->base_addr; + objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + if (!objcg) { + spin_unlock_irqrestore(&pcpu_lock, flags); + return; + } + + bit_off = off / PCPU_MIN_ALLOC_SIZE; + /* find end index */ + end = find_next_bit(chunk->bound_map, pcpu_chunk_map_bits(chunk), + bit_off + 1); + bits = end - bit_off; + size = bits * PCPU_MIN_ALLOC_SIZE; + + if (charge) { + obj_cgroup_get(objcg); + obj_cgroup_charge(objcg, GFP_KERNEL, size * num_possible_cpus()); + rcu_read_lock(); + mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, + (size * num_possible_cpus())); + rcu_read_unlock(); + } else { + obj_cgroup_uncharge(objcg, size * num_possible_cpus()); + rcu_read_lock(); + mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, + -(size * num_possible_cpus())); + rcu_read_unlock(); + obj_cgroup_put(objcg); + } + + spin_unlock_irqrestore(&pcpu_lock, flags); +} +EXPORT_SYMBOL(charge_percpu); + bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr) { #ifdef CONFIG_SMP