From patchwork Sun Jun 19 15:50:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18205C433EF for ; Sun, 19 Jun 2022 15:51:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAFB88D000F; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A60018D0006; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 901398D000F; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7F5DB8D0006 for ; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 609CD34A3D for ; Sun, 19 Jun 2022 15:51:09 +0000 (UTC) X-FDA: 79595424258.10.5E96B31 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf29.hostedemail.com (Postfix) with ESMTP id 17D6012000A for ; Sun, 19 Jun 2022 15:51:03 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id m2so492213plx.3 for ; Sun, 19 Jun 2022 08:51:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QAskb+5yNIJsaldLr50aWhyLhYS5r5QAog9O7C8QAr0=; b=KVtvmyag+k7GB/fCkk8AqkzTGDWXZtmlQ+PT8NnSmeq2wzVE6RIirW/ZsVSL0WWl65 r/F0p777dkMTZPtEYW0XSiRRz5FeDX1WCndPeYgV5jvIuU4HH1HmzNOprr3x38QJih41 /DVbvpKY2WZujqIpOlmgwxyc+85XJbvqSIsK1n2vY4Jo67si1AG5+OHzdizLQQEHW700 uWHc1PExNAaByKT6ttxpzOxXL+OeK0CLn1S7pL0mI73iHpByUDSEephJTCx7J1VJeacY TTjatlqZe7ts+Vs2hs68NR/gdLC+OZnyxmvFQ2Hx3FZhcKdTpBFYsXDEU2cpSj5g5/6l UeHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QAskb+5yNIJsaldLr50aWhyLhYS5r5QAog9O7C8QAr0=; b=uHnBfszCygN6CaZstG0IXAos0ozLgbNg8EdWhMNoWiiACimGU0kX5wV2OZn3UCh+jR 64NLspiUiqR8yySKbpw+f0QjbPl6j+aMSQYab4bIQJTWhg2fV5fKlAge4rcyCEt5Kf+y Im8+xuPX8215B+VXHB6LFRFhGQezTQZUBOixppU9cKR604zslaNrETJIC3hnxjZu36Ea NOANbi5L93KPArAQqNAhR5aeUKOjWHWUf1vy6X4bJ7JNVgcBxYVF8wWsc3ytA2Maz/5y G5birqf+BKTS35JJI7dMPyBAPaimB4xqd0GA+T94eSHRSUhR4ctOPaC+RMyk0kawX/+N sgtg== X-Gm-Message-State: AJIora9G3bKfLP4Y3lIia47KiXs6q/VOB4tpOimJKlBrQOYzTCwexs2G DNme74ysErH2FVfUPx81yyvCc/5OHHQolDJUISU= X-Google-Smtp-Source: AGRyM1tZi5Zt0G1PIz0scQfO4Nr+NU/o7ujG423ZsaI0GC2fomWJFdewsZiWfRB2CETXpaFwWsKygw== X-Received: by 2002:a17:902:b597:b0:168:d8ce:4a63 with SMTP id a23-20020a170902b59700b00168d8ce4a63mr19745970pls.57.1655653863216; Sun, 19 Jun 2022 08:51:03 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.51.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:51:02 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 10/10] bpf: Support recharge for hash map Date: Sun, 19 Jun 2022 15:50:32 +0000 Message-Id: <20220619155032.32515-11-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653864; a=rsa-sha256; cv=none; b=yjcvzGy32VUthjQkQMbUhYDpv7MK66S5nmtUF3PCNyL6npZcA6dGz+/0WiWzqcmE3cRXvv 2Xl5lreEAhuS7+zMSFcakc/LbXma90CSmpQm4CKaojeErmVUEOebMq6YiveCaE2rslRD+j XEmYZ8iMdFYZt884LMyhL0MHGCstD+Y= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KVtvmyag; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653864; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QAskb+5yNIJsaldLr50aWhyLhYS5r5QAog9O7C8QAr0=; b=obcSZ0YW/nPHNNuqLYajDzaYtFIZQOlx9xCIhlASZXqGDBOt9enUJxypAhGJJpuhU6hmYx n7hLcFD1DpUbkT7Vl1yvLOUKxiUg6i0AYar0Ja1oWgeUrvhuairNFXaVEq7RM1qb+00MXs 2/982OBeHB/9sHhKtBhZ7sinzZAMO+c= X-Stat-Signature: b3ocojrsmkjinhk4dazggpax5fgix48o X-Rspamd-Queue-Id: 17D6012000A Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KVtvmyag; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1655653863-199894 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a helper to recharge pages of a hash map. We have already known how the hash map is allocated and freed, we can also know how to charge and uncharge the hash map. Firstly, we need to pre charge to the new memcg, if the pre charge successes then we uncharge from the old memcg. Finnaly we do the post charge to the new memcg, in which we will modify the counter in memcgs. Signed-off-by: Yafang Shao --- kernel/bpf/hashtab.c | 74 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 17fb69c0e0dc..fe61976262ee 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -11,6 +11,7 @@ #include #include #include +#include #include "percpu_freelist.h" #include "bpf_lru_list.h" #include "map_in_map.h" @@ -1499,6 +1500,75 @@ static void htab_map_free(struct bpf_map *map) kfree(htab); } +#ifdef CONFIG_MEMCG_KMEM +static bool htab_map_memcg_recharge(struct bpf_map *map) +{ + struct bpf_htab *htab = container_of(map, struct bpf_htab, map); + struct mem_cgroup *old = map->memcg; + int i; + + /* + * Although the bpf map's offline memcg has been reparented, there + * is still reference on it, so it is safe to be accessed. + */ + if (!old) + return false; + + /* Pre charge to the new memcg */ + if (!krecharge(htab, MEMCG_KMEM_PRE_CHARGE)) + return false; + + if (!kvrecharge(htab->buckets, MEMCG_KMEM_PRE_CHARGE)) + goto out_k; + + if (!recharge_percpu(htab->extra_elems, MEMCG_KMEM_PRE_CHARGE)) + goto out_kv; + + for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) { + if (!recharge_percpu(htab->map_locked[i], MEMCG_KMEM_PRE_CHARGE)) + goto out_p; + } + + /* Uncharge from the old memcg. */ + krecharge(htab, MEMCG_KMEM_UNCHARGE); + kvrecharge(htab->buckets, MEMCG_KMEM_UNCHARGE); + recharge_percpu(htab->extra_elems, MEMCG_KMEM_UNCHARGE); + for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) + recharge_percpu(htab->map_locked[i], MEMCG_KMEM_UNCHARGE); + + /* Release the old memcg */ + bpf_map_release_memcg(map); + + /* Post charge to the new memcg */ + krecharge(htab, MEMCG_KMEM_POST_CHARGE); + kvrecharge(htab->buckets, MEMCG_KMEM_POST_CHARGE); + recharge_percpu(htab->extra_elems, MEMCG_KMEM_POST_CHARGE); + for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) + recharge_percpu(htab->map_locked[i], MEMCG_KMEM_POST_CHARGE); + + /* Save the new memcg */ + bpf_map_save_memcg(map); + + return true; + +out_p: + for (; i > 0; i--) + recharge_percpu(htab->map_locked[i], MEMCG_KMEM_CHARGE_ERR); + recharge_percpu(htab->extra_elems, MEMCG_KMEM_CHARGE_ERR); +out_kv: + kvrecharge(htab->buckets, MEMCG_KMEM_CHARGE_ERR); +out_k: + krecharge(htab, MEMCG_KMEM_CHARGE_ERR); + + return false; +} +#else +static bool htab_map_memcg_recharge(struct bpf_map *map) +{ + return true; +} +#endif + static void htab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m) { @@ -2152,6 +2222,7 @@ const struct bpf_map_ops htab_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_release_uref = htab_map_free_timers, .map_lookup_elem = htab_map_lookup_elem, @@ -2172,6 +2243,7 @@ const struct bpf_map_ops htab_lru_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_release_uref = htab_map_free_timers, .map_lookup_elem = htab_lru_map_lookup_elem, @@ -2325,6 +2397,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_lookup_elem = htab_percpu_map_lookup_elem, .map_lookup_and_delete_elem = htab_percpu_map_lookup_and_delete_elem, @@ -2344,6 +2417,7 @@ const struct bpf_map_ops htab_lru_percpu_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_lookup_elem = htab_lru_percpu_map_lookup_elem, .map_lookup_and_delete_elem = htab_lru_percpu_map_lookup_and_delete_elem,