From patchwork Tue Apr 15 02:45:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 14051380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39DA6C369B2 for ; Tue, 15 Apr 2025 02:48:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B901280028; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94017280024; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79424280028; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 59254280024 for ; Mon, 14 Apr 2025 22:48:18 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2BEB3BAEFF for ; Tue, 15 Apr 2025 02:48:19 +0000 (UTC) X-FDA: 83334744318.01.7422FA8 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf03.hostedemail.com (Postfix) with ESMTP id 38CE320009 for ; Tue, 15 Apr 2025 02:48:17 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bQs+DKaT; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744685297; a=rsa-sha256; cv=none; b=NRW816o3/6bwIZFjZYSwccifmSvhDn7TZANlgCiiBNa+g0Aw8RpFJz2ZWZ7V9Nzn86EfM5 dUuFLtH/R+cBFSwMiHr1AWsTlpN9lEWmbqWfxilVklHylXnBBuEH64iODo8zxRByllbfAr chPcbTolPCf4TUHDNASYoefgXpN90yk= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bQs+DKaT; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744685297; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=chHLX5G0loxHSohYhxyRHh7WVRRPsaZ78MDusal3hIs=; b=VUM/KRQsulnoHciah+vneKwIrq1bnWYl6rRsENLEaRv8Qme8Vf8cqZzdB8DuGdFJdVt2Mw LYOoPfBLC5t2Z9BnS5GYUu8D4i0QYi8rDN7miRzSa3/z/8Y2BC/G/q/w5umnxZEo7LYEFO QlNDdCyBVv4q3Z3zL7vGabpWA7Wtals= Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-af590aea813so5551302a12.0 for ; Mon, 14 Apr 2025 19:48:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1744685296; x=1745290096; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=chHLX5G0loxHSohYhxyRHh7WVRRPsaZ78MDusal3hIs=; b=bQs+DKaTVdAZJvwf7ECYvjdnVlIzmsF23PHff10sZAFSN9+pCYNqNnkM/4QLLpG/3h B+N52oLTVtxleHypr63dwEkbGaVUy5pykPt8RBtslnlwZ2/YkREI4U8VmlI+pMcg9vwK 7obr35cfiHSXmYVoMr7GtEIeBnsmkwyYg/qHX5dICDeMYYmb97Id4irOHVATM1Y/+pyM nAgFdELrQkfYguMw5BarFZ1Yq197FYunt46g1cDP7gCMeuu/Psa+R0z4fweCcUuCD37J 1Vm3TymO851Bx1UxigjWUQ4inqglIGAuUJiwDG8QbEZzlFI0DuPICiQHUKFdE1/tEfog rsGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744685296; x=1745290096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=chHLX5G0loxHSohYhxyRHh7WVRRPsaZ78MDusal3hIs=; b=lvzoQ4Q7bTUZYorI5Khc9/AeY+kXvYgk5dNfCPVR9aszghtnKtGt6zx2pyVxDeLhHp 6vwDgKOlgHm/77HtqHogZ1Xkk/059D2tl537lKlLlvf4wd9x4i8c8+05l5h1OdbXoy74 fivI05CxcflsaQn6f7+UfEzRKAc5vwWyQ/S7eI8rGWNaYpAy34kIPwbWPyY3HvI6xqC2 FfGOi8V/xEeXNXy8c0rJkIJxQjONyTp4a7hiRpGyGUihEzuK0m1pejthxBpPmWW8qBuj Zfz2fR4a9LYIVRg18bIk8NWruq739U5EraSUT/Md9MCYfBNbmqgsogthcDBb5d+MNncn z/uA== X-Forwarded-Encrypted: i=1; AJvYcCWd/yMuw1y8RnQ5KoEdjZdP2MVg/ux6XKfEsBhobuoIdE3qZjXQX1ncdK9We4rwxZNIyG1+9SyM1g==@kvack.org X-Gm-Message-State: AOJu0YzYVSLu21vWHkFJT4NY08TdtZcDRh4VD0JreulEpnjgRLUM/fcz Kq8sZ1wIMLyh9T5ToVZ3lDyCF5m0fnoiaufw1okZt+M7DCe9C4zqK/ZwRwr3uUlO4+jVJIIZ0iP YSPqqYA== X-Gm-Gg: ASbGnctcDdqJ+g2yTO2fvodyY2b9e7MgZo3M972d0EevD6Yp+bl6CD87Q/Vbu37y9UF Ohz3/yIp6pGuFsxT+oiPUKIXv5Pm1z7PNnB06TJXdUvKXgU5wEJ5VlUp6wnxdc7E+cvzFBQDl82 fOTouQg4X1KlhFg5Vu3jhvPp5YjV7YOUyzc7/7KfbIShYEeuiaZzRLoz53orA7L+uUD9pwZWqrd NxkR2Kklw2/Fr4F9CoONlc+dqfsrVLZWXzTsQAv6Oq4b98B80X1ReHhPnspRNW6lmLWBlNuffvC i3ILUYMENOOM7f5ddSVXFegY3zHbzSRuMVBKb5vJ7HbxmwpABgUamSYpotw/EeFqpiBLJt0X X-Google-Smtp-Source: AGHT+IHfdYPb0mTyQBEggcIEtJXMdJHS98DVkuG5qNfvvUIjwBW9edwqUb5MNEzj4f0Vk/I+xU8TBA== X-Received: by 2002:a17:903:3203:b0:215:58be:3349 with SMTP id d9443c01a7336-22c24987312mr27942235ad.14.1744685296075; Mon, 14 Apr 2025 19:48:16 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.5]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22ac7ccac49sm106681185ad.217.2025.04.14.19.48.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 14 Apr 2025 19:48:15 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, Muchun Song Subject: [PATCH RFC 26/28] mm: memcontrol: introduce memcg_reparent_ops Date: Tue, 15 Apr 2025 10:45:30 +0800 Message-Id: <20250415024532.26632-27-songmuchun@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250415024532.26632-1-songmuchun@bytedance.com> References: <20250415024532.26632-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 38CE320009 X-Stat-Signature: 7nza646ycoyki9h81pyxy58sn5y573qa X-HE-Tag: 1744685297-12912 X-HE-Meta: U2FsdGVkX1/f1qE/g92qKe4kKTJorn1lRRCvwk++bkL8xs2pJqDP575M8wMQxHSIQtGhDpRAKdKa63Y2ER9cKfAAzgKi1H7KcF8wDkuDf78o4ZHBWUARGyTYTA+5e+n5VcSDZQn23TUwBfbrXJEfJnDecDurIAmHmgE07kkXOLrRLeiQ99uTTGBLHvHatNOMRoyX6jcTGu2sOZzQfT5cKjuTsvBFaCfTeMvu6Gtktakf+S81sSZ+SGsboboRE5ZF9vvjX5wOhcJT9w3gznLcrkRd+Cs+RiJmmQe+cJbJ70DwWkiQvT1vSr5yt/SpK60Seaa+xoQ8x+b/ldKl8w3T6CLAM2Q3fcWWE+cCIP4WzvmyLhhXe+mJfBK0/Nbe8JaRiS7ft1xlxmHXHvtoEsdkyM/SzfyQgbhpScggmDVD98hNH6SaelV/qa/FZCp1g0145pNczEZ4dSiCqVwQ/FKuwNCo0IJNErh8mOLxpNcxvlmHm/hLrI/a6N2T1CvjKBs6fZDhWiB2Ji9zDr5C8LLMzRD55TONFEFv0fPrbe4cJi8Qj54d3HDPJngsioFlBjzTrK5SaOHbFTPdqzBp2JHqpEyK0Qal4eHXk7BYtCDBAJ03xZTk36IaJar/1P9KEfARkYn3E37iPkXadKirfcWdIP8+qKxXlQE+IT/fgkFXPcmgy7YB6bxioI3vWAOT42HSoFruRkKo/N5LxqaEcP8+nIhWoSUdMygRHzLc+xhG0OVWkxC/yQhPAUVV+0ganZ3iPG1XjwxGZ0V1akWQSueC9UozNK/wfUZpsBPv4NnWmEquRyIgtx7Z5fo60m/+bkqP6NRna2VrUcRWnqPRPCklj8W6nzvLJCewTd3WDXgLIDaMt2W5rGzGl1s1msR+LX+M72d3KBYFKMpGjtuHcxZgXVPZh+rzrZe6HIPpvtPcia1HT40M92Mhi7Z1CUuYbh5CWdxbpknuRwlXJIi9JT5 lc6TY0XB YeEAoh8qvK+7Ro1Xf7eLJj0ox7UpYmUKZnf+ynntRahzOdviAuJ1j8XBGc4lB8Wl98Jq+5NEDUFfekegZrk93UkwlrPrRRpsL5UDkBi+Yb/XtZ3ggdNFGqBJbkErleex8bxiEDaV7S9bPJxuY442ziLp3i/3R/qGj3wcDXumx1tT2CnL/hn6uWbm9LOHx1NVCxKFSjobVFugTIlvJxCoyBe5u/Ppd+/E4L7ZalWNRL9hQ52L3ojXrJyVG+vdyvV0D5loDSgwPVXhJrsVgDcsheZKsAA3J63XFAeVW7MR+1IUaija+V6W5xUXBlK5YHnO5rWMtuKIlRY2ULYXJxibiF0eTDXsqsFpox2V7gdu6aDNk9vU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the previous patch, we established a method to ensure the safety of the lruvec lock and the split queue lock during the reparenting of LRU folios. The process involves the following steps: memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) relocate from current memcg to its parent // Move all the pages from the lruvec list to the parent lruvec list. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); In addition to the folio lruvec lock, the deferred split queue lock (specific to THP) also requires a similar approach. Therefore, we abstract the three essential steps from the memcg_reparent_objcgs() function. memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) relocate memcg_reparent_ops->relocate(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Currently, two distinct locks (such as the lruvec lock and the deferred split queue lock) need to utilize this infrastructure. In the subsequent patch, we will employ these APIs to ensure the safety of these locks during the reparenting of LRU folios. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 20 ++++++++++++ mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++-------- 2 files changed, 69 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 27b23e464229..0e450623f8fa 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -311,6 +311,26 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; +struct memcg_reparent_ops { + /* + * Note that interrupt is disabled before calling those callbacks, + * so the interrupt should remain disabled when leaving those callbacks. + */ + void (*lock)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*relocate)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*unlock)(struct mem_cgroup *src, struct mem_cgroup *dst); +}; + +#define DEFINE_MEMCG_REPARENT_OPS(name) \ + const struct memcg_reparent_ops memcg_##name##_reparent_ops = { \ + .lock = name##_reparent_lock, \ + .relocate = name##_reparent_relocate, \ + .unlock = name##_reparent_unlock, \ + } + +#define DECLARE_MEMCG_REPARENT_OPS(name) \ + extern const struct memcg_reparent_ops memcg_##name##_reparent_ops + /* * size of first charge trial. * TODO: maybe necessary to use big numbers in big irons or dynamic based of the diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1f0c6e7b69cc..3fac51179186 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -194,24 +194,60 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } -static void memcg_reparent_objcgs(struct mem_cgroup *memcg) +static void objcg_reparent_lock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_lock(&objcg_lock); +} + +static void objcg_reparent_relocate(struct mem_cgroup *src, struct mem_cgroup *dst) { struct obj_cgroup *objcg, *iter; - struct mem_cgroup *parent = parent_mem_cgroup(memcg); - objcg = rcu_replace_pointer(memcg->objcg, NULL, true); + objcg = rcu_replace_pointer(src->objcg, NULL, true); + /* 1) Ready to reparent active objcg. */ + list_add(&objcg->list, &src->objcg_list); + /* 2) Reparent active objcg and already reparented objcgs to dst. */ + list_for_each_entry(iter, &src->objcg_list, list) + WRITE_ONCE(iter->memcg, dst); + /* 3) Move already reparented objcgs to the dst's list */ + list_splice(&src->objcg_list, &dst->objcg_list); +} - spin_lock_irq(&objcg_lock); +static void objcg_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup *dst) +{ + spin_unlock(&objcg_lock); +} - /* 1) Ready to reparent active objcg. */ - list_add(&objcg->list, &memcg->objcg_list); - /* 2) Reparent active objcg and already reparented objcgs to parent. */ - list_for_each_entry(iter, &memcg->objcg_list, list) - WRITE_ONCE(iter->memcg, parent); - /* 3) Move already reparented objcgs to the parent's list */ - list_splice(&memcg->objcg_list, &parent->objcg_list); - - spin_unlock_irq(&objcg_lock); +static DEFINE_MEMCG_REPARENT_OPS(objcg); + +static const struct memcg_reparent_ops *memcg_reparent_ops[] = { + &memcg_objcg_reparent_ops, +}; + +#define DEFINE_MEMCG_REPARENT_FUNC(phase) \ + static void memcg_reparent_##phase(struct mem_cgroup *src, \ + struct mem_cgroup *dst) \ + { \ + int i; \ + \ + for (i = 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) \ + memcg_reparent_ops[i]->phase(src, dst); \ + } + +DEFINE_MEMCG_REPARENT_FUNC(lock) +DEFINE_MEMCG_REPARENT_FUNC(relocate) +DEFINE_MEMCG_REPARENT_FUNC(unlock) + +static void memcg_reparent_objcgs(struct mem_cgroup *src) +{ + struct mem_cgroup *dst = parent_mem_cgroup(src); + struct obj_cgroup *objcg = rcu_dereference_protected(src->objcg, true); + + local_irq_disable(); + memcg_reparent_lock(src, dst); + memcg_reparent_relocate(src, dst); + memcg_reparent_unlock(src, dst); + local_irq_enable(); percpu_ref_kill(&objcg->refcnt); }