From patchwork Mon Jun 24 17:53:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13709940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1216DC2D0D1 for ; Mon, 24 Jun 2024 17:53:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6351C6B0295; Mon, 24 Jun 2024 13:53:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E4F86B0297; Mon, 24 Jun 2024 13:53:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45E8D6B029B; Mon, 24 Jun 2024 13:53:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 243506B0295 for ; Mon, 24 Jun 2024 13:53:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D393941204 for ; Mon, 24 Jun 2024 17:53:46 +0000 (UTC) X-FDA: 82266530052.06.16F39F9 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf10.hostedemail.com (Postfix) with ESMTP id 08FD3C0002 for ; Mon, 24 Jun 2024 17:53:44 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dU32gZ5D; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719251609; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OVbj5lGZ2MXAdaurvv+B/YwNudp43tdHqr1WyLtvH38=; b=LZLC4cW33N8decxd2x11fG97u2aKz5e56Wn2r8DZlKgyZ5M6PoKG2T6jMuAvVV4sAKWlXc fy+gf1sAgOFEy8xiWMkR1DI+DDr89xxXI+wZykZp2euVxkONEmn36UrznuBi/AaO0SMeTQ g7lkebYaXsGvMXqyTFkOhAh5wRk++Vw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719251609; a=rsa-sha256; cv=none; b=pb/paGfAhuI6oAJGwVJxrW+yyS546ljwp0s86p3R8I52ggClcv6MxL7x4w1q3YlFW87Lpm r68xAAgm7DpCs8CzQ/Fottlr5kz50vCH2kug2gnqzZuNB35JSCUhchcnQVmOfK9NVEyFtR xevcinPLuXq6QNno2Sp+VvrqJxZOzTc= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dU32gZ5D; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-7066a3229f4so1476733b3a.2 for ; Mon, 24 Jun 2024 10:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719251623; x=1719856423; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=OVbj5lGZ2MXAdaurvv+B/YwNudp43tdHqr1WyLtvH38=; b=dU32gZ5DteR0cvx4Ow9x0cCn8wLRLMpIi88Y9us7Z6oDZVI0+vHWKAl/229CXBR1tl 3E+bmtstB5Z6gik9wFB+Ck8NQLRyBBhgRVJ5jYN8nNj1oRWEes1Dzq6zgORsrmCRNeZI I9K87GoddXjUgC1R6RmQOKv5+QluIViD/SlFJCUGhIEyDg93xLutvXNMkZle9gT8MUeQ b53HQeVGOIg1rfXKfurOGm3aYgrW6u2dvp1QL0i2S6BEWiQOjr4m2MGlIOzByS+7lWa7 6ud1kmMplO+JaGNlhHwosfMrOipSOk467syuRNRLmpgm0YUT3ZNfPjgcuq041G7rDmw+ fw8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719251623; x=1719856423; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=OVbj5lGZ2MXAdaurvv+B/YwNudp43tdHqr1WyLtvH38=; b=fnJXimo7rl83iUnMtbayzFRgU5G2Rw3i8hgCursVM3W/ewlSF5fsrRz/JKW1RA0fbC BHbwe7BOiXiRDnkdp0S5n4Ipg5u2Ah/kd6xXIYqNG0BpfEAe9glesUfD45tOw3TFd2pE clXn8V9hBn2YkE/Cpobzf6EvQVPjU8bzTU51uptIo3DIul+ib2ZdhTL4Ex4JwMQEY2Y8 GnF2drpswRtpozvb0A6GGEWK0qoknYCU6+q6llHMCB9aDlF1QZZa4O9z1PcZcDTpJEgf ie6WHQB+5KbaC8JF3NxLzTpa6BxI2sP1pk2pFg/AhiU8m/JxxOkVJ57VVnKQl9b4lFX0 GoJg== X-Gm-Message-State: AOJu0Yz2ZDkd5ko+rQgTnYvGzQEEpMpUW4x8y+IPVK/oRdPtyqz0DFAJ FyWssG7rPpvR+iCZRHxFFb3L6IqmOr5cONouS4zi9yp2mPt1pbQj2oqlU5vuwb8= X-Google-Smtp-Source: AGHT+IEJZZQ7bFfrIJhEvsZjquIMc/eeKkcoozXNlkD7oYTiQEBK5MC9t7olbcYhZATA7FNPVYQh7A== X-Received: by 2002:a05:6a00:8d3:b0:706:6bdc:4dea with SMTP id d2e1a72fcca58-7067459c699mr6968912b3a.4.1719251623205; Mon, 24 Jun 2024 10:53:43 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70662ee117asm5008049b3a.211.2024.06.24.10.53.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 24 Jun 2024 10:53:42 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Nhat Pham , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Chris Li , Yosry Ahmed , "Huang, Ying" , Kairui Song Subject: [PATCH 5/7] mm/list_lru: simplify reparenting and initial allocation Date: Tue, 25 Jun 2024 01:53:11 +0800 Message-ID: <20240624175313.47329-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240624175313.47329-1-ryncsn@gmail.com> References: <20240624175313.47329-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 08FD3C0002 X-Stat-Signature: 67jf3tafzikoguu56zwk749s5nod6e6t X-Rspam-User: X-HE-Tag: 1719251624-56559 X-HE-Meta: U2FsdGVkX19tIWNTf8w01lZ3qiGDgYHSxoxHXO6tVk7zc0wFqZAlYjB/BaqweF/HJBfjvIACD2oxwAuC+XGwtFOmREtnh9t0QiXAVmbblqJ4wvOVJ4+Y+RinoOmIixummEgPFSygHZ8pla1nlRa/q6slhIWRdfJj+4q+6AyIwhlrlwZVar9NdBqQPTeXWFW26IQB9FWdU15u7Cli2OwPCudE10/T/TRmqFISJvodWXkGFeSv7Qdnvj42+y8R3svTQ0ZjK9UrG4+fe1m5HCyqaUUEGmvWYrhmK5z4Wdg30AxWfUMQbMo2JkK5+a0BlWvPCDeJOR30A/DFDPDHRkDGov+NPJsoBpNMW594ZFZKbkQ7QL2vR70H5AMCa4InG8dnQgihEdbZj5nSFGfmD+1P5TiZ6AQsOxEGA4xQz2LU66moR+Y9vJ6nYA/pNj9I8zKFCyHAnOppzSfyMqmg1dGptIhbeC5ey7Iu9jU9gq4MkdsD1m83sye6p7W4dtBSOR0O3M1CI0tCYpCgEH2LTxrsmg5/ii1sg1DXZl2VwldMMTpcUeMbkqiYqvvHa/78INacSvbYwt9nib632bjtCD8IGdJU/35c4ypMa6dLFHpYYqvBpT6a8x8G6902Khvh3tppQJgBTA0FWnugjBh+FZZ9TOJ+sxmWejhY2bhDawH1skdIhqV0CrjbgYimAdcyWmXOR3aOvjx0Iu+RG1NuAACZyYIfLKMZbxB3XWW7oL/uxcAcrlf+FDdw+ne8agvrZT9z+cBWGNHuxJIleCRcb61kMXgqZrWeioYHGZSSv8OIKgAZGyu3vxRLz4/vpJrb3B9372meMjXfUfVlqz7lUdO1iTlSZJYij9jJ646lzWkyazeWHhNB+Rz66bmHaBtDIAaVTntP078SuJCWQ5FLdEPzc+TUGnN8SieCutNNfo7X2YCfGpQaLo6l8+L9Or39z3poq+iSjcc4hLzpTNQk34U z5SIcM6Q DVYQBT6MDD5vCKNLxiEGzAbqX8oqOsJ6RJ1KnG0ujYr5hWeshoUq7GkggVe8TnTsZyuEWlPKVgCodQxJu8sAU7ZuluBopfNskQjSkgaJ7UDnutdXbmtTWuz7oOBeBGNdT5EMzwmGWgegVoKp23X20pEL2G1nIyhrW93qnxWSo9J8gU+iuKRKz0Ui133lfRkOVnnHBnbkJrncutnhd41uojwzSQuABXmg+VBk01FIhoNp7cSE5sBO1yCr233JBhv1Q3TT8PlH/Ssy6q+LHENfj904PAIADa8VVrrOQDJlXLjatc5Sxbe6VQlmQNtY7ON6u2v2EoUVf9ncIRy/Yjo8d6XwkAFUBzUXc1wuy3KGIr5aINloKE82Po9WIOuG3XMsCMPJn2V48tCaDI5D1m3DI3bYssyXrmScOGTjA0e/uN289DWJQJcGFj3X4Q0b9xTBvXmdTZ5FvnNLtaZZ7VOn2reNnkMoVXv4Qv4oTRZ0/l2TKpQjxHDlVeHzIUbDeAyL/F9tb/oxGOqMPPdnDkbqGzaNYp30AAvlc2HMqDzs525NW9H+1fFAHa77Bv2Pd4b6j85RU3nZKRZgRZ00= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, there is a lot of code for detecting reparent racing using kmemcg_id as the synchronization flag. And an intermediate table is required to record and compare the kmemcg_id. We can simplify this by just checking the cgroup css status, skip if cgroup is being offlined. On the reparenting side, ensure no more allocation is on going and no further allocation will occur by using the XArray lock as barrier. Combined with a O(n^2) top-down walk for the allocation, we get rid of the intermediate table allocation completely. Despite being O(n^2), it should be actually faster because it's not practical to have a very deep cgroup level. This also avoided changing kmemcg_id before reparenting, making cgroups have a stable index for list_lru_memcg. After this change it's possible that a dying cgroup will see a NULL value in XArray corresponding to the kmemcg_id, because the kmemcg_id will point to an empty slot. In such case, just fallback to use its parent. As a result the code is simpler, following test also showed a performance gain (6 test runs): mkdir /tmp/test-fs modprobe brd rd_nr=1 rd_size=16777216 mkfs.xfs /dev/ram0 mount -t xfs /dev/ram0 /tmp/test-fs worker() { echo TEST-CONTENT > "/tmp/test-fs/$1" } do_test() { for i in $(seq 1 2048); do (exec sh -c 'echo "$PPID"') > "/sys/fs/cgroup/benchmark/$i/cgroup.procs" worker "$i" & done; wait echo 1 > /proc/sys/vm/drop_caches } mkdir -p /sys/fs/cgroup/benchmark echo +memory > /sys/fs/cgroup/benchmark/cgroup.subtree_control for i in $(seq 1 2048); do rmdir "/sys/fs/cgroup/benchmark/$i" &>/dev/null mkdir -p "/sys/fs/cgroup/benchmark/$i" done time do_test Before: real 0m5.932s user 0m2.366s sys 0m5.583s real 0m5.939s user 0m2.347s sys 0m5.597s real 0m6.149s user 0m2.398s sys 0m5.761s real 0m5.945s user 0m2.403s sys 0m5.547s real 0m5.925s user 0m2.293s sys 0m5.651s real 0m6.017s user 0m2.367s sys 0m5.686s After: real 0m5.712s user 0m2.343s sys 0m5.307s real 0m5.885s user 0m2.326s sys 0m5.518s real 0m5.694s user 0m2.347s sys 0m5.264s real 0m5.865s user 0m2.300s sys 0m5.545s real 0m5.748s user 0m2.273s sys 0m5.424s real 0m5.756s user 0m2.318s sys 0m5.398s Signed-off-by: Kairui Song --- mm/list_lru.c | 182 ++++++++++++++++++++++---------------------------- mm/zswap.c | 7 +- 2 files changed, 81 insertions(+), 108 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 4c619857e916..ac8aec8451dd 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -59,6 +59,20 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) } return &lru->node[nid].lru; } + +static inline struct list_lru_one * +list_lru_from_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg) +{ + struct list_lru_one *l; +again: + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + if (likely(l)) + return l; + + memcg = parent_mem_cgroup(memcg); + WARN_ON(!css_is_dying(&memcg->css)); + goto again; +} #else static void list_lru_register(struct list_lru *lru) { @@ -83,6 +97,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) { return &lru->node[nid].lru; } + +static inline struct list_lru_one * +list_lru_from_memcg(struct list_lru *lru, int nid, int idx) +{ + return &lru->node[nid].lru; +} #endif /* CONFIG_MEMCG_KMEM */ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, @@ -93,7 +113,7 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + l = list_lru_from_memcg(lru, nid, memcg); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) @@ -124,7 +144,7 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + l = list_lru_from_memcg(lru, nid, memcg); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -339,20 +359,6 @@ static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) return mlru; } -static void memcg_list_lru_free(struct list_lru *lru, int src_idx) -{ - struct list_lru_memcg *mlru = xa_erase_irq(&lru->xa, src_idx); - - /* - * The __list_lru_walk_one() can walk the list of this node. - * We need kvfree_rcu() here. And the walking of the list - * is under lru->node[nid]->lock, which can serve as a RCU - * read-side critical section. - */ - if (mlru) - kvfree_rcu(mlru, rcu); -} - static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) { if (memcg_aware) @@ -377,22 +383,18 @@ static void memcg_destroy_list_lru(struct list_lru *lru) } static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, - int src_idx, struct mem_cgroup *dst_memcg) + struct list_lru_one *src, + struct mem_cgroup *dst_memcg) { struct list_lru_node *nlru = &lru->node[nid]; - int dst_idx = dst_memcg->kmemcg_id; - struct list_lru_one *src, *dst; + struct list_lru_one *dst; /* * Since list_lru_{add,del} may be called under an IRQ-safe lock, * we have to use IRQ-safe primitives here to avoid deadlock. */ spin_lock_irq(&nlru->lock); - - src = list_lru_from_memcg_idx(lru, nid, src_idx); - if (!src) - goto out; - dst = list_lru_from_memcg_idx(lru, nid, dst_idx); + dst = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(dst_memcg)); list_splice_init(&src->list, &dst->list); @@ -401,46 +403,45 @@ static void memcg_reparent_list_lru_node(struct list_lru *lru, int nid, set_shrinker_bit(dst_memcg, nid, lru_shrinker_id(lru)); src->nr_items = 0; } -out: spin_unlock_irq(&nlru->lock); } void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent) { - struct cgroup_subsys_state *css; struct list_lru *lru; - int src_idx = memcg->kmemcg_id, i; - - /* - * Change kmemcg_id of this cgroup and all its descendants to the - * parent's id, and then move all entries from this cgroup's list_lrus - * to ones of the parent. - */ - rcu_read_lock(); - css_for_each_descendant_pre(css, &memcg->css) { - struct mem_cgroup *child; - - child = mem_cgroup_from_css(css); - WRITE_ONCE(child->kmemcg_id, parent->kmemcg_id); - } - rcu_read_unlock(); + int i; - /* - * With kmemcg_id set to parent, holding the lru lock below can - * prevent list_lru_{add,del,isolate} from touching the lru, safe - * to reparent. - */ mutex_lock(&list_lrus_mutex); list_for_each_entry(lru, &memcg_list_lrus, list) { + struct list_lru_memcg *mlru; + XA_STATE(xas, &lru->xa, memcg->kmemcg_id); + + /* + * Lock the Xarray to ensure no on going allocation and + * further allocation will see css_is_dying(). + */ + xas_lock_irq(&xas); + mlru = xas_load(&xas); + if (mlru) + xas_store(&xas, NULL); + xas_unlock_irq(&xas); + if (!mlru) + continue; + + /* + * With Xarray value set to NULL, holding the lru lock below + * prevents list_lru_{add,del,isolate} from touching the lru, + * safe to reparent. + */ for_each_node(i) - memcg_reparent_list_lru_node(lru, i, src_idx, parent); + memcg_reparent_list_lru_node(lru, i, &mlru->node[i], parent); /* * Here all list_lrus corresponding to the cgroup are guaranteed * to remain empty, we can safely free this lru, any further * memcg_list_lru_alloc() call will simply bail out. */ - memcg_list_lru_free(lru, src_idx); + kvfree_rcu(mlru, rcu); } mutex_unlock(&list_lrus_mutex); } @@ -456,78 +457,51 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg, int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp) { - int i; unsigned long flags; - struct list_lru_memcg_table { - struct list_lru_memcg *mlru; - struct mem_cgroup *memcg; - } *table; + struct list_lru_memcg *mlru; + struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) return 0; gfp &= GFP_RECLAIM_MASK; - table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); - if (!table) - return -ENOMEM; - /* * Because the list_lru can be reparented to the parent cgroup's * list_lru, we should make sure that this cgroup and all its * ancestors have allocated list_lru_memcg. */ - for (i = 0; memcg; memcg = parent_mem_cgroup(memcg), i++) { - if (memcg_list_lru_allocated(memcg, lru)) - break; - - table[i].memcg = memcg; - table[i].mlru = memcg_init_list_lru_one(gfp); - if (!table[i].mlru) { - while (i--) - kfree(table[i].mlru); - kfree(table); - return -ENOMEM; + do { + /* + * Keep finding the farest parent that wasn't populated + * until found memcg itself. + */ + pos = memcg; + parent = parent_mem_cgroup(pos); + while (parent && !memcg_list_lru_allocated(parent, lru)) { + pos = parent; + parent = parent_mem_cgroup(pos); } - } - xas_lock_irqsave(&xas, flags); - while (i--) { - int index = READ_ONCE(table[i].memcg->kmemcg_id); - struct list_lru_memcg *mlru = table[i].mlru; - - xas_set(&xas, index); -retry: - if (unlikely(index < 0 || xas_error(&xas) || xas_load(&xas))) { - kfree(mlru); - } else { - xas_store(&xas, mlru); - if (xas_error(&xas) == -ENOMEM) { + mlru = memcg_init_list_lru_one(gfp); + do { + bool alloced = false; + + xas_set(&xas, pos->kmemcg_id); + xas_lock_irqsave(&xas, flags); + if (!css_is_dying(&pos->css) && !xas_load(&xas)) { + xas_store(&xas, mlru); + alloced = true; + } + if (!alloced || xas_error(&xas)) { xas_unlock_irqrestore(&xas, flags); - if (xas_nomem(&xas, gfp)) - xas_set_err(&xas, 0); - xas_lock_irqsave(&xas, flags); - /* - * The xas lock has been released, this memcg - * can be reparented before us. So reload - * memcg id. More details see the comments - * in memcg_reparent_list_lrus(). - */ - index = READ_ONCE(table[i].memcg->kmemcg_id); - if (index < 0) - xas_set_err(&xas, 0); - else if (!xas_error(&xas) && index != xas.xa_index) - xas_set(&xas, index); - goto retry; + kfree(mlru); + goto out; } - } - } - /* xas_nomem() is used to free memory instead of memory allocation. */ - if (xas.xa_alloc) - xas_nomem(&xas, gfp); - xas_unlock_irqrestore(&xas, flags); - kfree(table); - + xas_unlock_irqrestore(&xas, flags); + } while (xas_nomem(&xas, gfp)); + } while (pos != memcg); +out: return xas_error(&xas); } #else diff --git a/mm/zswap.c b/mm/zswap.c index a50e2986cd2f..c6e2256347ff 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -718,12 +718,11 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) /* * Note that it is safe to use rcu_read_lock() here, even in the face of - * concurrent memcg offlining. Thanks to the memcg->kmemcg_id indirection - * used in list_lru lookup, only two scenarios are possible: + * concurrent memcg offlining: * - * 1. list_lru_add() is called before memcg->kmemcg_id is updated. The + * 1. list_lru_add() is called before list_lru_memcg is erased. The * new entry will be reparented to memcg's parent's list_lru. - * 2. list_lru_add() is called after memcg->kmemcg_id is updated. The + * 2. list_lru_add() is called after list_lru_memcg is erased. The * new entry will be added directly to memcg's parent's list_lru. * * Similar reasoning holds for list_lru_del().