From patchwork Thu Mar 14 08:39:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "zhaoyang.huang" X-Patchwork-Id: 13592226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CEC5C54E5D for ; Thu, 14 Mar 2024 08:39:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7B0B80088; Thu, 14 Mar 2024 04:39:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C030080073; Thu, 14 Mar 2024 04:39:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACABE80088; Thu, 14 Mar 2024 04:39:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9C5D380073 for ; Thu, 14 Mar 2024 04:39:48 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5EFD0813C8 for ; Thu, 14 Mar 2024 08:39:48 +0000 (UTC) X-FDA: 81894996456.26.A09C5A5 Received: from SHSQR01.spreadtrum.com (unknown [222.66.158.135]) by imf04.hostedemail.com (Postfix) with ESMTP id 5917840012 for ; Thu, 14 Mar 2024 08:39:44 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf04.hostedemail.com: domain of zhaoyang.huang@unisoc.com designates 222.66.158.135 as permitted sender) smtp.mailfrom=zhaoyang.huang@unisoc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710405586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=gUEoVcnfJS6BVXWMMbtY0IFiCLbL9OKVJbiUoWcPqs8=; b=d9SZ86BJEOkQwHedU6xHGWhcJi0GuyxknZV3+vCTBeUka6SbATrU74FQ2hgEpsldiGXMSb YESV4GWsmv8xOCn1HCCBqLKCoK6K4HWeEaGDhP9jBGf+xdj8aRDI79wgZu6v5Zgk0n2sJE acZ4aTEdkOGAHo5Yusn+RdntHvj9Q28= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf04.hostedemail.com: domain of zhaoyang.huang@unisoc.com designates 222.66.158.135 as permitted sender) smtp.mailfrom=zhaoyang.huang@unisoc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710405586; a=rsa-sha256; cv=none; b=UOx+nK6I9zvY6vGTZbEvVEOX9uGIzOyYvUt6zqaNPHDM6V49Kd1nAh30dFCA+T+68ZoOy1 BbRntr1cpNBJ872UlBdF1hVTW+E/78zqViosAH9TuIuSk/OmpA3RHiyqRZ8fLPSJ8PpZKV YXY7bV9oQy6RP+zKRTKhLOACuTXvdBo= Received: from dlp.unisoc.com ([10.29.3.86]) by SHSQR01.spreadtrum.com with ESMTP id 42E8dQ2t042813; Thu, 14 Mar 2024 16:39:26 +0800 (+08) (envelope-from zhaoyang.huang@unisoc.com) Received: from SHDLP.spreadtrum.com (bjmbx01.spreadtrum.com [10.0.64.7]) by dlp.unisoc.com (SkyGuard) with ESMTPS id 4TwLPF1QZbz2L2Bvr; Thu, 14 Mar 2024 16:38:09 +0800 (CST) Received: from bj03382pcu01.spreadtrum.com (10.0.73.40) by BJMBX01.spreadtrum.com (10.0.64.7) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Thu, 14 Mar 2024 16:39:24 +0800 From: "zhaoyang.huang" To: Andrew Morton , , , Zhaoyang Huang , Subject: [PATCH] mm: fix a race scenario in folio_isolate_lru Date: Thu, 14 Mar 2024 16:39:21 +0800 Message-ID: <20240314083921.1146937-1-zhaoyang.huang@unisoc.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Originating-IP: [10.0.73.40] X-ClientProxiedBy: SHCAS01.spreadtrum.com (10.0.1.201) To BJMBX01.spreadtrum.com (10.0.64.7) X-MAIL: SHSQR01.spreadtrum.com 42E8dQ2t042813 X-Rspam-User: X-Stat-Signature: ncx9p999dm83j1do7b9g413n4b6wuqck X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5917840012 X-HE-Tag: 1710405584-207702 X-HE-Meta: U2FsdGVkX1/aLIFpqALseNn8TmvEVia3a25+5nivEgfpha4+ITUi2bXrIuGUYCZ1fx6MXHq+bd2259Cr7wBSeDk/JaDbkPZ1kgoL4n/Y8xIzDG4bbPit9nOhtl1T47f/IGYEbl/GBbWZI0Q7hSOqshN9BMPN3nteXW4utHZmpDX6hvCEZSJphwriO1oJLvMaU9dVd0yiP2HDDml6e1Iz65UV/LFfJzcPNZk/2RDbmFQoGNBY627eNZ/y2PsbfzEjPp3Fibb0cAvwyc2xAgiSQyP5vY/kgk6rdax3nCVLjry2SSdaCLNLCCv9SeAtVs2f8qqyfEMaRasBP9lA3TEvvjbk8WIz0AJiTBshXW84KpHu/YsSepMnh07MaE9s76xamFv0VmSDFFcV6Y/RHttp8S4bNMCe6AnDCpmpc4v+rY6WepIgARw+Lo0szmeOXTIfFhQ3QAHJz5XManCNmML4TawYgOLfr0ffyKt4QnoorORiAHI7zzEh0NUMdNlBNbPwdh2uEm+o3P5EpX8UM4EyKavsmD5vmovfly5XV86wvD77ZYKf1gPOy7UP8+6oFR80lY/E/wc0P4nv4m2SXpr546pKthU2Mc66G+Ku7JycOXb9yl4MR66WxXfF9lLb9+xE8CxyDzGVmUxfS6IpNONxwJ1zLLq+fiJr5h9jIVqdLSsS8UEs7tqSQX5kXvS2E8iE4CPX8cHlnvNQrXwjR0BdMRdOGOWJZbYryirrFJUbDp82w6hko5KmVqVCGuxZyT0QHCi9I5SsyrasBuVwXEY49nwSTn8oKDIsvm45vd4mPj0A/T7yp7DgImaqZV1n0/g3OjnYqMF2TN6kpt9lmtXwCLhEunY8DyqZ5xy7I/HJjvy9eY1spoYERWs99miUFQQbC3s9M6Qx4JnU7uO5Oa5Gyg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zhaoyang Huang Panic[1] reported which is caused by lruvec->list break. Fix the race between folio_isolate_lru and release_pages. race condition: release_pages could meet a non-refered folio which escaped from being deleted from LRU but add to another list_head #0 folio_isolate_lru #1 release_pages if (folio_test_clear_lru()) if (folio_put_testzero()) if (!folio_test_lru()) folio_get(folio) list_add(&folio->lru,) list_del(&folio->lru,) fix action 1: have folio_get prior to folio_test_clear_lru is not enough as there could be concurrent folio_put(filemap_remove_folios) to make release_pages pass refcnt check and failed in delete from LRU #0 folio_isolate_lru #1 release_pages folio_get(folio) if (folio_test_clear_lru()) if (folio_put_testzero()) if (!folio_test_lru()) list_add(&folio->lru,) list_del(&folio->lru,) fix action 2: folio_test_clear_lru should be considered as part of critical section of lruvec which require be within lruvec->lock. #0 folio_isolate_lru #1 release_pages spin_lock(lruvec->lock) folio_get(folio) if (folio_test_clear_lru()) list_del(&folio->lru,) spin_unlock(lruvec->lock) spin_lock(lruvec->lock) if (folio_put_testzero()) if (!folio_test_lru()) list_add(&folio->lru,) [1] [ 37.562326] pc : __list_del_entry_valid_or_report+0xec/0xf0 [ 37.562344] lr : __list_del_entry_valid_or_report+0xec/0xf0 [ 37.562351] sp : ffffffc085953990 [ 37.562355] x29: ffffffc0859539d0 x28: ffffffc082144000 x27: 000000000000000f [ 37.562367] x26: 000000000000000d x25: 000000000000000d x24: 00000000ffffffff [ 37.562377] x23: ffffffc085953a08 x22: ffffff8080389000 x21: ffffff8080389000 [ 37.562388] x20: fffffffe05c54180 x19: ffffffc085953b30 x18: ffffffc085989098 [ 37.562399] x17: 20747562202c3838 x16: ffffffffffffffff x15: 0000000000000004 [ 37.562409] x14: ffffff8176980000 x13: 0000000000007fff x12: 0000000000000003 [ 37.562420] x11: 00000000ffff7fff x10: ffffffc0820f51c4 x9 : 53b71233d5d50e00 [ 37.562431] x8 : 53b71233d5d50e00 x7 : ffffffc081161ff0 x6 : 0000000000000000 [ 37.562441] x5 : 0000000000000001 x4 : 0000000000000001 x3 : 0000000000000000 [ 37.562451] x2 : ffffff817f2c4178 x1 : ffffff817f2b71c8 x0 : 000000000000006d [ 37.562461] Call trace: [ 37.562465] __list_del_entry_valid_or_report+0xec/0xf0 [ 37.562472] release_pages+0x410/0x4c0 [ 37.562482] __folio_batch_release+0x34/0x4c [ 37.562490] truncate_inode_pages_range+0x368/0x63c [ 37.562497] truncate_inode_pages+0x14/0x24 [ 37.562504] blkdev_flush_mapping+0x60/0x120 [ 37.562513] blkdev_put+0x114/0x298 [ 37.562520] blkdev_release+0x28/0x40 [ 37.562526] __fput+0xf8/0x2a8 [ 37.562533] ____fput+0x10/0x20 [ 37.562539] task_work_run+0xc4/0xec [ 37.562546] do_exit+0x32c/0xa3c [ 37.562554] do_group_exit+0x98/0x9c [ 37.562561] __arm64_sys_exit_group+0x18/0x1c [ 37.562568] invoke_syscall+0x58/0x114 [ 37.562575] el0_svc_common+0xac/0xe0 [ 37.562582] do_el0_svc+0x1c/0x28 [ 37.562588] el0_svc+0x50/0xe4 [ 37.562593] el0t_64_sync_handler+0x68/0xbc [ 37.562599] el0t_64_sync+0x1a8/0x1ac Signed-off-by: Zhaoyang Huang --- mm/swap.c | 25 +++++++++++++++++-------- mm/vmscan.c | 25 +++++++++++++++++++------ 2 files changed, 36 insertions(+), 14 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..287cf7379927 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -968,6 +968,7 @@ void release_pages(release_pages_arg arg, int nr) for (i = 0; i < nr; i++) { struct folio *folio; + struct lruvec *prev_lruvec; /* Turn any of the argument types into a folio */ folio = page_folio(encoded_page_ptr(encoded[i])); @@ -996,9 +997,24 @@ void release_pages(release_pages_arg arg, int nr) free_zone_device_page(&folio->page); continue; } + /* + * lruvec->lock need to be prior to folio_put_testzero to + * prevent race with folio_isolate_lru + */ + prev_lruvec = lruvec; + lruvec = folio_lruvec_relock_irqsave(folio, lruvec, + &flags); + + if (prev_lruvec != lruvec) + lock_batch = 0; - if (!folio_put_testzero(folio)) + if (!folio_put_testzero(folio)) { + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; + } continue; + } if (folio_test_large(folio)) { if (lruvec) { @@ -1010,13 +1026,6 @@ void release_pages(release_pages_arg arg, int nr) } if (folio_test_lru(folio)) { - struct lruvec *prev_lruvec = lruvec; - - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - if (prev_lruvec != lruvec) - lock_batch = 0; - lruvec_del_folio(lruvec, folio); __folio_clear_lru_flags(folio); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 4255619a1a31..13a4a716c67a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1721,18 +1721,31 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan, bool folio_isolate_lru(struct folio *folio) { bool ret = false; + struct lruvec *lruvec; VM_BUG_ON_FOLIO(!folio_ref_count(folio), folio); - if (folio_test_clear_lru(folio)) { - struct lruvec *lruvec; + /* + * The folio_get needs to be prior to clear lru for list integrity. + * Otherwise: + * #0 folio_isolate_lru #1 release_pages + * if (folio_test_clear_lru()) + * if (folio_put_testzero()) + * if (!folio_test_lru()) + * + * folio_get(folio) + * list_add(&folio->lru,) + * list_del(&folio->lru,) + */ + lruvec = folio_lruvec_lock_irq(folio); + folio_get(folio); - folio_get(folio); - lruvec = folio_lruvec_lock_irq(folio); + if (folio_test_clear_lru(folio)) { lruvec_del_folio(lruvec, folio); - unlock_page_lruvec_irq(lruvec); ret = true; - } + } else + folio_put(folio); + unlock_page_lruvec_irq(lruvec); return ret; }