From patchwork Wed Dec 1 14:30:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12650039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EF50C433F5 for ; Wed, 1 Dec 2021 14:39:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24A876B008C; Wed, 1 Dec 2021 09:31:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FAF56B0092; Wed, 1 Dec 2021 09:31:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0259C6B0093; Wed, 1 Dec 2021 09:31:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0120.hostedemail.com [216.40.44.120]) by kanga.kvack.org (Postfix) with ESMTP id E6B5D6B008C for ; Wed, 1 Dec 2021 09:31:08 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id ABB3F180DA185 for ; Wed, 1 Dec 2021 14:30:58 +0000 (UTC) X-FDA: 78869462196.07.6B7780F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf12.hostedemail.com (Postfix) with ESMTP id EF3F210003DD for ; Wed, 1 Dec 2021 14:30:56 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1B1EJZTM012062; Wed, 1 Dec 2021 14:30:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=qySFw+X1GPAl5BW1S+EiG6vLqLNDW3/J6JndGMz+0Ro=; b=QCECo9sp/evHhGCK7gemERuVLInG8z//+QkvfJOMfS/lxvEuvrEUN5eqHLZy36zGkEVt 45lgysF6Up3LG8TKMJ9tdTMhHcHDesGPdn8Wb5n2mHgLN29r66WAjDElFbz1s8q8MmAv w60ksH+J5YQUOo8qCw6xO/16gwYRph8ACEIvgs5P9ygchPgmPtwdxBJ+Pcfca7XIvOtS i8yIr89XKp6q85iYG5d3clt2bMq139iqGA8IW4WkvDzQKloBRSE+10lTLHZ8HoEy0RvC pk9fQQ2jowzW5VTJvGwnqJaeDPjVrgSnXj2rAzDTz6mt2yrmxT+NxPNfI7PbnizoM0F5 jQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3cp7t1h0wv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 01 Dec 2021 14:30:47 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1B1EBov1037539; Wed, 1 Dec 2021 14:30:42 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2171.outbound.protection.outlook.com [104.47.58.171]) by userp3030.oracle.com with ESMTP id 3ck9t1v6xx-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 01 Dec 2021 14:30:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S726Pm1pxxVQ7rfRbuOIQ2oH3uLMkh6chZpZUu8vAnU39fdGl6VOHgl2T4LYgwNyrQ8iJOd5F79qMNu1tqMLNA3rQUKReMVP5nRick9K3xBR9kEMarGJipxDl6JuWOngzpLaOjzzOJt2aLC2Uq5hdUlZjSSqhdzZO3ohW3s2RivMbePd07w1D2DFviBEk4NrbiUxuD1QFVbvClnkRvUVgBKWVW3OMC6J5TDUAH4ZHhr0Bn/I/Gzez4+rNZlMBSY84OEtg07uS3tC+oPreq3twlgj2MQVCXDucwt7ghVSkhlBrji0D5NKSIUl6HSfufH4/M1eREutmwz1BtTCJFbO4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qySFw+X1GPAl5BW1S+EiG6vLqLNDW3/J6JndGMz+0Ro=; b=b9M94+q0g97ef8y71mebiPb+POmpR0uYMJIoQ6kxB8Cr3OAxatI0qPEt0ezS5BlpFfqxAaVWl74YHl0+GdaFvDV4cqRP/Q60p3Sc3PlhiVIA3gqxmK/zyyWYsI2OerSyWZzvaNrYL6vBC+U/nC7ZYB4JvVWnbdMjeZ5V+EzaddRV7NGogmxK0blC1Tu/hEp/Bz7kNpUPtFODzzf9J4aU5pRd7DL9kkAvjuau2Y/qqArC/39/2tCa73eWuiqNpqnpdMaMT2pbdHrdEWxZbiiMUL6F8QYPtif+8pWCsCQO5lxG439pMYuIrj8wgN3D8G/uZXmsP4d4mIYZGWRxxujAUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qySFw+X1GPAl5BW1S+EiG6vLqLNDW3/J6JndGMz+0Ro=; b=gUZKvh9kvs+hZExoqYzlWIiUm+Jm4/iAQbm/SYH4CgK5LD/Sp9Cetd6pJnUzT/azpHRFBCphz+aPfZYMd2BzQzt9gr8ghVHDDMSaR6p6iNVY2okmXWoHXwnmG68x82/8mJiGdXLYBQGpsDYTmFcdptXJ4FtTUmZN5G0WRbqyquE= Received: from SN6PR10MB3022.namprd10.prod.outlook.com (2603:10b6:805:d8::25) by SN6PR10MB3021.namprd10.prod.outlook.com (2603:10b6:805:cc::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4755.11; Wed, 1 Dec 2021 14:30:40 +0000 Received: from SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::c4d0:8291:84d:ac66]) by SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::c4d0:8291:84d:ac66%7]) with mapi id 15.20.4734.027; Wed, 1 Dec 2021 14:30:40 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton CC: Song Liu , Davidlohr Bueso , "Paul E . McKenney" , Matthew Wilcox , Laurent Dufour , David Rientjes , Axel Rasmussen , Suren Baghdasaryan , Vlastimil Babka , Rik van Riel , Peter Zijlstra , Michel Lespinasse , Jerome Glisse , Minchan Kim , Joel Fernandes , Rom Lemarchand , Liam Howlett Subject: [PATCH v4 52/66] mm/ksm: Use maple tree iterators instead of vma linked list Thread-Topic: [PATCH v4 52/66] mm/ksm: Use maple tree iterators instead of vma linked list Thread-Index: AQHX5r/0oKsiVYoWv0SiYWzYqFoOkg== Date: Wed, 1 Dec 2021 14:30:13 +0000 Message-ID: <20211201142918.921493-53-Liam.Howlett@oracle.com> References: <20211201142918.921493-1-Liam.Howlett@oracle.com> In-Reply-To: <20211201142918.921493-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.30.2 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: cebf8358-451c-457b-b35b-08d9b4d72656 x-ms-traffictypediagnostic: SN6PR10MB3021: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:612; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: US5kbistU3+sLZ4EPBbNO2g8L5pQdKzo74NQ4/+I/LWXj/ja4JdRqDMZFUcVZRYwmRr1fS204kizn2Y7mUYUBYqt5tY/PI7e6vl0KPZpVXtKgij/H2fL3cM+1dRIoCwArodnGig64jeComOG9RGAJo7T35/oR5fai8IJUyCADqIJCBd/wL6tNPMbCikB9hbb/Sex5GtWiX9AGJ8Cape9P5EddQRzWUwrtcFadIVfTkrqL987r86yfsf2SZyuPWeM6o9vsHe6VZ0ZJ/cAjR8JtqtalDtAD0OrMEWYstI/TraPA1caWuFWzFQfcB/a9qflMilU2PNKzxAHYdaG5udoS8/LfcJVUtO7lyNbebzT/EWINA1UesQ+jPd0870wS7kXm/Bq6ZPdLggXzhQiEBZC3qha/SqEClLr29/GzwGE3VvnIzbCByqLNT2N9PWOnPmpzDGlO+HYcqfph9mWwxom43lXJps2Q8wYZNcl/w39JeaqEKt6oPL1vBotdZfYhFowdWimprQ0sRjRBZCkXeMDg+oHsIiLstmieDHkPd8PenZDMx68FchHvCmdalyyrg+zjvmKHu8nPX1jDM/Ad2BqMhVbiHC4uLH9XE7JZTQujlyzRAFL/RNAjC72KTFp+9HSJXANpXoE8ECSAox2l1ihuUgGkJ21bDw0x/8NdMZPBFfZH9/jBgReA9J0HIv62YIIFX+Sr8wowBLsz9PJ/9YzJQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR10MB3022.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(66556008)(7416002)(38100700002)(107886003)(38070700005)(316002)(66446008)(76116006)(64756008)(66476007)(2616005)(122000001)(83380400001)(2906002)(6512007)(91956017)(508600001)(66946007)(71200400001)(8936002)(54906003)(186003)(86362001)(44832011)(4326008)(6486002)(6506007)(5660300002)(6666004)(36756003)(8676002)(110136005)(26005)(1076003);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?U5IQnYbaBqzpJ1HJyum+5jr?= =?iso-8859-1?q?NkzbQ3ICsg5tgcBE+XmzsbcAhoj+ZlU51oTV59EMpF/XodlQB9sT5wgC2TjX?= =?iso-8859-1?q?1XyarWXrCzPb2BLUlYIctpbv87CvDSWAMVq3yPWBFNtT8Kh6+PgtZiIoi1dj?= =?iso-8859-1?q?EUooCCt2D9ZqOrTDP2iQ0NuxoosfR6ReXjjAXE1HSGjJ1an6JZ5DuTImtGMS?= =?iso-8859-1?q?dHGvJHI2wK1VLhS8a/oA5pR6SStwpWypWg7m/8PdXb0M39qs3EwOilf876AY?= =?iso-8859-1?q?lvkliuU+S5lOGdNrBbIJP4lD++ULWwVguhMU+Qd3Sg3VvAwiLjsFjzWaC2k6?= =?iso-8859-1?q?JtowwG1kkfUwrZmrI6ay9nJ4ENk5nj0mizo7t6eCmg4z3/ZHEY87AUko8kQs?= =?iso-8859-1?q?WIz3R6fX8qU3vdQWXvaTAJVr2IrIZaFdE6HoxoTZtFcNgccFb83Ilk7BClDb?= =?iso-8859-1?q?wU3nfetx1xEaUDCMv2ix/9FXBMHhWmLKM/y+cWi3pqcDkSqU/ce0T0mleefr?= =?iso-8859-1?q?7xNd5TvB79SbZDK7p2b74SJrulbH+/7zlHZhe7M5q9bZRK/ulz9P251L8ar2?= =?iso-8859-1?q?TiHp/NaDHPfhjrodn2bSKEyJTV6CzriKgfGF18gA3wSIUEmGmrS3fLfGCVCo?= =?iso-8859-1?q?Owwvd8YwxpQN+h/yUrMAth6UgTGSp1lpKCo7Y7uhKkQTx4RWjsmeLxdHJiS2?= =?iso-8859-1?q?w1szzMgsa2jB0Q1x79e37XfqjypPHpNnGO04ehjzKaFDSjcfoeHw+Bc+8+HA?= =?iso-8859-1?q?Lch95ujsRCFfvG3QvCEL2yPTvo8Eg1nGEult4L3bizOja/3s79Fp9Fr9ODgv?= =?iso-8859-1?q?915FhinZGzV6U8hz5In2xTQgqKZR9UPlqIc7JLz8FImL5koGK6a9+fBECNaA?= =?iso-8859-1?q?ddpmYItpADyfrB9iwMh44Gqrcl149gQfRW7KYljr7jnYRhClajg14FzWo4lP?= =?iso-8859-1?q?tiPD0ucflEwMIL53f+En1u+hbvb0vHrvB1LSk5oGOd4dF9Bt54/9prlPonCB?= =?iso-8859-1?q?kYHEveVhv+iEfbsY/YDUkv9t1XDn4x9PYF8Qw+4J7evU/opQRuB1p86bh+BN?= =?iso-8859-1?q?bfXKTeItpXCK8nNI34utp71TKjSjT50LlNO+G9DmRiK4cLIrtxkIzEne/1L3?= =?iso-8859-1?q?7yXhEem4KuQAC9tc5vmrhrTP6KA/HHJM0WpJzEbCS71ZP4FtPBvQ4Koggdir?= =?iso-8859-1?q?t0vmLVYCYbpAbHWlZa09LS/6PBec5pfITBZgImXLSQrXg+vdiOMdmEgREiRb?= =?iso-8859-1?q?MKIBor6Rxzju4M6LPOTv9cFPWT8fH5h5LyMX28TsGY5ELxoC47p/elersIyA?= =?iso-8859-1?q?vVjPhvh6rqta/TnRhD0tAOyHyv58FjkE6QZZdCrxRocXWL83fSi0RNd4iu1M?= =?iso-8859-1?q?fPEjI7ZCeXeVQDkAZy/DrNS7aYSKtCU7ZxkAYRoSjJ5twC2+1JYTY3qfGcub?= =?iso-8859-1?q?Q/D+h72A90lcAh7on74HNrJxLWcvjFPHNBewcKVotXLu13j1puB8YY/B8ySh?= =?iso-8859-1?q?T/phq04XoN8gYJH6ipdhJRSBQzF1DsX1GPfviwXs7mXpsJo1QI4Ui/l09Isd?= =?iso-8859-1?q?Jh2srTiFcAspYs3q96G2WOBWY0Ak/T9gQJZmYpaXpu9KVtpVY5kqIlac98zv?= =?iso-8859-1?q?fxuUxJx9NIOquv3merBM4jCEpNr9gA6Izx9q4RwRh3reSv+hWSFSClnWd2Ww?= =?iso-8859-1?q?=3D?= MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: cebf8358-451c-457b-b35b-08d9b4d72656 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Dec 2021 14:30:13.8081 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qXPbWqvXB3+m5GqthlQoQTCajD5dL45n7XwCKehhgmF/3oFHHSBE6ozRRo2L0/F1Sua4qJF8zx4dWxtaOTegZA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB3021 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10184 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112010081 X-Proofpoint-ORIG-GUID: Z-W-5-LCph-dkD4vIemhy-Ga_n7YVyg6 X-Proofpoint-GUID: Z-W-5-LCph-dkD4vIemhy-Ga_n7YVyg6 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: EF3F210003DD X-Stat-Signature: 7yegnfijbiji33xkz9izhpz4km8qpco5 Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=QCECo9sp; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gUZKvh9k; spf=none (imf12.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=liam.howlett@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-HE-Tag: 1638369056-184416 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Signed-off-by: Matthew Wilcox (Oracle) Signed-off-by: Liam R. Howlett Acked-by: Vlastimil Babka --- mm/ksm.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 0662093237e4..d309249c95a4 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -973,11 +973,13 @@ static int unmerge_and_remove_all_rmap_items(void) struct mm_slot, mm_list); spin_unlock(&ksm_mmlist_lock); - for (mm_slot = ksm_scan.mm_slot; - mm_slot != &ksm_mm_head; mm_slot = ksm_scan.mm_slot) { + for (mm_slot = ksm_scan.mm_slot; mm_slot != &ksm_mm_head; + mm_slot = ksm_scan.mm_slot) { + MA_STATE(mas, &mm_slot->mm->mm_mt, 0, 0); + mm = mm_slot->mm; mmap_read_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) { + mas_for_each(&mas, vma, ULONG_MAX) { if (ksm_test_exit(mm)) break; if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) @@ -2226,6 +2228,7 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) struct vm_area_struct *vma; struct rmap_item *rmap_item; int nid; + MA_STATE(mas, NULL, 0, 0); if (list_empty(&ksm_mm_head.mm_list)) return NULL; @@ -2283,13 +2286,14 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) } mm = slot->mm; + mas.tree = &mm->mm_mt; + mas_set(&mas, ksm_scan.address); + mmap_read_lock(mm); if (ksm_test_exit(mm)) - vma = NULL; - else - vma = find_vma(mm, ksm_scan.address); + goto no_vmas; - for (; vma; vma = vma->vm_next) { + mas_for_each(&mas, vma, ULONG_MAX) { if (!(vma->vm_flags & VM_MERGEABLE)) continue; if (ksm_scan.address < vma->vm_start) @@ -2327,6 +2331,7 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) } if (ksm_test_exit(mm)) { +no_vmas: ksm_scan.address = 0; ksm_scan.rmap_list = &slot->rmap_list; }