From patchwork Tue Oct 5 01:31:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12535313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75AB4C433F5 for ; Tue, 5 Oct 2021 01:34:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 12EE7610C8 for ; Tue, 5 Oct 2021 01:34:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 12EE7610C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 3AC3394002D; Mon, 4 Oct 2021 21:31:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3344A94002A; Mon, 4 Oct 2021 21:31:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09C8F94002D; Mon, 4 Oct 2021 21:31:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id EB23394002A for ; Mon, 4 Oct 2021 21:31:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A62048249980 for ; Tue, 5 Oct 2021 01:31:51 +0000 (UTC) X-FDA: 78660657222.06.F9FC387 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf28.hostedemail.com (Postfix) with ESMTP id 30C2A9001B5A for ; Tue, 5 Oct 2021 01:31:50 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 19508Mdq004481; Tue, 5 Oct 2021 01:31:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=cdxTr/4wZGvSNvNGJmWUPSav5cHPr8YVCcTtWirbX9o=; b=UleLUtfKrOyAI5GFgkWSa3UQ7Ifwgah+vFfkThD6k5cCujavg/UmHM77RTfTTqTEMv+B CyBr0eJh3xOKZ50iorBFeyDdLILbLJdrAx3TWRM0XV47wcp6spL9IJzjWvH1XsIHljpC mxzQ0DSOFAvM1bqJ8pjVo2mttEWd9eCLoe0NCu0Str9blhuag6/q3m53HQZAQ4iiJ0da 0qPSqoYKjWaGIlP5hHnaXfZ7GzYRABDyP8UA5D59ITYE4CnjWhiHhgSAbBky1kBfZUvp yjzGMUZc47cgg7UEVvMCvg84RBJKcBJOawxOrBHPjtZqrjlzGbNx5iQqB42WXLHavchF Ug== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3bg43dumr8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Oct 2021 01:31:41 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 1951UICl178521; Tue, 5 Oct 2021 01:31:40 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2174.outbound.protection.outlook.com [104.47.58.174]) by aserp3020.oracle.com with ESMTP id 3bev8w03ek-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 05 Oct 2021 01:31:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SZrsY257lI3tHSZ8EdSzqi3dNftHNMQXf9LlqGI+gXkuE2Icad1LTN1QYolMcDPF+4z9NEQtGUOeRAdUDjr/S+R7ng0LkRL+cQSQAOkzFz1o82xNeDm1FwUd/5G3/PGXH+Fg4h+OmD3fU9OZ49zuAz7Ay5wbL2ELXbmirrnay6VHkfNXN40HELEQKArTXre7qcTWs1RiP4avUbd5nvtTeaQn6YOUC64K/DayCjlsBvOPLOrpc9qilj8R5Xs3LNq4EvC71SACH6oGgLL4d6B+IzOkWNBEzDPQFZdTARnabPJZKu24n4RSciglLMAZEAGC6oD2oEDsV+AxMewEVMSiFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cdxTr/4wZGvSNvNGJmWUPSav5cHPr8YVCcTtWirbX9o=; b=JYGEvkt86SnywRBbq9cauNOKAb+fsDttZctgGWWhuPx38YdYyqm1agwGLffwFEFsEKz8tJW+wIGSK5rIq0Cf6VfO2As8GH3koer39qg7zlYo5Dl8A06NalIiGciZmx6iktFivKXW1QhUCleUUAfsr/sGg79R2eKYh5tJA0dGseKRRXUPOJCsX+yw0yIY34VRSRrBeJdJCidSjY21I9rfC5jk2ypwTiuMWA5aT7XAtnlZC357S60U6d3qk251ofZiInnpwFcATI6s4TIj8i6YiXneMunAtlr3Eq9AeruXrKoMAdRWTKMLurvk64inHDEJ50MODpmeVere7nbMlL6OUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cdxTr/4wZGvSNvNGJmWUPSav5cHPr8YVCcTtWirbX9o=; b=AZMnBaPzucBYpmlAiq2BQ9+AQqm6MrK5iCR+XJGN0/pRmvNGDcvU4bhW7D1cC8nm3Izuk6NcxBZuFrLhe0vrdS779+rz0tkR0SSFUPs7amaJJ2G89e74Et+g1NJg1yUP3izgNHgMMSTCX1M8z5k8pa2Wo1vcdIBLEsvv/KB/sM8= Received: from BL0PR10MB3011.namprd10.prod.outlook.com (2603:10b6:208:7e::29) by BLAPR10MB5044.namprd10.prod.outlook.com (2603:10b6:208:326::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.15; Tue, 5 Oct 2021 01:31:24 +0000 Received: from BL0PR10MB3011.namprd10.prod.outlook.com ([fe80::6d61:54c2:40f0:93a]) by BL0PR10MB3011.namprd10.prod.outlook.com ([fe80::6d61:54c2:40f0:93a%5]) with mapi id 15.20.4566.017; Tue, 5 Oct 2021 01:31:24 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton , David Hildenbrand , Douglas Gilbert CC: Song Liu , Davidlohr Bueso , "Paul E . McKenney" , Matthew Wilcox , David Rientjes , Axel Rasmussen , Suren Baghdasaryan , Vlastimil Babka , Rik van Riel , Peter Zijlstra Subject: [PATCH v3 50/66] mm/ksm: Use maple tree iterators instead of vma linked list Thread-Topic: [PATCH v3 50/66] mm/ksm: Use maple tree iterators instead of vma linked list Thread-Index: AQHXuYiqXeCDqSn200+jDvcxOUgcfw== Date: Tue, 5 Oct 2021 01:31:05 +0000 Message-ID: <20211005012959.1110504-51-Liam.Howlett@oracle.com> References: <20211005012959.1110504-1-Liam.Howlett@oracle.com> In-Reply-To: <20211005012959.1110504-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.30.2 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 89843487-efee-4c04-48a1-08d9879fd860 x-ms-traffictypediagnostic: BLAPR10MB5044: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:612; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: qIwpSj1fsbH8H4u3YPP6W/No7IWap1b+7WSDmET3W1wxIyT5QNBGb+E6XZbWa/EMBThgHb4sO2C4eGak3L3Nub2N4MBZvUyrXXBZpQfqsAAIjkV0KvzUNMrdgTrjW32y2GrJtwO/jjXqWY4D9SkShVr/eHi3KCc4FY99D34jhkw207W3A88yH9qEU3L8RYdAcVojhB037EEr9lizv5jDL8P9ps4ZsVFG0KqiC1fsqL+rObJ7+7DgeUaoqNtwxhNEjDPYS9v9tbbLzFiQcOKU4lFhjNKBmx+m+7G2/K8Y7HVnJEd05m5BVvueCC32rCeNB9YvdM4/+tVw0+GUprm5xQiDxIeCkyW0VE8gaYs8P3Gb/L1Hdf0pNTypvQYIK+2HWtcy0d7ogmwXlrk5vT3KxXQYE+UBsYAJdUTanPYN6NnZHqHlgMFP3eKxgObWjMdEXO/tovXm7AbQ7ktW9UV+KS5UeWEduciEcmcY/5wCp256Hn2StOEH0yaAAZ3Sh3c6na1njvEgPrNSI6Y4mMJnE5FU23llWb50sXLKZCOU2DVHy61XnyfdwjdpkDn0hEW6F+vu3WdYJhH6rgZHOpDRkz46awXIRWzptUr+9ti3laMYnmMEWUJJaFEwW89+HhyPAOfXo0tK2CxVH7nJIu8tkIFvedjO3LU3S0ZL85TXafT55Nlq7Qc42T5SkFcdu1pYfurPqgQixcmGB+2ZIn97yQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR10MB3011.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(110136005)(71200400001)(8676002)(86362001)(6666004)(6506007)(6512007)(36756003)(316002)(8936002)(54906003)(2906002)(83380400001)(186003)(66446008)(122000001)(26005)(7416002)(6486002)(4326008)(508600001)(2616005)(66476007)(44832011)(38070700005)(76116006)(5660300002)(91956017)(64756008)(66556008)(66946007)(1076003)(38100700002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?+c4Vwfa0p6QGPPxP6d+lwry?= =?iso-8859-1?q?eGF2ztWizrjYBaJefB0bTVOiD2xJhsr7a015dv4wOFPrTrhmnkMxT3mV4Llt?= =?iso-8859-1?q?u/aiw+UaWjwPsZL3VZ9PidNkyywK7BEkK3uTJDCXAD13QZ354HK+0mv6DdPL?= =?iso-8859-1?q?O0zDjjulgTmaiaY4WIGJomT/PkwPXcR7ItdiHpct6gy9c8Nkq7SRVDgLlydj?= =?iso-8859-1?q?UGIyVrGc3D32MufOtNTfjQXsg+50wdMWx2vugE4slyRP416dc8zB8KGI+EMI?= =?iso-8859-1?q?VQd+TKqeIyH3GnzAU0SJcOZURN2k/L4i/beuvmCG4twAzq1BmMRY3G3b7zdf?= =?iso-8859-1?q?fd9gWi8UXwUIvcV4pPmQPkFCFRF58iCo0JjRO5DbIO/7n8MoyaCbe3g0WSRe?= =?iso-8859-1?q?lewcJ1dFJL7mCkJe1Nww04fIpv0pcvBpVwKjy7Qloitvt8QALGVRG15Ll0M2?= =?iso-8859-1?q?XbTGZpAPbTYmTb5zhO5Ge24QxP5GvgIHsgQ96xkud2kIYNdMwymzi5I4LrQb?= =?iso-8859-1?q?aANelGBt7nMNpUgIU00AKjqd74ZsgBRD6cCGaPSHVlinZ/dAqLwI7bkej4hs?= =?iso-8859-1?q?FNg+GKRlSyrfhCLX6J4wNr9VcbAuvBw5GY7c7/1DyxQOhephq6tIjOFiwwhR?= =?iso-8859-1?q?JNbK3Ehde0jrK4N6JEzdJ2xgFOs3cy9T6LNjXcljDRFSrxT+ivNN2KWalRaP?= =?iso-8859-1?q?I0SX57Vfh4WHDDtXoWJ2+ELsMBvj3IOi6PxKd46HhF/P7k49649QLlt0vpzS?= =?iso-8859-1?q?SH4H1arYBCLxISsCCpWfkHXKSilpGX+ef/EixXdHyFyH1dQjMAhWcGIIOoxw?= =?iso-8859-1?q?+kwfuZqJoiAg5ybRLUBdb63ilWOu1699APHaDjTwOVYjNA+/lh9atAF2dCOe?= =?iso-8859-1?q?UMZEya1iPJePRqVMs56oxcYpEjYCVDuFGOfOkwh+CaPd/Wnl3u1GqaNlFhrL?= =?iso-8859-1?q?5xvI3+EFfFyLNsPgQ0d+QMp6R5inURKWeLxciz+OJE0OXpGUy1JORAWNZ57V?= =?iso-8859-1?q?T86EPWSTyY++X4iBQrq/BQ+k6cMPXX3UjDLvHoYQBlMXX+53ucCgG1hvb7Pn?= =?iso-8859-1?q?ESwpi2IXkm8YnAS0FUGkgqchrrK85PZ6RvJA7aY611YZUmmZEI9K/weE60MY?= =?iso-8859-1?q?PfEIQp0AnaogW/MgEZW3eGjUQGzfcMBmTROudzh2bdf9D77Eu940jpzleocu?= =?iso-8859-1?q?OysOlJAvTvTr5k67+WLdxvZGV+f85IgHkCsz/JLG8BoTk0qQ9dY3BcowgH/m?= =?iso-8859-1?q?lHnJBGxuUYzyLWmqcOcXBcyUrHMhse1R9NGRmC4qN6Wdrdg1anP+U4h/gr9a?= =?iso-8859-1?q?uGtRuD6RiW39tRhCDFb1vNdTap/9byhTfMaA6rXXVGIdJUB/3dLWX0+WZmT4?= =?iso-8859-1?q?M?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BL0PR10MB3011.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 89843487-efee-4c04-48a1-08d9879fd860 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Oct 2021 01:31:05.5124 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zzEc2xGOUh8byU2XRU+Rq0q9JgJw9S5Gnplcbk4BAK68XJej/35Drmp3EW/kjt+s1/I8arAROyouZTOMuYykfg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5044 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10127 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110050007 X-Proofpoint-GUID: vqUr1ZZ6J67oiUUGbmOqltLC-vofpcfA X-Proofpoint-ORIG-GUID: vqUr1ZZ6J67oiUUGbmOqltLC-vofpcfA X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 30C2A9001B5A X-Stat-Signature: hi8atae1a8wxouq7paaoun6f4san7eee Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=UleLUtfK; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=AZMnBaPz; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf28.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=liam.howlett@oracle.com X-HE-Tag: 1633397510-830064 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Signed-off-by: Liam R. Howlett --- mm/ksm.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index a5716fdec1aa..fe93a55f5158 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -968,11 +968,14 @@ static int unmerge_and_remove_all_rmap_items(void) struct mm_slot, mm_list); spin_unlock(&ksm_mmlist_lock); - for (mm_slot = ksm_scan.mm_slot; - mm_slot != &ksm_mm_head; mm_slot = ksm_scan.mm_slot) { + for (mm_slot = ksm_scan.mm_slot; mm_slot != &ksm_mm_head; + mm_slot = ksm_scan.mm_slot) { + MA_STATE(mas, &mm_slot->mm->mm_mt, 0, 0); + mm = mm_slot->mm; mmap_read_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) { + rcu_read_lock(); + mas_for_each(&mas, vma, ULONG_MAX) { if (ksm_test_exit(mm)) break; if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) @@ -984,6 +987,7 @@ static int unmerge_and_remove_all_rmap_items(void) } remove_trailing_rmap_items(&mm_slot->rmap_list); + rcu_read_unlock(); mmap_read_unlock(mm); spin_lock(&ksm_mmlist_lock); @@ -1007,6 +1011,7 @@ static int unmerge_and_remove_all_rmap_items(void) return 0; error: + rcu_read_unlock(); mmap_read_unlock(mm); spin_lock(&ksm_mmlist_lock); ksm_scan.mm_slot = &ksm_mm_head; @@ -2221,6 +2226,7 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) struct vm_area_struct *vma; struct rmap_item *rmap_item; int nid; + MA_STATE(mas, NULL, 0, 0); if (list_empty(&ksm_mm_head.mm_list)) return NULL; @@ -2278,13 +2284,15 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) } mm = slot->mm; + mas.tree = &mm->mm_mt; + mas_set(&mas, ksm_scan.address); + mmap_read_lock(mm); + rcu_read_lock(); if (ksm_test_exit(mm)) - vma = NULL; - else - vma = find_vma(mm, ksm_scan.address); + goto no_vmas; - for (; vma; vma = vma->vm_next) { + mas_for_each(&mas, vma, ULONG_MAX) { if (!(vma->vm_flags & VM_MERGEABLE)) continue; if (ksm_scan.address < vma->vm_start) @@ -2312,6 +2320,7 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) ksm_scan.address += PAGE_SIZE; } else put_page(*page); + rcu_read_unlock(); mmap_read_unlock(mm); return rmap_item; } @@ -2322,6 +2331,7 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) } if (ksm_test_exit(mm)) { +no_vmas: ksm_scan.address = 0; ksm_scan.rmap_list = &slot->rmap_list; } @@ -2350,9 +2360,11 @@ static struct rmap_item *scan_get_next_rmap_item(struct page **page) free_mm_slot(slot); clear_bit(MMF_VM_MERGEABLE, &mm->flags); + rcu_read_unlock(); mmap_read_unlock(mm); mmdrop(mm); } else { + rcu_read_unlock(); mmap_read_unlock(mm); /* * mmap_read_unlock(mm) first because after