From patchwork Wed Feb 2 02:42:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12732583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 647D8C433EF for ; Wed, 2 Feb 2022 03:27:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C93308D00DA; Tue, 1 Feb 2022 22:26:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95F288D00D9; Tue, 1 Feb 2022 22:26:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 525328D00DB; Tue, 1 Feb 2022 22:26:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id 3764A8D00DA for ; Tue, 1 Feb 2022 22:26:57 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EBC5B8245578 for ; Wed, 2 Feb 2022 03:26:56 +0000 (UTC) X-FDA: 79096403232.19.E27BE61 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf13.hostedemail.com (Postfix) with ESMTP id 4410520003 for ; Wed, 2 Feb 2022 03:26:56 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 2120bDdr015804; Wed, 2 Feb 2022 02:43:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=qA8vMoB7l083Q63xAY6AychJP+sRI03WI1rkY8W9gKs=; b=aRxfw+9VLfM++09krNfvt7Z/+wBXgmjEaWChU9ck34fB2JWBSHxhnImz9eHVU3r0Wjt7 yrcNSGQmEql3mfCrmizfBYyllY5cSO6bKmnTOOQwlI9rqUz+DV2vABqG9x0TUeT7BOWy IzPhmhSQln12Nqu8a/xE+69wkWwm9tQ6QiNBo9bJV4iLWR1AoqsmJ0N+05YYRegbYuyu jv43Q0RGrYZp6OIAwmHsTHWasrru/QUrYgP97yTtIdeBuQg/ANzKmogjx/LhoE7p3+za IIZBPFTEctduZ4rlAh5BAOyP1ASych7uEyTs6msvY3qs5oNQZwqaoc6klKx5j1o8bXrW xA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3dxj9wcnwg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 02 Feb 2022 02:42:59 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 2122Zwrl015281; Wed, 2 Feb 2022 02:42:59 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by aserp3030.oracle.com with ESMTP id 3dvumggud1-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 02 Feb 2022 02:42:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S+ABBl/h8RMU3VxjsFftBjP2QUuoKbMWxtTjReVMmSl+OhRm1GrwtTcv2mIHkU/xsCM+4S9LN6COcZOyATZbEShY2ooX1dAHG3KGo8dWwQXHocAUkuxCPUesxEzNTtsAv2Ltg6Ktjt2AFCeCoAWc4JO01SdBRGVzVoF9AGaki1Va6Ljch3N2f6aCPwNO9H7cWJY6XXhlEai5GmW0oFsiY7jVwJLI374kvQPDdb6C21QqAxT/GEjSKY5FU3r2jYJ5HylPYdT67RVuDlGvsqE9tF8CGQSL1ySXMHUmDPZil94Re33ZhFKqA4dcvhw6FTl7o8qet1OqiP/jenPfYhcQFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qA8vMoB7l083Q63xAY6AychJP+sRI03WI1rkY8W9gKs=; b=iAGpPp0rjPAJ60JsGEW5MBze+4aQCyHMYq/0PtxWZf2aIEzJWzjDbBaB3VdUSpYGd1M09F4LVcTE22YYzVRIEzIx6RalSyBsyGPHpHzb2w9u66xTWerq0alyZPKIQBaVyM+dlJfbt9kepuLwroRE3lLxW9KkoMVVpiSuiP4XasJZ1eQJk3MHuzNW6uJ/2oFPz8nroUansFtivwh5GBdhQLXxC51KZibp3E6UgWqd7nQNMijR528CeC02MT0lK5VaBix6KL6v13L2KsGoCA01zGhkDTdXRfjZfggdwqRCCBoEys+uWV5S1B4Jv5R+nq9EXEb3YEvEFhhPHRU1njS4aA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qA8vMoB7l083Q63xAY6AychJP+sRI03WI1rkY8W9gKs=; b=SAcnIfFPRS5licEXb4w2P+XYXLhSCT2G2cvtzLoXY9ktFflUHZF/RrBHq1RaYAjsUJJisp7GCZu8/WndCRV7MkLwSC4y2UN1UcPcs2P4ToN14QskFBKZFd91wMLpUP/E8zdAy/dcQnPB2t5UoTOmgMKW029WbtggZwpqbbdIKRU= Received: from SN6PR10MB3022.namprd10.prod.outlook.com (2603:10b6:805:d8::25) by SA1PR10MB5865.namprd10.prod.outlook.com (2603:10b6:806:22b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.7; Wed, 2 Feb 2022 02:42:57 +0000 Received: from SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::3857:3a25:3444:fdd3]) by SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::3857:3a25:3444:fdd3%5]) with mapi id 15.20.4930.022; Wed, 2 Feb 2022 02:42:57 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton Subject: [PATCH v5 57/70] mm/mempolicy: Use vma iterator & maple state instead of vma linked list Thread-Topic: [PATCH v5 57/70] mm/mempolicy: Use vma iterator & maple state instead of vma linked list Thread-Index: AQHYF96A1IHWfKJi3Em6i+RYxd8Hsg== Date: Wed, 2 Feb 2022 02:42:21 +0000 Message-ID: <20220202024137.2516438-58-Liam.Howlett@oracle.com> References: <20220202024137.2516438-1-Liam.Howlett@oracle.com> In-Reply-To: <20220202024137.2516438-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.34.1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 33152de1-b202-4d45-07df-08d9e5f5b870 x-ms-traffictypediagnostic: SA1PR10MB5865:EE_ x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8273; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 0qw7vfbAEkXnLXSq2ukvX3JVxTFHBq7FanZqvnvuXwJKMk5k3X/dBPUJnX5R2i7P9FooruWnv3CTGEWw4JT5QNhFWSIfoZRiUXBoJnvgI69CLtrSxQL9xQBqHU2+PzaYNSvjRgtraor5Jwtdd6PCBg7jNvyySiI0E2CLWLtgjFJCq6IyTA94o3J8HjcukSKxhTPX83V9o6i71xeK6Fm2cUBu17FbABrEdrlkXIDaZJUGj5zssjFXSXszMzUAxq+l8eOLfLkneFHxaHH3S1PEgWA/JEPfLnq+8q0ibM1EUm9rfFBOthGgl5pZDRGwYtRflvgtSBjF5rP52YS6y3ziY8SP+4c0LuRxCEf9uldE+/XElr7d5yM3YCZbFe5cLx8zXUqDjQZvK0JfA8+Emasbn0vjVG7x3NN9hKJd0mvJy4q7twBm/vAsCTM5tTh+1S6n1RvxLN3YBNVXfgRCL8SKzwkokwMWljQTXwm2q/Bmibm6Vd4SA5kUugrgahK5Uf4EFvYcGRNa1uYi7wuyAn4V8vC9exommMsy1wGz9AjjlzmhJpdl/qETbKSulv6yQgVNlpFi/P8oLHkX+5sSBMy1Skb/ZqDONvJRWvluGw944j5FzyDq1wtlnrMoi877zYHKzi7+YC6vxy11+bJ8zbkq9bmFgsK8zQ/dX5wKNxcHJ8KqDwpxstgJ/+TPAlXgGtK6Pu4+MB0+s8pcxU85nsF3aQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR10MB3022.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(66476007)(5660300002)(36756003)(83380400001)(6666004)(508600001)(110136005)(186003)(6486002)(66446008)(26005)(316002)(38070700005)(64756008)(76116006)(66946007)(86362001)(44832011)(8936002)(122000001)(1076003)(2906002)(91956017)(6506007)(2616005)(8676002)(71200400001)(6512007)(66556008)(38100700002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?UL1faLCM0dyFylemI5R0KmQ?= =?iso-8859-1?q?llSMZeUgzXpcrzRrzDlaLPX7ftUZXPmEWz3R89OZBUV/3Pf372RgG/N7lqJd?= =?iso-8859-1?q?SITo4394Zc0CPpOmeUKoNfcZ8yccL0UyFbANyjyhlVZNpcOKLBPSSaS2Ywzv?= =?iso-8859-1?q?tibkddhPdEP9XP5wz1FF2huxo1FPZE+jKTCgQ6eHsUHNiib0bR3/KJYMegXF?= =?iso-8859-1?q?baqsg1aACuS+wBXL7LYxhk2qhppQFXFCziglsqNk5TNnXt6gPhQ4AZ6ukM/P?= =?iso-8859-1?q?AzYUbT6T3beAtkMWEGs/LGlfgeBGGgsAspIN63DbKhto7JOadmxReZL9VBr3?= =?iso-8859-1?q?j7o2DAz1W+5IgDlvoCOwtAmwjFfjmCYvnmM1FESn19Dau/cACl//Bt3u5WGC?= =?iso-8859-1?q?QxKpz/zPFuP6jfMVhEkOcZlmA42N1ga56UXw3yv59QKRaAjuYThJYbAkmLxf?= =?iso-8859-1?q?8rHNKMDq7XSJbqI/p7BN+ZBpySwFvim4ddTyBQ94ZufxLJmw3r+NtjVksp9j?= =?iso-8859-1?q?V9QTgcaHeauHmlH5BsrxEQ46zad6HZn6ZEJ5/Yn/PQvxNy8zqYAuolYCPQoH?= =?iso-8859-1?q?piGn+CTgyaGm67BSxU+mj/LziUSQcbVrP9O5IKpCfszQxBotV7UtvVsm8juE?= =?iso-8859-1?q?DlJ/N76FZXWQgo8REziPZdduxiroQSGJOkTkPDWgebpS1r4qCYKRdZq3CNI0?= =?iso-8859-1?q?Shz3wI5BKIUHxUQpdXR7BypIcwNQRh0rrC/GBV5eHE2YIsJLTOF7iZOc41uM?= =?iso-8859-1?q?zul/VJZtaeL2c+xurTNjDpfeIhDzEhUXN8Qi8opifm7jorBl5p0LoSXWHSaX?= =?iso-8859-1?q?EmJqnL8dy8TsTblgeALcxUjMpbm6BNQnKVDCE48NKMM9TMjbW4pxFFXoDXwi?= =?iso-8859-1?q?bIl/3fQ/F5KsnGdM2F059yaHh9QZcJiqg+eQkvtNw9ibdiQsnQtDIUsarGpD?= =?iso-8859-1?q?JrmWyWhpfJO0etRGn2SyS/oekMGsctYlqiTsK5nlJAeS3JrZP+kklZTgn1gA?= =?iso-8859-1?q?bx7igYN4zdrRDejVJjgOFn+/gl1yXAkV+/xH6IWDgS27FB3IZyEboTyVg/Ty?= =?iso-8859-1?q?4doJ1rq1YiSgyMfI/ND2i3/hwXnBHOCi+cJalhsFKnDvKLcSA/7ynDDicS+B?= =?iso-8859-1?q?boMR3cIsWkg+pkaYs13/qlcrFpxHc5fVuZoGFNe5SmXzJcyx/k+YSnz1vIpB?= =?iso-8859-1?q?sOazHR6kR30YgQCmpIC+3MQi+sQW4Xg23G9bWYQpVTljvGbjOOD3F8xU107W?= =?iso-8859-1?q?bV2WmtV5XIi0CURnnb23uI91AidUi7OSi0FzfqXuwtw7wSJ4jccw1L4oB/Jd?= =?iso-8859-1?q?+kSi6sEmBis38lVwsbF2Nnt5zU7CLudCeld6e4iObhHmA71u/UPmw28icQpU?= =?iso-8859-1?q?FlY5Vex4QSjWEXCsXrLoDfzNiOaFZo3/61hsH2/fatsVBrtPL2DJwPiiQ1KQ?= =?iso-8859-1?q?jpRKc69TrayBbwR7ZN3Vgvh4WXT0OAOmMR0ivv+vZg17y5gNc1pDBjS8LxY+?= =?iso-8859-1?q?2mnwdbKsJGpp+UVqiD09AM9d+8CvJOXOU9E3nfmFR4cPMRmWIxz1AatgtVC6?= =?iso-8859-1?q?yfDge+PncFFQrhOnOqpLL+kvMN8qyhe+VeZn4MgBbPJtHA9FqY79iTSvsEZi?= =?iso-8859-1?q?fhaoaxhTsT496R6yPuSsXCZdmpbpX/1XJA7ZsZUoQVTwqCKVYuOrxUn1LWdY?= =?iso-8859-1?q?=3D?= MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 33152de1-b202-4d45-07df-08d9e5f5b870 X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Feb 2022 02:42:21.7976 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: T+LpikStG3IAAOXgz8lDsgwPjs2GsfX1B3YTxj+7h0d68zDxy+X7rng0GXS8hbf4mU51/Of/Yx8aZeArlR8kfg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB5865 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10245 signatures=673430 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 phishscore=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2202020010 X-Proofpoint-ORIG-GUID: c_SFG_bPRHCwFUujJVOsg_tosaPCxe6s X-Proofpoint-GUID: c_SFG_bPRHCwFUujJVOsg_tosaPCxe6s X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4410520003 X-Stat-Signature: crbd6pzpsuirbgknwh8mb7rthy39ip94 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=aRxfw+9V; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=SAcnIfFP; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf13.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=liam.howlett@oracle.com X-Rspam-User: nil X-HE-Tag: 1643772416-746075 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Reworked the way mbind_range() finds the first VMA to reuse the maple state and limit the number of tree walks needed. Note, this drops the VM_BUG_ON(!vma) call, which would catch a start address higher than the last VMA. The code was written in a way that allowed no VMA updates to occur and still return success. There should be no functional change to this scenario with the new code. Signed-off-by: Matthew Wilcox (Oracle) Signed-off-by: Liam R. Howlett --- mm/mempolicy.c | 58 +++++++++++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 27 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 028e8dd82b44..d058b2be8655 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -380,9 +380,10 @@ void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) { struct vm_area_struct *vma; + VMA_ITERATOR(vmi, mm, 0); mmap_write_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for_each_vma(vmi, vma) mpol_rebind_policy(vma->vm_policy, new); mmap_write_unlock(mm); } @@ -655,7 +656,7 @@ static unsigned long change_prot_numa(struct vm_area_struct *vma, static int queue_pages_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk) { - struct vm_area_struct *vma = walk->vma; + struct vm_area_struct *next, *vma = walk->vma; struct queue_pages *qp = walk->private; unsigned long endvma = vma->vm_end; unsigned long flags = qp->flags; @@ -670,9 +671,10 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, /* hole at head side of range */ return -EFAULT; } + next = find_vma(vma->vm_mm, vma->vm_end); if (!(flags & MPOL_MF_DISCONTIG_OK) && ((vma->vm_end < qp->end) && - (!vma->vm_next || vma->vm_end < vma->vm_next->vm_start))) + (!next || vma->vm_end < next->vm_start))) /* hole at middle or tail of range */ return -EFAULT; @@ -786,28 +788,24 @@ static int vma_replace_policy(struct vm_area_struct *vma, static int mbind_range(struct mm_struct *mm, unsigned long start, unsigned long end, struct mempolicy *new_pol) { - struct vm_area_struct *next; + MA_STATE(mas, &mm->mm_mt, start - 1, start - 1); struct vm_area_struct *prev; struct vm_area_struct *vma; int err = 0; pgoff_t pgoff; - unsigned long vmstart; - unsigned long vmend; - - vma = find_vma(mm, start); - VM_BUG_ON(!vma); - prev = vma->vm_prev; - if (start > vma->vm_start) - prev = vma; + prev = mas_find_rev(&mas, 0); + if (prev && (start < prev->vm_end)) + vma = prev; + else + vma = mas_next(&mas, end - 1); - for (; vma && vma->vm_start < end; prev = vma, vma = next) { - next = vma->vm_next; - vmstart = max(start, vma->vm_start); - vmend = min(end, vma->vm_end); + for (; vma; vma = mas_next(&mas, end - 1)) { + unsigned long vmstart = max(start, vma->vm_start); + unsigned long vmend = min(end, vma->vm_end); if (mpol_equal(vma_policy(vma), new_pol)) - continue; + goto next; pgoff = vma->vm_pgoff + ((vmstart - vma->vm_start) >> PAGE_SHIFT); @@ -816,8 +814,9 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, new_pol, vma->vm_userfaultfd_ctx, vma_anon_name(vma)); if (prev) { + /* vma_merge() invalidated the mas */ + mas_pause(&mas); vma = prev; - next = vma->vm_next; if (mpol_equal(vma_policy(vma), new_pol)) continue; /* vma_merge() joined vma && vma->next, case 8 */ @@ -827,19 +826,24 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, err = split_vma(vma->vm_mm, vma, vmstart, 1); if (err) goto out; + /* split_vma() invalidated the mas */ + mas_pause(&mas); } if (vma->vm_end != vmend) { err = split_vma(vma->vm_mm, vma, vmend, 0); if (err) goto out; + /* mas_pause() unnecessary as the loop is ending */ } - replace: +replace: err = vma_replace_policy(vma, new_pol); if (err) goto out; +next: + prev = vma; } - out: +out: return err; } @@ -1057,6 +1061,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, int flags) { nodemask_t nmask; + struct vm_area_struct *vma; LIST_HEAD(pagelist); int err = 0; struct migration_target_control mtc = { @@ -1072,8 +1077,9 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, * need migration. Between passing in the full user address * space range and MPOL_MF_DISCONTIG_OK, this call can not fail. */ + vma = find_vma(mm, 0); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); - queue_pages_range(mm, mm->mmap->vm_start, mm->task_size, &nmask, + queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, flags | MPOL_MF_DISCONTIG_OK, &pagelist); if (!list_empty(&pagelist)) { @@ -1202,13 +1208,12 @@ static struct page *new_page(struct page *page, unsigned long start) { struct vm_area_struct *vma; unsigned long address; + VMA_ITERATOR(vmi, current->mm, start); - vma = find_vma(current->mm, start); - while (vma) { + for_each_vma(vmi, vma) { address = page_address_in_vma(page, vma); if (address != -EFAULT) break; - vma = vma->vm_next; } if (PageHuge(page)) { @@ -1491,6 +1496,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le unsigned long vmend; unsigned long end; int err = -ENOENT; + VMA_ITERATOR(vmi, mm, start); start = untagged_addr(start); if (start & ~PAGE_MASK) @@ -1516,9 +1522,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le if (end == start) return 0; mmap_write_lock(mm); - vma = find_vma(mm, start); - for (; vma && vma->vm_start < end; vma = vma->vm_next) { - + for_each_vma_range(vmi, vma, end) { vmstart = max(start, vma->vm_start); vmend = min(end, vma->vm_end); new = mpol_dup(vma_policy(vma));