From patchwork Wed May 4 01:14:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12841017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11F28C433EF for ; Fri, 6 May 2022 11:59:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8316E6B0072; Fri, 6 May 2022 07:59:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DF936B0073; Fri, 6 May 2022 07:59:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6801D6B0074; Fri, 6 May 2022 07:59:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 59CE66B0072 for ; Fri, 6 May 2022 07:59:23 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2F87E2E4CF for ; Fri, 6 May 2022 11:59:23 +0000 (UTC) X-FDA: 79435173006.02.89155FB Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf29.hostedemail.com (Postfix) with ESMTP id EFD05120026 for ; Fri, 6 May 2022 11:59:16 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 243KctF4024988; Wed, 4 May 2022 01:14:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=rtEgJ6g/+NrMFge7rfz5iJSFQ9yYY7v2UpbI0NJG4ec=; b=uk3s/AncAFsFZ101mH5kz2+rB9OJ9EbgpQjowFzBA6wRpN9+Dunjgn23tYzbUK/pQ7Gv t27nuqN0bbG6LQql0GLjQHcKL38cWWbqeAwIUdDjyhmZSmuITuEn6UZd+wwyapD/UZXS WE04TCsVOf5+WAOUpuhcLVemxrrpah+qGt+a7azi36+tAwshLSbA2xt+kT0tQ8LXCqbt 09n8ax/SOej4J8BT0WydTqlqW6q730jXQ/5AJplxbAdbqq0OSyc8cAdAYHIqETTeJSS0 Hl+zbU7FgoIpj9WejAq6o87KEfPtu2ciQONhl70iaz9/Pa0JGRHGVyF5lEqAx0RuE43N Rg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruw2f6c9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 04 May 2022 01:14:42 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 2441BXR1013207; Wed, 4 May 2022 01:14:42 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2049.outbound.protection.outlook.com [104.47.66.49]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fsvbmv41r-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 04 May 2022 01:14:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S2aVQtegsGWqSBXMESLSY8gRJmKJTSNJEzr/8VXeBmJFHxp7WmKkWO6KrWFKrwQsXMxG0hwuiZLoClJX9MuIUNX0VUqaeFte1F2bQAc7SU4OXEkKxDa9geYl2VbcAUGpffCJbFWHh3h86fypdEyTGcYqS/jj8Rn/r1oXrU1uuQSqu1ygtixZ+Pl60rX+KO+dE7cc8oLRW3GNyj/ZOVfoHiiC/u+fRts/lynV3ZxGiZjX7Mb4Ok52iKqeb6BrOMn1fd0oxWVJ1hZUlamyI+C5y7b/cCh8ukiJbMn1fvIu/ywmjJALiz5B2rKZAZfbPxz/880QVKb7Esmb9RDV1VdyiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rtEgJ6g/+NrMFge7rfz5iJSFQ9yYY7v2UpbI0NJG4ec=; b=mzV7RJIe+nCuqmAG1jje6SeSV0ixP+wakmLMVsmMh+NkbyqiKtcGLO92CQaD4Ogu3QJjYktbjx535mtbFzeuUvSm2mgkIvBoJQH/jVVC2NpdJx3DLqhtdYNltr3jWA12AXcO4FmBM5p1RciuZsE3cjEaWilnJZKfmV9VH2H4gO/tWEcsbkiaajtQOwHh85TR7Ft8tOoeSdfjXNlMBQSViFL3yyOEw7Ay0ED74/w4yFDd+T3bIdI3eLWzytax3V+17TnmZhCjUsHDoYg2+4QBeWhHgfZJK/X4J1AhISMKjZ9PVT0Q1AQuRMJxvydFHsAowyPj2E0LczXhCNTL8c2QmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rtEgJ6g/+NrMFge7rfz5iJSFQ9yYY7v2UpbI0NJG4ec=; b=v2R65lp2We/Qkr4O5HYFdAztf4b9g6sI6KDtZGTwxrHs8dU5+j4QLLIcLfSdb9XpsLooDH8UF/fsenJCWNlMPoUraVIBxjuEqUnU3Nm3v4aaWSsTxEK+/h9YuJ1WUSI0SVDFs7EFPvyMJTZ+vIK5IVXWC29w4aSYLF13fCd6Uzw= Received: from SN6PR10MB3022.namprd10.prod.outlook.com (2603:10b6:805:d8::25) by SN6PR10MB2688.namprd10.prod.outlook.com (2603:10b6:805:4e::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.13; Wed, 4 May 2022 01:14:38 +0000 Received: from SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::318c:d02:2280:c2c]) by SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::318c:d02:2280:c2c%7]) with mapi id 15.20.5206.024; Wed, 4 May 2022 01:14:38 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton Subject: [PATCH v9 56/69] mm/mempolicy: use vma iterator & maple state instead of vma linked list Thread-Topic: [PATCH v9 56/69] mm/mempolicy: use vma iterator & maple state instead of vma linked list Thread-Index: AQHYX1Q+3skD0a2xEUKL+wRRlZUrrw== Date: Wed, 4 May 2022 01:14:03 +0000 Message-ID: <20220504011345.662299-41-Liam.Howlett@oracle.com> References: <20220504002554.654642-1-Liam.Howlett@oracle.com> <20220504011345.662299-1-Liam.Howlett@oracle.com> In-Reply-To: <20220504011345.662299-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.35.1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 38142693-32ae-467a-6bd0-08da2d6b75e0 x-ms-traffictypediagnostic: SN6PR10MB2688:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cdGCs9gFe/Iz0ODEhIWymObt7Mn2V1DwMUpSj5u5uZxZcMRFupbq1v9l9mxlR0EWw8I0JxzoxeO/sztmeMJL9ztSn/8Zs0FmI0b6WxiqifQHNMn92Gc3LZouMxewgamBl390VUjECQNb91mi/vMf7nDCyr3cwkZHqFUIn/Rv2W5Y/mS+iKlEPzi0VtTUwJjlRP+f41IynEIFBYhzqqhyKCSJA+SHNK7gnYldxPvMMFRjYJm8oyoFsgNGQ+0+SqO+4y5ZrWDtIddJZ/MgaCbNSjECeVRZz3rgoihYiKQlGDsAuRMPuNbmYmUcg3dc6lkVPdak3nplcZdl+st7WrBy54ocZcMB4RSJX9CleEEaJABSuVlJsjhez3+stJwcFKe+lewwkX0AemkyMMHi2GoCVqZEKDzcu7qQBo71xYmWxsZdjreJ+Z2jjTLY0QaD7M632cdwSjmDXhwTr7A1wythB9e+koAOTEcD4x8Pm9W1GCNC9GKXNaqd+XDxIrATZuKOItQADVkAwHMB2ME9OPAhXCmywqhjSheldewRZkdnRu7MuCoQ55EJz9dzvJI5fkOGAqs1QAdJccr3Sz+AcWdzia+E+x0OA6PW5qPfXo1LZmYJqrfOe2LJc83YhfQ8U41lcYyDpyfcapaZRKnSw0hRrBmtTexucgiKbOd2NNDRNVDPTUAAuPbGMVvi9CF+4m2RssBlc4SU+6R679AH4WO7Yg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR10MB3022.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(186003)(122000001)(1076003)(2616005)(66476007)(8676002)(66446008)(64756008)(38070700005)(36756003)(316002)(38100700002)(66556008)(66946007)(76116006)(91956017)(110136005)(26005)(6512007)(71200400001)(8936002)(6666004)(6506007)(2906002)(86362001)(5660300002)(6486002)(508600001)(83380400001)(44832011);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?mDQ+2VW0Lf03DlPWL3GJsmd?= =?iso-8859-1?q?SYByUZlFVgw4EvtCGZnPDdcAf6r4mTGRYinR2vHmK09EGtqrIVMmQ8p/Nli+?= =?iso-8859-1?q?ddzxt8DiPvPHXld9ElMwLK8QAj/zebITYb/AS92CLim4Awr62chu5xXwpIZx?= =?iso-8859-1?q?6gd9M7ec3fDLn0vrYwAW6V/gkM4XEZVbZ9E9Q8eLKJJuYa2sZyTFrDt9uJIu?= =?iso-8859-1?q?3z9H0l340hGa/TpyP/T33Qn4WAqxq4aRCUccghV2xz27mh7mT4w99g9iffo8?= =?iso-8859-1?q?doLsXYUZ2VkqSIsk911rzJM2TCFed+LtmRXnbnyxwtgDi1lnztBGYxxrmQtG?= =?iso-8859-1?q?wcjigqi3ibDTBUQGv06fgby565O59ExZYJbcEbBEGXHj1O6bzF4hvmM1C5Ne?= =?iso-8859-1?q?mASy16xIqSzMXUUeo2DHlFI92EstFuctVaVxS7gjXvz34yQJ/4YZzD97Ccfm?= =?iso-8859-1?q?ABeCg7J0dSHSReX8box/ZBhDSMrRuVb9Eqaa+mRhGhKQKj/r0zEupv0Y9/8K?= =?iso-8859-1?q?nds4wHZ/ouEd5AbO2g3JhdQz1sDSbBgr+noF5lopisYdc1ufFyC+zgcs8n+C?= =?iso-8859-1?q?sGsLVDFNsBupBgm/h1VyXkFfdY2yY5rJS/yrwSlqadAuRk21yP6YQ6pJL7Mn?= =?iso-8859-1?q?pE+2ABDnbg9gPtUeucoG3u2kEKZird1aNchjegFFoc5wkXZT9KJoKnSRJ9Ot?= =?iso-8859-1?q?ljk8exx7iJNRfqxstldM1UeOCTCYhNB3FRj2Bs6YPBmm7nGd1GSnaI9unhXX?= =?iso-8859-1?q?PqtUjG5xGX4h2Z5L9PMuA9nptFN2hypZNeHxQrLFGgIPkOxiVnS5Vmh8qe4g?= =?iso-8859-1?q?fehP/cv77VwcddKSjRz1ieZHGW83HsaxIGipoB2jvzqx+9V5DTQaM3ZAAv9m?= =?iso-8859-1?q?dFl+99YwH484eAyLj4QFuYRsEGbi3ZIyF2TdVJmwQjbN1yHGlDIllLzVCRA9?= =?iso-8859-1?q?WSMRL915bvjvaPEa7tpVU1AtNBOIwA/dP2i3vfLkOpbJAcYNcQJq2UP1RsuB?= =?iso-8859-1?q?z8aANFk4iFhJgsPJcOI/IzZaxU5LAtSwJe9J6yXV56ZJ6+XumhCBWhUs1Shs?= =?iso-8859-1?q?82TTto0HIIhU6CxjIiDIsadg2wquXLMdk6kAfLIwtyR84vddfuXpD2Er4NWw?= =?iso-8859-1?q?BtyqckuTz6c2mtnbIvuZIUE2SMUNgh/66VYfRkZvwAWe0M6gIis/3Zbupa4G?= =?iso-8859-1?q?aTzskKywoSEnm45YSux26BGXiaRmy3+NTpkkq4sqygE0hE3m8Qj04Sx6N+gQ?= =?iso-8859-1?q?ZapvoFNa6AsAZLTpOMR/5wrgRUAoAaXJ6TspY1KK/K3ZAE6TTeTZvt7ozRmZ?= =?iso-8859-1?q?AvSJnRtpSZoNldyEOawIaXgj0qLhI6wcff6n0FX3+ooVSdLIiQPlYI685Uze?= =?iso-8859-1?q?BLihc1RwQvRX+K8utOCYEgXucxvoiJww2oEMJdlNdAoHwPpdtkhoX2bF/dOR?= =?iso-8859-1?q?ZEht7gqlTs8hS0f8zdkmK4n8w9Dx0dVIi6DO8WLkYu2QWEfWfAGZEscbZmpC?= =?iso-8859-1?q?ENLAXhWvsfPkl5FAxE2F7+pPWJN2tUu0K9lGust9EW8aHJiI/7DE58ds5A+0?= =?iso-8859-1?q?TmgIQk7Gpcc2pEMgSSwdHX6rIahzN3mL8wgMubxU7tdNYNGchjFpcAvTSls7?= =?iso-8859-1?q?lqqYrYQXEFTRaWo3xeTcstb15OMGgL50gZxXmjnRRG0ed8349LFjP98nCgxc?= =?iso-8859-1?q?AzMjjnoiKTAvKLce/kbidOA5Ij2Cn7SN7PigKJXwj2WZ7zaLYfQEcEih1nok?= =?iso-8859-1?q?8b9XFuLzBHgZnfD9oG7gBxGBca9kobEsRDkga61XGi9nVNJHURnx1nZI2HmA?= =?iso-8859-1?q?begZQPRc=3D?= MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 38142693-32ae-467a-6bd0-08da2d6b75e0 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 May 2022 01:14:03.7668 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: EkdO8VACHl6HmSM682MgZgGI7yPkuNNscfkfFkQFrqOm35Hf889BnxsIAVoyfHC1TzML/5Tl839Mj+ufAGenxw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2688 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-03_10:2022-05-02,2022-05-03 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 spamscore=0 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205040006 X-Proofpoint-GUID: AYWXTTPaw17bSyjIlsM0ypeVdQfYeMP2 X-Proofpoint-ORIG-GUID: AYWXTTPaw17bSyjIlsM0ypeVdQfYeMP2 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b="uk3s/Anc"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=v2R65lp2; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf29.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=liam.howlett@oracle.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EFD05120026 X-Rspam-User: X-Stat-Signature: ogs434jcmcwur4ku3ac7tfecjr65t9se X-HE-Tag: 1651838356-772649 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Reworked the way mbind_range() finds the first VMA to reuse the maple state and limit the number of tree walks needed. Note, this drops the VM_BUG_ON(!vma) call, which would catch a start address higher than the last VMA. The code was written in a way that allowed no VMA updates to occur and still return success. There should be no functional change to this scenario with the new code. Signed-off-by: Liam R. Howlett Signed-off-by: Matthew Wilcox (Oracle) --- mm/mempolicy.c | 56 ++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 0288ffaea064..df4487767259 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -380,9 +380,10 @@ void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) { struct vm_area_struct *vma; + VMA_ITERATOR(vmi, mm, 0); mmap_write_lock(mm); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for_each_vma(vmi, vma) mpol_rebind_policy(vma->vm_policy, new); mmap_write_unlock(mm); } @@ -649,7 +650,7 @@ static unsigned long change_prot_numa(struct vm_area_struct *vma, static int queue_pages_test_walk(unsigned long start, unsigned long end, struct mm_walk *walk) { - struct vm_area_struct *vma = walk->vma; + struct vm_area_struct *next, *vma = walk->vma; struct queue_pages *qp = walk->private; unsigned long endvma = vma->vm_end; unsigned long flags = qp->flags; @@ -664,9 +665,10 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, /* hole at head side of range */ return -EFAULT; } + next = find_vma(vma->vm_mm, vma->vm_end); if (!(flags & MPOL_MF_DISCONTIG_OK) && ((vma->vm_end < qp->end) && - (!vma->vm_next || vma->vm_end < vma->vm_next->vm_start))) + (!next || vma->vm_end < next->vm_start))) /* hole at middle or tail of range */ return -EFAULT; @@ -780,26 +782,24 @@ static int vma_replace_policy(struct vm_area_struct *vma, static int mbind_range(struct mm_struct *mm, unsigned long start, unsigned long end, struct mempolicy *new_pol) { + MA_STATE(mas, &mm->mm_mt, start - 1, start - 1); struct vm_area_struct *prev; struct vm_area_struct *vma; int err = 0; pgoff_t pgoff; - unsigned long vmstart; - unsigned long vmend; - - vma = find_vma(mm, start); - VM_BUG_ON(!vma); - prev = vma->vm_prev; - if (start > vma->vm_start) - prev = vma; + prev = mas_find_rev(&mas, 0); + if (prev && (start < prev->vm_end)) + vma = prev; + else + vma = mas_next(&mas, end - 1); - for (; vma && vma->vm_start < end; prev = vma, vma = vma->vm_next) { - vmstart = max(start, vma->vm_start); - vmend = min(end, vma->vm_end); + for (; vma; vma = mas_next(&mas, end - 1)) { + unsigned long vmstart = max(start, vma->vm_start); + unsigned long vmend = min(end, vma->vm_end); if (mpol_equal(vma_policy(vma), new_pol)) - continue; + goto next; pgoff = vma->vm_pgoff + ((vmstart - vma->vm_start) >> PAGE_SHIFT); @@ -808,6 +808,8 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, new_pol, vma->vm_userfaultfd_ctx, anon_vma_name(vma)); if (prev) { + /* vma_merge() invalidated the mas */ + mas_pause(&mas); vma = prev; goto replace; } @@ -815,19 +817,25 @@ static int mbind_range(struct mm_struct *mm, unsigned long start, err = split_vma(vma->vm_mm, vma, vmstart, 1); if (err) goto out; + /* split_vma() invalidated the mas */ + mas_pause(&mas); } if (vma->vm_end != vmend) { err = split_vma(vma->vm_mm, vma, vmend, 0); if (err) goto out; + /* split_vma() invalidated the mas */ + mas_pause(&mas); } - replace: +replace: err = vma_replace_policy(vma, new_pol); if (err) goto out; +next: + prev = vma; } - out: +out: return err; } @@ -1042,6 +1050,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, int flags) { nodemask_t nmask; + struct vm_area_struct *vma; LIST_HEAD(pagelist); int err = 0; struct migration_target_control mtc = { @@ -1057,8 +1066,9 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, * need migration. Between passing in the full user address * space range and MPOL_MF_DISCONTIG_OK, this call can not fail. */ + vma = find_vma(mm, 0); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); - queue_pages_range(mm, mm->mmap->vm_start, mm->task_size, &nmask, + queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, flags | MPOL_MF_DISCONTIG_OK, &pagelist); if (!list_empty(&pagelist)) { @@ -1188,14 +1198,13 @@ static struct page *new_page(struct page *page, unsigned long start) struct folio *dst, *src = page_folio(page); struct vm_area_struct *vma; unsigned long address; + VMA_ITERATOR(vmi, current->mm, start); gfp_t gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL; - vma = find_vma(current->mm, start); - while (vma) { + for_each_vma(vmi, vma) { address = page_address_in_vma(page, vma); if (address != -EFAULT) break; - vma = vma->vm_next; } if (folio_test_hugetlb(src)) @@ -1473,6 +1482,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le unsigned long vmend; unsigned long end; int err = -ENOENT; + VMA_ITERATOR(vmi, mm, start); start = untagged_addr(start); if (start & ~PAGE_MASK) @@ -1498,9 +1508,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le if (end == start) return 0; mmap_write_lock(mm); - vma = find_vma(mm, start); - for (; vma && vma->vm_start < end; vma = vma->vm_next) { - + for_each_vma_range(vmi, vma, end) { vmstart = max(start, vma->vm_start); vmend = min(end, vma->vm_end); new = mpol_dup(vma_policy(vma));