From patchwork Fri Aug 16 11:13:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bert Karwatzki X-Patchwork-Id: 13766001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BA80C3DA4A for ; Fri, 16 Aug 2024 11:14:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 713506B0151; Fri, 16 Aug 2024 07:14:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6967B6B0152; Fri, 16 Aug 2024 07:14:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 402C68D0070; Fri, 16 Aug 2024 07:14:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 164F26B0151 for ; Fri, 16 Aug 2024 07:14:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C41C6161ACF for ; Fri, 16 Aug 2024 11:14:34 +0000 (UTC) X-FDA: 82457850468.12.495BD65 Received: from mout.web.de (mout.web.de [217.72.192.78]) by imf13.hostedemail.com (Postfix) with ESMTP id C9E9420014 for ; Fri, 16 Aug 2024 11:14:32 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=web.de header.s=s29768273 header.b="Ct60w/Lw"; spf=pass (imf13.hostedemail.com: domain of spasswolf@web.de designates 217.72.192.78 as permitted sender) smtp.mailfrom=spasswolf@web.de; dmarc=pass (policy=quarantine) header.from=web.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723806836; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O94QRspLCb1Ter1jcWYZ79/yZW2oVM5cQef3hvy6a+o=; b=ttHHhyCJr4lpLYekYmm9nyK3emS71UvwW4Xc6KN92iv4K5ybWPqEXOc1qsgi4dozEMgrIw Ix/MZ08SAejtmW7HrcYbawXZTczZ4MYkLNjBbGXIl037fuPxC56f8eBATv/0LfZ+X0bps4 ggYKL1DQlz8SxSOw6G5Au0fwopZnJCQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=web.de header.s=s29768273 header.b="Ct60w/Lw"; spf=pass (imf13.hostedemail.com: domain of spasswolf@web.de designates 217.72.192.78 as permitted sender) smtp.mailfrom=spasswolf@web.de; dmarc=pass (policy=quarantine) header.from=web.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723806836; a=rsa-sha256; cv=none; b=h4vkouQF5evVNmVMOFUUoLYZF3ED1H2d99EZE9gK17/nS5pM20xYwqHFyC3mU1KikX+nG/ TkntudyC0V4p2UqQNGY2MxjLQKEwHKonMLUZa2oAX25mFdEqkCec33Jgvwhx0p2isbB3Qn A2Bs76BptogVi4wJDaok12z3cpmZlRo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=web.de; s=s29768273; t=1723806865; x=1724411665; i=spasswolf@web.de; bh=O94QRspLCb1Ter1jcWYZ79/yZW2oVM5cQef3hvy6a+o=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date:Message-ID:In-Reply-To: References:MIME-Version:Content-Transfer-Encoding:cc: content-transfer-encoding:content-type:date:from:message-id: mime-version:reply-to:subject:to; b=Ct60w/Lwqy4o/CiqVarKJpz/BZNpHxc8rnDQDS0jM2EggTw5K0icYoDzM5efaME1 PaAn0gr9OzHRMFbZKuFuKLaa/wXUPVZiUQtgVJUJ17n8LZ5kkgwpen1oBH/45R3+O k/dc0KnrP2wke/EsEH3k+GhMOgC5oNfaz5myT36Xzb/FPoLoN5DylhjH9oURL7FtW Ij13b8IO/CeO1UuhBGN+uoWOaht8dMZR9odCV021ddL25UclyPk7raZfE2MoE7jFy uDFBc8MwGcxvEskoOKhKy35ulalEsiLoXx+6MgZqGsB3pO1rCz2zclJykvNTEuRZU PIiLmAsygSzoOLxbrA== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Received: from localhost.localdomain ([84.119.92.193]) by smtp.web.de (mrweb106 [213.165.67.124]) with ESMTPSA (Nemesis) id 1M2Phc-1sb4d51SAw-00DZT2; Fri, 16 Aug 2024 13:14:25 +0200 From: Bert Karwatzki To: "Liam R . Howlett" Cc: Bert Karwatzki , Suren Baghdasaryan , Vlastimil Babka , Lorenzo Stoakes , Matthew Wilcox , sidhartha.kumar@oracle.com, "Paul E . McKenney" , Jiri Olsa , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Kees Cook , Jeff Xu , Lorenzo Stoakes Subject: [PATCH v5.1 05/19] mm/mmap: Introduce vma_munmap_struct for use in munmap operations Date: Fri, 16 Aug 2024 13:13:46 +0200 Message-ID: <20240816111405.11793-6-spasswolf@web.de> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240816111405.11793-1-spasswolf@web.de> References: <20240816111405.11793-1-spasswolf@web.de> MIME-Version: 1.0 X-Provags-ID: V03:K1:Xow6P1CPkv9JrFgOzkYeaVlz5A5aXI9IPW6ca4u3EDYV+mRkkdZ L6Dg2AQnE6KRQjW5Yp6edvS/sQZKPYTYy8A+LxRdVZ3eJo0j6GYc/K/7V/wG4rn/RHYJkyr 36rmbdVbCKbBsbYi8YC/jcUkt4uEeB2sz04s+YlXsEji5due2wWJcWhQOE6Ui83SPjUsSk5 xchSGYI5N5yH1HrU90Ftw== UI-OutboundReport: notjunk:1;M01:P0:11uR2nMLhjc=;FW3YfgVlt+qM5+KL6eDuIuCMv3a 3u6qUP9IgPVWokBcxNBGReXjgEaP1qMLkKfKehaspiexAQCFNUIbitE6lHu41gfXWiy9vmUsV fUbwNY/8yv8exex9+JnONdYa2BhSU8cYPAZMoUs/wp9qJJP19yuva/Lz4CoYNj/8MIr9TqRmI L1n1eFHZezNWQfKy9QZNJl6qaojp3THGq6kQtPeZvMPK5M6Ae6eI56XWLclkkZ53nCl7a39DE TLDp/2ylFG31K+g6LGNEj+XdyJ8oTcBTmp6aHhyQAaTTVZ/0YHovJ+v9nn66QCaZUfBIW8ZG3 OIJ9QryAOgoFIPP+4wg3u1fu4rcGKVWOz2tY5yPvyT6Zfwlj2tZNQk69QJK31RAOmLGTTS+uz w4VtjiOIYf+SpKzHCqTYePSeNI5itKUyVAewdXGD8QNcXtOytDTjLE7qMjcFOs6+JwRmMazjH +e6uj9dBomVOGqBk4XEt/Kc2Y05i03Ba0Q80PkQIpImpqdtWvECriwpfTwYXp0Xos1GO6J/Tv GgnR9YlmdCqVwyBqafdvDTlxcyqrys4ANXnvLwJsSQ2TQm6REMe2o/iHZGqcUJUWRBKXqF2Wq P5MwlQDrl1gChorq2bGDA0BBniDejITQPvdWbAtRV3PMlJ8cCD7qEsWyUxnPOnshJmk0xmPIq VOLNDDExZz08TPGRkk/kcIDbacU/0gIjOzpknlkfe36GxtB0NeG1oWg8yQsUH0qP8MlhV/DOU dpc2+F6kcVCIfwgVuxNAHG8dmGBZqehzowfBi9iSVQMVxJbrkf6CMbC4zRZHEeWkR93rp31Sy mZjrmgoLmAMkiQ3x6J7LdTdg== X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: C9E9420014 X-Stat-Signature: jwa75qgr1po3zrw7iq41nhmw9aj16dz3 X-HE-Tag: 1723806872-891069 X-HE-Meta: U2FsdGVkX1/OxnUBB2889dEPVheuFxdMcRTqH/ssJC9pwoIqI1a8YHURarcBLYcPTogKiB8x3U9Y7j9mcs64PQQrUSh107/1UvD3a7aOZTr++AzFSo01aPczDB3H7fN6EGxx/rKX1WO/x3ZKs/MNUprxFJhiVEtnE7NKOY4HIGoOosp+9PGaJ4xPwwmKaU6VUjm7RQ1StasDMKwVSU+F+aQ17HrCUkyBiyJFoVuC1B39hEfEvL7A662VTUEY6W9eSEnUNgrS8QlV1B2BiXYjmURmajK2nmkA77+18/tHQ35DWhBLUrWUdaRykmSBjBSsKco1Jp9vTq0zJrvZ3/W+JxS6fSOt3ff2dRDoPBbGBrctIsvnSddxAxDFu26qDksOHOkL0uq62n4WArar8wa1e+ErIlV5rfiqX13pNm+BvhnnYvEb8SgMq5d7xE2zSm3lCBQw8VZ5hwFmfNasLiziKTfDf2/fornclZy46WtYlX83y0CSfglAkR5aeLroNkNKhiY7dWzLTe3yEVnTckDBLZOChLoNqANwbXOjltUFPA4w+VlWDppHxOIBdORhnjXXTGFfb09wURSnxAek9FF1/IyxeIMNWhdu05CH+fU4lzwcOdn/9G1bqLIuvFP3ECtX645fW+1lQ6vFES+6OXRoAwrKxoyZ84tBnABITNS93IU/2jLTaKlzRzPzNNn05KAwHz8M57fh5JHaU9ScJJI7CezmrOPCIJTkhkXWPcfovTE9rjwk5lPfasyGHGGfxsYFD63MtmyqHES1TCFbRkGv2QM2waxBSvzSAy7pTK2MXFT7jwcaMrFuSUttdkaPKjLfvrXZCLz+3LFTIzvHVqqcObjRoR2xOuljXneOcQ0FORRDIsOXLezumb1qEqHeMLwCBnAlAcPnfJnR5/mXSS0jS0Ga2e5cXygtm5H4fli+aqDmnxhPcPdNX3gUzZ2/SykswMn65GhlnJcrcNoPEPn YL92w4xb PVfo/7IKvyuo1BcwwK601MkQzB67Kex0rJvWbxk5DxratzR1HMsCizm+fvZrzl2tCVpXVtOJZCLoEoNs5h38Uq0wK8V7H3OnKUsPjDp7ABGiFq/T/Rxhk+/mjYeaaxYiB7njZVGgHVZ8mN8oDvdppBz6Rke4v3xOlOS+TrOBbxk6MoMZJZwsZQFsVF+EJLtBRHYIEdTPOMDcN+YXvatn1VfExGepFmb4ZuRDlxJ0zEANRQ0sZTvmZrEg4/IBrOtCvJ4gMATshfZ1CstGBKSgXesiiTroNCSiUn822Pqp1HQ/lcsWZ7CJpmmkqM82f3ZHpCPL/Cw3YoEMXWGH4L/r6j4lsjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use a structure to pass along all the necessary information and counters involved in removing vmas from the mm_struct. Update vmi_ function names to vms_ to indicate the first argument type change. Signed-off-by: Liam R. Howlett Reviewed-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- mm/vma.c | 140 +++++++++++++++++++++++++++++-------------------------- mm/vma.h | 16 +++++++ 2 files changed, 90 insertions(+), 66 deletions(-) -- 2.45.2 diff --git a/mm/vma.c b/mm/vma.c index a980837eefd7..9495230df3c3 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -365,6 +365,32 @@ static void __vma_link_file(struct vm_area_struct *vma, flush_dcache_mmap_unlock(mapping); } +/* + * init_vma_munmap() - Initializer wrapper for vma_munmap_struct + * @vms: The vma munmap struct + * @vmi: The vma iterator + * @vma: The first vm_area_struct to munmap + * @start: The aligned start address to munmap + * @end: The aligned end address to munmap + * @uf: The userfaultfd list_head + * @unlock: Unlock after the operation. Only unlocked on success + */ +static inline void init_vma_munmap(struct vma_munmap_struct *vms, + struct vma_iterator *vmi, struct vm_area_struct *vma, + unsigned long start, unsigned long end, struct list_head *uf, + bool unlock) +{ + vms->vmi = vmi; + vms->vma = vma; + vms->mm = vma->vm_mm; + vms->start = start; + vms->end = end; + vms->unlock = unlock; + vms->uf = uf; + vms->vma_count = 0; + vms->nr_pages = vms->locked_vm = 0; +} + /* * vma_prepare() - Helper function for handling locking VMAs prior to altering * @vp: The initialized vma_prepare struct @@ -685,81 +711,63 @@ static inline void abort_munmap_vmas(struct ma_state *mas_detach) } /* - * vmi_complete_munmap_vmas() - Finish the munmap() operation - * @vmi: The vma iterator - * @vma: The first vma to be munmapped - * @mm: The mm struct - * @start: The start address - * @end: The end address - * @unlock: Unlock the mm or not - * @mas_detach: them maple state of the detached vma maple tree - * @locked_vm: The locked_vm count in the detached vmas + * vms_complete_munmap_vmas() - Finish the munmap() operation + * @vms: The vma munmap struct + * @mas_detach: The maple state of the detached vmas * - * This function updates the mm_struct, unmaps the region, frees the resources + * This updates the mm_struct, unmaps the region, frees the resources * used for the munmap() and may downgrade the lock - if requested. Everything * needed to be done once the vma maple tree is updated. */ -static void -vmi_complete_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, - struct mm_struct *mm, unsigned long start, - unsigned long end, bool unlock, struct ma_state *mas_detach, - unsigned long locked_vm) + +static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, + struct ma_state *mas_detach) { struct vm_area_struct *prev, *next; - int count; + struct mm_struct *mm; - count = mas_detach->index + 1; - mm->map_count -= count; - mm->locked_vm -= locked_vm; - if (unlock) + mm = vms->mm; + mm->map_count -= vms->vma_count; + mm->locked_vm -= vms->locked_vm; + if (vms->unlock) mmap_write_downgrade(mm); - prev = vma_iter_prev_range(vmi); - next = vma_next(vmi); + prev = vma_iter_prev_range(vms->vmi); + next = vma_next(vms->vmi); if (next) - vma_iter_prev_range(vmi); + vma_iter_prev_range(vms->vmi); /* * We can free page tables without write-locking mmap_lock because VMAs * were isolated before we downgraded mmap_lock. */ mas_set(mas_detach, 1); - unmap_region(mm, mas_detach, vma, prev, next, start, end, count, - !unlock); + unmap_region(mm, mas_detach, vms->vma, prev, next, vms->start, vms->end, + vms->vma_count, !vms->unlock); /* Statistics and freeing VMAs */ mas_set(mas_detach, 0); remove_mt(mm, mas_detach); validate_mm(mm); - if (unlock) + if (vms->unlock) mmap_read_unlock(mm); __mt_destroy(mas_detach->tree); } /* - * vmi_gather_munmap_vmas() - Put all VMAs within a range into a maple tree + * vms_gather_munmap_vmas() - Put all VMAs within a range into a maple tree * for removal at a later date. Handles splitting first and last if necessary * and marking the vmas as isolated. * - * @vmi: The vma iterator - * @vma: The starting vm_area_struct - * @mm: The mm_struct - * @start: The aligned start address to munmap. - * @end: The aligned end address to munmap. - * @uf: The userfaultfd list_head + * @vms: The vma munmap struct * @mas_detach: The maple state tracking the detached tree - * @locked_vm: a pointer to store the VM_LOCKED pages count. * * Return: 0 on success */ -static int -vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, - struct mm_struct *mm, unsigned long start, - unsigned long end, struct list_head *uf, - struct ma_state *mas_detach, unsigned long *locked_vm) +static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms, + struct ma_state *mas_detach) { struct vm_area_struct *next = NULL; - int count = 0; int error = -ENOMEM; /* @@ -771,17 +779,18 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, */ /* Does it split the first one? */ - if (start > vma->vm_start) { + if (vms->start > vms->vma->vm_start) { /* * Make sure that map_count on return from munmap() will * not exceed its limit; but let map_count go just above * its limit temporarily, to help free resources as expected. */ - if (end < vma->vm_end && mm->map_count >= sysctl_max_map_count) + if (vms->end < vms->vma->vm_end && + vms->mm->map_count >= sysctl_max_map_count) goto map_count_exceeded; - error = __split_vma(vmi, vma, start, 1); + error = __split_vma(vms->vmi, vms->vma, vms->start, 1); if (error) goto start_split_failed; } @@ -790,25 +799,24 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, * Detach a range of VMAs from the mm. Using next as a temp variable as * it is always overwritten. */ - next = vma; + next = vms->vma; do { /* Does it split the end? */ - if (next->vm_end > end) { - error = __split_vma(vmi, next, end, 0); + if (next->vm_end > vms->end) { + error = __split_vma(vms->vmi, next, vms->end, 0); if (error) goto end_split_failed; } vma_start_write(next); - mas_set(mas_detach, count++); + mas_set(mas_detach, vms->vma_count++); error = mas_store_gfp(mas_detach, next, GFP_KERNEL); if (error) goto munmap_gather_failed; vma_mark_detached(next, true); if (next->vm_flags & VM_LOCKED) - *locked_vm += vma_pages(next); + vms->locked_vm += vma_pages(next); - count++; - if (unlikely(uf)) { + if (unlikely(vms->uf)) { /* * If userfaultfd_unmap_prep returns an error the vmas * will remain split, but userland will get a @@ -818,16 +826,17 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, * split, despite we could. This is unlikely enough * failure that it's not worth optimizing it for. */ - error = userfaultfd_unmap_prep(next, start, end, uf); + error = userfaultfd_unmap_prep(next, vms->start, + vms->end, vms->uf); if (error) goto userfaultfd_error; } #ifdef CONFIG_DEBUG_VM_MAPLE_TREE - BUG_ON(next->vm_start < start); - BUG_ON(next->vm_start > end); + BUG_ON(next->vm_start < vms->start); + BUG_ON(next->vm_start > vms->end); #endif - } for_each_vma_range(*vmi, next, end); + } for_each_vma_range(*(vms->vmi), next, vms->end); #if defined(CONFIG_DEBUG_VM_MAPLE_TREE) /* Make sure no VMAs are about to be lost. */ @@ -836,21 +845,21 @@ vmi_gather_munmap_vmas(struct vma_iterator *vmi, struct vm_area_struct *vma, struct vm_area_struct *vma_mas, *vma_test; int test_count = 0; - vma_iter_set(vmi, start); + vma_iter_set(vms->vmi, vms->start); rcu_read_lock(); - vma_test = mas_find(&test, count - 1); - for_each_vma_range(*vmi, vma_mas, end) { + vma_test = mas_find(&test, vms->vma_count - 1); + for_each_vma_range(*(vms->vmi), vma_mas, vms->end) { BUG_ON(vma_mas != vma_test); test_count++; - vma_test = mas_next(&test, count - 1); + vma_test = mas_next(&test, vms->vma_count - 1); } rcu_read_unlock(); - BUG_ON(count != test_count); + BUG_ON(vms->vma_count != test_count); } #endif - while (vma_iter_addr(vmi) > start) - vma_iter_prev_range(vmi); + while (vma_iter_addr(vms->vmi) > vms->start) + vma_iter_prev_range(vms->vmi); return 0; @@ -886,11 +895,11 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, MA_STATE(mas_detach, &mt_detach, 0, 0); mt_init_flags(&mt_detach, vmi->mas.tree->ma_flags & MT_FLAGS_LOCK_MASK); mt_on_stack(mt_detach); + struct vma_munmap_struct vms; int error; - unsigned long locked_vm = 0; - error = vmi_gather_munmap_vmas(vmi, vma, mm, start, end, uf, - &mas_detach, &locked_vm); + init_vma_munmap(&vms, vmi, vma, start, end, uf, unlock); + error = vms_gather_munmap_vmas(&vms, &mas_detach); if (error) goto gather_failed; @@ -899,8 +908,7 @@ do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, goto clear_tree_failed; /* Point of no return */ - vmi_complete_munmap_vmas(vmi, vma, mm, start, end, unlock, &mas_detach, - locked_vm); + vms_complete_munmap_vmas(&vms, &mas_detach); return 0; clear_tree_failed: diff --git a/mm/vma.h b/mm/vma.h index 6efdf1768a0a..f65c739cbd00 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -21,6 +21,22 @@ struct vma_prepare { struct vm_area_struct *remove2; }; +/* + * vma munmap operation + */ +struct vma_munmap_struct { + struct vma_iterator *vmi; + struct mm_struct *mm; + struct vm_area_struct *vma; /* The first vma to munmap */ + struct list_head *uf; /* Userfaultfd list_head */ + unsigned long start; /* Aligned start addr (inclusive) */ + unsigned long end; /* Aligned end addr (exclusive) */ + int vma_count; /* Number of vmas that will be removed */ + unsigned long nr_pages; /* Number of pages being removed */ + unsigned long locked_vm; /* Number of locked pages */ + bool unlock; /* Unlock after the munmap */ +}; + struct unlink_vma_file_batch { int count; struct vm_area_struct *vmas[8];