From patchwork Fri Mar 17 21:58:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 13179524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C7CC6FD1D for ; Fri, 17 Mar 2023 21:58:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 787546B0080; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 735DE6B0081; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 588F06B0082; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4AA746B0080 for ; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 16B18C06ED for ; Fri, 17 Mar 2023 21:58:37 +0000 (UTC) X-FDA: 80579755074.07.31CCED0 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by imf21.hostedemail.com (Postfix) with ESMTP id 2183B1C0018 for ; Fri, 17 Mar 2023 21:58:34 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="ZTWp/dwX"; spf=pass (imf21.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679090315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0WRGTUCFXkHGDXIpCCchpoD4urIMZfIjv9HBnqnD3s8=; b=BHaV4Y4JyT/KGUTAvaczOjY0nGeuSQgPzihdybflZS8r/3aXyE+E5ms4SUkHGLK9o/NUIG +RrwcB3tE8/ngW1RYWvaixLF83djExrGqsseiaf4tfFawPv7ir8ddiA8U8206ycuqYrD7o h25mI1lKWpB2lv/OoONpOJdvpMBPxJg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="ZTWp/dwX"; spf=pass (imf21.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679090315; a=rsa-sha256; cv=none; b=RueX535t6IhVtjWziTpv5+9Nw4bukCVqr9zJAIXrWXCzFUGZiStZ5L3x8GfkoN33kUXSEL 9Yt8pCdDn0GqaZulsF4UbsEpq+ZwIvGzR3CYlrLthvChfm+GQWo5FNI6/Nkt/YZwhvsicj eifiBadZSFv7J8SvBzhUNvD56OMzi1o= Received: by mail-wr1-f43.google.com with SMTP id r29so5622095wra.13 for ; Fri, 17 Mar 2023 14:58:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679090313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0WRGTUCFXkHGDXIpCCchpoD4urIMZfIjv9HBnqnD3s8=; b=ZTWp/dwXxkIJIy+sbetjHxGbG0sO2V60Lc6mgkm/ewopNB3etrn+nimKaBs9LZa5oK s09Hb5HBEPk6X2C62wDwpIYPR5YI48oITA9/DKyfpdaWn8BwBJalIS6Ey+pZPsCJt3Io hpucBdirZtxnd9VMFC4qtthIk6X5y0S03dKANyqGe4sgtSpgwVAbnGedB2t6t+Fe/B/a N27QqhyVa7oEnyqFzwfh3ysCWlZrg90Y7BiV246iEUpdMlwGY8VYLSC2Zg4Lwef5+2oA 2oyXGyEm1ndLdBbzbbS/zNzczh1IrcNT4Safuzctvy7quwGA6/wrUpwLBF2Ychrgqyb/ kreg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679090313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0WRGTUCFXkHGDXIpCCchpoD4urIMZfIjv9HBnqnD3s8=; b=KbvAK+vi4xBcNwcstSQOj5+HnYrVbNYf/OQpqi2IsolTMmP6j91QTQe22c08QL2EWM Dk+hpOSdNRIbPNE+HXUw4RA5TgOBXDpWhJqlOt4fERRczUzjfKb1AhC0CDcZfelSWb+9 8JhCioBA8TpzqQ7vP25MKFaTaqwX2P087+0FavHrHzXz/ni054D+DVMI1T+zM2oye0j0 fM04LAOlCGF+dRmWJfbYncmrApAbmNV5DxLD+6A3EPdkoBmTdFKx+XhHk6FpikXGuBnM wBTz1xqygVw9xvggiIgXPdQ6spnWG/HG45En8000hXHiPT+CAUV963AkygPSrEr8Comh GoOg== X-Gm-Message-State: AO0yUKU8Fg7FRb5eyBf/LCKbtLTPMOfKRMjbUtqfewe5DPLlcemfleEf CbdD9fKBVfc2Ti2f87jpuZndiwPbtHM= X-Google-Smtp-Source: AK7set8pr8elo4x2KKbOBJVOAwcf4Xa9fosjB+N4AOwqHw6oQ4zMx+Q4dkmlOhI3zZUFpRUaUI8ZTA== X-Received: by 2002:adf:e8cd:0:b0:2ca:175b:d850 with SMTP id k13-20020adfe8cd000000b002ca175bd850mr7718855wrn.11.1679090313217; Fri, 17 Mar 2023 14:58:33 -0700 (PDT) Received: from lucifer.home (host86-146-209-214.range86-146.btcentralplus.com. [86.146.209.214]) by smtp.googlemail.com with ESMTPSA id c9-20020a5d4cc9000000b002d21379bcabsm2849757wrt.110.2023.03.17.14.58.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Mar 2023 14:58:32 -0700 (PDT) From: Lorenzo Stoakes To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes Subject: [PATCH 1/2] mm: refactor do_fault_around() Date: Fri, 17 Mar 2023 21:58:25 +0000 Message-Id: X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2183B1C0018 X-Stat-Signature: xw5yku4ta7crtuydf1ffn7nh31ajfccq X-HE-Tag: 1679090314-816043 X-HE-Meta: U2FsdGVkX1/krD6E95ROieTtEuIKPdVmbyL1hFMCE4e/zuYGr+A7LGbT5/f2HL+yotvE3ze6bpqBrlp5CqVRrEtbF29wAqjEPAKoJf5NF7jVlUeCHhN7Zd8sTDxFiempi3mrtTRWudOak30p8FMdWm1UztwImo5V/5PAReVs4tanJGM3l6GxY/5ofWgPEaEb3zfRVJ2vWzicp10kbFPUbdz+QdIM8x3gEBIu9ot/FbV6RBx3qSdIgaKxgD8P2o82Svw+OiOBavZ0yjHfZGCK+IZA0uvdGLGehFA0psPSQcJkNr0c1p2xT0F6pm6RTN9Uw2HvVX4J+HAR2r6xv+47xBtgrnNpGfIRH+j9QXZJjqPlOzD6Zl0Ob8kawMcx4qVT1TavpA8TBEa2gSxFEoq+rmp8RXTqwkJVqgt0DRHqECSody/dhFWgGoHvuuY+E9L4EuXtLpynlEiq2A8m6jACs2HccrvpjD4WundC2ovOvhxtwvzTZRiF18NGoy7MYveTZ32B1esDZgsqOO3fiBCLcBBqpqXz5ZahZx4rgHaNFJDkDx7qeLUGl0i/HamdFws2tphxZyrJkgOmDn0NdlBxiaBerTb/tqGZGrbVWmwQmGJfiz3zgALN2xO8zNm+JAssU/nVOqnZHdJUgLFgct3+Lj7njpmDtsnJM8J2l/vr7NFe/xNR7g/UTx7kMSNvI1X1sRBcbHJioqM9+OfJZhyTzaAwo+Ru5YG76MMeqwQW4RR/ZfDbCDtoC7MnLJ9xzF5RHS6XeyVKyTrjzSDK/CN5Tf/JSxBhhjPupIt6BkYMK99mQkkNQbVjbf9cB5TQucBmXmAz7rSVgtlphhEw678SCUZEGbReHz5e+8Xc4jXYN7N7H53EfCk3DGi8ltsX1RkFbItSFpsNXqefGCKr+mXOM249Kza5DtIi4v46uGuc1l6x0vUj8QgkNP1AtTzx5tzj+S+jUuoCYDs1cWYGNy5 Ing67KRa fxlbxNWBeG0yjEC55flb47/KaCzHs38Y6yJVdwNp6opiL5bf1lMoYQUPl3FQyFR7ZSUWDQGud5rtOggnle656bb7wyhmt6gLZK5XohDaU60ZKNfKEpFRHNjwjgwBPaloZxP/ri5fhWB0OuZhvJpp1ZP2pyWltmDl3l0MHlW8roiqGEQ952MW8T5n02V8y/fZ/xO/lGcInzk1NGqZfVTMdYX8mVDPtRh1SVSpGJBpgQ5qIL3Q/jbbW2a5smgp/ipa6XxG0/14A1cSB+Sr0DC5maLY6kcdF/FhXYlkhCK98uK8+UH+MW2DPKVglQm2bJz0uvfAzpkFubWdeo3Qs1uL36SaNLveizQFKYbrfdDbOeYq+NY7x7pp7BIBcgS/YxkjkRpDSxSzoQ6zKsCmhOCgL32oIg0U2bsxVIstn58JlvisGQp28mIJkr74L7nr1sA2B8p16F9v07MXzJ0GJ6DBvajnWfiimyY/1VONjhgA7mvFyClh0C14JoqvPP2G2AoWdptK1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The existing logic is confusing and fails to abstract a number of bitwise tricks. Use ALIGN_DOWN() to perform alignment, pte_index() to obtain a PTE index and represent the address range using PTE offsets, which naturally make it clear that the operation is intended to occur within only a single PTE and prevent spanning of more than one page table. We rely on the fact that fault_around_bytes will always be page-aligned, at least one page in size, a power of two and that it will not exceed PAGE_SIZE * PTRS_PER_PTE in size (i.e. the address space mapped by a PTE). These are all guaranteed by fault_around_bytes_set(). Signed-off-by: Lorenzo Stoakes --- mm/memory.c | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c5f1bf906d0c..3d85aa7106b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4437,8 +4437,8 @@ late_initcall(fault_around_debugfs); * It uses vm_ops->map_pages() to map the pages, which skips the page if it's * not ready to be mapped: not up-to-date, locked, etc. * - * This function doesn't cross the VMA boundaries, in order to call map_pages() - * only once. + * This function doesn't cross VMA or page table boundaries, in order to call + * map_pages() and acquire a PTE lock only once. * * fault_around_bytes defines how many bytes we'll try to map. * do_fault_around() expects it to be set to a power of two less than or equal @@ -4451,27 +4451,19 @@ late_initcall(fault_around_debugfs); */ static vm_fault_t do_fault_around(struct vm_fault *vmf) { - unsigned long address = vmf->address, nr_pages, mask; - pgoff_t start_pgoff = vmf->pgoff; - pgoff_t end_pgoff; - int off; + pgoff_t nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; + pgoff_t pte_off = pte_index(vmf->address); + /* The page offset of vmf->address within the VMA. */ + pgoff_t vma_off = vmf->pgoff - vmf->vma->vm_pgoff; + pgoff_t from_pte, to_pte; - nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; - mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK; + /* The PTE offset of the start address, clamped to the VMA. */ + from_pte = max(ALIGN_DOWN(pte_off, nr_pages), + pte_off - min(pte_off, vma_off)); - address = max(address & mask, vmf->vma->vm_start); - off = ((vmf->address - address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1); - start_pgoff -= off; - - /* - * end_pgoff is either the end of the page table, the end of - * the vma or nr_pages from start_pgoff, depending what is nearest. - */ - end_pgoff = start_pgoff - - ((address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) + - PTRS_PER_PTE - 1; - end_pgoff = min3(end_pgoff, vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1, - start_pgoff + nr_pages - 1); + /* The PTE offset of the end address, clamped to the VMA and PTE. */ + to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE, + pte_off + vma_pages(vmf->vma) - vma_off) - 1; if (pmd_none(*vmf->pmd)) { vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm); @@ -4479,7 +4471,9 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) return VM_FAULT_OOM; } - return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); + return vmf->vma->vm_ops->map_pages(vmf, + vmf->pgoff + from_pte - pte_off, + vmf->pgoff + to_pte - pte_off); } /* Return true if we should do read fault-around, false otherwise */