From patchwork Fri Mar 17 21:58:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 13179524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C7CC6FD1D for ; Fri, 17 Mar 2023 21:58:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 787546B0080; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 735DE6B0081; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 588F06B0082; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4AA746B0080 for ; Fri, 17 Mar 2023 17:58:37 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 16B18C06ED for ; Fri, 17 Mar 2023 21:58:37 +0000 (UTC) X-FDA: 80579755074.07.31CCED0 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by imf21.hostedemail.com (Postfix) with ESMTP id 2183B1C0018 for ; Fri, 17 Mar 2023 21:58:34 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="ZTWp/dwX"; spf=pass (imf21.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679090315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0WRGTUCFXkHGDXIpCCchpoD4urIMZfIjv9HBnqnD3s8=; b=BHaV4Y4JyT/KGUTAvaczOjY0nGeuSQgPzihdybflZS8r/3aXyE+E5ms4SUkHGLK9o/NUIG +RrwcB3tE8/ngW1RYWvaixLF83djExrGqsseiaf4tfFawPv7ir8ddiA8U8206ycuqYrD7o h25mI1lKWpB2lv/OoONpOJdvpMBPxJg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="ZTWp/dwX"; spf=pass (imf21.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.43 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679090315; a=rsa-sha256; cv=none; b=RueX535t6IhVtjWziTpv5+9Nw4bukCVqr9zJAIXrWXCzFUGZiStZ5L3x8GfkoN33kUXSEL 9Yt8pCdDn0GqaZulsF4UbsEpq+ZwIvGzR3CYlrLthvChfm+GQWo5FNI6/Nkt/YZwhvsicj eifiBadZSFv7J8SvBzhUNvD56OMzi1o= Received: by mail-wr1-f43.google.com with SMTP id r29so5622095wra.13 for ; Fri, 17 Mar 2023 14:58:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679090313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0WRGTUCFXkHGDXIpCCchpoD4urIMZfIjv9HBnqnD3s8=; b=ZTWp/dwXxkIJIy+sbetjHxGbG0sO2V60Lc6mgkm/ewopNB3etrn+nimKaBs9LZa5oK s09Hb5HBEPk6X2C62wDwpIYPR5YI48oITA9/DKyfpdaWn8BwBJalIS6Ey+pZPsCJt3Io hpucBdirZtxnd9VMFC4qtthIk6X5y0S03dKANyqGe4sgtSpgwVAbnGedB2t6t+Fe/B/a N27QqhyVa7oEnyqFzwfh3ysCWlZrg90Y7BiV246iEUpdMlwGY8VYLSC2Zg4Lwef5+2oA 2oyXGyEm1ndLdBbzbbS/zNzczh1IrcNT4Safuzctvy7quwGA6/wrUpwLBF2Ychrgqyb/ kreg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679090313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0WRGTUCFXkHGDXIpCCchpoD4urIMZfIjv9HBnqnD3s8=; b=KbvAK+vi4xBcNwcstSQOj5+HnYrVbNYf/OQpqi2IsolTMmP6j91QTQe22c08QL2EWM Dk+hpOSdNRIbPNE+HXUw4RA5TgOBXDpWhJqlOt4fERRczUzjfKb1AhC0CDcZfelSWb+9 8JhCioBA8TpzqQ7vP25MKFaTaqwX2P087+0FavHrHzXz/ni054D+DVMI1T+zM2oye0j0 fM04LAOlCGF+dRmWJfbYncmrApAbmNV5DxLD+6A3EPdkoBmTdFKx+XhHk6FpikXGuBnM wBTz1xqygVw9xvggiIgXPdQ6spnWG/HG45En8000hXHiPT+CAUV963AkygPSrEr8Comh GoOg== X-Gm-Message-State: AO0yUKU8Fg7FRb5eyBf/LCKbtLTPMOfKRMjbUtqfewe5DPLlcemfleEf CbdD9fKBVfc2Ti2f87jpuZndiwPbtHM= X-Google-Smtp-Source: AK7set8pr8elo4x2KKbOBJVOAwcf4Xa9fosjB+N4AOwqHw6oQ4zMx+Q4dkmlOhI3zZUFpRUaUI8ZTA== X-Received: by 2002:adf:e8cd:0:b0:2ca:175b:d850 with SMTP id k13-20020adfe8cd000000b002ca175bd850mr7718855wrn.11.1679090313217; Fri, 17 Mar 2023 14:58:33 -0700 (PDT) Received: from lucifer.home (host86-146-209-214.range86-146.btcentralplus.com. [86.146.209.214]) by smtp.googlemail.com with ESMTPSA id c9-20020a5d4cc9000000b002d21379bcabsm2849757wrt.110.2023.03.17.14.58.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Mar 2023 14:58:32 -0700 (PDT) From: Lorenzo Stoakes To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes Subject: [PATCH 1/2] mm: refactor do_fault_around() Date: Fri, 17 Mar 2023 21:58:25 +0000 Message-Id: X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2183B1C0018 X-Stat-Signature: xw5yku4ta7crtuydf1ffn7nh31ajfccq X-HE-Tag: 1679090314-816043 X-HE-Meta: U2FsdGVkX1/krD6E95ROieTtEuIKPdVmbyL1hFMCE4e/zuYGr+A7LGbT5/f2HL+yotvE3ze6bpqBrlp5CqVRrEtbF29wAqjEPAKoJf5NF7jVlUeCHhN7Zd8sTDxFiempi3mrtTRWudOak30p8FMdWm1UztwImo5V/5PAReVs4tanJGM3l6GxY/5ofWgPEaEb3zfRVJ2vWzicp10kbFPUbdz+QdIM8x3gEBIu9ot/FbV6RBx3qSdIgaKxgD8P2o82Svw+OiOBavZ0yjHfZGCK+IZA0uvdGLGehFA0psPSQcJkNr0c1p2xT0F6pm6RTN9Uw2HvVX4J+HAR2r6xv+47xBtgrnNpGfIRH+j9QXZJjqPlOzD6Zl0Ob8kawMcx4qVT1TavpA8TBEa2gSxFEoq+rmp8RXTqwkJVqgt0DRHqECSody/dhFWgGoHvuuY+E9L4EuXtLpynlEiq2A8m6jACs2HccrvpjD4WundC2ovOvhxtwvzTZRiF18NGoy7MYveTZ32B1esDZgsqOO3fiBCLcBBqpqXz5ZahZx4rgHaNFJDkDx7qeLUGl0i/HamdFws2tphxZyrJkgOmDn0NdlBxiaBerTb/tqGZGrbVWmwQmGJfiz3zgALN2xO8zNm+JAssU/nVOqnZHdJUgLFgct3+Lj7njpmDtsnJM8J2l/vr7NFe/xNR7g/UTx7kMSNvI1X1sRBcbHJioqM9+OfJZhyTzaAwo+Ru5YG76MMeqwQW4RR/ZfDbCDtoC7MnLJ9xzF5RHS6XeyVKyTrjzSDK/CN5Tf/JSxBhhjPupIt6BkYMK99mQkkNQbVjbf9cB5TQucBmXmAz7rSVgtlphhEw678SCUZEGbReHz5e+8Xc4jXYN7N7H53EfCk3DGi8ltsX1RkFbItSFpsNXqefGCKr+mXOM249Kza5DtIi4v46uGuc1l6x0vUj8QgkNP1AtTzx5tzj+S+jUuoCYDs1cWYGNy5 Ing67KRa fxlbxNWBeG0yjEC55flb47/KaCzHs38Y6yJVdwNp6opiL5bf1lMoYQUPl3FQyFR7ZSUWDQGud5rtOggnle656bb7wyhmt6gLZK5XohDaU60ZKNfKEpFRHNjwjgwBPaloZxP/ri5fhWB0OuZhvJpp1ZP2pyWltmDl3l0MHlW8roiqGEQ952MW8T5n02V8y/fZ/xO/lGcInzk1NGqZfVTMdYX8mVDPtRh1SVSpGJBpgQ5qIL3Q/jbbW2a5smgp/ipa6XxG0/14A1cSB+Sr0DC5maLY6kcdF/FhXYlkhCK98uK8+UH+MW2DPKVglQm2bJz0uvfAzpkFubWdeo3Qs1uL36SaNLveizQFKYbrfdDbOeYq+NY7x7pp7BIBcgS/YxkjkRpDSxSzoQ6zKsCmhOCgL32oIg0U2bsxVIstn58JlvisGQp28mIJkr74L7nr1sA2B8p16F9v07MXzJ0GJ6DBvajnWfiimyY/1VONjhgA7mvFyClh0C14JoqvPP2G2AoWdptK1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The existing logic is confusing and fails to abstract a number of bitwise tricks. Use ALIGN_DOWN() to perform alignment, pte_index() to obtain a PTE index and represent the address range using PTE offsets, which naturally make it clear that the operation is intended to occur within only a single PTE and prevent spanning of more than one page table. We rely on the fact that fault_around_bytes will always be page-aligned, at least one page in size, a power of two and that it will not exceed PAGE_SIZE * PTRS_PER_PTE in size (i.e. the address space mapped by a PTE). These are all guaranteed by fault_around_bytes_set(). Signed-off-by: Lorenzo Stoakes --- mm/memory.c | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c5f1bf906d0c..3d85aa7106b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4437,8 +4437,8 @@ late_initcall(fault_around_debugfs); * It uses vm_ops->map_pages() to map the pages, which skips the page if it's * not ready to be mapped: not up-to-date, locked, etc. * - * This function doesn't cross the VMA boundaries, in order to call map_pages() - * only once. + * This function doesn't cross VMA or page table boundaries, in order to call + * map_pages() and acquire a PTE lock only once. * * fault_around_bytes defines how many bytes we'll try to map. * do_fault_around() expects it to be set to a power of two less than or equal @@ -4451,27 +4451,19 @@ late_initcall(fault_around_debugfs); */ static vm_fault_t do_fault_around(struct vm_fault *vmf) { - unsigned long address = vmf->address, nr_pages, mask; - pgoff_t start_pgoff = vmf->pgoff; - pgoff_t end_pgoff; - int off; + pgoff_t nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; + pgoff_t pte_off = pte_index(vmf->address); + /* The page offset of vmf->address within the VMA. */ + pgoff_t vma_off = vmf->pgoff - vmf->vma->vm_pgoff; + pgoff_t from_pte, to_pte; - nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; - mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK; + /* The PTE offset of the start address, clamped to the VMA. */ + from_pte = max(ALIGN_DOWN(pte_off, nr_pages), + pte_off - min(pte_off, vma_off)); - address = max(address & mask, vmf->vma->vm_start); - off = ((vmf->address - address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1); - start_pgoff -= off; - - /* - * end_pgoff is either the end of the page table, the end of - * the vma or nr_pages from start_pgoff, depending what is nearest. - */ - end_pgoff = start_pgoff - - ((address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) + - PTRS_PER_PTE - 1; - end_pgoff = min3(end_pgoff, vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1, - start_pgoff + nr_pages - 1); + /* The PTE offset of the end address, clamped to the VMA and PTE. */ + to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE, + pte_off + vma_pages(vmf->vma) - vma_off) - 1; if (pmd_none(*vmf->pmd)) { vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm); @@ -4479,7 +4471,9 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) return VM_FAULT_OOM; } - return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); + return vmf->vma->vm_ops->map_pages(vmf, + vmf->pgoff + from_pte - pte_off, + vmf->pgoff + to_pte - pte_off); } /* Return true if we should do read fault-around, false otherwise */ From patchwork Fri Mar 17 21:58:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 13179525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA9CDC74A5B for ; Fri, 17 Mar 2023 21:58:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 674E96B0081; Fri, 17 Mar 2023 17:58:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FD846B0082; Fri, 17 Mar 2023 17:58:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49EB56B0083; Fri, 17 Mar 2023 17:58:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3B5C86B0081 for ; Fri, 17 Mar 2023 17:58:39 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 06C0B40403 for ; Fri, 17 Mar 2023 21:58:39 +0000 (UTC) X-FDA: 80579755158.30.7D62019 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by imf11.hostedemail.com (Postfix) with ESMTP id 2733A4000C for ; Fri, 17 Mar 2023 21:58:36 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="p0sz2Q/6"; spf=pass (imf11.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679090317; a=rsa-sha256; cv=none; b=LdGFKPr0kP6ASUg+YXXrD4hWGsa6X22vpnn1xB5TDp9mxEgwexsIjRA7Sq0w0HbytHy8Jy FnyYetYuLzfhbRrdUK0fWL5a+rqZDDBgIuOFXtkvnHhHMQMFdPTVF8B39HjwPwRUFLU/xy MsQUXjyJuhs3WemDpTAnZY8JX8UI1Ns= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="p0sz2Q/6"; spf=pass (imf11.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679090317; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E73zG66hPCj2y2LLQHqVtdvTmlQn/mfxXz04vMaqlmQ=; b=EnwlSjI1k9kuoXpdWoCpVwV1fHG1ceshjacD7vbO1l+YwS5Tlxf4b9WUtAyheg7y1DuRTE y3mwsVsDOUIg0XwRChCy6SkpoG1g6CaYj6kKi1vi4RBG1IKrLiOKJEfJBwDqqByXnS+nA7 OyEjwabE1Wo43lyDm6k+ZfN/covJTrw= Received: by mail-wr1-f53.google.com with SMTP id i9so5647971wrp.3 for ; Fri, 17 Mar 2023 14:58:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679090315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E73zG66hPCj2y2LLQHqVtdvTmlQn/mfxXz04vMaqlmQ=; b=p0sz2Q/60JExrD8KLivrs4RisQT/t4knswKiVldLNc181jBlMOow+MU/gMnc4CDC16 g3ISFEdAk7O6o1WyLtOOu7w61OMFjRDGqPRR/1Y84rL1fl9f9dBAroADdsvGkx1pVXP9 6p+wOqskB3Kyl9nHhD/WfUx6Tv1gSBBHI0KzizJ6fn/kIxUjOhB6kaoVDaeR1b6uQHzb suEORJV8evSP7NUNk+sEywUCvlA5pJoutPivMPazdNJx2ogfxvsqQYH8TWVpGyfn+29z 9EIpmSwCC74XqZaDtXAHCn/yPhr/yV2ow5ve2TmN3mABWz0rk6fE23lFhM+dIm5q50zl YXKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679090315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E73zG66hPCj2y2LLQHqVtdvTmlQn/mfxXz04vMaqlmQ=; b=A4H77FM5sBRUbPXfmzWxXJcdZX3KZO0jX1L+szCWQsvRn3d78KKrbd34aOi3Eye/hv RzZgZTXiauUMl0ii0d2ptL75rVpyLG2aZFRxT9cfpRmVZemE+BkWHoCLvNxUHmot+r8+ yh2CCEjZ8doTsfYccdjQuDxyKouoQA/bwZMtTUKwMPVkTQNs5AKwmS4u+Wj0o569fnoU fjVgjSzIhNLMj3wVYq45AotRyh7XV+JB7V0zGOFvB7e9CHGVWqBCB/yHa6mjBNydJ5jq adi6Y6dK5PerizH3W8pOEzJ3jG8MJymN4YB4Zpc2EtqCtamgUw3UApaMPaY9lZA62as7 +0qQ== X-Gm-Message-State: AO0yUKXcrQj73PM3ry41oAXB4Qj4FHqloPFxH/3vTukDGBrlupfSz11C XqsuFOcXj/OZXNThqyoGSCwPY3qgMxQ= X-Google-Smtp-Source: AK7set85J+fF2ekkIZVxIUpMfYu6GSJeUFX4eACMqsfBv9LTucAUS/0tN4bnCc4vBud9r4nwdVk3jg== X-Received: by 2002:a5d:638b:0:b0:2cf:f312:6e60 with SMTP id p11-20020a5d638b000000b002cff3126e60mr7547549wru.21.1679090315367; Fri, 17 Mar 2023 14:58:35 -0700 (PDT) Received: from lucifer.home (host86-146-209-214.range86-146.btcentralplus.com. [86.146.209.214]) by smtp.googlemail.com with ESMTPSA id c9-20020a5d4cc9000000b002d21379bcabsm2849757wrt.110.2023.03.17.14.58.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Mar 2023 14:58:34 -0700 (PDT) From: Lorenzo Stoakes To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes Subject: [PATCH 2/2] mm: pefer fault_around_pages to fault_around_bytes Date: Fri, 17 Mar 2023 21:58:26 +0000 Message-Id: <4995bad07fe9baa51c786fa0d81819dddfb57654.1679089214.git.lstoakes@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 2733A4000C X-Rspamd-Server: rspam01 X-Stat-Signature: decey38o71byykk5u137k4749sxwkwzo X-HE-Tag: 1679090316-4758 X-HE-Meta: U2FsdGVkX19+2tE4xuEzVsLouwKIFiiGaNvjLonUTTpsmr0mntKLG547uXJuIHAyTKdufGM5Zcyl0S42S7a60oo0MlrAFusq6LPhm2L1xIL3O6CmAc6odX6eDw6tDGXBXUgGrPO3DqZ+lohPzKRbcCtoPMxP0/m/mxu03Ms23MgUJXRl/pagsn9l58K69/0qRRh4A5cTT6ipXNIGOgar+wxJQyayTBkpZ0l5IpOoDz7QV8NsxS9z4Y8rX4A5wYV1V7mp5JyqMIqHlQ4/FvwTTrJq7dMk713mX93OEqNpSHXjs3jrJb+aNk2fPA2Ho0KzbCznLi7qZpQEXYrTddeL/LBYpfJOMrkN2VckVjQJQ1YWHFnmUEwsnpMdW59AK8jd8hq6RvdJ22mmpehOOs1WYgh4RXawCCeYuaNdvsb4n5IXsW/XI4GAiFnBGbUX4uzFddZ6FslIUbElsRMV3lyJWUURLIUsfbGpxSU6zxToL3YlMHwPOqKBoMOOCG3z3D4XRdolBdL3akG0ScefgdB3EY5xhJMmcg5sAt344yKmq6tuQd8ASjEOFAjB13romqvnuXYVprxADygRSj7GmY7HUrNYzP+JtMrVfBhRajAy9N53DQGihFamqFYyzI5u4PLzpjdnjvbT8WbF660SDwplGHOTplx7nk+dnDHtvkHtUUWBzSRWsDeZLJL7EYBMlGYYO48kfp0qp5WAmLcVsdXhdfimC/E7y8lmv/CkxAGxltAC6PDYSDb+C8OcJ1cHtEpXijk0XJl5rnAoAjbevaAh5/X7Sb6ti3UtAScbQmnh5OkwMS03dBRYZU0KzIuAiDAIYlZfoPdYtna0FBgBtfR3npJwlbeGZxaloVEEu98z5CyoyceQsOLmb/cPdIVyZUAwF3luAgIGIu9GlbsixRwZgKDnFiXbKpXs44rCjp5F/4UO2KTTXtP3zcij+NgqdI4luFm07okMIrke2ziEhNX 92zVa865 HP9ZVGNM2bUT0kaIcXSPNzpU6Wtsih3SjTz29TE6ipVqAAdCmv/2EOjY3MagNqDAGMYy52hVL5EB2L/Ij0/Rk8Zawq7fyP4ib6x09Vnnc76tHO04084vGhcfQRktmBaoner2CobnhmsL72/Uv93kZ9Z7P0XYTB+ZFa329HTVufR2EuaQPiMD10cKhzEoi+YDe1itFygMLvbK+InUUa4Avb3k5nilNmWf4BS9Q2tKr5/j1FzeqKBmHIKwemLuKhhAdYQLPZAz5pqqtNru3FGqKvdr5j+cLCiF9UBfv90U0qFkDTzcbYtlioQMw5xasCdZXZipZ3DpppjBlX7VC8dBZ9FRhuuD1DY5KAlJ+DEcb7cjebrGFIhy86J+gZTyY7zjW+P6YRgXYUH7RlE9d6suqQf3I5Ssx4CCQr48zgaqZlQu0jUQQKzBDfcqNKAXZLJ6Prs0s1H5KjAiquBGXeGrRCWInCvZ8ka0UKrUf8to1LgHqXlSov1LTaCBXf6NfkvmZJL48 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All use of this value is now at page granularity, so specify the variable as such too. This simplifies the logic. We maintain the debugfs entry to ensure that there are no user-visible changes. Signed-off-by: Lorenzo Stoakes --- mm/memory.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3d85aa7106b0..ae01f541ad30 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4393,13 +4393,13 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return ret; } -static unsigned long fault_around_bytes __read_mostly = - rounddown_pow_of_two(65536); +static unsigned long fault_around_pages __read_mostly = + 65536 >> PAGE_SHIFT; #ifdef CONFIG_DEBUG_FS static int fault_around_bytes_get(void *data, u64 *val) { - *val = fault_around_bytes; + *val = fault_around_pages << PAGE_SHIFT; return 0; } @@ -4411,10 +4411,13 @@ static int fault_around_bytes_set(void *data, u64 val) { if (val / PAGE_SIZE > PTRS_PER_PTE) return -EINVAL; - if (val > PAGE_SIZE) - fault_around_bytes = rounddown_pow_of_two(val); - else - fault_around_bytes = PAGE_SIZE; /* rounddown_pow_of_two(0) is undefined */ + + /* + * The minimum value is 1 page, however this results in no fault-around + * at all. See should_fault_around(). + */ + fault_around_pages = max(rounddown_pow_of_two(val) >> PAGE_SHIFT, 1UL); + return 0; } DEFINE_DEBUGFS_ATTRIBUTE(fault_around_bytes_fops, @@ -4440,18 +4443,18 @@ late_initcall(fault_around_debugfs); * This function doesn't cross VMA or page table boundaries, in order to call * map_pages() and acquire a PTE lock only once. * - * fault_around_bytes defines how many bytes we'll try to map. + * fault_around_pages defines how many pages we'll try to map. * do_fault_around() expects it to be set to a power of two less than or equal * to PTRS_PER_PTE. * * The virtual address of the area that we map is naturally aligned to - * fault_around_bytes rounded down to the machine page size + * fault_around_pages * PAGE_SIZE rounded down to the machine page size * (and therefore to page order). This way it's easier to guarantee * that we don't cross page table boundaries. */ static vm_fault_t do_fault_around(struct vm_fault *vmf) { - pgoff_t nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT; + pgoff_t nr_pages = READ_ONCE(fault_around_pages); pgoff_t pte_off = pte_index(vmf->address); /* The page offset of vmf->address within the VMA. */ pgoff_t vma_off = vmf->pgoff - vmf->vma->vm_pgoff; @@ -4486,7 +4489,8 @@ static inline bool should_fault_around(struct vm_fault *vmf) if (uffd_disable_fault_around(vmf->vma)) return false; - return fault_around_bytes >> PAGE_SHIFT > 1; + /* A single page implies no faulting 'around' at all. */ + return fault_around_pages > 1; } static vm_fault_t do_read_fault(struct vm_fault *vmf)