From patchwork Fri Oct 20 18:33:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Vishal Moola (Oracle)" X-Patchwork-Id: 13431021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F331BC0032E for ; Fri, 20 Oct 2023 18:33:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39FF1800CA; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3518280044; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F097800CA; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0959D80044 for ; Fri, 20 Oct 2023 14:33:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D679CA0882 for ; Fri, 20 Oct 2023 18:33:50 +0000 (UTC) X-FDA: 81366688620.06.296988C Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf02.hostedemail.com (Postfix) with ESMTP id 162E080022 for ; Fri, 20 Oct 2023 18:33:48 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hHmOFlfD; spf=pass (imf02.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697826829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=Re7idhDgGRNh9/tOJGq89jRRg9yjSnl7lkEGKkT4Bo0dUVC0HMNEXUH5xmO56mGldKj0Bs 0Rt4xTg2WNjzocrxzRX8p8CfpCUwVnJR8UlW/ovrTl0pBut/MxpTQfdjJWHIXruX9SSeXb v7oheC75gFn+cmjD0iUxEo2eLHU5xSw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697826829; a=rsa-sha256; cv=none; b=GeKf7LDKi/hirdj+pJKqlH4NgTCXeWMJOWKCKri9zl3Ua2K4igXzpD+Ptx46JZZcEyK9Nr SXXa9IXknYh7yQ724C0YipAa7Fe8RzUM6JdNBtYI8k+DCTuJ6PkeyvjU24OPfov9Re3ONU YxG2hYOAlAcxAOXYRFc0HJ5/Pd7JjGI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hHmOFlfD; spf=pass (imf02.hostedemail.com: domain of vishal.moola@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=vishal.moola@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6b1d1099a84so1066413b3a.1 for ; Fri, 20 Oct 2023 11:33:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697826828; x=1698431628; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=hHmOFlfDlbqklrfab9qHnY0CQNI5g5ayjRBBP8gQfvnJHfetoKpp43pfd2Qm9zfQAp CEOT5G8GgzPXhCxqJixNl3rMfFwYV/Cz1uG5upE2bhlWzqjZBshxV9SWdKVmLPm3kWCp CN9nfPyIwfzqN7ztMd8ShSMTbXiN9xDX+HUNFSEGqeFBkGMjjGvBjB9a+xcdKU7gnAs7 P/vDRxwZCmIkm+2XVm+RXkmBItuP9itCRtIeDa3UOe8aDWDwI5gqgMA+H3nsXXe9zsKq CZaGu0gwNLEpiZXBDbmY80iLGg5CHXR88sx3ld4uXiMzwoT+fkvVHeDOFqfHP5DdvN+i rlVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697826828; x=1698431628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=shwuj1hNaDkCqNReHb+lM6WWmRCPMvGlFZhV2p81nh8=; b=BjuS8+t382Tb7z2URqH6/Ne0uDxmckge+ZxpKr3D5z2akClMNZUtf3qijD1HBjPBsu ij1fRzWRx18am+lcoterofMr+DmFd/cFa3e1GMbxx3SWZs0YL29D8V62nxw7IbetQOjL wHpGh3f2fxCywDJzN02ob6Wo+8FPxCWBa5TlqrVgtL5dGkhbaAr4NiF1akMXryxWotD/ YE7pa6ofA3JRPudPxmAucV3p3LER+HIt67Qi6ZaH+znHPcHkwx0fkl6W4eOvcAlQHdRA BTzBvTV+zbNd0xNrIuVF+u0mi37Jr0YcLZVkj1J8puk9Q9Wi9+s9LtkOh15RuLWBb07m dCbQ== X-Gm-Message-State: AOJu0YzcCFtpWCYJb7EghSXj11Um6lBRfqwXXANtVTIsgsYndZUemlWt PJHkRuK1Af1nYcAZEPu4N7FJ96m9fTw= X-Google-Smtp-Source: AGHT+IFoA9q8+nleOsVaWYGKJMR5PUXVnKaxoWBcmjk+PdUFvUSMwibx6e8j5iJCbuEQcfjDcMuY/A== X-Received: by 2002:a05:6a00:1ad1:b0:6bd:a7ea:5c7e with SMTP id f17-20020a056a001ad100b006bda7ea5c7emr2751158pfv.11.1697826827696; Fri, 20 Oct 2023 11:33:47 -0700 (PDT) Received: from fedora.. (c-73-170-51-167.hsd1.ca.comcast.net. [73.170.51.167]) by smtp.googlemail.com with ESMTPSA id z14-20020aa7990e000000b006b4ac8885b4sm1911345pff.14.2023.10.20.11.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 11:33:47 -0700 (PDT) From: "Vishal Moola (Oracle)" To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, "Vishal Moola (Oracle)" Subject: [PATCH v3 5/5] mm/khugepaged: Convert collapse_pte_mapped_thp() to use folios Date: Fri, 20 Oct 2023 11:33:31 -0700 Message-Id: <20231020183331.10770-6-vishal.moola@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231020183331.10770-1-vishal.moola@gmail.com> References: <20231020183331.10770-1-vishal.moola@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 162E080022 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 3jor3tr4w5nm4wrz4nx86iac8j6qgraq X-HE-Tag: 1697826828-604334 X-HE-Meta: U2FsdGVkX1+DVhrfUp+KioBTN7/8xFUi5n/DmQSbwMoMpW+lflMMeJgs22Ti8ILU/nVO4oRj68G36QJgd5fetkiR+D56Ft3sYqgalUi9600u7w3BOAbXU/fSR7r8ERhLX5q0yVFnn5yvPnGNwzym1OS6h0TIQKkVK1zQ97OEpQSc/svegY/S5Dgwt6oKP1WkrinoqrCgH6NjmYo57kc3cHzH1660AXMA/a31othmWHBo4bD7VdqBUCyIl6Pf0j+pHQRHHXsVO7skaejy+S68T1M0b+nGGO2uNbafdjBoSWrTTthrVfUr90edq688WEzw0xvSm1VOM7AxuhutN9ihO5ZpUccCzAhtxZ87DVlmYqAWXBB91MO0AjGLKsjU/n1KSBhC5KKQ1ptEQUTJigGyuTH+2iD2+5wFCmajMLVHfsZ8YN5OKLta+Jxrt1qVOVpF4jIYOR/YWA400pI72Wpc9Q0DfJME3RzzeNUo+uGNi+6Gk6N9vcHrm7N9PGL35zYIoB9aa5VzEk8dP36m/n4Z7qPLXsc1aZ8CQON9N7grMKlbTBuWWt7LFkYeb4EI6sRbjzHKFjJfRyyq+t3tx4zZxb78X7zQbMM1UJDIy/ypJcgbxn1Hz+7F5texU+Zy3PUuDXNhloEC7rmDkmcJuKQBfrcUWAuHvMt9KrhmMiEvJLOsGGjoizflb5052lOhn2DNhH6CbU9wCsHIxqR4otUewKotX/5fSYqABn664hQLKZq3+UWZIvuoyCMqdBM8H+xhXJEhLu/M5X7zBlWuNDJfkKq/ME3YlS49Qv0zwhbJ2JfeKZQ5CPKWLJUOeEE+HJweJp3JNBXk1OKzv5d7VaKHY+t2RwB/zzU+MSfuQODGk2w1mAWG91r5GcLrCpZKrA9dMTOY0TiATm5jGWkb8sugicf+70YxjUBIe4gxxrCyx7dh+q9IRCeLUhalxKxaht+c7gDfSF4+hB1itq0sa30 +dqoT1E/ BQvefgK/XWbDzvbleTH/gpb/e+bEfr3KWlEJ0gObgTJo9s8KhzzilL4JX7kl3f0jFRv+/NCZM4UYu2pT4ikz+qfTSyUio5mrN58QkfiUGaFUy5Ju3tgngWlpIi/2RP98tTDvH4hdnyeGDRpec9jM5v+VtyjcK44ddm889MHOfcjJ0wdABvWOPdmKzcWlXvEl0mT8I9thTkaEfroFD9qBNtdUdlrNR+IPCzGu/JbB+UNJMSjTW0PnBKV3/R4exVBodNF1aLA87lep1ps0QaT72mGvmialJuKPn8y5zYnft+lyKu3xw7m/c+VbLPMX+1ASJ3vgG7FKkFf1QEfctlfv8fbHU4bBGlUGGIuqlEbNw3yVI09zoE6AWwa6Sr3FPybSURRAcx3lYv9lj9sc4iQ2XOsFWvoTeEPlphmHSOuMlMu2oz1B+zIWg69CI0bjDvlaPLzwWHIxVJEsy1ebxnqQ2ORB8c9UT/WY+xA/yw+GCmK1QxlTWSlPCseayaveR2Sr90EZ5zgGM2TH2y+tpwTMp5RimqyxZoUZ7eBtY9gYwGjGb89zXUmvmJq7G38WoawykTSIIWSt7Gsxc5JRmK5PlJJxvhQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This removes 2 calls to compound_head() and helps convert khugepaged to use folios throughout. Previously, if the address passed to collapse_pte_mapped_thp() corresponded to a tail page, the scan would fail immediately. Using filemap_lock_folio() we get the corresponding folio back and try to operate on the folio instead. Signed-off-by: Vishal Moola (Oracle) Reviewed-by: Rik van Riel Reviewed-by: Yang Shi --- mm/khugepaged.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6a7184cd291b..bc2d8ff269c7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1477,7 +1477,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool notified = false; unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = vma_lookup(mm, haddr); - struct page *hpage; + struct folio *folio; pte_t *start_pte, *pte; pmd_t *pmd, pgt_pmd; spinlock_t *pml = NULL, *ptl; @@ -1510,19 +1510,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; - hpage = find_lock_page(vma->vm_file->f_mapping, + folio = filemap_lock_folio(vma->vm_file->f_mapping, linear_page_index(vma, haddr)); - if (!hpage) + if (IS_ERR(folio)) return SCAN_PAGE_NULL; - if (!PageHead(hpage)) { - result = SCAN_FAIL; - goto drop_hpage; - } - - if (compound_order(hpage) != HPAGE_PMD_ORDER) { + if (folio_order(folio) != HPAGE_PMD_ORDER) { result = SCAN_PAGE_COMPOUND; - goto drop_hpage; + goto drop_folio; } result = find_pmd_or_thp_or_none(mm, haddr, &pmd); @@ -1536,13 +1531,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, */ goto maybe_install_pmd; default: - goto drop_hpage; + goto drop_folio; } result = SCAN_FAIL; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ - goto drop_hpage; + goto drop_folio; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1567,7 +1562,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * Note that uprobe, debugger, or MAP_PRIVATE may change the * page table, but the new page will not be a subpage of hpage. */ - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; } @@ -1582,7 +1577,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * page_table_lock) ptl nests inside pml. The less time we hold pml, * the better; but userfaultfd's mfill_atomic_pte() on a private VMA * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of hpage does not protect from it, so we must not drop + * So page lock of folio does not protect from it, so we must not drop * ptl before pgt_pmd is removed, so uffd private needs pml taken now. */ if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) @@ -1606,7 +1601,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, continue; /* * We dropped ptl after the first scan, to do the mmu_notifier: - * page lock stops more PTEs of the hpage being faulted in, but + * page lock stops more PTEs of the folio being faulted in, but * does not stop write faults COWing anon copies from existing * PTEs; and does not stop those being swapped out or migrated. */ @@ -1615,7 +1610,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto abort; } page = vm_normal_page(vma, addr, ptent); - if (hpage + i != page) + if (folio_page(folio, i) != page) goto abort; /* @@ -1634,8 +1629,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 3: set proper refcount and mm_counters. */ if (nr_ptes) { - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } /* step 4: remove empty page table */ @@ -1659,14 +1654,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd - ? set_huge_pmd(vma, haddr, pmd, hpage) + ? set_huge_pmd(vma, haddr, pmd, &folio->page) : SCAN_SUCCEED; - goto drop_hpage; + goto drop_folio; abort: if (nr_ptes) { flush_tlb_mm(mm); - page_ref_sub(hpage, nr_ptes); - add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); + folio_ref_sub(folio, nr_ptes); + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); } if (start_pte) pte_unmap_unlock(start_pte, ptl); @@ -1674,9 +1669,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, spin_unlock(pml); if (notified) mmu_notifier_invalidate_range_end(&range); -drop_hpage: - unlock_page(hpage); - put_page(hpage); +drop_folio: + folio_unlock(folio); + folio_put(folio); return result; }