From patchwork Thu Nov 16 01:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B0FBC5ACB2 for ; Thu, 16 Nov 2023 01:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BB6F6B03F3; Wed, 15 Nov 2023 20:29:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6171A6B03F5; Wed, 15 Nov 2023 20:29:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43FC66B03F6; Wed, 15 Nov 2023 20:29:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2EE186B03F3 for ; Wed, 15 Nov 2023 20:29:38 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 09810402CA for ; Thu, 16 Nov 2023 01:29:38 +0000 (UTC) X-FDA: 81462085236.09.463704F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id D6FF6180004 for ; Thu, 16 Nov 2023 01:29:35 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GcNuMlIM; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098175; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=y3oQ83CkEPukgdGG/qaYWqeukz6ZRNSCX0zmBCJAKNwbAdviSCOBpGw0pVwymn6Plo6pap 5q/qjxqHOf1F2PaEcjozVvgHDjt4kqTpFJZwnY2wbslTD8A3rsVjuOLdxGFt/cXED0IWmJ HFPsBfx2XQkNA8AR8lV11XNh26pGd+0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GcNuMlIM; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098175; a=rsa-sha256; cv=none; b=cyBdfP47ryDHAf87AuoxYxnAVG3o5LaQ2UGHoAR+2jZxTCCvIVGM7ScbPdPB2SP+y28cBh sd6nyXlXhwPh1kbzvtrM0lweifZlL+iyLzQO7gtnHd1V6yIKiTV67iNqq1VSdqtgBF6lVV Uk8EX9RIQZT+NldjUfpG5fuwOL28P9Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=GcNuMlIMDy0ZbF4dYiiZzR1SqgbTNuB0FhHx2OHFT1d6nhRqnvxV/FNe1IG1eoeWYImY7p /tgoJ4XX3PhXhpwuC4D8TyU46bVyNC1gHwVzl5kNk56t2HXx9eMpguP82YIqPzYZR+k1HK n0r5idvp7WwbLCirNLhmcTv6tXIe6dU= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-389-jCoTNCXgObma1NLozEOjCA-1; Wed, 15 Nov 2023 20:29:34 -0500 X-MC-Unique: jCoTNCXgObma1NLozEOjCA-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-45fb8f2166fso12484137.0 for ; Wed, 15 Nov 2023 17:29:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098171; x=1700702971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=B52wZ2in5PKhS29Iic4IlIebP8SfYnxRtFEc+ncy5/SSiUD5dtVdvTNkdcC7tFji49 e4YZxvciik4nQScyiCyiqdyHNgyWk2fZZuub+eayiKusDmqJV56SU3FtDjwS9wxFPj9Q OwRKjG7EKbUrsULfVzrDdsep6cez77kuIsPPbxEETX67pkYPIxZz9k2oiHQQm0H6piwZ 2g5Z9GZPUoxAFb81O4DjilyK8btw7kx7ZuoM6OMTrYfpNxR5D5+Y/a9inHaMaYNvKim4 mz+FMZenO7Nbqz0eo7r1mtHX1VXMyqyVZkE1xTBJYtULFRPeOqndWVdM0A+wQYVW+c+E +jPg== X-Gm-Message-State: AOJu0YzJy0s8dckssDlO0LTMc6wipKI0GM5MuChZ1oRGnbhydnmqLE8Y 0ZHKqH5fjGO41KsbWnnWjoJ+/QcZiF1SZbWvPPoFRO0cLfhlmxCswKiFvo98oxRH7wldgxxFq44 Ci74244wnLIg= X-Received: by 2002:a1f:a017:0:b0:4ac:3694:7011 with SMTP id j23-20020a1fa017000000b004ac36947011mr7269393vke.0.1700098171275; Wed, 15 Nov 2023 17:29:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IFewvoOFObJgHzAs8T/gdoGZhF8fsIKsBFTyxvttvo0eTXlsdk7Fy6qUthUD5DfsgEESegFFg== X-Received: by 2002:a1f:a017:0:b0:4ac:3694:7011 with SMTP id j23-20020a1fa017000000b004ac36947011mr7269371vke.0.1700098170969; Wed, 15 Nov 2023 17:29:30 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:30 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 12/12] mm/gup: Merge hugetlb into generic mm code Date: Wed, 15 Nov 2023 20:29:08 -0500 Message-ID: <20231116012908.392077-13-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D6FF6180004 X-Rspam-User: X-Stat-Signature: oh4x5q3terc4i5ob31wz9ddwo6f7kksr X-Rspamd-Server: rspam01 X-HE-Tag: 1700098175-22472 X-HE-Meta: U2FsdGVkX18cQdkLlJtIcebiGEpXTC8cWo0r3JAszeDx3rjVWPRw9lJgvhojFWSXitan/+UPSzMjOZ6nORHDCuGbrarA76VMc4BtRl8XR1diZlBjkR+ZXzxfzVawlgS7IC0pXZj7DRqstBlfvdpnSOtN2l5rFZijZ1BQKqH/EGX8BZWFedyNTBArstizplfeh67OTmLMd+20hrq8D8esT7npdivAhdUWMG7mBNCoUXc9Iy4Bpc04OMPY844YY3Mg9MOHqGNeCbMuVKkBZb/J4hELiu0zJhxbiqO2QPc0nB405aYzRRaJQdFXgy8xrxg9YtNiz5XrQBIac1Ih3+GwB27I6fOSvJaizLuoKyCzAR9EgzuUAXSgm4t36Jzp0LXDiDnGAlmqpeM3wtL0t8nx9ryUpE0ZQdZd7JC7zhPd5wen0zY5AEm5ak27+rDxejrGm22daJccrwwXKdoKDACfvb7QW/iT1niDp+YzUvl/ji8HjRasDzPNnki+UEpuHTdo72WV7BClcEBXArAIXTbk8T4MZg/WKWlT35MXuCOE7yUO82HojmMLmZtE+pPdqVoNgTZqnCNZIfz0X8TZ/wY6pNJa5xpGbpYwyb3Tlf1o/pLht+geFm4XzPH91V2aYSkv0EEYEe2DZmsxcDNm0yIu+udAKPAvaboHU+kLUgaLmVKK+zYgCSOzbBerHRy/vyzBAli2bGa+DIhMPCf5fcCgMUJ3k9HbWzYDm1dP6JxUAtVImRCqCoRaP6waNZK1SByHwSJmbw642k1vD44I6JEO0283R7/p0wQuPNLNIikMpHdVjagG1fNJ7X6n+kWtkmU93FVFUx6y+iT1ujJDhpHhUSIna8t2zpwCBM/IrTrwMhk50v2YwhYFIk8INgbERdswsn5NVkTIgl1ChlYWV2a00RW21Ac+QydZMh37aqM87WSyYEM7aidR5+a9ovBApuNWUlc1BOXQcwVwUy59e0v e65W7Ukw 8b80CsdpnKbFHW6OkBZxYJtGGwSUr7sABN0eO97kLl6qnFfFo9/NhK1qMKICTGXBVn/tKBIZEN/o8O7iV/6zRQvN3NmkTqgeIcgu8hs+0lEi6tZmNXoXEmLfdx1QZZ/mCbe6j7ZevWYDMQe7OyzQstAgG2I27+xYPj3nAK+40blFNmA+bg4jy79syujlmG5jF2yn9F28tu6QFk0I5bSWYGG0yDqiT9evJOh/O4x8H/hapPuKiMImI42Pk/m/AQd/uPWup2ykET16MxxnX6n91Y7ujMKKiuFjspwkPQq8I2sBD1TN1a/ZAzrAclbuRK3shJK8tm4Ts1S9EIjQVmouKj4b4rg+fkAWEvLAT3oRYlyXIuRINkwOQHL7q/VtTYnd/prTcMmBtqKz4Gpg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing GUP over a large hugetlb range on either ARM64 (e.g. CONT_PMD) or RISCV (mostly its Svnapot extension on 64K huge pages): each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. However, the performance difference should hopefully not be a major concern, considering that GUP just yet got 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL""), and that's not part of a performance analysis but a side dish. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(), for example. Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index bb07279b8991..87630a185acf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -332,13 +332,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index e635278f65f9..23fcac5aa3db 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -830,18 +830,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -853,6 +846,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 29705e5c6f40..3013122a739f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6783,77 +6783,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags)