From patchwork Thu Nov 16 01:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 974ADC072A2 for ; Thu, 16 Nov 2023 01:29:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A19F6B03F1; Wed, 15 Nov 2023 20:29:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 511E36B03F3; Wed, 15 Nov 2023 20:29:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 306906B03F4; Wed, 15 Nov 2023 20:29:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1987F6B03F1 for ; Wed, 15 Nov 2023 20:29:36 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E699CC01D6 for ; Thu, 16 Nov 2023 01:29:35 +0000 (UTC) X-FDA: 81462085110.22.A6D2CEF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id D5809A0005 for ; Thu, 16 Nov 2023 01:29:33 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MSVFoLe4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=2b10LAAj4x5KUwQnmz0gbjjDswJmb52XSfIYsQVJ2Y58gEQQf3BSIbu36Jackh5W+KKNHi WEoDVSwOYXWQHeGbfHaX2cecIVO+hr1a30KORqYabHU8eQj0NDf4F1P4RZjOIDTBK8+uP4 utXd9K5tqsa2oajap8ThfjyKn2Hwbcw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MSVFoLe4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098173; a=rsa-sha256; cv=none; b=lu0FLmn+hgGZsNTiIKN074RM7mpO2DtUFn9Vu4jnb1aEzAV6dwSlP71/Rb7xSjHoA0kDNR TqbEuJPr55yvam3KAQa8uRr92v9RNBG7cVaFCo/ARC5IztQLOjIPT+lcfyZReU4NPUUaJj p9JGgZoIC+gcywBCUTPYX0I/F8NQmBY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098173; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=MSVFoLe4543+AAleX94WuN/MXL1ArtQ7HP5/l5Wg0bIFtcesqhDRGs5/I7LVPTondLkx1c 2m9s2FC8dt3lA2zIciuFxPOSoe23P5WIu+MkUc1yQilbr5P/YCzRhfbbwEMqXSisRWiIfl 472cMgd+vgiZFJpJrjW9ChQUye/mlB4= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-136-Y70yDB4gMPCRN9jjRwrrJA-1; Wed, 15 Nov 2023 20:29:30 -0500 X-MC-Unique: Y70yDB4gMPCRN9jjRwrrJA-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-46107bc7936so12676137.1 for ; Wed, 15 Nov 2023 17:29:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098169; x=1700702969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=VJb+rcMORmwgNNFC7mLDdp/Hxb/iFA4MfsgWSZqn7LD3ZUpTAVUCXe0uIn3gyco1MT Wkl1FBNyKhApJAhWBkgEdTYlygkyVMfe7M8mDAQo3jhc7sbMw6hgkgcx+R1n0sNXBmiK wA8Hu2BMzn645vS/l4OgSn56t7LqA0RAAjT7SFAkyBtT4bYexzcc8SU+SXhnqiUAfsoT mkbS4ys6xtAlu8SPaox/VLN6sEq1HJDA3tHoFdSt0ihnHEXvxf9va2+kHpxDndt6hEF2 z/t2H1hder18JPl9Er/gQU3DvGVMBAMybqlujXkg4FQa2Uar/CrZnTCFXg1yYWk4NcqM FKjw== X-Gm-Message-State: AOJu0YyFYkDNLvO5hOqiEeVKEXC6kLZ+dWRVSfZ8zAYQazMfFmH5KP+q 1mtZVp4SmPS2lvRwxYeL+imTwzG4uc5JfCKK++jV82RbXhbaB3Dp2SY4Oz3wXz1zePPb1moKYjR WPqmGZ6VTVErhBhUhYTk= X-Received: by 2002:a1f:1f95:0:b0:4ac:3094:fc14 with SMTP id f143-20020a1f1f95000000b004ac3094fc14mr7205521vkf.1.1700098169361; Wed, 15 Nov 2023 17:29:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmlcK0wHn56i8+DaE6cZQcDPiXYda86sHdfw6XVjm04MvVaxCRJjJBn7Y0oU98rdXaw93h4g== X-Received: by 2002:a1f:1f95:0:b0:4ac:3094:fc14 with SMTP id f143-20020a1f1f95000000b004ac3094fc14mr7205497vkf.1.1700098169016; Wed, 15 Nov 2023 17:29:29 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:28 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 11/12] mm/gup: Handle hugepd for follow_page() Date: Wed, 15 Nov 2023 20:29:07 -0500 Message-ID: <20231116012908.392077-12-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D5809A0005 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 5k6g6wd7sz13rrm46w4op54cq5fgzwen X-HE-Tag: 1700098173-178794 X-HE-Meta: U2FsdGVkX1/XTpApeG9qY2oi0mhqodpqWeeIE27XzQm66D+CtvqHztfUk/19Uhx2Slk+uRb/LRPMTn0xhmx6w469AxfPNhlTje1udJwvc5baKJ4kk20lkAEKyYxYxKzKys2guqrfqKjp8u2Llimwm5/w1Q8NPoSyL6ZVDFH8ZPMhKUAY/esEvbeDGP/rJouJeKUf41l3f5qJv67f6vmJhYuT0RuRgD4LS78kE+gtehMx2TqQiNCBH+RteW0wK/itZPr8C9cwqCbDir0gvD1d3teMg5grKnsY91leyXmrdP+gBRxQtYO7h/RRDc5z7kPaa7VIntR5FpGCh7g/kSwOKBTYK0hFrW6wMsg269Ry0lp3mo6nwL5HYFZP/MOkARLlC7WIWbcGggVpnmdqq141VUtBPDUWadSbnivN1EzIuwOh8A7L7lQYZpvjPwCdLKqn7G/xS44nK9+ZccnrC4TSRuhdHE8Kx/HNgMIYdyJ3hek0vfdN9RFAfSLDJ1BUrKYZg0t32Iae7z8/usfsXerRkRutv7D86lvIVrd7ouqcyflMkwBysJcD7NTvPcUQtqM0EnSafsehz/BBFvUnMdxBqSdCcuGI3ZjpYtWnx2jo9X6D2f1ERIm1pBA+8VCs9WBBWG8ujWVY1q2ToH20IkziJf1vH1E0nOkSAjywVpL99bYqgylp1CRLcFGFsEKRw/pT3FeddWdD5mIAuMXTTPZ0QpIqp2pobyluNHwbdhSnZik4SrhFfwaVgkhgc6h/tCFtp4pwbJARgn8odWwYn7VMS+ju7kuy8epJTNvCa1FwpgwX8HGGHYJET0gTboY+eEMTmS9LfLWr3ka1eegwMb58cxy8G9n9XASysW5x7IiFJZVAgcD4u1ZegEBuN8cbq4b9+ZyFwHgG3D67YG4aURhjpDb/E/XEo7vkZYXEevIulKLVpVRVTTFsWiYzbCwytahz8VIpjAZB5ePDws0M0dk CNncc4HO 0t1bj5qDWunHIHqGMo1JIrGcJJ5VP25/EGgGfCPWuWhxELcWtGpszF4zyIqbY6TtHtAjqjNMpd4ylXnvJvc20rJ6c82xMv2td4TwLC+z9IVx5MvB/Dv3QsN2XJZf77b+Yvu+k4F6b/jwK0siYpT9c08Ug1ShfPx4mWyZx1OzdB6+OUntLH/QfJ05eAnq+fNZMbXnJr7DpwAR293Y0j+iJHuFOj6jp8guyLhRi93UNnHEHdNr1wRQWAk+xe0gSdX8E0OofaEHTlvqcDHpqWnOU7M5UjK+j5MyEhOU472StpnKzids7D3a4t01D9IZV4Ks2p72Vqi3DpIyTi6VvUNGTI6rxrESS1KQvdtGj6MCH+cFDDI1JAJj2x3fkCmkT7M1G2hLcE/mCrCP/F5qvkkZiV7bx8xsXRiwmbsnhelajBgvG6T6qOGXgg5EnlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hugepd is only used in PowerPC's hugetlbfs. follow_page_mask() used to leverage hugetlb APIs to access hugepd entries. Teach follow_page_mask() itself on hugepd. With previous refactors on fast-gup gup_huge_pd(), most of the code can be easily leveraged. Since follow_page() always only fetch one page, set the end to "address + PAGE_SIZE" should suffice. Signed-off-by: Peter Xu --- mm/gup.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7c210206470f..e635278f65f9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -25,6 +25,11 @@ #include "internal.h" +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx); + static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -713,6 +718,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return page; } + if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) + return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), + address, PMD_SHIFT, flags, ctx); if (likely(!pmd_thp_or_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); @@ -764,6 +772,10 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) + return follow_hugepd(vma, __hugepd(pud_val(pud)), + address, PUD_SHIFT, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } @@ -772,15 +784,19 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - p4d_t *p4d; + p4d_t *p4d, p4dval; p4d = p4d_offset(pgdp, address); - if (p4d_none(*p4d)) - return no_page_table(vma, flags, address); - BUILD_BUG_ON(p4d_huge(*p4d)); - if (unlikely(p4d_bad(*p4d))) + p4dval = *p4d; + BUILD_BUG_ON(p4d_huge(p4dval)); + + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(p4d_val(p4dval))))) + return follow_hugepd(vma, __hugepd(p4d_val(p4dval)), + address, P4D_SHIFT, flags, ctx); + return follow_pud_mask(vma, address, p4d, flags, ctx); } @@ -812,7 +828,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct follow_page_context *ctx) { - pgd_t *pgd; + pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -827,11 +843,17 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, &ctx->page_mask); pgd = pgd_offset(mm, address); + pgdval = *pgd; if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags, address); + page = no_page_table(vma, flags, address); + else if (unlikely(is_hugepd(__hugepd(pgd_val(pgdval))))) + page = follow_hugepd(vma, __hugepd(pgd_val(pgdval)), + address, PGDIR_SHIFT, flags, ctx); + else + page = follow_p4d_mask(vma, address, pgd, flags, ctx); - return follow_p4d_mask(vma, address, pgd, flags, ctx); + return page; } struct page *follow_page(struct vm_area_struct *vma, unsigned long address, @@ -2850,6 +2872,37 @@ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, return 1; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct page *page; + struct hstate *h; + spinlock_t *ptl; + int nr = 0, ret; + pte_t *ptep; + + /* Only hugetlb supports hugepd */ + if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) + return ERR_PTR(-EFAULT); + + h = hstate_vma(vma); + ptep = hugepte_offset(hugepd, addr, pdshift); + ptl = huge_pte_lock(h, mm, ptep); + ret = gup_huge_pd(hugepd, addr, pdshift, addr + PAGE_SIZE, + flags, &page, &nr)) { + spin_unlock(ptl); + + if (ret) { + WARN_ON_ONCE(nr != 1); + ctx->page_mask = (1U << huge_page_order(h)) - 1; + return page; + } + + return NULL; +} #else static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, @@ -2857,6 +2910,14 @@ static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, { return 0; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_ARCH_HAS_HUGEPD */ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,