From patchwork Wed Jun 28 21:53:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D258EB64D7 for ; Wed, 28 Jun 2023 21:53:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 871268D0003; Wed, 28 Jun 2023 17:53:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8200A8D0001; Wed, 28 Jun 2023 17:53:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E7D18D0003; Wed, 28 Jun 2023 17:53:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5F96F8D0001 for ; Wed, 28 Jun 2023 17:53:19 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 106BB1C84F6 for ; Wed, 28 Jun 2023 21:53:19 +0000 (UTC) X-FDA: 80953508118.11.F7D0503 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id D6F9E80012 for ; Wed, 28 Jun 2023 21:53:16 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WSPuBJpk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989196; a=rsa-sha256; cv=none; b=tQsUzhOV0W0vTuTQDrD5b393u5QHaVdkJ9rFz7/kPt3uSyP3QezWjz7BmYUxwpgYb0+t+e S8FWXg4Ya7RpG1EHeBKdnHK4n/KxBZyZWoqQOOAF5v2VZ9o0/iRj0zE6HO+Fa4FEbElyHY G++ZMZ+G//ReUkf+DaStpiPFVcuUT00= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WSPuBJpk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989196; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=HKf7L+kJkw0uf+saAZmNihoRob1tGByavpBN8qnxGhn0x93XksJD8Zo5iiEHdKVZWunJcm yIrows5qPpXLkfAuHgMr2onHs1VJIB683dfQt8wr4TXljmMgeInqfleFcs9ercph7j3Ang aqkWjh8JyDM0wJcU9h+KVktmlE5eWGs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989196; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=WSPuBJpkqpfxNLq2KR3ZPe+7i5u1JGRP0drjAzDEf4auzWtedPi5QV+Qx0P/QLXwVJZJ6e eaKrFfII8eZaFEhf7kfJwjOdkjBpG6HaZ4UpIka97RIXqsk65MFz7Bz5FIFdEqe2U7bC/H CkOE2PrXpd/VRNKmFjPsaOVvpmHv6UQ= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-613-kKFWf2xGPU-T0iVRLtUZ6Q-1; Wed, 28 Jun 2023 17:53:15 -0400 X-MC-Unique: kKFWf2xGPU-T0iVRLtUZ6Q-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635de6f75e0so20516d6.1 for ; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989194; x=1690581194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=iU1JbhXbgsutykap4CN2CYofJGu++5/gMqkLgku/bNbrsNNyjxstztTHmDfA7NRKvt 3Im+cnQ0vQX7uY/Fqo0nF/TT3kaYMz0mRvkBOwOHwp8B2eQevlxFKiae33igXDvusurW YD0Rr+srvWYa+/5XGkGlPWz1N1QsKR9o3EnjJwWkaHRHwpWHm07EsQTUh4/q0v60RcMg nxDocDPqYS6HgqlDyMkk9hDPK3z3QzcUdRLp16z6lFtA4NCxS7Mwp9t3NG//B15ZSxst V25NoW2GU3edQge9wqQ5tl+9t+9AS3y0AKvxRWXVNZ+s5gSQajfq+MyMjMdOOg625rjR 5U1Q== X-Gm-Message-State: AC+VfDwj6nBy8RcHa4qlRcxWDkioAh/ohNh9R4GG2Y/ETjhTrRkGa+AM TAb3hEvAvtclUSBu4vImZQCniM90GYAufXWVUeRrf/YEPYiPzfrn77EtJBnKRa4P5YaBG1MyBFr EVPtblMdvrW8GPB5eRIF/DeX3FnvYO6Y5F1BV4EXAxHhRZT/STGTOtrh0gTdFJu46wzr+ X-Received: by 2002:a05:6214:401a:b0:62d:fdc4:1e8b with SMTP id kd26-20020a056214401a00b0062dfdc41e8bmr42375766qvb.2.1687989194368; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4MC34MgzekDp0X74lVrIdzH9S5zruaiTrJfdeBf4bPnhG07fp5o87gyAcA985czOX/TxVoqw== X-Received: by 2002:a05:6214:401a:b0:62d:fdc4:1e8b with SMTP id kd26-20020a056214401a00b0062dfdc41e8bmr42375738qvb.2.1687989194020; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:13 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 1/8] mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() Date: Wed, 28 Jun 2023 17:53:03 -0400 Message-ID: <20230628215310.73782-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D6F9E80012 X-Stat-Signature: 96rs3qw1cg6eecpx45pj5axii1cp8ude X-HE-Tag: 1687989196-829846 X-HE-Meta: U2FsdGVkX1/KhUNMLcT6dULalvEh3lgb+SsJJIVN/o4WfL5x/ySRdLRv+zXgIvSwtVHWj4jlY2DHBoTBA9Uh7u419ybgUxS98Sec3hMEq2hoFNtBo86pIR/psk3nxtEAF75FZB5/Ak1QBxYQrJyKUx7TBkdXCI0E2jy9mQ8TAHAxk0X6ctgeoDL+2grVvOOg7Iv2U0uwXIurBq4VxcvOqBjL9HGYwcvNZAyhaQ1Cj6rMx4VHsX41EodKJBv/M5HWwN0cyVCJ4n2mdy1kC2t0rixQFWS1TMfx4dSYGp50TGQMw0ZSw+No8i97+zu4T5seFuz5Eh1s+uBJOV2wgHLYwFbfD0xAbVaWlo+EEXXAiPUvr5eA+Xs9jh32Xvrh/uIyODYMn0UO7t3SgBPY1z6xYk2AsFOcyIFDZQ2gDHEE0le9sLGErPc50FeNM6mJVQaznkV6FMpXB+NLVVrSJZe+KW+88NF9eYGeUunYxvURpjtoLd7JHZF07RxClt/LfzZUzZcsYdWCpPAKdQwefjoF+UdMqKkpKooNf5cfdCuPB1+RfR6SOLN3Nri7osI5Wp3IONd9amxhqx47d/ujhCk3wqo4LOwjh4DYXpn8mUbwD/cnqA2ksMvPV/9MZBfI4lArkxub/jrIKI9XM16V6Q5YkIVLul/JmSCmxY/A/VsG4QRUFkygSrJJE/RTuxhyPSLqNPik59mYhC07UIUZTQM4Bq6aWf0pGM2LqEpx2FlOW/q01KDw+RZqyJzTVyFfb7Tr4MgVxNIIB1cnCH+AvwYrPqkKOIq+IchjiUdJud8tmK5fYo6Q1d/UmyLIQqT7gVPCPevDtCodn/qEFwY5N0mhVRhrXA3Itp6blxxsXGGr8lq5PrFYxPy/ooDnbdG6utpvu9/kpS7vOzAy+Vb1DkXof/wWY788GZuzHmojIQfZc178BB0yPLuB6GcCwWZPNn3QsXTNmWWKtOuy13El6Bi weSuGA9J sfJHmX8Xbc61CziJ1SoRffc/Awdan0Wb9BbV1Dc4HsrAZaRlZ/L6OzqUS7nwxE7KiWEPvhQBBFPCQKjq1rbE0FGdkbV7IPsb7BNvlKxQAeNEIVPLkldIEFh4OunCaJ1Nic4puNdd9abv/t1tdpTf09hIfhxBS31DduYidxnyQe1l4EyCsYlZj2doU7fXeOpb076r9V7OaT7nh1uyXWcJPZLPRZtrdnXQZAq2Lutunlo9B2HHluwwoIkiAy+C0mOEofsHYfuVsANOieRMrClqnJ847EFAqOVMDfpZo3aNyk3IBbqnFqU3ExShmoT3lGmUj6iHYobMy++ix8PKxISKkgbV0zZ1C0jOvRzRhnMP8JRfHSNeugqIXtpLZwtBWWJZJiOESG2VHHBt4HDlYQf3FwdWOCj2xQ+SPP+JZB4U0nAGfdBMJkSQRZQx9F77h/g6jTsp6zDmkqyP3n5A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies: - vma_is_anonymous() == false - vma->vm_ops->fault != NULL So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable. Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb. It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a8336b39d6b5..06bc2e31f209 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -767,7 +767,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, struct follow_page_context *ctx) { pgd_t *pgd; - struct page *page; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -780,12 +779,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ - if (is_vm_hugetlb_page(vma)) { - page = hugetlb_follow_page_mask(vma, address, flags); - if (!page) - page = no_page_table(vma, flags); - return page; - } + if (is_vm_hugetlb_page(vma)) + return hugetlb_follow_page_mask(vma, address, flags); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bce28cca73a1..d04ba5782fdd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6498,6 +6498,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spin_unlock(ptl); out_unlock: hugetlb_vma_unlock_read(vma); + + /* + * Fixup retval for dump requests: if pagecache doesn't exist, + * don't try to allocate a new page but just skip it. + */ + if (!page && (flags & FOLL_DUMP) && + !hugetlbfs_pagecache_present(h, vma, address)) + page = ERR_PTR(-EFAULT); + return page; } From patchwork Wed Jun 28 21:53:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296390 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14106EB64DC for ; Wed, 28 Jun 2023 21:53:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7880C8D0005; Wed, 28 Jun 2023 17:53:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 710DF8D0001; Wed, 28 Jun 2023 17:53:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B16E8D0005; Wed, 28 Jun 2023 17:53:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4DF158D0001 for ; Wed, 28 Jun 2023 17:53:20 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1783A14028A for ; Wed, 28 Jun 2023 21:53:20 +0000 (UTC) X-FDA: 80953508160.05.DA04D2C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id EC32B140024 for ; Wed, 28 Jun 2023 21:53:17 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SBlEFL7z; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=mYL/4lfIEpcowrIIiznWswZcUjglaRSS6T+hiMMzdlNl7amNdyHSQCAnmOGIMU1zzfxNna J+WSRdYcDhU66+75XmRDW/4o2jFEdPxQIlfWZEmVA6Lqfsz+vPYjNBiCEHPBnH76C2cJ+9 G3QWohlw57tgPS3v0ai6aFgPGmXSxVE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SBlEFL7z; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989198; a=rsa-sha256; cv=none; b=ZBojrXN/+b+D8yPCXMD1ZW6oah8zXNO+24L8SWbd1l76YPA9rviYGc7yz9HWbavbvIGw91 kJHQ6f94RwLCE51/o2udVmSFszp/89kChoGfQvbSMTkIpCGvOx1UeFvkKJoHrkAk3sdDR3 nKaFVWmL50TcUMuhw81+LrDbYxvJ3v0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=SBlEFL7zaU18F4KSLlvbV34f9ZF4NjwhLF+xGGAxBRw46sXIoIFGjMOvjS1/rZcPzGV0Y/ Pc0GoAiktSeqxAMl1BKHEkDDFqtbQJkRuAsC3/+EpSr2LjJmk3JP2WN1KxnzeWxvy6RUq0 2ZKkq+rjjU8XK93DeIpUi0Yq58wtRQU= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-253-9YXeYkbbOhOq8lWKj-ZYDA-1; Wed, 28 Jun 2023 17:53:16 -0400 X-MC-Unique: 9YXeYkbbOhOq8lWKj-ZYDA-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-635eb5b04e1so36536d6.1 for ; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989195; x=1690581195; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=e4hf5e34qiOOHefs2a1++7++VnmyYG+y5ave/Bn5HFrz1tlN8b0Lc6dA/wYXDNhOU5 nDdQXx+yGlcY9tGSSsGivwoLgyHc3yRpNfiZU9Fg6kteGlSPGjKud9BidvOzwfalYm/v DyA19k4P+pceF0eaFLvF43tSvUfNQgpP7WWICY0+3Zo7V+EAVmcuY7YmrTqY3Vn8FTaJ 0gFgzcw16os3zfadb3cma4ePcXtAnQA+Yv0Nykcd6rfI3uS5f1XQCG/KPW484rlcwS0U GlWM4gGFYICBr1Zx7JACm7POxAX6QrHzJtEaasT2k/skpam0QBAZU5PWZzUeiI58flLM DhdA== X-Gm-Message-State: AC+VfDzjtPOhU2tbXFrS/Zo2SLw4mHLHxpLZ+qvm9nqeFgKOHy6aimiw boPFFDoxmwGNtsGc6dZWzHMF32wSq/0pLO4CxbZUK7hNnBxPBpv9GE7Wqt6u+JyO0UDrxZuAyF+ AjeFxAZnW+6pL5J3tolfBpyR5PGtapajf0ptIfiGIyvBOBadt6Gw5TO/DPX9GgBbhIWHB X-Received: by 2002:a05:6214:f2f:b0:62b:6c6f:b3e3 with SMTP id iw15-20020a0562140f2f00b0062b6c6fb3e3mr45427380qvb.3.1687989195661; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6HBU77CP7xJIw4KayPpNq4pGfYosv7bGjCnvBLq1S/WNi+e+thj4Rrto2qPbX4dx7awzouIw== X-Received: by 2002:a05:6214:f2f:b0:62b:6c6f:b3e3 with SMTP id iw15-20020a0562140f2f00b0062b6c6fb3e3mr45427346qvb.3.1687989195254; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 2/8] mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN Date: Wed, 28 Jun 2023 17:53:04 -0400 Message-ID: <20230628215310.73782-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: EC32B140024 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: gkeerrd6znbp1suimma6x9xxd1om7cjf X-HE-Tag: 1687989197-857084 X-HE-Meta: U2FsdGVkX19sR2R0b2ruz5y/1T4ONFzrnRLcz3RI2145saj5hjkY1yAI3gZOtsKCAH6D/hPc4TEJHwk+ALj8hsJKcNQm8rr/9dGhvP46f4AkUGGkBF0fOaP+9/K23Ey9PNbwqh83w5tunR7AdYMVuMFVJwoC+rI89iapZDsE//lEGgDXE/o/MJyXOZzm50SAbLCtDF7I76/owiDEh/r32+dGWPLq5xOL2gmFYC52gzJWQkjWy4AgiyiADChQMQ9QU/XWmywL0hU+lHKgruIvWotaUx2ffwjFtjlZE/OlAaVDP1c+7KEo+BDN6CmojqXB9LNE3KXKmNXwlSlLaXSuCBmHC/+XI/uJIbF3J6b5M8hsdktK46xHls95iX/6ovLe7hD8aCgD2tRTpUZ1lh68pl1rk4F2l4MiGEvPqjYbwvloMyR6I//2/R9qikmt14oKknrjrb7rRRobMN3lKDH7m7I/oPvNzlrwny+VsnUHlCsWMGD/cTZyBnLUgExPEFhLwJIFHclcMc3pilN7L/Y5jrP28bICZUDb2rgvIETXnRvj0xWsCKbUl7cWWqbYqt4yabuiPzY86Gw+MdpBnaFwIhfPEMYUzXicXuA7ZO14JZNOuzmAoWZu8T0sM5tm0P2FGRGozQwZnXoW14IuvhtR/5YgJn9rg5G1k5wCVsGQo/fmfVHhKLiY2/9ogfwSdiYuHO/5QKN+yIuQ4w7lbiF8t0wJbw4ABTGoRyUf6VD6I/Y4ZOtY4p1DaaBQqzOtc1DN+UvD0YPc7ET136hLnTuVuRwmX2XmtOy0DzU+O1bCOU+7oerq3ij4VwGAuzmWIL4+eBLaVjw1z1b7VO+xpHdh8JyOAXjMAdvByfWHl5gaQuvKOuqgJSFnyKrpEdVTIVkQwsFb+E7NWGe/Lmxf9xnvyShoyBn21XwflHKv0QDhrmostd3LumL5IxDRvo1Zvdy+3iyhuEwD1SuvWD/pPQw IkTch/kx oK1KAHDRFj9HV18CPqWzpUgrnOSHP+4NCU4umVr+qo2qpWIrL4vXsDpn1f9Lg17rIy16wvGB+4/CxhPjpr77WoBUWNNGgvrO/tAZrWLxD+VDzmzTFgLwQOdlxTgbRrZfHQsJurYh6X4lkPXIvqYW046e/88GPHrm81JLzRVqRPjB28uKGEa7gKndXtar/4Dd58ie5Nn2cv9esR5ismfQ0GhTvF4OOTJZjExpHmakYLRkYy/GTpD0+ac0JU3lCYEjR4210WxkIkFGeAF2lov795d3qGw6KgCwFkRNf1j1kwbMMX+xaAdC4CKWT2Mz4eEhtzF6HGvA+BafQSMU6oSOloUqx1y6ar/n3dIvDuU3IMkbC3Qk+JRGOYiGbtLwxCx1R/MulFWoGNZL4Xzc1ymsZ3oc0kbZAlauxFXmfwyuqfa1DW0pM4f2RoY5ohEHCPfGVciJpm0OiqcU+Sjg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't use FOLL_PIN, meanwhile hugetlb seems to not be the target of FOLL_WRITE either. However add the checks. Namely, either the need to CoW due to missing write bit, or proper unsharing on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page(). So we don't care before, and also for now. But we'll care if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK properly, as that's the gup internal api to mean "we should unshare". Not really needed for follow page path, though. When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. When error happens, instead of setting page==NULL, capture the errno instead. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/hugetlb.c | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d04ba5782fdd..4410139cf890 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6462,13 +6462,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, struct page *page = NULL; spinlock_t *ptl; pte_t *pte, entry; - - /* - * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via - * follow_hugetlb_page(). - */ - if (WARN_ON_ONCE(flags & FOLL_PIN)) - return NULL; + int ret; hugetlb_vma_lock_read(vma); pte = hugetlb_walk(vma, haddr, huge_page_size(h)); @@ -6478,8 +6472,23 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); if (pte_present(entry)) { - page = pte_page(entry) + - ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + page = pte_page(entry); + + if (!huge_pte_write(entry)) { + if (flags & FOLL_WRITE) { + page = NULL; + goto out; + } + + if (gup_must_unshare(vma, flags, page)) { + /* Tell the caller to do unsharing */ + page = ERR_PTR(-EMLINK); + goto out; + } + } + + page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + /* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only. @@ -6489,8 +6498,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, * try_grab_page() should always be able to get the page here, * because we hold the ptl lock and have verified pte_present(). */ - if (try_grab_page(page, flags)) { - page = NULL; + ret = try_grab_page(page, flags); + + if (WARN_ON_ONCE(ret)) { + page = ERR_PTR(ret); goto out; } } From patchwork Wed Jun 28 21:53:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D22A6EB64D7 for ; Wed, 28 Jun 2023 21:53:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1FA68D0006; Wed, 28 Jun 2023 17:53:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A6FE8D0001; Wed, 28 Jun 2023 17:53:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 822D88D0006; Wed, 28 Jun 2023 17:53:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7480B8D0001 for ; Wed, 28 Jun 2023 17:53:21 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3430BC0336 for ; Wed, 28 Jun 2023 21:53:21 +0000 (UTC) X-FDA: 80953508202.22.FE6FEFB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 1A5DD160004 for ; Wed, 28 Jun 2023 21:53:18 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Akw5Raoq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=ehgaq0HXFPI6RMR/NtSX+VsXzPdcbiTPLHn01foWC6E08bXilk+5EhSzL/o3qCsclyyqkM vn43qrn0ZP4n5bitDja9ZT7s1nL7hFH8+k7eE5cPOdj6R84TV1spkszgaTSpzME9ytDPXv cZkjjNShEyAGaoFcUW9DHj1m2QKAgvQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Akw5Raoq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989199; a=rsa-sha256; cv=none; b=HBicJ0DFXe6oAJdCzNHqbWrpkDZPgPUve0GhUfX26Wkus4uJE8UM8Q7KNG7L4unnHlvqPC 2eBUQ8Lez84bkjr1P+aeYsIQKueIU3x1yOqKjcTc87d77UxceiGbeRA4H0+ljn1ziv4K9K z9wjlHVX8Sz+HLz9N4/Ejw1VX45ybUU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989198; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=Akw5RaoqS1n+uI2Aj0PTKAoAMl+ZPR0DIHSaMZY/bi11XeUREbDsg8sjBNTr4TuuFra8iz 4+zogOxeSeW9lgdEhEMN+EPZxB8dwCr8h7qySrF3++sl2D4AqxqsBc5NIFQ1F4Qv05m5/J NYBeIQqBOU8nsdW/k+OoeFhn9DHcJ/Y= Received: from mail-vk1-f199.google.com (mail-vk1-f199.google.com [209.85.221.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-649-o1kgDH27MVWLJRsznwe-tw-1; Wed, 28 Jun 2023 17:53:17 -0400 X-MC-Unique: o1kgDH27MVWLJRsznwe-tw-1 Received: by mail-vk1-f199.google.com with SMTP id 71dfb90a1353d-4716b1f55b9so2893e0c.0 for ; Wed, 28 Jun 2023 14:53:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989197; x=1690581197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=kjGY3J/4RHI2kDKi1/Cs1ubUIAxZ3tUEs4q3nARcRZzRnneFkDUQPl8Ulr4SOEtwSi EXcKIo84QMC1W+S9+AzoZbopWP0TzEAR2w9KSSaVn9juRTDEao39y5SFJDLLenbjfjJW R/dgZOV633Q4JXeMhn0Ll79vf0f+SRxqnBFtEO7TQ2rj/Sp29u2QQByd0O4z6e9Azohf F3n22rn0ZlXcACC+Wex1x5QisyfKFGmBusVDijXrmPh8oQyVXXaD7+X7wVw0BosJiV3w ruSemJHSUapAu+TnyInkwOsQuL7pNjvLHUgHNx4GIczJjtBPHnxNF5xGB6RUiLGzfxrG nxCQ== X-Gm-Message-State: AC+VfDwp0fqHJam4m8v4S9BYaWeHzjyYWt9XMU8BORNuDcNgjKo6VSN1 nYFu6O7q4Y4kKJFhR17+cA675S4c3N0L52zrVCu+TEicYFVg4PVnz86/pxvKzjcjzSpxbkwExqf BV4vDuDcb9HLUBF+4LGxbCC3KiZY7Y4CUPq5BUVi2WEYn1ewk43dqL37NhdL9nGynk1px X-Received: by 2002:a05:6102:358d:b0:440:b763:a69d with SMTP id h13-20020a056102358d00b00440b763a69dmr12045555vsu.2.1687989196849; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7xAhw0w/TD6i35B4jAPAsfvCfbDcR0SS33V5o7YLBIW4QDiUOfL0qQyToMemXcWuE4mGgWTQ== X-Received: by 2002:a05:6102:358d:b0:440:b763:a69d with SMTP id h13-20020a056102358d00b00440b763a69dmr12045528vsu.2.1687989196447; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:16 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 3/8] mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() Date: Wed, 28 Jun 2023 17:53:05 -0400 Message-ID: <20230628215310.73782-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 1A5DD160004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 66znfbm55a9rbtj3sotbugksx3x7znb9 X-HE-Tag: 1687989198-240217 X-HE-Meta: U2FsdGVkX18I3K68TVZL6V5bPKlcVTw5Kr/MJteVx2LVPeAm0xqWXp5sK1cErJjIgdP2VWRDMw/I48VC2PSdBxIPNnb4EnGDBsYlsUrYO8qDOcgn4mDv8mwsF5gAksZEW/vhBnge9jHBxjolCTJ+PTB1HirU0YovvNLXs7i2NmhyvwTUk6lfEZf6LmajWr37NO26ipNpibwuCl1uzjwNBWSHuC1yKnEwO3oI51HjIV29ftOyUtp6YdMC+ksFFGZpvXDFMPnEMrD28/1BKHAvgYF3gYo469MaomakAl01M1ocpXGjc5734a5YIt6t7LJWJNIKzSiRPXra0BcgNvRfcAbelZpnta030NrNZHioEh6nltQzteRS/hhxzDPR+/P1AaGZVaLkb/EDcWBK3Hy7DmSD7z1z/+eU79sBVZmgYEG73rQ3a0XbL9xpz0DHglpBDOTzSH/xJhln59cNC3gF1odoGbHXgJ+RCKJZzxr76Bj29z6n9BuhFqaxRiWWwL/yq0CwiQOeRlR4Cmj8yoVL2IAdM5jW2gLO2+fhaOPhHDgOoMAj4M7w89YAA2TtmOHYgdJ0wBFN0ZXhdfv/dUkIAa5EW55Axqf0PY8Kl8LF4owbXuZrr+oP2HpSAqDgx377agqUA11gtMkvulJ5rQ/7zYICMym/HKATHFI9EmZsntO4Y9kBh/1X3Jw4I0ESaCVibKUUKGc0A1+ndNkRmoPGRvGEZakTg/EoXT7HgQMxRWWUYYoHJKHjbROr8QjD33EmgRtLCzCqD+uu0MO8oKERTmJ/2fYtPeSncp63VCYBmX7s1hWBeflUGKUis/P5EkU5GuftLVIPVh3MN0fbst8kGehgVd3vyOG3gof6jTGsg2Hrg+RGaKXESCxuH6wjqtHGoBER9L4syTiRIHbQkgK9Lp6RC848PmxkwfNUyJAbzE0vD9WY0Rm/ZxMJ17l+kd+HP5bh4RU5p9iGcFdu2Ib WfrU5r7F rL+6fKs+zKCgtAdQdN/4dIVfGCQaUJB8sciGOCtH5D+RWNmb9uPXftlIL9kavgYjKD9AW0hy/+eGugdFus95fLekqxcCa4myiv+4I3XsbftkEK0kK+ehmBJMnJQruz9uuLZg1/al0r+9ODhDyT53H27ZAqiGivfw53/Toxv33lJ9j+snlHaKfNVV1pxndGO+d78wAgzNDJtf0NudpQ2BXMlf4SuuCqjcHX/knZjyTNYFexVO6OLUiRbkHwQlsHNl0ud6HoTgt7mt0tHRD3YA1pMXMVxEGAACTDzZjhhWWCRSBpXDyGda9d78j/+i3enKQZY8hMElvjoIDM7WVWVIRo9IilwPDRoX0Ak9EHH6MtRhQ6wq5Ah9UyIHL0Z9r4eyheQQ5SWn36XbXMlIbu6OOdDJr5Xq/a6zvDC/L7wNRia8GbGg8e5nTvMCdCQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb. Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 5 ++++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..9f282f370d96 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *, struct vm_area_struct *); struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags); + unsigned long address, unsigned int flags, + unsigned int *page_mask); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, unsigned long *, unsigned long *, long, unsigned int, int *); @@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } -static inline struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static inline struct page *hugetlb_follow_page_mask( + struct vm_area_struct *vma, unsigned long address, unsigned int flags, + unsigned int *page_mask) { BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } diff --git a/mm/gup.c b/mm/gup.c index 06bc2e31f209..1e2e23084f3c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -780,7 +780,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags); + return hugetlb_follow_page_mask(vma, address, flags, + &ctx->page_mask); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4410139cf890..15e82a8a2b76 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6454,7 +6454,8 @@ static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, } struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, + unsigned int *page_mask) { struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; @@ -6504,6 +6505,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = ERR_PTR(ret); goto out; } + + *page_mask = (1U << huge_page_order(h)) - 1; } out: spin_unlock(ptl); From patchwork Wed Jun 28 21:53:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1888EB64DA for ; Wed, 28 Jun 2023 21:53:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A80B8D0008; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 058548D0001; Wed, 28 Jun 2023 17:53:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9F428D0008; Wed, 28 Jun 2023 17:53:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CC0328D0001 for ; Wed, 28 Jun 2023 17:53:26 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A8D211A030B for ; Wed, 28 Jun 2023 21:53:26 +0000 (UTC) X-FDA: 80953508412.21.84BE2E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 6FBECC0023 for ; Wed, 28 Jun 2023 21:53:23 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KeboY+yW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=Jn11f6bAlphf/J+xebsfidABl2WedJCfMYWWbVAD8AsMoNTQKYDV7FYt8V9SrSawuZnkFS o3trCo68OaNtZyC5jFq4Y73q4p104AeMzUBKpex4Dyn+PzUDkel5Pv15EQdLIalyjOI0Ty LIAiIXqUJ6sgfDcv8PjB1X2kgh/sTNo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KeboY+yW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989203; a=rsa-sha256; cv=none; b=0OS0QhtzuWOLsxPBWl1oUICWrt+NENXAJ2f+d0PA2xRo/EmFOwd4HSzTc06sbb4ybTZwda uwybgIkaYt5TZeGENCmIMVkN5SuMhlV6hQMcxTe+gLV0cIKYetyk5WArcGsY9GGKXFqaCe jrxvHBDhzhn4kq+oP9ejAImCG8qLbOI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=KeboY+yWnMKPcw2c9tAxR3tXXzaV350QaS30sgj7axrBoxG85ZTstcCfZmBmz/o0rM5vA8 2rgQmsKBs5+E0i+G2kuCvmUrakR8NcS+Qt5pc0Yv3nz5LoOK60fPUtE9W84zZOrL+1P7ht +HEI33J9VrqWaGGFw53Cyu9qIlj+9nM= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-619-RVyHkiDJPFOXqajzmPxoTg-1; Wed, 28 Jun 2023 17:53:20 -0400 X-MC-Unique: RVyHkiDJPFOXqajzmPxoTg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-4009ad15222so27711cf.1 for ; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989198; x=1690581198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=ZBcbwG2VFSZPVlk/QXDvfqG0JhfB74wGE2Zzu+nXVGzJU8w4V6bOKKjNkz50cTblxr LgYBWHpXdNMBgHnkORlfQrkTS1Okl4LodllAbZXdrg0dqQFgBx6foh+VWeYvJDeWGLpt iYWO/zMK3si4s6lQRGOfv1dcSglj8JwkLq/T+4l+l83RC/BN/EUB7n0EdO9P4Wr/0zS6 j1VZoBbNd+gwDB9KLslQLDPJGmnhYAnzDyPi0JIVEjAERtw10JxeRYy7Mzgqgo/hq2em i6tAnYpK6zomBV2zY5Pml45m5CS3XDAO0oH1feJPMloTCQ3bX50PX2kd8PV17efaIOl7 TTcQ== X-Gm-Message-State: AC+VfDxsXUGak9PqnnR/XhcJrXytAz0sk+tHxcx2/h2o8WisDXmwhlM2 POx/NxGY2+eI9PrQQxye1paeyuRcozbNcyxT4TRxiEFVpGOFc/4aDitSQm4727aLAYNMrwC0Rpn BVFif42IboAy0eOYjhzv32EZc5zF4pgd0GzERibx7b+U8EE2ppjJ+owFj9dLJFzoEv2Ty X-Received: by 2002:a05:6214:301a:b0:635:da19:a67f with SMTP id ke26-20020a056214301a00b00635da19a67fmr13098208qvb.1.1687989198128; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4bWthzbIpu8brAzYWFel0vezukPFHw5JtwGCHRd23ZlsKMqKd/fjN3mRP22Dyi1lIzaJO53Q== X-Received: by 2002:a05:6214:301a:b0:635:da19:a67f with SMTP id ke26-20020a056214301a00b00635da19a67fmr13098180qvb.1.1687989197705; Wed, 28 Jun 2023 14:53:17 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:17 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 4/8] mm/gup: Cleanup next_page handling Date: Wed, 28 Jun 2023 17:53:06 -0400 Message-ID: <20230628215310.73782-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 6FBECC0023 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gxaoc6kjtk9shmgpw1f7xi44puywj739 X-HE-Tag: 1687989203-197959 X-HE-Meta: U2FsdGVkX1/s2J7bvXe0no06nxoE9t5mxpx0kYnbuK4jHeOPEH6odA6pQjbPHAjRrNfBkTyK+nkufq6p1V+KOTk+V+6i0x1LH7L0z+oC5RRT2d8Y9JcW87RiGp+8VWb1AEBxT58N1TccLxyB7YZFPnUPMlJz5iorlCmNy3R+TNoEEh3M4ZkcYr/soKuBgCqmCJCfP4sKLnupOPeLDM3o4Ftp/9RU48Bl3fJZa0T/XrRtkLj8htbKo3HSI7O2RB8FyoW312TvLdJH26JBWKaGb5fd9Z36mLuSvaG2urf4L0+cfSUNKvxnak2sNmdBpq7ixGqyH6Wicv3V2bzbKjZSz+iiF865/7HDhbGf5HyAU4kBqz1iLEo4t0E/kmSBtwHG8OXnRa9N4xIws+i+4WyK9fJNuhbtLCC+yQJgdfGXZlgPrk2Q5Vk9GitoI5DLSN29SzIAutt9wcTCMtBTVeyzYq9yf997oRIsMj0gPSba77xe7HfJ+pUi8tVt54krUURvFYfhHM89NEJ0ad1ngKAzrcnJkHX0V5CqV3pQ4eNH6hvGx7MUtx1Twzj5KbMsQju3N0xcu+HFdd1Cyl2Td6IFEqG+s3jXJ2553uGKEGgcTbUvczLi4XlnaDL0i3IVFhBHwY3OawSpw1/7vh2v1JM/PO4z3w6DAwu5NaQaVQXuAlDKGmgO/KuwBnqjUDBvfJ/lQygrdbfOeVuzZyDQYzIp2GGOgppKY+rSGlJRTvHfGxegH+eRYFLmg6exCCJT2byPBYmJFWS/OO3NTO+O9fnFqbiFEAf2mM1vcYnxZFWFgmHnU2BUtiTLowUxZEsthE1z0koH5jSD+Spt5XmA6w2lRw+tJ5oXnFKhKD7Itaq+QoDXPZ8Jq5cjlAfZQDEGcCJopt+wJwkRg+eQC72KemMm0KEaSkPC67PYwwfOPCVv/k2Bq6o8mZ8AozUD1P0DZUUo7rwGq81ctImsmn8EWwY +d9b9Kq3 sIbc3N3sPZAkoFWQCoVhKzFkVRAIYoLJBXFWsk0zjFgK8WX3MhdA9XreC3tApqQBfh55E5OfJYlRJcZ9mOEW2tctiFP/9T6WXSt+tIZJcETWuL3Xc4SKFs5l0gvJyR82DCy7DOUTugpLP+cQMhSGncz/gkXoKlUgrlV35GUrvZ6xd5VCXm6VNGgveYlLODb7VjPwy4cDJAeiByvjD1qkG3zOiauIDmz/Rz/Y/eV6LNGQF1MXxwhC+gvVBJmMz0aiqcVp+aQ9llIaLoDJRnCO7w8xuvQrjjVl1sIENZeDaB2DekT4FtzhtMBkd4VqAxiRRInKJoxrdpaqBNBiMWRGXgcAGbGQUctwZu+hB82aBARsg1OqL2phJAGE5Mzui292O66AYAtxD2vNg8sasWU2lO1Gipwh4hvEPsA+66INadflTKxA6NnZOsm/0vwMgowdqXuQC578c4NJYbHo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages". Reviewed-by: Lorenzo Stoakes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1e2e23084f3c..5af1be81390b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1124,7 +1124,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!vma && in_gate_area(mm, start)) { ret = get_gate_page(mm, start & PAGE_MASK, gup_flags, &vma, - pages ? &pages[i] : NULL); + pages ? &page : NULL); if (ret) goto out; ctx.page_mask = 0; @@ -1194,19 +1194,18 @@ static long __get_user_pages(struct mm_struct *mm, ret = PTR_ERR(page); goto out; } - - goto next_page; } else if (IS_ERR(page)) { ret = PTR_ERR(page); goto out; } +next_page: if (pages) { pages[i] = page; flush_anon_page(vma, page, start); flush_dcache_page(page); ctx.page_mask = 0; } -next_page: + page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; From patchwork Wed Jun 28 21:53:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D458EB64DC for ; Wed, 28 Jun 2023 21:53:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F27318D0007; Wed, 28 Jun 2023 17:53:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E621A8D0001; Wed, 28 Jun 2023 17:53:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDC128D0007; Wed, 28 Jun 2023 17:53:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BB39D8D0001 for ; Wed, 28 Jun 2023 17:53:24 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 75A8B80354 for ; Wed, 28 Jun 2023 21:53:24 +0000 (UTC) X-FDA: 80953508328.04.9B27A63 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 60E024000F for ; Wed, 28 Jun 2023 21:53:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gc5+DCG9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989202; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=w2cZr3FyzlYqF98noECfezA9CYWeKS6E8VMnQnxxJWhUQQIbEN64gdJJfi19UANKH3u5hM ueR6/nRcEMqhvyDzptXZktvLqVExhNzVyfM7ZkSNNLQUZ1RbJzRW19Fm9ucj5u4TGich/h lLyAr6yyGFl8W6qzkiJ4HkuUoRB0Dqg= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gc5+DCG9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989202; a=rsa-sha256; cv=none; b=wLgiyrbgJgC4MJ40CbppWAVMUbjZr/dAUfIe9MArcBFafufjj3VqEu0KuGXtAkvtArGBBS 1wbSqx1l21TzC4Vy+PcJphyayPNDGOWLExIPA7vhgvqg5cF0gDW+6BbuGFD72NYtgHjrgi 8/UBJiyC2eG0xY5lJh7goVWVi8kdWLw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989201; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=Gc5+DCG9htzZwbnRi3edq97hbW3GjAjfp296NpOYML+b6jzNe9qFSUdvnHE53UCEpUw7// OdEVpH0wiSC+P2nl/ajOzXusOmHA8r00nTvP57mNxtM6GfVSvuHEDbMFRueyMucJJBstwO 7xYBHSfJBNDXWGr0n4Q/eHrl9n9bNuI= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-633-UF5u1aa-MS6MWIx3cqPAfw-1; Wed, 28 Jun 2023 17:53:20 -0400 X-MC-Unique: UF5u1aa-MS6MWIx3cqPAfw-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-635e2618aaeso42536d6.0 for ; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989199; x=1690581199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=TIeLifFjp4JvvCFek+htg99GO4dsgh51FChlmXL2v55OY3e+w212a4VK0YpuiRjv/9 N5diKmVwnxX4V/ls5EvyarzQCEHYabEfTSz93KIg84lOYFQvIFbuPu2oVKkWMDO5zeXv Ie4AX9fSiAdjfXg2BmYnFhnDhtarxrlSuUgYc0YSpFVWCUP8V6uxdBMkq0Nxsp4AQ5m4 V2i+7IdkjkPqO4cO69yki25OtUzAGbVE7U6vc5uy65/A+lLhw5S8odL52a6n3gs+Y4hO 4G3O8nxUIjF1vZcFb0q26kd3AGkyErMRXzUyVBMx1qABurNLaOwfEiIP3xe11IW196TR qhFw== X-Gm-Message-State: AC+VfDzkNX5Au5RHgGwDKp8LkUg4NTEX/sZjDI0e98ohvcjS0pPWxDMf j/xNHUMwH/V4F0qpUA9+ysVp5zZcFuD1ReNpWqBNitZ6sSB7/GTP10Ox6kHA9P1/kDLA2DK0O0f +E7AnAei5uDpcDIUZYUf+iSL7DjhjHbJX1ovUjTKzvZGtVKYYQyogo6do7wOQnAvg2tva X-Received: by 2002:a05:6214:501d:b0:621:65de:f60c with SMTP id jo29-20020a056214501d00b0062165def60cmr6650332qvb.3.1687989199502; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4aH92Xekg+ZXFKVELiAnLIK82VapCJUdD+ZE2LzSIKvWv5j/RuoE6PplAHYjC4on1+uql7Bw== X-Received: by 2002:a05:6214:501d:b0:621:65de:f60c with SMTP id jo29-20020a056214501d00b0062165def60cmr6650303qvb.3.1687989199149; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 5/8] mm/gup: Accelerate thp gup even for "pages != NULL" Date: Wed, 28 Jun 2023 17:53:07 -0400 Message-ID: <20230628215310.73782-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 60E024000F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 19estdr99xjds5by6jm7uf958zc9k37z X-HE-Tag: 1687989202-295002 X-HE-Meta: U2FsdGVkX19m5Vaq4tsscf0OmKgW3r9tTgLx638ijOJe14jmpkmtmjrIFgJPtG2RFkjoR5Z6y80BQJeakAjIGgcZFwHoxnulZWe8rhEcjr3pzA/j83ZPgQLQ8+IQDACO6bbqfsEDoAp0UmQ0mMQhANF1Z+GnCYDYzFL5We+lYWPZ/IpYG33077INEqKmaDd3jqDbMxBUTjmzD0qkEmZWAjEihoVo5y0xyv3+gkOb0PcU0hAgmMrkIU2buaNDoxT0vgn2QE4LexVQ5q3xkoebLLt9ovDvmHeKBc+VeY0vp4JZfIpd7miLV6N1FiwhOOFM8fgPf0kFuVK934NsceA9hLgGfP8ebymtuWaWcdelKq0HGXx0flXdXQGE1zvQRpVWT2bpskyoLanW11wM+Vclf6XK8wSkMor2GHwRFgRTrOTXvNPEHTFiKM/j1KIZeqRVXswQjkRmSIr+QBi0C/5cODTzLFt0eqbqsP+9tH+5yk+EBJgTivT+Sg0cEVpHUADzqnJPw2+9fz5TcnREa/BJmuQKI1UGdpn+UWQ/bWD7srOu4yxeZBeiv8Wk4MXLSuTwqxSxpvILjHEbohSqJNDbegxUNGRJ7cIOJgi3BXInNxw1rEwYSAx05wti4iD7RohrJb0tZwcDj4l2xvTUtEbKzabPQwLrSVBxBrUSlgsQ0OPu2Y3vXt+u5FUw2ukXLNCf9djIejfcCt5hVDNFIEbt2nCDzkTRQn6YktqO0i7qpFkaaLOBumfLzm6qcbk+pjQ4BRT3TQFPO0AHJx/gb92kfXC6j0OyMkXJRl9fz2ROc8WvDUCQgm/8Rk8Z1gcVVLmgCwb+920R4U4fWmgkOC6Abm+83Sct2gCLPAd7OuNhE4EMbSUi2B/CT/2C1mw3Ezwb4o+hBZs/z8JkizAwk3ZQTOfZjYWJJ9VQk95Ye/clchbuWRQMwoj3DJZvUkCoLskDhjVjTfzwZwMum5NafPB uYkZdKsX nCXuSY8Aw5i2VqSO4TRoTbeyQzuCG560KYmgeYhyBgp2qptAZ89I0eWY/QpbHZOluP3dmjKLLpBsHgToaL12DyjpAZNgey5Te69nVV7ZLhn6jg/btWbSvAGz7jPJqPvIcxbqBjPrEuomlrFFbBpIWTI+axh9soLZqoD7ZvHGITDsj6Kg1IvhiMqbP9e7pxSREO4S7PguQXdDH3sovPsvS5LBL8bzdUcBhSVKSeD8U2s52enGuzwEljhy0ZQ1RP5AeAl+qkDMzVUGGXClFGq6MXWBdhS5kw/I5ZfX578SYsKLbjKQmSWLzGW5BPPz7vkzppuqmX0AzWodNmOFpD5Tw5B327Rl8wPr9hagg1IGs1ed7Gnz3v4UgZT8CpTIxw/0NXgJjJUT8QI+y2RQgmOK4cHn7YC0fqqOmuEe/AdFT12iRBUd1m1frGg3x0rZnap9b8Nvj5Ou+Cf0LQCY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL. The old optimization was introduced in 2013 in 240aadeedc4a ("mm: accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then. Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go. This can be verified using gup_test below: # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%) Reviewed-by: Lorenzo Stoakes Signed-off-by: Peter Xu --- mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 5af1be81390b..0e2b0ff1143a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - if (pages) { - pages[i] = page; - flush_anon_page(vma, page, start); - flush_dcache_page(page); - ctx.page_mask = 0; - } - page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; + + if (pages) { + struct page *subpage; + unsigned int j; + + /* + * This must be a large folio (and doesn't need to + * be the whole folio; it can be part of it), do + * the refcount work for all the subpages too. + * + * NOTE: here the page may not be the head page + * e.g. when start addr is not thp-size aligned. + * try_grab_folio() should have taken care of tail + * pages. + */ + if (page_increm > 1) { + struct folio *folio; + + /* + * Since we already hold refcount on the + * large folio, this should never fail. + */ + folio = try_grab_folio(page, page_increm - 1, + foll_flags); + if (WARN_ON_ONCE(!folio)) { + /* + * Release the 1st page ref if the + * folio is problematic, fail hard. + */ + gup_put_folio(page_folio(page), 1, + foll_flags); + ret = -EFAULT; + goto out; + } + } + + for (j = 0; j < page_increm; j++) { + subpage = nth_page(page, j); + pages[i + j] = subpage; + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); + flush_dcache_page(subpage); + } + } + i += page_increm; start += page_increm * PAGE_SIZE; nr_pages -= page_increm; From patchwork Wed Jun 28 21:53:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 194F5EB64DA for ; Wed, 28 Jun 2023 21:53:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8B4C8D0001; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F9C48E0001; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70E718D0001; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 585E18D0009 for ; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 210F4AF83F for ; Wed, 28 Jun 2023 21:53:27 +0000 (UTC) X-FDA: 80953508454.07.BBE9557 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id D316820024 for ; Wed, 28 Jun 2023 21:53:24 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Iz1r9vs3; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=oZXJKvHyVZaqtSUIjPyQWBEyVEe59iKgNjsh4sMFF48JuWzE7ZcwmqJeDviYcTFSfOuZ1j d2M9QSClQMp1pVxAajREJyffGmgkpHRvUAagiNqJDEbkSgCUD63V3Hyaes2P9hWnNjet9L lr1I8bSL85AgYCb3+eDLAjHxG9lmORM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Iz1r9vs3; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989204; a=rsa-sha256; cv=none; b=jkO12dCNVvY9n3p+tcaH9ftOfppqcvJvYGrn4MMGCA7tjDdvgosp5RASkcFMGbN9Wq1jzZ B4XKp8EvBzCTXFC8cebEeUTvUNUO1y2fHO3//+hrE7TPk1TjQ1o9lZV07D6alqGOLtUg0x m8ReC6YJ7OJ13NNXhWylT1GkDSvBbAc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=Iz1r9vs3tZZCJSGW8voEBK2lXrUCteONfCtRFIazClZEinAZQ509TNH8/wEpuKMTUPudZo YDCqmk3wcVrUN3WW+V5w6LHf3AFm4R/B4XnirtOGrguwTm+8vnk3xJtDrwJPeCldDlmwc+ REqJh4vMwSqIWw2fpQvj3Ctgcsv3RjM= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-138-RcnVuP2TPe-poR__NZlNVw-1; Wed, 28 Jun 2023 17:53:22 -0400 X-MC-Unique: RcnVuP2TPe-poR__NZlNVw-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635e2618aaeso42626d6.0 for ; Wed, 28 Jun 2023 14:53:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989201; x=1690581201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=CpW8k+W2bZZ/SHcm77kP7JxSsWRsXbmpiSBGQReKXO+/SUbyF1ywbzCSCfnwGaDz5d ZxK+uPYxHm6WSXj7iJy3Cx0bK+pAYk61ti9jzCg0wKUWYXTdxX7/cEGvhFNPZDGTamK9 uOke/6HY+r0Fwi4XxCGWFb6N+3MF/p+kZJ6oywxO040zII4AMPt1/ieHufMshX/ymk24 Ny5gOQe4uWrzasLsue4w1GohN8cTRG4AhqX7wslRIcDx0es/ZpaoeI5CEfaWg6SfOPxH 9jW1GYD9VZr8+khbZEZAcMdXXhByOubIFrBKRSlZNvsA1+9tTMlS2f8EijXI0Zvr7+kx p7KQ== X-Gm-Message-State: AC+VfDwUcVTjl3myF7IwMGcP3RRVPvRsVQpAh4Ca1OfHKZBovOHBe0iZ Js5NmwXYQ/rDbbamwZgP7pqrS4XdzxUPyqZ4G+SucnU7NwCo2EOhjs8YzIBYFAW29m741xjxQ92 ihpMNdjxBu80kpnILTUbB4pHz7gJ9/+pZGIc8UFSWtWO6nO1ZlnP0QvfCI0bvP2VITRCz X-Received: by 2002:a05:6214:c6c:b0:62f:f6ac:abf5 with SMTP id t12-20020a0562140c6c00b0062ff6acabf5mr43014729qvj.5.1687989201160; Wed, 28 Jun 2023 14:53:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6IPT/qUgAkouDwxzob72SaB3s+apQpeAgKJRetCHfNucooMDrCa2F7cSJb+eYJVhrci9iAmQ== X-Received: by 2002:a05:6214:c6c:b0:62f:f6ac:abf5 with SMTP id t12-20020a0562140c6c00b0062ff6acabf5mr43014693qvj.5.1687989200642; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:20 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 6/8] mm/gup: Retire follow_hugetlb_page() Date: Wed, 28 Jun 2023 17:53:08 -0400 Message-ID: <20230628215310.73782-7-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D316820024 X-Stat-Signature: eb6sffbxejtdm7pbzxesszj5x9uenh7b X-Rspam-User: X-HE-Tag: 1687989204-507762 X-HE-Meta: U2FsdGVkX1+VI181lb5nBgt+E2f58bZgN/9i5zQ9e7Hqtyx5pcF7lCWTDvETyaTiUWmYUfQQUih0q9dfJwkBgtOzcSJVg/kxhpMN+DdeNXb1sErAJEkEUEBmeqVqq0Zzk/AJe3g1+LV/KEg+FlNNBtk3wrWm72WQ4ML2NTm1reog0dycks8R8Vo69p5s4i6Otomifgg7QGKXqzQdmvjvUbYC8ZmkHDgCdloTmIgqHO0AJGvuBT7Cb2UzI8eLJ1Bi9puZO9ulUo/GsN+dtrQUBfkdQU6Ww5uBvL94Menz84/vv0sfIkBIyraV1JfxtqSEvpnkuyeGMDcigy9pDu1/9OiMeJaFVCVioQjvKnJDvccegH3sFs9/8f9D098bnBcBG6+5dalUigDp8DDgNRZxcVRl7V4qBsjma5O0rvaR+Osb/eah/Y4JoFcRwW0QimzEr/AGAkX2LXoqdMaHQsnhpG+D5SqeZI82v6jCi55ClKVXn1bLrClP7bWuW3Hl4zWdd/M0Rq9tl1Z29iBywu+XHyJ2Gjjc2lVzzROtPQWM24yQA1RcHUyTTQ41pD6us7KN9NF/bVOPKbCx6BmU/NrEsY2UEvd1BGctVuiRcSFWU3EaqCxkPCHNtkRtqwB4uDEfXrU6rjx5gnY8bSyp7JRjEP2oVOTyD8GaS+UZwo69017n4MRt3B7T1/aq/+WoiuP6yqUuhja7QKt+hXNgCmwIIdvzeyY/jwyJZwGokV/eZheyT7zf1pxTKBLaqhhsFbiz4qyQHlg7uVC/AUfS3HKX+L1CvoFBbORbEB29a2bL4uX45g5PUm/bvsCc8kUTPkHo7vk4fwNSSHsDRug/0AbBu6wbmqmJwJdIf+8h1zFxDQ/ufsxv/bGlcaOGdACBV35stH5IUmEE48eaX+y2tENEdy/XuC5KYI6ba9W5bTxxZyqRZDK5sP2qjg1Z4wKn9wXK6YA9G+g2pZ8ZlHC47BZ UW7w0QLO 2uOaU8fy9ng/C0c3Lr83pnbQ2G7/gB7Qj9kZnRHI+yGIaHRfqgLaLfQYEvxYt+Uaquo9SUsK6RMc9QvbBUgNX026sQtNoWKW4vTZvoM1ZMB5Rko3ewhfLPg0mUsN6t7rf76jYn6WewbrjvwUvr7k1U/pcwlKT2Q0jNWbRFaye+CO/rVLBpKwbyo2FIpbZPZrS0z7Eu3usIgpLzuKPKEMMDWo2hAxQR7obm6KmAhJzXUQ1vGKK/35h1l9DLq4fb4twIgNGbuTG1prlAkyz019EDpLY7J42fZfbP3yfdM/s8b/9uMrFyEYQnjVjmbfEhrr27nObE50ukt9MyZWS285s5TPq75J8ypUMmHDpMzeK8JKPnALggK1Wk36xWGJ7HeOVvSjHj7cI4OAbgOV0YKKY3oRLx3dI73g5e5sSmSYzK46UmG+Emct0HrJmJA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path. Time to retire follow_hugetlb_page(). Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal. Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 224 ---------------------------------------- 4 files changed, 1 insertion(+), 256 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..ae711f1d7a83 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -427,7 +427,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) * * We also don't do userfault handling during * coredumping. hugetlbfs has the special - * follow_hugetlb_page() to skip missing pages in the + * hugetlb_follow_page_mask() to skip missing pages in the * FOLL_DUMP case, anon memory also checks for FOLL_DUMP with * the no_page_table() helper in follow_page_mask(), but the * shmem_vm_ops->fault method is invoked even during diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 9f282f370d96..9bc3c2d71b71 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); -long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, - struct page **, unsigned long *, unsigned long *, - long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long, struct page *, zap_flags_t); @@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask( BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } -static inline long follow_hugetlb_page(struct mm_struct *mm, - struct vm_area_struct *vma, struct page **pages, - unsigned long *position, unsigned long *nr_pages, - long i, unsigned int flags, int *nonblocking) -{ - BUG(); - return 0; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 0e2b0ff1143a..a7c294de6ae5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -775,9 +775,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. - * - * hugetlb_follow_page_mask is only for follow_page() handling here. - * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) return hugetlb_follow_page_mask(vma, address, flags, @@ -1138,22 +1135,6 @@ static long __get_user_pages(struct mm_struct *mm, ret = check_vma_flags(vma, gup_flags); if (ret) goto out; - - if (is_vm_hugetlb_page(vma)) { - i = follow_hugetlb_page(mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); - if (!*locked) { - /* - * We've got a VM_FAULT_RETRY - * and we've lost mmap_lock. - * We must stop here. - */ - BUG_ON(gup_flags & FOLL_NOWAIT); - goto out; - } - continue; - } } retry: /* diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 15e82a8a2b76..2f12da409a19 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5721,7 +5721,6 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. - * Caller follow_hugetlb_page() holds page_table_lock so we cannot lock_page. */ static bool hugetlbfs_pagecache_present(struct hstate *h, struct vm_area_struct *vma, unsigned long address) @@ -6422,37 +6421,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -static void record_subpages(struct page *page, struct vm_area_struct *vma, - int refs, struct page **pages) -{ - int nr; - - for (nr = 0; nr < refs; nr++) { - if (likely(pages)) - pages[nr] = nth_page(page, nr); - } -} - -static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, - unsigned int flags, pte_t *pte, - bool *unshare) -{ - pte_t pteval = huge_ptep_get(pte); - - *unshare = false; - if (is_swap_pte(pteval)) - return true; - if (huge_pte_write(pteval)) - return false; - if (flags & FOLL_WRITE) - return true; - if (gup_must_unshare(vma, flags, pte_page(pteval))) { - *unshare = true; - return true; - } - return false; -} - struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask) @@ -6524,198 +6492,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return page; } -long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct page **pages, unsigned long *position, - unsigned long *nr_pages, long i, unsigned int flags, - int *locked) -{ - unsigned long pfn_offset; - unsigned long vaddr = *position; - unsigned long remainder = *nr_pages; - struct hstate *h = hstate_vma(vma); - int err = -EFAULT, refs; - - while (vaddr < vma->vm_end && remainder) { - pte_t *pte; - spinlock_t *ptl = NULL; - bool unshare = false; - int absent; - struct page *page; - - /* - * If we have a pending SIGKILL, don't keep faulting pages and - * potentially allocating memory. - */ - if (fatal_signal_pending(current)) { - remainder = 0; - break; - } - - hugetlb_vma_lock_read(vma); - /* - * Some archs (sparc64, sh*) have multiple pte_ts to - * each hugepage. We have to make sure we get the - * first, for the page indexing below to work. - * - * Note that page table lock is not held when pte is null. - */ - pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), - huge_page_size(h)); - if (pte) - ptl = huge_pte_lock(h, mm, pte); - absent = !pte || huge_pte_none(huge_ptep_get(pte)); - - /* - * When coredumping, it suits get_dump_page if we just return - * an error where there's an empty slot with no huge pagecache - * to back it. This way, we avoid allocating a hugepage, and - * the sparse dumpfile avoids allocating disk blocks, but its - * huge holes still show up with zeroes where they need to be. - */ - if (absent && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - break; - } - - /* - * We need call hugetlb_fault for both hugepages under migration - * (in which case hugetlb_fault waits for the migration,) and - * hwpoisoned hugepages (in which case we need to prevent the - * caller from accessing to them.) In order to do this, we use - * here is_swap_pte instead of is_hugetlb_entry_migration and - * is_hugetlb_entry_hwpoisoned. This is because it simply covers - * both cases, and because we can't follow correct pages - * directly from any kind of swap entries. - */ - if (absent || - __follow_hugetlb_must_fault(vma, flags, pte, &unshare)) { - vm_fault_t ret; - unsigned int fault_flags = 0; - - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - - if (flags & FOLL_WRITE) - fault_flags |= FAULT_FLAG_WRITE; - else if (unshare) - fault_flags |= FAULT_FLAG_UNSHARE; - if (locked) { - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_KILLABLE; - if (flags & FOLL_INTERRUPTIBLE) - fault_flags |= FAULT_FLAG_INTERRUPTIBLE; - } - if (flags & FOLL_NOWAIT) - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_RETRY_NOWAIT; - if (flags & FOLL_TRIED) { - /* - * Note: FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_TRIED can co-exist - */ - fault_flags |= FAULT_FLAG_TRIED; - } - ret = hugetlb_fault(mm, vma, vaddr, fault_flags); - if (ret & VM_FAULT_ERROR) { - err = vm_fault_to_errno(ret, flags); - remainder = 0; - break; - } - if (ret & VM_FAULT_RETRY) { - if (locked && - !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) - *locked = 0; - *nr_pages = 0; - /* - * VM_FAULT_RETRY must not return an - * error, it will return zero - * instead. - * - * No need to update "position" as the - * caller will not check it after - * *nr_pages is set to 0. - */ - return i; - } - continue; - } - - pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page = pte_page(huge_ptep_get(pte)); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - /* - * If subpage information not requested, update counters - * and skip the same_page loop below. - */ - if (!pages && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >= pages_per_huge_page(h))) { - vaddr += huge_page_size(h); - remainder -= pages_per_huge_page(h); - i += pages_per_huge_page(h); - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - continue; - } - - /* vaddr may not be aligned to PAGE_SIZE */ - refs = min3(pages_per_huge_page(h) - pfn_offset, remainder, - (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); - - if (pages) - record_subpages(nth_page(page, pfn_offset), - vma, refs, - likely(pages) ? pages + i : NULL); - - if (pages) { - /* - * try_grab_folio() should always succeed here, - * because: a) we hold the ptl lock, and b) we've just - * checked that the huge page is present in the page - * tables. If the huge page is present, then the tail - * pages must also be present. The ptl prevents the - * head page and tail pages from being rearranged in - * any way. As this is hugetlb, the pages will never - * be p2pdma or not longterm pinable. So this page - * must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, - flags))) { - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - err = -ENOMEM; - break; - } - } - - vaddr += (refs << PAGE_SHIFT); - remainder -= refs; - i += refs; - - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - } - *nr_pages = remainder; - /* - * setting position is actually required only if remainder is - * not zero but it's faster not to add a "if (remainder)" - * branch. - */ - *position = vaddr; - - return i ? i : err; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) From patchwork Wed Jun 28 21:53:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BAB1EB64D7 for ; Wed, 28 Jun 2023 21:53:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FF4E8D000A; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7ABE28D0009; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64CD88D000A; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5467E8D0001 for ; Wed, 28 Jun 2023 17:53:27 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2D98F80341 for ; Wed, 28 Jun 2023 21:53:27 +0000 (UTC) X-FDA: 80953508454.08.8BDE0B0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 10F96100008 for ; Wed, 28 Jun 2023 21:53:24 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gwyS9Hp2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=dU645e2eNXXldCosHnUuUa4f7DNbojiKuchm8GvV8v+Dsq/lCsYu7yR16ttLrdZwKIh7Ot dqGGDAR1gsCwXxQvczGVdQoLpinNM59DssmbVfcz9A5Zp8Ywkgi9ErDsiTme0QCOHR8bLt YD1X9No/r0fzzy/yEekPnHBrWNvutbQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gwyS9Hp2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989205; a=rsa-sha256; cv=none; b=gEOhi8CmIXeZautQPLeSpI1B6zQ5OVDY63vLtKk8IBwe2XMcqU5aktFRU4d1/5F+Wpe4Z2 3piiM8Y3D72BRwgzqf/OpzyJASQ9OegH1B8kCXBUfv5cDm+j79nqCQohV2peDutAfOvd6T bVKHq2ckMbjkVEaCzJncESKyiyWWDLU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=gwyS9Hp2zlF6QLICMBjVFDcX03eJS7s/sme7dVRih7yaxHZftU+Fw4LNorMQlsrtQySZ3P N1+QT5Zk0zLDj+qq+IRtvYKejV3dWcS+EnFKzaX8e37IIad8tgn6z7CrDVv/BVQk7ESC++ YIFUEZ2BaHhzVpTNrRB8mlxPFWKEjlU= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-56-yFJiDXqNMDGdz1x2H4Wdtg-1; Wed, 28 Jun 2023 17:53:23 -0400 X-MC-Unique: yFJiDXqNMDGdz1x2H4Wdtg-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-635f8ae5f88so39536d6.0 for ; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989202; x=1690581202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=fEENmdMfROokhF8MacK1yYNmmzyxXaTV4oOZkfh3tOkyuk6WCg+4lARMDrBXEbT8LD igtWa0CvpzVVGV0koKyn76rebspM/myVWX4VSKaR8QStR9EOkOVptkNZ9WAnM7IASyPR vHs3DDTAItSTsMg2FcQz5JCMzHNm/UDfZ3L03lDHSmgDjLf99FxiEzkI6svlfGHVY/Pc L9zvjBbHb+pozny8j7PjKqOUvrlY6AEXW6Bjjhv/g0tZQzD94cttiMckd+sDcvQNZNAH tJpg6htunrsyXyDbELpRgSKaf99Nj6/G3FifPhrLVlDXmqMeA3IG4aZQ9R7qpXQxY7tb hANQ== X-Gm-Message-State: AC+VfDwEFacoKsPeQD4sAs73Mj9Jxd4DPtU5BUPgnd6nHQWY0ecHBGsT AXdaBfbT3epRDuMVTGD2Zf/OoVdzY0FPM4lsUI+dKlqjrUOowVuh5AKgXkLibSay22vdOzYD56t y9PHcTC6AudyK430WwF1dyYr7VpIw1sBKa/4dmQEro/ED/0vkKuHlwiVDpm0Zn6WaaLcx X-Received: by 2002:a05:6214:260e:b0:626:2305:6073 with SMTP id gu14-20020a056214260e00b0062623056073mr6706631qvb.4.1687989202386; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4hmEwEfHLsDUveWKtt2eKqu3uz2U6qRFVCfBKaJKRVOW25O/2xvrvScvNnG1pcxxvjL3xB5Q== X-Received: by 2002:a05:6214:260e:b0:626:2305:6073 with SMTP id gu14-20020a056214260e00b0062623056073mr6706604qvb.4.1687989202071; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:21 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 7/8] selftests/mm: Add -a to run_vmtests.sh Date: Wed, 28 Jun 2023 17:53:09 -0400 Message-ID: <20230628215310.73782-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 10F96100008 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 97esauzabxwepaffqigge3xr61oaccm9 X-HE-Tag: 1687989204-566313 X-HE-Meta: U2FsdGVkX19gCxB3RRCqOMOJl+MHVefb3VyVNol4jml4TClAfj4vBrs2HTy4cWIbeTx2ZaI2+lJ7K6JoJLUzIFDdIijB5pbPp/jrQJHIPjr9+jTEKDVbdKnnX87lZaBfbt9Jmubl17SeVx0Xkro5cxespjH4A9ofmRZEyGdOYJIhGtxtvegMej+QykIsgBYouD8cMPpAY9HqYzbPMXZfnYDRtgmBW38ax/PwLbBw8e940xq1D/YjIOKHOIB7c9m2b0Y8Gy97Ey5esIJC1i01sGdRXtXK3rH79YQgqVkjTurdMkqe30umKGun75ZaTDollveOmtOthPg76NDGxNBK8Y6s5VrglNdVFVkIvlosGTUhYVBOMtwQ5B+hwfUTUKdpfMaeJSsQSb0TwxLTpvyVP6aNGL1sAnOylShIcvou00jNvsC0Nj7OIdj5ekZImrTDW6k+rbLgUPITb11keGeabioaXFhVYWdSoSEyPx0K6UQl11QBf1Z6XZeOg921P8dM2FmrDFOHuB2LycsRnBNBKGUevH2fdB1RPFA6fKkp+TtXE5UYic/HgxXI/0KFjFIfP45PHDxwzHuBLx9qiUZ8B1wbkemHWmua6iaMJiY2xyWugF1S5Z3BGvvf2tZUsopvkmqbX0ZmFU/Qfk7THPHhNs++Y+Bhn78Fv7BcmUWbD9lAIsG94JCzTfQj+JCB53H0V9KueOdMBB0X0ERcl4gyKqVlCnz6EvqSv68JrrUCOSWUpgWLofvQ618JxcsGbjrndeLY5x/mJ6+qr0QAMaBfCX0COHKLkxdne5MxZ04VBHLj0HO3w5u5Fpq+FYijKj7F/dm5UPb5TjIH3sFWpyotDQ7Ey+DcON++ysxqJipyPlvPgURKY9vbLPYbb4FdFJBJQaO8tzi4yZ1F+jAvcJ+6lcCS1giDt/rgrEZROuDwKewlnELWKzXx09rMWoM57469peJGvwwOpugC1PaZ0/U 1sebNaRG tpRqKDfmnoNGxNGWD7pOSJMFcqdld5giMMSpuNTk4bQPfrd29vubuNQxvYRiQeyA28NtTqvmmOHJYqQAj60ZxcnToCkXSlNNKd5rP2llZz4MTUS+90kJcsCGCcll2iw8IghmldtEAoz4qDnIvsdcmXSCBSCOFbsebO3aEPOms3mZ7j4PdVomLg7kH5Pwv55qbQJggqclNOh+jsvw82WHD1Z3hgnBRHP3ddTWFbTtJASK9WvxD8rRmTsXGrcJLH95pgC+Lp2SWP3xaqKQWgSOrhKqhTHbbjK2sFHR7VdJ81fPLs/iiN/HhEX8vHWhSAtpdNa5wAlo5fnifWlTNw/rxZ1SqZxh34xboSpwfj1MeFXitBSVC9N2oESVkzAJcT97dXXQz5AB0yoYMTZctOaskcCx/+WpVbbpC0jxmEvt5WLxpS4fduGLaAuqOdA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allows to specify optional tests in run_vmtests.sh, where we can run time consuming test matrix only when user specified "-a". Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 3f26f6e15b2a..824e651f62f4 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -12,11 +12,14 @@ exitcode=0 usage() { cat <"] +usage: ${BASH_SOURCE[0]:-$0} [ options ] + + -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message -The default behavior is to run all tests. +The default behavior is to run required tests only. If -a is specified, +will run all tests. Alternatively, specific groups tests can be run by passing a string to the -t argument containing one or more of the following categories @@ -60,9 +63,11 @@ EOF exit 0 } +RUN_ALL=false -while getopts "ht:" OPT; do +while getopts "aht:" OPT; do case ${OPT} in + "a") RUN_ALL=true ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=${OPTARG} ;; esac From patchwork Wed Jun 28 21:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13296396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7FFFEB64DC for ; Wed, 28 Jun 2023 21:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 675128D0009; Wed, 28 Jun 2023 17:53:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FE038E0001; Wed, 28 Jun 2023 17:53:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 477B78D000B; Wed, 28 Jun 2023 17:53:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 37D708D0009 for ; Wed, 28 Jun 2023 17:53:29 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A3313120325 for ; Wed, 28 Jun 2023 21:53:28 +0000 (UTC) X-FDA: 80953508496.20.930E575 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 8F3132001C for ; Wed, 28 Jun 2023 21:53:26 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WOJXluuh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687989206; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=wXYmEXhOgM49zxeWFbeUrCYH5ZbnU52orSUVQVM9Zui5ikFEahOsPFLqhYEQ9fm0YvY1Ng 82IFFOu3hY3KVqBHR46VWsXPHtY7/paqXuSUit2Hnbt3Znoi6Kb7uZK6X+2TvbfxNtbm4A W7VkKkSdfuA27tkd6bVXGx0bZLvUGAM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WOJXluuh; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687989206; a=rsa-sha256; cv=none; b=OcNz0R20XBD0i3eAxWz5frXcCeTv8OOV/BMjexH0b0nWGz9TpuzBBqafmTlj2nkxNuOxBm +9xyXsRcbGZ83GtxAPR6X7MsQErgzdzzBCCjn8hFbNp6DLV5uXvXvvsusI5lW4JDXrmoRD WibOQwwiyrWoA/voHuMS9fXA+S6+zck= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989205; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=WOJXluuhvY5SE7U2TL8m+i9Q5kZWASrn3LgUVtKG8hWGL+ZzWPDayJacpXIVqiEAs5jZ5s 1RF62Iu4nY7Ozf4oKspsXLpkpn9StThKt4Pu+UCAP9WoG7Y7DVgTreMojJzHJQjRf4fDcz 2/vGMHG3owV6KMySk85dfCG+IHJRKjQ= Received: from mail-vs1-f70.google.com (mail-vs1-f70.google.com [209.85.217.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-265-geVrfBogN2i-8Qyt65LTIg-1; Wed, 28 Jun 2023 17:53:24 -0400 X-MC-Unique: geVrfBogN2i-8Qyt65LTIg-1 Received: by mail-vs1-f70.google.com with SMTP id ada2fe7eead31-440bb217ba2so3078137.0 for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989204; x=1690581204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=X0U9Y7IaetB1IjlxUl45A9okA7XN8AmsMXYD0vYiAC49Fh5j5Pi3T1Ma7lzl1rAPe7 ZLB90dEc3XlwFX3yI3oxfGlmzqotTW8B87e63bcSAy40nuxhc4hVVsBuuTkFH11CVs51 JO86+JuRCsza04uN5/x+iW44ztEDqiO5HeNS87MRenR5ZE4dZoWIsE8KVBJ+aO7hNCr5 rWjk+wp+WZsQLtfin8gj9AiBaGbiqgMpYWVZrgzFsnE9J3fwgdE1DZkEU1Ac5x+cSHXB 8TqMjTU1pUcyYaW/nbp3QQ4D5bGxqxbVyK6R8IRa6hrN6qCWxYgO9rgk/w7/PiUbPVTl 0zGQ== X-Gm-Message-State: AC+VfDymF9Rbjw8weeNnpkgc4998F6B5WKcA7v0xeqOM54hbVzZEfJoF p+Wu8Di0bjSdzp+6kesDulD8UhCO0nTBtIrcl5PQjXhxJsjCpJe/4YTqa6Hn9YoQ2E9E2RCb/kG ItrDG355tDjbZ2RfAiLpmR28chpXQFBIaU9SKhFvjmruOLEq/ZuDIUjP8D1dn2X6zZYyB X-Received: by 2002:a05:6102:3e16:b0:440:9794:fefa with SMTP id j22-20020a0561023e1600b004409794fefamr960345vsv.0.1687989203721; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ593pf/LYdqYwensAF9OSWrZV3jL3XksL0J6jNBIon+rk3QrdtClhhmqvv34L8EI9C3xofIng== X-Received: by 2002:a05:6102:3e16:b0:440:9794:fefa with SMTP id j22-20020a0561023e1600b004409794fefamr960312vsv.0.1687989203304; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:23 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 8/8] selftests/mm: Add gup test matrix in run_vmtests.sh Date: Wed, 28 Jun 2023 17:53:10 -0400 Message-ID: <20230628215310.73782-9-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 8F3132001C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 9kyp7fcipoi9a8qw699usb453amnrkqs X-HE-Tag: 1687989206-630673 X-HE-Meta: U2FsdGVkX1/BVIEh8B1Xz0frBYh/z30sIbH+gSxuhyfVrXAmySnAACty+L7/kttTtxz7hU0hMd1scRfPe0i+1cFZIu8eyN9UZStgJE9SbTL79azYEcY4dGzmRuIvbYi7OWxTBpmqAHoyj1usit/t312HDEyRDGIkjhFwqRcfVdZr8HsiAmjV6Lrh0ZXbWaqOL2+C2l0aDdVBRBFwCITO9RlcN/9Gcib/fM9M+lUdWCw9Q/w1W7NJiGsv9SWTNeZdLQDEzGbvNVt+b0ZFJ/HnVoODAKPPCcDS9mTyZy/s4KYcjTEkSEfUd09YkyCYCWwq6zSoiqENWOzBMhf4sCGIoIHFv6Af2rJAJWfPFe/9gHvTANVoZuES8GjIwC4ODy4RLps4zhRu+gvr3IEa3vqKthN6PMkZyCf1FRUU6zgk6zEMMB++Z17tJfgHs1fU1U7xjzPdR+8mFKf7z3x2ralHfTsijIzsVYpmJtEgYU2DwceYQUrpO126VgF9H7wRdXbww3Dj5YpijjwDidNFNa5M4YMSog00h+Nq3CkExVo88orWZ63aLTM2iz+0pa+rm6LIcG1RkQBW3CQBge0mhO3iflnfUHHYZY1yi0To/IcaTe0gBSNMw22UspB2kXJN1rz2sQVtyA98LwZvhnPNNVKjlSZcjig+5ZBoTS4A5mLPqOxzgaZ8omV/UFMLpVkFmAS/JvSQ96yaeUSRIenVtDS7aYBTQ1z1TYF0o2JR84JY04BuWuAihdw10BTwk5v6KFrlt0ZADRBezyqWtSbUuoZEaZn5cOjhHTATL6uwzifLZWJ+91MAm+cmvMNAWIuGjlA64yitOMbsjQa0++6jjqXG00VxEk5T6xgDPXhX4A9v+C/O12sC6UFUqTo8J4/24VmIF11fo+TzfIdyjTx2WA/X3K0h/Eo2aQNebx8BftUdTmozEbtAsO3uojoIxODuDhedA7k4EPjkKIA+y7FnGZ6 G/7mk89W Mf7sgssQsRj5ZuFrwYq1CvPsTm7p8gGrom5wB2bQKS8N/Cf9P/SK6RBNODx8IHSNZRDBDHmWV3V/bzd4Q94PYPJ6XyWkFtIZxiYVUrzAS60xokv9yv96gP88staAx7C0z7q2v5pqF9Ko056bkuDBliQptFOpvttCpmeWQX00XSsFzth8HuUZrZiCEJ9POQ+2auHNyucagq60INXeZpVTzv6K/RPX6XeZK3WO4aB9fesk8U6XR33ufhLMqxPu1KXE0DXFXxWBV/eNLAalRMHWND8fWi5Em1U+H+UAF1F33KB6Zup7KJfX5uHxTYe/H1H+wXT7UBRFAw2fL+/PABY6T6HBpZrgwtEmi+TQs2itI+02h/cbaXI2J+pCnea7FZ0EZ05S9bVkjmNwWtqvOJVdad87oZ555EISbsoCIx55wWaJNPmIoERQFQuBIfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a matrix for testing gup based on the current gup_test. Only run the matrix when -a is specified because it's a bit slow. It covers: - Different types of huge pages: thp, hugetlb, or no huge page - Permissions: Write / Read-only - Fast-gup, with/without - Types of the GUP: pin / gup / longterm pins - Shared / Private memories - GUP size: 1 / 512 / random page sizes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 37 ++++++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 824e651f62f4..9666c0c171ab 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -90,6 +90,30 @@ test_selected() { fi } +run_gup_matrix() { + # -t: thp=on, -T: thp=off, -H: hugetlb=on + local hugetlb_mb=$(( needmem_KB / 1024 )) + + for huge in -t -T "-H -m $hugetlb_mb"; do + # -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pin-longterm + for test_cmd in -u -U -a -b -L; do + # -w: write=1, -W: write=0 + for write in -w -W; do + # -S: shared + for share in -S " "; do + # -n: How many pages to fetch together? 512 is special + # because it's default thp size (or 2M on x86), 123 to + # just test partial gup when hit a huge in whatever form + for num in "-n 1" "-n 512" "-n 123"; do + CATEGORY="gup_test" run_test ./gup_test \ + $huge $test_cmd $write $share $num + done + done + done + done + done +} + # get huge pagesize and freepages from /proc/meminfo while read -r name size unit; do if [ "$name" = "HugePages_Free:" ]; then @@ -194,13 +218,16 @@ fi CATEGORY="mmap" run_test ./map_fixed_noreplace -# get_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -u -# pin_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -a +if $RUN_ALL; then + run_gup_matrix +else + # get_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -u + # pin_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -a +fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 - CATEGORY="gup_test" run_test ./gup_longterm CATEGORY="userfaultfd" run_test ./uffd-unit-tests