From patchwork Fri Jun 23 14:29:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 624FCEB64D7 for ; Fri, 23 Jun 2023 14:29:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC6A58D0002; Fri, 23 Jun 2023 10:29:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A4FA78D0001; Fri, 23 Jun 2023 10:29:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C9548D0002; Fri, 23 Jun 2023 10:29:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 79D1F8D0001 for ; Fri, 23 Jun 2023 10:29:47 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3BE2A80281 for ; Fri, 23 Jun 2023 14:29:47 +0000 (UTC) X-FDA: 80934246414.30.62280A5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 59FA340018 for ; Fri, 23 Jun 2023 14:29:44 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bGhZw3U1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KBG5VmVtD6hD7uqVIm/KwEJm6kTIhRuBKiuOHix9n+4=; b=Dyl/EiZSTjIoNKI5/WG17M+a/F3FtCw9rVYjip161rgldH/OSlipupZ2M5sccGdPDaeFjF jPxQs1q1DFlM/xokQh3I3EE3K4/0qU5UtswSnKw+zuXDvDfgxhS1+TtfdsQU+OjLa5dVqO 89zkVVcQT2Vi4+EUMH5BB6gxgqD3G2M= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bGhZw3U1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530584; a=rsa-sha256; cv=none; b=0p18H3Y7n4K2dWH5LaE/cCtatF9yMP2P7PQjwptaUuJmGOwPXZ1ZHKhWsFH0afwwHAXHC3 d9rFiTdWw23ce2Tkrvlj9S9hsTifv0CQcoWGeaXn1s6KtADCW9Etm2eXrC6tQumwO6EIKD k+KiD1T79ARHgkDpty3tzg3RjgAPGx4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KBG5VmVtD6hD7uqVIm/KwEJm6kTIhRuBKiuOHix9n+4=; b=bGhZw3U1rf9Qt+Fj57KQ3Yh0HFT7hi4M9YYUzLBryCXpa9ppjHs092278toSDYAYOffq64 BmcWTXGH6QCMRRp8wj6VMwrtJu0YB6nn47ZuJX8FRlXQ4im4LXXdIWW6ZdtfdE1azp3DN1 6LQizlXg6+JuSk8mU9W4KTtoBkscbH0= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-192-4jgGoOJCNbmkgSJiXRLP2w-1; Fri, 23 Jun 2023 10:29:42 -0400 X-MC-Unique: 4jgGoOJCNbmkgSJiXRLP2w-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6340023ffbbso477486d6.1 for ; Fri, 23 Jun 2023 07:29:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530581; x=1690122581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KBG5VmVtD6hD7uqVIm/KwEJm6kTIhRuBKiuOHix9n+4=; b=CTIMFWmdo5utyTRVK6ECM2alKlWd4tQypWetaaa78cD3nTjPiwUpcLk3wr9Eg314zn eGfgWYztdEesjnUIRM3tG78caWQHmsX9ebADhwy1NjMsCZ5McePTq7ZvqhfKPcDWPNK1 Otu44ryuI1i24LRRcFTHb/ox6vBna6uK5olgSo11Y0zvZ1wD6hzqhSStVo3M6a7VCJ33 12GKzZZRACN6PVBgDk15EwwFUYO6iYgH8F7A5a97InmLPai2WlLMEVF8zDjzHr5eOpN9 YGp95Jsz5WTnD5SX/0pwjIjYtVbsKfk04tOhQWRpqnM005snimU0vjeSqUFztECK1idW lEqg== X-Gm-Message-State: AC+VfDyIY5TEW+d5yw/mUnt+nj77up6iNU3uHOpCaIoQBOevQ2ZurEXN 0DsGgkJSOgImFnx3QEm2YpewJSEQ/C0h6Oi3U7asHcfFXqZ3FNNP8Hv3N52wAlfBOogx9DV4hV6 ppBzsnFyo58K6WZRNrGBB8Vb2iJ6v6WNPJSjo+EYDwa71NuvlbAULrDDQmFoPoWMb9q1n X-Received: by 2002:ad4:5b8a:0:b0:62f:e4de:5bed with SMTP id 10-20020ad45b8a000000b0062fe4de5bedmr25208874qvp.5.1687530581325; Fri, 23 Jun 2023 07:29:41 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ77EhCcf/GM+yw2Xot5giAOKEx2zhLBubqKJqeheIkq+W5o91Ui2eWJpfYc4H7mEvuNOB2yeQ== X-Received: by 2002:ad4:5b8a:0:b0:62f:e4de:5bed with SMTP id 10-20020ad45b8a000000b0062fe4de5bedmr25208839qvp.5.1687530580951; Fri, 23 Jun 2023 07:29:40 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:40 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 1/8] mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() Date: Fri, 23 Jun 2023 10:29:29 -0400 Message-Id: <20230623142936.268456-2-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 59FA340018 X-Stat-Signature: a3euosmihpsxz9aexr5fwijfqjt77kid X-Rspam-User: X-HE-Tag: 1687530584-734619 X-HE-Meta: U2FsdGVkX18A1RuLwjzOB7MkwpJMZKAKQmS/OuWyzfI9T7bLYbotVBLEdMlMfLVJA/5cqOMzo/firNW7fgSBLMeg0WvvYiXlIGHWYX/LGk0xvMJQLMlLuKIBHXjtHHZvbgWggwZSloGpZDj/kaQ4T3oUTY+FWz7/5m5JWAwOXDx34K52NgG6/mQGaukz7uesPBPHoZ2E6Pj88FgkN8FGoBIDNyQWIdHkdR8kNStvRT0nLnY+MauMtE6dfum/U9bDNDAcPc84aibRpqZ3GtIaA+sFWIABpaBgVfO4Dbe49kGHxEvGez7Aklm1OdelOmFW2DyB0EoslqdRXnrJkTPklKmB+WrOfwlC+8pg3NTj2qpco63Jp2vjDspS7t7JBpNWVvZ4uU7nwZKkaF5PiblOw6is0naRpqffvUEIVBAyKU0aNPuUaQImaORx0WF4G2ePvo328sePOi0U0w7Pd56GNFArzjZkk2FANrK5OgDGw3W9nCdXR00m3U3kYi01e3h5tn4jUt244wPRHsWl13ULNRa/QgJL3lpldQA4Uica4+yB7E8yAHCdNlAkpZ6rM2wtkpYV7KW0mnhNlW1hv6++wPdRGxf3eODI32CzSDDnIL6loySvxpQrJ77hicNiKfB/u/1bwZeLtzZIxe2LfHy0+AqCLBKbNvWUmPJQE9pNXgsmGwZmdYAxiYX1vp7FYbPJkNSRn7HVNJ4PimNPm+NTuem3nAjD+Za+oZhrnHrK9DJpb/sYgmT+irCOVWkECOa99JPQtBdb6kuiqnv0LppJVwAobr7ww7UQlb4t2uJv9S9NrPrGuA6ueWmhygCi/nh1MrBIzDNaJ4k3ruPDgbtez2SHWc1kBA93Som1RgeWYrp189zMAJP/mkKXL8oNT99V43+2eMKlTSQAD1TzFsC4y8hwMG5kGUAORXtnRJn/g8rGU6gvpDdKL0uDjAPZbqhUqt35dzSuYQOUSnTWZlR +AVfw4tJ awud2HlIf2uv5mGgh0UvF+2FaupuBh7lszCLaVpMFgTxJEsouEMoI/BVGA+ybwlkf3W2i+4NNPTFaxq53vj1pY0ZaIZ9q299F32GJyl5o+cY7RFDc39ZsxyhSJHp5NRJtOd69sLS9vFGstGnatZQ8NGXr8jq8vxpGL80YS7KOSIfayydV38qdDjoWAAj/v3qQSe33PhXmzc2pwULJCxAa8ZB1zYESjo+VQ2SbdZdT7nsKa1akTlFyVI1BrOxdAhd9nTQ1C4MmyWnMsOKISsoie2tw/pwXhiNPSAwo9Obk1zaFTDUaaLbn/NcmLvEPtr1mgZaFVq/kwQNPSfRieIU4i80/6m6EHkaUJ+jm/aBqfy0CE4hkxYPAy1KFgHW4DDVMAdBIQwsc0x6n1gX4+HDsVzNFltD2uRuPM7k9EHNfTxw0wdSUAhjpIQiQcRqDV1ySSJsjbl04DT7Os3Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies: - vma_is_anonymous() == false - vma->vm_ops->fault != NULL So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable. Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb. It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index ce14d4d28503..abcd841d94b7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -767,7 +767,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, struct follow_page_context *ctx) { pgd_t *pgd; - struct page *page; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -780,12 +779,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ - if (is_vm_hugetlb_page(vma)) { - page = hugetlb_follow_page_mask(vma, address, flags); - if (!page) - page = no_page_table(vma, flags); - return page; - } + if (is_vm_hugetlb_page(vma)) + return hugetlb_follow_page_mask(vma, address, flags); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d76574425da3..f75f5e78ff0b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6498,6 +6498,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spin_unlock(ptl); out_unlock: hugetlb_vma_unlock_read(vma); + + /* + * Fixup retval for dump requests: if pagecache doesn't exist, + * don't try to allocate a new page but just skip it. + */ + if (!page && (flags & FOLL_DUMP) && + !hugetlbfs_pagecache_present(h, vma, address)) + page = ERR_PTR(-EFAULT); + return page; } From patchwork Fri Jun 23 14:29:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290734 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2522BEB64D7 for ; Fri, 23 Jun 2023 14:29:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC4EE8D0005; Fri, 23 Jun 2023 10:29:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C005E8D0001; Fri, 23 Jun 2023 10:29:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A06028D0006; Fri, 23 Jun 2023 10:29:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 779E08D0001 for ; Fri, 23 Jun 2023 10:29:49 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3F58A1C8F54 for ; Fri, 23 Jun 2023 14:29:49 +0000 (UTC) X-FDA: 80934246498.08.2D6D293 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 7FE7814000C for ; Fri, 23 Jun 2023 14:29:46 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BupfHnaM; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6gk9nBOPG6j2+jM6FIDDaVYVwIkM1l3Tetg7sVWCV7Y=; b=jPDJN/chZUucEzVIPXLslsOG9lVRTrGgsdB3ot3xrWA7oDbs8keH/GiK31VljfdBs2o2P1 HF7/RcVnCBB3X47E01GOYLq0o7MMQ5gi7u96Zv3S/N2TaWHcQ561cUUiwyNEA2u2z3NtUG ybGDkg0mxC5Zf6TjOhKeZ6+MOF6xWrc= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BupfHnaM; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530586; a=rsa-sha256; cv=none; b=oH3qRLKuuPgzvdakCiTIzv5Zhrv8nOMm6UMApdbYmkg0An/imYzYx1KRkJ6pLQ7L8gQb3p O4XHrCmz8dzXTS3bO0xO69d5myXwMmQKl4QI9kGskxXMSPZ+PvOBBpqd25V4MW6VrqpUCr 3mdvCtJF6905jvZTr/73AJG+GuJdFGU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530585; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6gk9nBOPG6j2+jM6FIDDaVYVwIkM1l3Tetg7sVWCV7Y=; b=BupfHnaM5CAo6ndolRUqy6fl+iS0EGf2lChEMbNJeZmrS2uYctos5I2A9XDJiQc+1GqtHw AJ1QsYtr+LkdIbkN4q89qOwLv4YCDmoqExtvbpZJIrpm9zJjQgk9FcYDoiwi/uXvZncyXJ v7YFQ4yadrI6XCUykYznic4p0bxupeQ= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-306-tC_h3gOHOni4Wm-6b6DpDA-1; Fri, 23 Jun 2023 10:29:43 -0400 X-MC-Unique: tC_h3gOHOni4Wm-6b6DpDA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-62ffa1214edso1381506d6.0 for ; Fri, 23 Jun 2023 07:29:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530583; x=1690122583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6gk9nBOPG6j2+jM6FIDDaVYVwIkM1l3Tetg7sVWCV7Y=; b=KEXGtRKC9IepgdtQWKmYcWTsoI2J2vqxH1wM27VuoD9zS60Dyq1Y3e7l8u22JgZ6sn uwQDlaF4pelRYaRLpqYIqfZggryNQTbzRZgVIVw+IRBJYX4itN01epXqeMiS9aMM9Lus KopXn3WBEHn3qmmMN+EUUWoLMCIQz2k77dFlCS2P3o75P/qIu/cPDrbXKh1eTWk5Iobb x9v2KnKK7wEnp6SRBYANeLnjAjwxmkRYUo2Y04gXhsFnL1HfjnYeglmuxLpN0QZm0jYk a8rczyal2pj+A2Qkj+s9QGs1fcOqdlE2xXozMwqwmBfDmSPJSp72Ev84b4mqQSeahVpl VFRQ== X-Gm-Message-State: AC+VfDxjGC9isUNSbQqHFwbXSNpBh9qkesAJCKT8DXW9XOjFJ+vRNrql LG+av0yDpZ+Y9lKsIN2//wwzgd4DWGdETvnhsOhAQm4MiHdABsmNp7uzf+GJswGvu2UkKnVK/tX UWYgU5d0XT2vrshpUNk91O/oALx8eHgWzigpk7/6QPUnK7HvfRhw4RYG3ot//UPb6pRq+ X-Received: by 2002:a05:6214:27eb:b0:616:870c:96b8 with SMTP id jt11-20020a05621427eb00b00616870c96b8mr25389420qvb.3.1687530582761; Fri, 23 Jun 2023 07:29:42 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6b6BfAOlLY4UY84oPtrElMpGPs2JooejEq2dKySJ7sqg4gO6FWfBJO1DRPK0VM93X3ONSVMg== X-Received: by 2002:a05:6214:27eb:b0:616:870c:96b8 with SMTP id jt11-20020a05621427eb00b00616870c96b8mr25389388qvb.3.1687530582350; Fri, 23 Jun 2023 07:29:42 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:41 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 2/8] mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN Date: Fri, 23 Jun 2023 10:29:30 -0400 Message-Id: <20230623142936.268456-3-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 7FE7814000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 55fzt3or1cf8johi7cq7e6d1djhp6e7j X-HE-Tag: 1687530586-517695 X-HE-Meta: U2FsdGVkX1/IYOmclCnZQDq2A8jm52lJ0JoCnR2bvIBII/jO/lMKsfsuWhyxfYvk12a4AZHN5FiXH9fn3QS50ROtd2hjzimVfIH0ORzo/nDBXnMV4s1IgD8zCpbCJU0tgXOia1NcnwXYG08PLAcZ81DOM/RIxzABDL+FrSApuKJoBXdXWy+ZOh64RqdSouQSllX0RJVd42VSyJXht60i2SypCzgOlCMc8SS0+PZktzBRRjcNUxKht52UaiNmIacgetE+tRSpRfwmkm5bmM0mbnfnPtirOrSOqMy9BufW7v1zPZq0wTiseZ/Mk2PB0I4cv5JQAzmNU5GX5daW2UxkoFHLiYX84Lhi3mebAIpmfsJ/K4shUgfjc4nvQSp4z3b7H+sgedAuNwLHm44PWxvqN5XXe84QWqkX9zaEeFf30rQpUT2Wlt/q6wUUkizNikdiW65bD8q4BaIwfwHMyhBk9gNMRpIMZ73meeThMIpXoYnVghwEZuxOtVYC71mOGxVlOYqsYt96tBtUzaZSJbV9/wnQ9U0/P188ndtOQULDKr6Z+dCS2eX14GXzZ0i0wV3vCgOoNuK+9Al06qk3J2oJXqsKqls+YcNtW7ZpDwNOHD0zPT5jyA9J+nRuYOkN8OFdt5wwUKwj43Vr0U3SrlNA1t1hr5krfe1xr4+4KYTdkhJjHM/UwOWIRg0z93gjm1Wwxp+gSMkO0+zs3SB5xo5FtYZMWoiRIcs4RBWp5u/iezCtGyMV6ZO7muasqYyk3lAIOrcH3pb7bqztG3ojSlf88a+FH2bsnGMsJRbDCBfABqsPG3IGF2prIod6QOnMIjZC1DMm/0zL9BDROewINY5fLfk2D/lreDlgxRzcafqguzVpKu0gRwTeCKtiFVBnzMnySR9S+IpT2nCfPhwQ9MbQ+fns6b8uXk0e8tcuyJXy9gTrPzKlXqE69u9d55gHA3KG/oWk+ORB/9P1BlAL6JJ a0awVM6q byaRoduZyDHRoEYHdxsQHL3QB/fOCRlzC4hEBfUOlgict70BnEF/IDjR4crBHqeMa/1QRRKNWToRWdCVU4Hj5y3d5zXFzgP7Fqsz0YGjUQWznager1gqiBEJTPw5GmSUsMfaSESPq+ZSrMELSUU35G3xe3qQ/i7mzAaGTPdMtNVtbU8v+mFbEO2dCBELlsJPNgTu0lUEQ30dKzwEUAM51p0FSyMJfiSwR5cdVUY7Vs/2raUTN3Y7Mk202QneBlJ8WDXqZLOydmWwamAswQ5wIvrrf8YeGBTijckogb3IAxOli92B1FmydT0TKjNb4fbk32PLT8M6maNeeiHJiMKxUDTABRKRz57AH2rRmUm+qvWqXsS3TMviC7Nbsg2OiSFQsx9xIDvCviXiwqlHCsnWTk2RZrONbW5i2rTuaVngbd3tMvg2lAo0dzkbIw+EhJXJj1CizRhRSKWuWS3E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't use FOLL_PIN, meanwhile hugetlb seems to not be the target of FOLL_WRITE either. However add the checks. Namely, either the need to CoW due to missing write bit, or proper unsharing on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page(). So we don't care before, and also for now. But we'll care if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK properly, as that's the gup internal api to mean "we should unshare". Not really needed for follow page path, though. When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. When error happens, instead of setting page==NULL, capture the errno instead. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- mm/hugetlb.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f75f5e78ff0b..27367edf5c72 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6462,13 +6462,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, struct page *page = NULL; spinlock_t *ptl; pte_t *pte, entry; - - /* - * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via - * follow_hugetlb_page(). - */ - if (WARN_ON_ONCE(flags & FOLL_PIN)) - return NULL; + int ret; hugetlb_vma_lock_read(vma); pte = hugetlb_walk(vma, haddr, huge_page_size(h)); @@ -6478,8 +6472,21 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); if (pte_present(entry)) { - page = pte_page(entry) + - ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + page = pte_page(entry); + + if ((flags & FOLL_WRITE) && !huge_pte_write(entry)) { + page = NULL; + goto out; + } + + if (gup_must_unshare(vma, flags, page)) { + /* Tell the caller to do unsharing */ + page = ERR_PTR(-EMLINK); + goto out; + } + + page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + /* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only. @@ -6489,8 +6496,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, * try_grab_page() should always be able to get the page here, * because we hold the ptl lock and have verified pte_present(). */ - if (try_grab_page(page, flags)) { - page = NULL; + ret = try_grab_page(page, flags); + + if (WARN_ON_ONCE(ret)) { + page = ERR_PTR(ret); goto out; } } From patchwork Fri Jun 23 14:29:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB8DCEB64DD for ; Fri, 23 Jun 2023 14:29:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 19F868D0006; Fri, 23 Jun 2023 10:29:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13CAB8D0001; Fri, 23 Jun 2023 10:29:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD0328D0006; Fri, 23 Jun 2023 10:29:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 93A618D0005 for ; Fri, 23 Jun 2023 10:29:49 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5EA73C0118 for ; Fri, 23 Jun 2023 14:29:49 +0000 (UTC) X-FDA: 80934246498.10.ADFD0F1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 4085080009 for ; Fri, 23 Jun 2023 14:29:47 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c5XzPguF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kqyY+x2QLggisqHxk6p5/e1iQd0i792KL+XWdM1gQLA=; b=zFLwdQBs9j5WG2uFKddmRmG4PxuOYKkRYygIG12Ni78iL1nPEtWRDJMpNaNJUW8dDVRzDR x3Z2JWUb0o6ceez0SbKot2hMCKjvJpQzACSHb4Et3+nP5+yYnG35R3GfKLsMlKxvqAj7YC Ua8bh26pQicv00BCvvtoUojJDYmlS6Q= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=c5XzPguF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530587; a=rsa-sha256; cv=none; b=Uim5ljzB1yBhLerbr4uZc3ACItbZcSmDFEon9l7CSYrqJhB39g9WbpiQvl0d0BakCYeNbJ I6Bs+eyKpmVFHJJXSWRbQ9Qcis8OlicExEtJu0irTU/2q2NDD/JGZ/JZTLjgebd9b44GwX QyA41lzg5xmwrzkKD6CHWpJEgjHjP88= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530586; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kqyY+x2QLggisqHxk6p5/e1iQd0i792KL+XWdM1gQLA=; b=c5XzPguFXA42xWpO8G4nFzgvX+KSpQwJyiCDnV5fW1nNPLfdJ6FSyW6yJXz5x3MTLgy5Bo UgTYSl/7U2/V8FVrHaxUHboiiO+Y/2LbxAzQ9thMKRlO/aC0sK6GFm8xQU/xVD+3nAiKKu XZverMjV95Z+wQXUHCWQt/l5gwMj6II= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-664-C1wuQSNMOPiWOECO9BXJug-1; Fri, 23 Jun 2023 10:29:44 -0400 X-MC-Unique: C1wuQSNMOPiWOECO9BXJug-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-3f8283a3a7aso1822101cf.1 for ; Fri, 23 Jun 2023 07:29:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530584; x=1690122584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kqyY+x2QLggisqHxk6p5/e1iQd0i792KL+XWdM1gQLA=; b=iAfjOlZqRqzdWpfAoKQPNT2pZOawwwr633ttuPVbZNnxC+5BjSxOzmQVdATE0GAKfH WOXQh2rRuri/oF22LBSWrimahXquuHZ9i1eigVqIb0gi0n08d5Vp5Am/sbEhUz5DFLJJ LKLa7fafWtQQyffWMqrT+bDD3r+zjMl96tBypWSVxFCB/FTvKQJ9cR5XXvPwxsrY3rrm myU8QfwU87c+RD222hlcJPTkxMi6iIpk9vLNCcjS/o9W8aWFGi2jLv9f0t4x6KfPQW6j Fymqhp1W4uY6OqOOUGQBVQqN8HN2FgvHjR2bdQMo+cdvyN7AiJ7zDEdHDPvzBm0r4wcd cjTw== X-Gm-Message-State: AC+VfDyqbOFJ8wRLskBwDMtHXZmXTFKRPzJzV0pUPfsporwj1jV6tIP1 8snt9DTjoi5/3NrqTE7TMM9vKSxLckLt4gbjA9I7OxdBfKC69/sE4leIez11LieAwzhlyhVnBj9 hz/+hn/08XrRrgw99VPDiYX9gQ5vtVnMZMy+3hUtvNA5zBqBEG3528XbBWF6//aVhiYhv X-Received: by 2002:ad4:5b8a:0:b0:62f:e4de:5bed with SMTP id 10-20020ad45b8a000000b0062fe4de5bedmr25209009qvp.5.1687530584036; Fri, 23 Jun 2023 07:29:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ73mR/vS6ro7yIhMCL42Ko8gAJI2pqYLlmvye+6zRW8sb2ZlI2HrV+rMo76OgVpAmu4iXJJhg== X-Received: by 2002:ad4:5b8a:0:b0:62f:e4de:5bed with SMTP id 10-20020ad45b8a000000b0062fe4de5bedmr25208969qvp.5.1687530583682; Fri, 23 Jun 2023 07:29:43 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:43 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 3/8] mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() Date: Fri, 23 Jun 2023 10:29:31 -0400 Message-Id: <20230623142936.268456-4-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4085080009 X-Stat-Signature: 9jq3mo8oeoj6mhu688u1tng6sq7q68uw X-Rspam-User: X-HE-Tag: 1687530587-419007 X-HE-Meta: U2FsdGVkX1/wcBBL+hNzysLJCcmskQ/wWix4UduhXrJocseJwWXtRlniAlIxyCIq4Pw6LafEr5DZIJONuKmtddBRpxLrTKY3IFqqLJ1tDpjA3LdBV5+kwNw7UKCHjxKzdAs0X1j8HV3x1kQN6NN6/5PzkybEWie/QuA91Rp/RhKpyHs70im+76sx8ftT7/pd2vJHWjSfSiE87fRu2nMoaCmp9dN2wfDJsCQ2d4DJ3xr+IX5equk6hBZUd6I+ccrNiwsdgLc4hclDPFwlO9SANhiDYCrppnNEPKUeO6HmKiOvBZlsKPrvSr8jcRJwnFd4U/BG+GJD7Gx4Xmt+G90RnhiLfVssego4ru4s6VIEys9o20tC1IzaoQSbZNPW65shTxmL13ksNdIn8uQrL9S+HJEeq7y2vGYINcgqv18X+607sLPIT5TWyrMPleZjTC4NkpoGoM+okc8kAg/vKLUyYrlENSbQgWTpfrpp36DOWbtSKqD2r/nhHnEF+zDba310qrq1xK9zBzTlsvTI0Om4/Xs+TDbelXnq24+QsqLUgYRBsHCIDCLBRo03uwwiPGrvqvTsb3i6jYloj33eRtiFyUteFpc1C3QqAtXT4pPkUt1ZdL/PL4H5FGvJQ0p0PPqMvFgLzbxezk7gPMRevYsVj8MiZZS8CBeEO4F+H1470TYa42lBuXsn1xCYIaREhmzhdLtQoC7CGp/zI2lxRD6bUIZjV3BN3JoAzaldiMyxq33YR0A0hwlDk0+XaSkju/el25XlEUyB5qNR8UwawZSnRAnnBvyU2PWPmZbgjwr5/cUPCo7O0Jby9TrQDiKDxqFhQmyXBLjCxJ/jbuz+AXqMrzveGVLHM+WVWl6uDNbAXnR+ugbGayZmwq3/NNH2ZK5x/cNcHUwzatQBIQeoY1YNjSPdqfWB3EzqnkukdJXaaV8iIqGqE9wbopYaXg6aQGQjHt8Z/ZpHWjBhC9ahKY3 oxv9hhPb zk3Fa8etF2P82QkdKLlLwRMrI0seKbzgyPTmcxfJyAz9UC5kmvlxRLZTQcGNLATV+4EBCJKQVmfM64yE0Rq+cMMvO9CpvWFWQ3KnZdlgtaAXyu0QuiahTwYt6TXSi7ONhDjkr7opZUVs/MSdPddwzMEHpYH4dFONiCesgE8QEaT/uR4hz7dsAwd3f/ftBzcZvQKNW1JurbcWLLoL/0l2b5CLBWyMlbh/frxV44WJiUiUyhKoCc9N0CTRbMImqiFes6GpqHg60k4twaAjf/Sg9iY1Q4AjWGsZkUXqAhSpAI1MIfW8XXvytrLCzSW24260NAngzPhOOE8xWNgN0bZzto324mcvDrU0CqH/c5mFwv0Rv67enUafAcWim3CiSBPEmMnognhRJdEI89pnCgBEgB68yycCFKlE6bNpejOUiUdoO4pyobEnub8+qTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb. Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 5 ++++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index beb7c63d2871..2e2d89e79d6c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *, struct vm_area_struct *); struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags); + unsigned long address, unsigned int flags, + unsigned int *page_mask); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, unsigned long *, unsigned long *, long, unsigned int, int *); @@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } -static inline struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static inline struct page *hugetlb_follow_page_mask( + struct vm_area_struct *vma, unsigned long address, unsigned int flags, + unsigned int *page_mask) { BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } diff --git a/mm/gup.c b/mm/gup.c index abcd841d94b7..9fc9271cba8d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -780,7 +780,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags); + return hugetlb_follow_page_mask(vma, address, flags, + &ctx->page_mask); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 27367edf5c72..b4973edef9f2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6454,7 +6454,8 @@ static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, } struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, + unsigned int *page_mask) { struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; @@ -6502,6 +6503,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = ERR_PTR(ret); goto out; } + + *page_mask = (1U << huge_page_order(h)) - 1; } out: spin_unlock(ptl); From patchwork Fri Jun 23 14:29:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52D4EEB64D7 for ; Fri, 23 Jun 2023 14:29:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E1598D0007; Fri, 23 Jun 2023 10:29:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76D318D0001; Fri, 23 Jun 2023 10:29:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D4EF8D0007; Fri, 23 Jun 2023 10:29:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2191E8D0001 for ; Fri, 23 Jun 2023 10:29:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5D91BB0773 for ; Fri, 23 Jun 2023 14:29:50 +0000 (UTC) X-FDA: 80934246540.27.1C492A1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 2C97D4000B for ; Fri, 23 Jun 2023 14:29:47 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=X4mGpUJu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eaaAtm4E0YzeLzCpC+sle/rLwyV4UWhhjWGc4/kvdNw=; b=aXmoQyiyCFQ04m5nCCSCzgvhcGTYSavhNQ8OzzVkusxrn3c3aeTKtApKaXJJeGHhVz4pGS VxSXI4LyOX66LGt5g395b4wYylv3DvqvlPrxpnbOQmeMXOS9KHdDgoJv7IHEHzFUeODwQx FBHZa3NQ/MX1ySQ0ZRyGt1EEXpIloNs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=X4mGpUJu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530588; a=rsa-sha256; cv=none; b=FpPGYBwYPHxKtl4IlB60hZ+i/Rl9T3eOgG64OY+fBgKyL9NlUNSrRzFaP19wj4yXZzXQ/D YwYJ575O68Q4CEgMfrLhJ4EhtHdRs8KfW6PM4+JewCpTpDKDrKTKbYwn5MLOrOwxGHf8ik mPFbpTNgo8teLjt266EM1zosl4Bs4NA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530587; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eaaAtm4E0YzeLzCpC+sle/rLwyV4UWhhjWGc4/kvdNw=; b=X4mGpUJu0HM6D/wdIQYuxUSGyIomYMKWFN3eXeeKfT4QajP7Y5U3tMc6B4T00EH9+kwAZE S4WX5VXQTJ5AK5sPTRFcro/hknHP5ab/pMYS7wYUAuVGzLA9jDvGAWiolA+kEV+XoD/hVP PBa9z5buZbfxdjyjPs8A+h2ezP1juqU= Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-500-4kPAkvUWMOCq6zKq1tIVyg-1; Fri, 23 Jun 2023 10:29:46 -0400 X-MC-Unique: 4kPAkvUWMOCq6zKq1tIVyg-1 Received: by mail-oi1-f197.google.com with SMTP id 5614622812f47-39f2b870dd5so48947b6e.0 for ; Fri, 23 Jun 2023 07:29:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530585; x=1690122585; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eaaAtm4E0YzeLzCpC+sle/rLwyV4UWhhjWGc4/kvdNw=; b=Akl0GEbCc/f993LjTQ7m79Vyeo7dJzElIx5iFFxwm+NG9cI5gbFWlKSXVuE1kFxeRj dZOO/E50rOGOVFiSSdt35Th/3PO8CmpKdmOs24Dz/V0PPuwQ48PQoi+T6ffxIDdM4qIr H4yipoOTjslGASmLTYbj+y3LJfd3awkXeEyGqtj2+rxcSSghoSXb66D4ttYYRSbPGajY P8BeOVMZX/5i3zkaP4XunJ5Hs5q2k4SVutKbWld3gJzarlNUzqmpf4P2/3cIX6TQE3Zy hFmTIAWqQKxkph369mveEGWbNta6jgzEVnjZ9KXdtX59T90ISYwOTY9y80fjO32Vs0if QGWw== X-Gm-Message-State: AC+VfDzOC1uY/6fAePbAWrf6xJGsLfXwO8jkkyLPcOF3sF2k+uUNm7gI n2BE4owF9tw6IayFnMt9vyvcMDBE2u/awahGbWZ89py5lCk+yV7+34v4Qr33Pue7VWxGRiPKUwf DUvr3rgqk+ufPxTzZrZJuhugYrN9byiQ3jcNgmJ+ImmKjRsNK/PwHSJx2P+QSkmxSVJgJ X-Received: by 2002:aca:e1c6:0:b0:397:f116:8f67 with SMTP id y189-20020acae1c6000000b00397f1168f67mr15159999oig.0.1687530585313; Fri, 23 Jun 2023 07:29:45 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6GO65T8HReZxqXh+dPcy+HPwi177tHnPRkclvjCV4jFXu/xj5JsB/zleCQUHHU+pcmmBAOMw== X-Received: by 2002:aca:e1c6:0:b0:397:f116:8f67 with SMTP id y189-20020acae1c6000000b00397f1168f67mr15159971oig.0.1687530584997; Fri, 23 Jun 2023 07:29:44 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:44 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 4/8] mm/gup: Cleanup next_page handling Date: Fri, 23 Jun 2023 10:29:32 -0400 Message-Id: <20230623142936.268456-5-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 2C97D4000B X-Stat-Signature: j7u9xaetaha884xy91uurhij8sfwhk5j X-Rspam-User: X-HE-Tag: 1687530587-173249 X-HE-Meta: U2FsdGVkX1/8juwosuYW826+V4KzTaJ6MKRbHxI3hfq/CIP+M7i0XVGUEQhc7nvkUKuwUq74LsRFaV41VDAvEX4lp5oZnyXCRYchq4CdfyE13gP6TOcbifaKfVJnkOI+XQCy464EsHA/dCO260jOhNzZnw6+i//CGepyVLVKLR4FaEc8jP9W0uluDXit9LQFK2Y5P2KgPHkpfgE/q4VA8jMHgPqzxmM6J7SD5R3rNRwGsHZNv/49h/Hoi9Kvd59sRKG0Nzjv14KwhJgaR6dRhay2HUCyXwv0DE5IXaVQwFoeNJCItQ7NTaR7Tnde49FXEtiJbZaL52RwUAAZE92Tcly5egNwu9itZ8EWx0NziZsnlPKIFkfL7jOcJ7721lOZKpZ7wx4Cp2qR0wnSEi+KhNOh6Djz2lAXWPZykSyF4NlD5diy9VUYS0Y+PZup54UqAKpNaEwHQJlAm9pMJ6L0X9jmeN94d/5v5N7v5RMAIRB6dDwaef2VMAQ83cUxzimpFJmPCZoLkiSq5VehLRSKtAwpK+8F4QdQBLveoUXOFGPIQAncuPpcS67Wx36rEJPsAoEcAaU0gpVdCk6a2AWyIHaFmnbVyzox8O91MEGcT/wJCjrpGLn7nFEmmu8uE8D0BBFgg/RCiFI08XXrbwHrweNhnKwq87UunlRDxpJ2R97WGnvwBI0v7BvluJu/WmQzKXDOA7KSBOWqaPTVQbNhT5APyyNQYOZ9fGkyeLYg7xNuPj1LFz6JjCnSr9FXD9eDHzijPgIS7j0B90lIGCjNjPYy2vzbEEngFt+5gnOCQW8MoAN9C0OYznohzkONtuAbg8SHEFW5lPoo6tfZPoYPbIr0D2C1BOAhDGKYYUKtOCjDm4p3ZM1p5a5KVq8KLe0sh4LjgsX5RwIzmorOY7c6xE+wd3KGVGrYvL1V4Bs9h2LONG09jfzTXYzYmqm+nAa9IkMWoVJcSU46R0Bvnj3 XYdlZUEQ izsRLFdct4SiIzuQbJoDg3VFdR45XlM3CHbXrI76+iIIEOLrsFU4EHl6OaHPPA4bLodZ5RnRSStH/0XQenNeBC/ddruAXnagM6jU19YP+5pJDPFoiz9MpUK1e9Fhfu/GXGG0ERQ3J/lag2tbYt4L2bSBQAhZ6Szx1q/sdrnFZ1sF8YB+/zldbJWCRk+iLkqCZ9f/s/xWoyTFdNjFpVWQjZLxUOtD9UGuqBngFU01kXBCpZTPkPnJ1EAmBNG5DY40xQIEbr4Lt6OpRl+kSiG7KEpNDobuKisLdE6yJ4z6AHlHtwu8TpVXguh0V0t/Jvyzr5ZHiRQaZNjVy5E1V89JStOnR3S9Zgb0kQeb9j9eATS6hp5ytl56Cs1xP8fIuvsZqYC9m1gCnLHiuP4zUtCnhBAlWmi+Gu4Yn4+G0SnJoc9bOXRp7DIcr1MHYWWtEJ2rJLNtjrLsSFC0LbZA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages". Reviewed-by: Lorenzo Stoakes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9fc9271cba8d..4a00d609033e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1124,7 +1124,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!vma && in_gate_area(mm, start)) { ret = get_gate_page(mm, start & PAGE_MASK, gup_flags, &vma, - pages ? &pages[i] : NULL); + pages ? &page : NULL); if (ret) goto out; ctx.page_mask = 0; @@ -1194,19 +1194,18 @@ static long __get_user_pages(struct mm_struct *mm, ret = PTR_ERR(page); goto out; } - - goto next_page; } else if (IS_ERR(page)) { ret = PTR_ERR(page); goto out; } +next_page: if (pages) { pages[i] = page; flush_anon_page(vma, page, start); flush_dcache_page(page); ctx.page_mask = 0; } -next_page: + page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; From patchwork Fri Jun 23 14:29:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FCD4C0015E for ; Fri, 23 Jun 2023 14:29:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2BD48D0008; Fri, 23 Jun 2023 10:29:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93FDB8D0001; Fri, 23 Jun 2023 10:29:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CD3E8D0008; Fri, 23 Jun 2023 10:29:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3C55B8D0001 for ; Fri, 23 Jun 2023 10:29:52 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0E1E8409A5 for ; Fri, 23 Jun 2023 14:29:52 +0000 (UTC) X-FDA: 80934246624.02.78E2FF9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 9152EA001F for ; Fri, 23 Jun 2023 14:29:49 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dsoLDDaG; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZmCvEzGFNb0TFjW2MTydxEZnQUnXGwjoOel6yGTr3Bo=; b=pq7GHVKcC3AqOgDKJRj9U0uaIFd4k1BCUz+gB0fZupL4HlbH7qtBtIOBLYrHiOERsdv5ei VcNQg4j2fSHgc+iN13lDtbDpS0S5GEEy04cb0mFEbMdZexQU2WdRTiWnymAT4uubmh5NU1 ELF3HlfMqkXFO+0BYsC+FyO0zQkaljs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530589; a=rsa-sha256; cv=none; b=kehkO35auq9p+Yno0Ih7pxxDgBsWLxWGMNI/wGesoFuhcmM9G1anUuqYxln972Egz8Io3V c2pgaFQERFNLQh6Kre/zPD8N/mvjQ0K+1XACi+wCjDPEUcsXFpVjuip0xCafv0txPVlOSj SM2tRTyeUIgtkrqrnA3E8yDq6yJNB58= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dsoLDDaG; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZmCvEzGFNb0TFjW2MTydxEZnQUnXGwjoOel6yGTr3Bo=; b=dsoLDDaGhwGMVpYiAVRuUvN+lCzxw878Ko3VE7deO1SDo69qCo//F5iWOhdT99mjJ/o1g+ 62M4R9fmQwQ55fRmgb91Elv1klwrGjoxqNICh/e44Q0S2GpUeFv6yMeX4m1v1d3Zg5lMaG UtljpoL0fRH/ZrtCKO36JWQtuT/2vF0= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-567-RjX9wvPHODGS26OFN69kAg-1; Fri, 23 Jun 2023 10:29:47 -0400 X-MC-Unique: RjX9wvPHODGS26OFN69kAg-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-62fea7a5de9so1325286d6.0 for ; Fri, 23 Jun 2023 07:29:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530586; x=1690122586; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZmCvEzGFNb0TFjW2MTydxEZnQUnXGwjoOel6yGTr3Bo=; b=jyi85oroTxWF0tbaEVRBb5UhVnxP8vyBR4yblVM7/qg7z/fY5XE5GytXvIqwMtKcTo +dJndyDhkA9v2FEZCm96eMd19KEhf0f2cUzIhvSloY/3W4BcYPn/YJJZ2OGa4ikQ6ql0 R+6DgMX7zuf6Ierj7YNxJA8FTE8DO3VhorRd8LFpKTH5UdtjFpl/3lI7nGBNAgt+aY9E N6PGRexN6D1/baRQR5Xm3HsdHN1H1RDfbIyHxIgYca4OwKdcrpQtL5cyLK49xi9CDwW3 ljzdOtrfH6QLmZuwb40xlFX3h9ynGiYUkR6MuSLZOd3TC5iYTvzDYfAFAOqbdV5w/3Y+ LDWg== X-Gm-Message-State: AC+VfDwx5KsX+4NxgcgmI4sS687cQmjZDUI4bfL115noLaY6F+TqDkJo iuB/RADm/isbkH23c6V35P3nqudhj6hdJTBpsjwmupLWOTkQymLj7ZkzWNFQsbJcx9HA+hxE/Z2 eKDwSDvaQNnVQsxy4/k9RRx+I41A4Q4ygx6SNC/jfdg9zuuUr6E+tNj+PrvDwncS6JgRV X-Received: by 2002:a05:6214:300f:b0:625:86ed:8ab4 with SMTP id ke15-20020a056214300f00b0062586ed8ab4mr4610266qvb.3.1687530586680; Fri, 23 Jun 2023 07:29:46 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6oG/I/D+2ou09cQcTlfX+z4F1YPxjXAbS6ijyg5lFwMxdYPO8Sr/TxyEq0M2ORmBAvTkxKmg== X-Received: by 2002:a05:6214:300f:b0:625:86ed:8ab4 with SMTP id ke15-20020a056214300f00b0062586ed8ab4mr4610231qvb.3.1687530586290; Fri, 23 Jun 2023 07:29:46 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:46 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 5/8] mm/gup: Accelerate thp gup even for "pages != NULL" Date: Fri, 23 Jun 2023 10:29:33 -0400 Message-Id: <20230623142936.268456-6-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 9152EA001F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 736se3i9s84ipfbo4rwwum95gycp84us X-HE-Tag: 1687530589-584240 X-HE-Meta: U2FsdGVkX1/ydBTirPl4QBHYebBDM5JYl03vdPULDvaDkJOvt6/pb+I7AhVVp9wIepeHgfyqfLbreLqJYP4y71rreZ0Gz+bIpMI1p4fUN7ol8ydfOynHCKG8sbso34LT6IBX5hN9j72VoMrzACfQ9adRbVj5kNhPt/0VsHGUTVRI1BoMNFimcaXQZbZttGxebRtYzx3leheFAtd8csgRyc79COgJn/HV177qctBFO6xFsYMsRjBkUNHZptVFItbdoPPpiDOYJhU/uE/ePKV/Jf2nYXKl8sVhBjJVIOdmCH4meQplSAVuDcvQwJLao6zhSH29mlVQkJ+ZyixEvrywY5f7tEAY+i183AwVcXLZuGoS4zWyHpsR4/KiL3jBuUnUvYJg38m9NO/MjOCFy+Ru3+61jO5dIuJAFTbGGt8J2tGaycRxnAApVfDeLRoag2JLQZ8SJileBrvK3eIVlbzjwhH5WuH5XIgyTTE5CbvnC/I63k+eKhIcyqpiJFWml1G/5Ydp5+u8a5PXW+WZ5TPm3SCsABOy9Aou0nbonjS7CYcYYVSLPgszahqOQs4DW4JiiODhwP4RCGfLhVWoeEJU8hfVHgMMSK/avy6ZoWgcIT18qIpTUq7++rMpRHkPWs7k8g2X/kP7Xhez7ng+ReZFhobxsYy/NsDWCCcEcdfZMm2jPrJBtql3BgxfxG6KCInMArtyZZwQVtN3GhFKcdYyQFAJLPGzyzLDSPRP18nhpbwIQrffm43oOghwQzq0exTgAjTgaqNrWH4Q4RG7Bcxak2uJcOiVfwx3VMKQ0zuBIQD/EWoICla4jxJRimSSiiGdXzUb5RUaXjF4FOMMlXKV+hba+674QhM+Fs810aJUNg2mTDtXdcTpPsbmmQ5METgBka2wYEQq3ChhZTFPCt9olCoMPU/AiV6alusV3Zvh7mt2ku2hh1XAf/SYdiWa5ntalR3mJR1auBZwKSHBElU XT0vXbyv 0mOXavvzYH96J+bNOLz2IYsWWHTQnNBY6/FzRHXmai6iYmWyrEVkj2gMr/Br6FaPU9yBLuvf9M2gAr1wo2uqUsEMvmUUtBwxSeUZNhDv2yLQN4iAETkXo8+PD09cJCbbGnjY3oGgZ2BT+rvkyFtDfWXEKsFnFvLKJ3EodhKTtkZ+had9ESjg0AcnRkDDcqXfA5+61SttuWhE4iXO5l0bqDLzfNtTnn8d5IAc/T/AHvfq73BW5HeV/3Hi6/C+2Gtv1eS9v/GKz50mD6e4AISKxRyYYymB1SgABWrDwNdQa7HPy76JqOejah746mArfdtURhx0laKzxGbc+noWJG/4HQrxp3SVxO00HO/HzXJ8FQ0HGEtjAw3KmAvoj5u/mY7w5NEsRKD3kj9GvZNijwARd/rYYiEX3AgbT/oqIfFRet8d58S/BXahwHhPxFuluMaf5nrFkZjhiM9REp0Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL. The old optimization was introduced in 2013 in 240aadeedc4a ("mm: accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then. Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go. This can be verified using gup_test below: # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%) Reviewed-by: Lorenzo Stoakes Signed-off-by: Peter Xu --- mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 4a00d609033e..22e32cff9ac7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - if (pages) { - pages[i] = page; - flush_anon_page(vma, page, start); - flush_dcache_page(page); - ctx.page_mask = 0; - } - page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; + + if (pages) { + struct page *subpage; + unsigned int j; + + /* + * This must be a large folio (and doesn't need to + * be the whole folio; it can be part of it), do + * the refcount work for all the subpages too. + * + * NOTE: here the page may not be the head page + * e.g. when start addr is not thp-size aligned. + * try_grab_folio() should have taken care of tail + * pages. + */ + if (page_increm > 1) { + struct folio *folio; + + /* + * Since we already hold refcount on the + * large folio, this should never fail. + */ + folio = try_grab_folio(page, page_increm - 1, + foll_flags); + if (WARN_ON_ONCE(!folio)) { + /* + * Release the 1st page ref if the + * folio is problematic, fail hard. + */ + gup_put_folio(page_folio(page), 1, + foll_flags); + ret = -EFAULT; + goto out; + } + } + + for (j = 0; j < page_increm; j++) { + subpage = nth_page(page, j); + pages[i + j] = subpage; + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); + flush_dcache_page(subpage); + } + } + i += page_increm; start += page_increm * PAGE_SIZE; nr_pages -= page_increm; From patchwork Fri Jun 23 14:29:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8D8AEB64D7 for ; Fri, 23 Jun 2023 14:29:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9418C8D0009; Fri, 23 Jun 2023 10:29:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CA6D8D0001; Fri, 23 Jun 2023 10:29:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A8708D0009; Fri, 23 Jun 2023 10:29:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 458618D0001 for ; Fri, 23 Jun 2023 10:29:54 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 069AA1C9062 for ; Fri, 23 Jun 2023 14:29:54 +0000 (UTC) X-FDA: 80934246708.12.B716809 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 51D5FC0024 for ; Fri, 23 Jun 2023 14:29:50 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OaOAJHnq; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P4y7Ae1fpCTtvuEzdDqcTV3xAT7ebtLUCI261jHMfiY=; b=E9PnpcuFh8rP91LbL7xsg1m87kOn0PCNlhrqKMYkMlpPQphoCJNRy+Qzlpx2tZLkX11KNg r2MhD3Tc23V4Wfkr6bN6558zkMraF1NIAc88HRIjA0jWoJB8rjnVcmmmVf7+ajCIkTe1aX G1SX1tkJfKFeVVTqnxy1hHzWvERqfDc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530591; a=rsa-sha256; cv=none; b=aUBsEuBinoq4VnIJclzLtXvIp9BZomezsumyrDfo66ijNRqiLfuC+ASyu/lRr0vVFJUuqo vS2GbIXt3I0lzqOjuAl3tE5lGNth8zzwJ/K6VmwNlZxBXHAB3bA0WIMyiZOY/JMt8WwmpJ j5zl5/CZKQpfGgQosG7GKbl2BZebV98= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OaOAJHnq; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530590; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P4y7Ae1fpCTtvuEzdDqcTV3xAT7ebtLUCI261jHMfiY=; b=OaOAJHnqBKfmRmcmO6CDsOddqEjX6OxyfBE02vWv25MyXqK5j7lRFSirqUJ1Prrwphfygb BX9Z486xlghrROn1V3MuJdSaJuYho1pF5uPMcFkR00kLcuRTjJzNMzE0uXfxz4NDj8HqBE ED2tiEkJsJygaJgnYc+bKnOMwkZNvc8= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-654-nFWoN1UIN3KVXz6QNZppjw-1; Fri, 23 Jun 2023 10:29:49 -0400 X-MC-Unique: nFWoN1UIN3KVXz6QNZppjw-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-3f9e7a1caf2so2096201cf.0 for ; Fri, 23 Jun 2023 07:29:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530588; x=1690122588; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P4y7Ae1fpCTtvuEzdDqcTV3xAT7ebtLUCI261jHMfiY=; b=TeJOANUDi8xhfrVhyOdRdEzyvsj1M+dk4hdxH/JukPxhz9JUTS1leSbhoVX6pjCDOl FRXGFdnwVcMMR4oZm0pQ0AIwjKr2eC4sMW7AOA/oOxNvOyceNEt8s86L61APbhxfW396 WLuktP++lHStsFYoL3PZoM687IOY1ZSDOMrDFtQ63uMY3iSExrkZgnnQVb4qMKYrIRVE ECoONOgt57bJerXnqI9S+1KWlPj8QrO5iCNHbLmDP2nZxRq3IZMOFjgYYYoxL7fDRFZT gzQ8wmVWedIWgoJU+7UM58k3MDIRXVY1SVZsNIY0SsSxkVBx2/f+b/lhAGJ0xrAbLB6E oz5A== X-Gm-Message-State: AC+VfDxTCZzi0GHFrcdpJlrfKrGnl2yBEWDl31fzf6TwZy/CQBM1BPiG MqTAJJi6R8bKonaPBvuuZ5gLp88gYPIwl+R0qeKkUXsxz2//MSCanCWlFfm8ES+V4jqBqA3FRec sKpsI3YAnXQthMqrFPw2xCndIDfPCK1+rC8rWNcHrGXXLGDfu0vFqGHc5F0cGunIvX7Y9 X-Received: by 2002:a05:6214:5186:b0:62d:eceb:f7ce with SMTP id kl6-20020a056214518600b0062decebf7cemr8898656qvb.1.1687530588334; Fri, 23 Jun 2023 07:29:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7xTPkKWLLEE9oaBVdmvOEL+EzktAffFIbQniQEAF//E5ihb/bIjet/WBRe0QH6WnGGRv1UOg== X-Received: by 2002:a05:6214:5186:b0:62d:eceb:f7ce with SMTP id kl6-20020a056214518600b0062decebf7cemr8898615qvb.1.1687530587775; Fri, 23 Jun 2023 07:29:47 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:47 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 6/8] mm/gup: Retire follow_hugetlb_page() Date: Fri, 23 Jun 2023 10:29:34 -0400 Message-Id: <20230623142936.268456-7-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: ififujb6dtc6xppjdk1pfnjfpyyxgeb6 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 51D5FC0024 X-Rspam-User: X-HE-Tag: 1687530590-962984 X-HE-Meta: U2FsdGVkX1+UlpMIlo0lo+CrQofQaxlOaggyRF0i+RIWxwhrOyUKtqBLnTP7oJ3zQEZn64okRJh4ebpKknHlnHFb5micni7NczCMs5wuFf3mHN8NqLZZzNbEDK9tkxKY127Kwmhf/yy4KbV97zaEjV/1naSIjzc9DBcbAkfaAzFvcFu7+nNMVZiRPy3LD7EfxKtveB90QjcIV/TyRLrrjOELbygLROT8vdAagrSmNZn6F3TA64wCfdYpQ4JDubvvWpE3ldMbkrol+SzC5pM+obl/TPpzPAMhznTyDEI9IVBoxTv2X08i7CfsA956CiVH7dhVbxH4oY/xvahdTWmYLePE3HLFBbE2N9DPaNjvR7Eg5hRZpIgkMV2tcMQJ743Nlo1w8jiR63q4dnX02JOwyXpkP7tlsZvRqz3lfc68IJBGf5qXEKr7Xrqb0AAlfWsAdeuaalgqd3qnQfU/4L8NQaanI1bAAi6D3QS8jNjQqkbgNRpx2d7h8jxIAvbbpAaJtiUsf8klpGPLmyKjDGfqixSznG0Fn3A795624i2E/M2P+233oUKUVEmTiQnCADdNbKVr2sgARQ74hBiPTytAzbHs4GPHXfMbghognFPk2Eap+iW2cRwTd4ujUjZEfhCPgk5MtAe6uqTdQA1ySgmFCGRG39LZDpSW2MZIYFHzL1GW4JxRk05KXBA9Q8fQSbPUeF4b3koBTsUqeUw8K1kA9HBwMNdLeGRT5OkDjxScCNQIVJcTmgvwiwtyvU1V/MCSspnM0H1XE7ulhKB2zVaR3k+1iOa0udyfghZR1C68x3R/nZhuuez4fVPPpDSV5H4EpPcu8hymMmFHe5m05wZ6F+KFg0PPTPFnEWx0Ln4jgQnfkToQkxTBmaf77i6Yj7FIM5czRs9snR7RXCqU+Qye0NhImIKijfLTZSkxqxeAkygDwHWJgR0mxLGs/6c+YmPK0oiaYss7VXPOZSDA15E 2Gbbc/ab 5YnrUJ8pQYEkWxqqYSK1e+tqpE/XipjaSiSS5uM/X7QNSOIm3IXo0s0tx0DwzjdiSBRPK5HhSbY/W8PGvh9T5/EuuevNoudrm9n/bQKB2T/Ph+P07hMGzlBqvbRDs+8iSlIj8Rw+kAhJzTcMZ8xrVyiDYCi1yKovkCXuHRpMFYqF46IXeJe2h1ESV9tKYA1inEPImcN16hOBK6xHtapmpQub9R2IzaBW6tmZPLJucvj9u4QVQKSAu/TA8SBOGV9REPRmYYLi8NrclxxyUkgQEPVYO9bAOxCV4M7XeBSeiw026UXH5vjQX7G2DNrR4cE2Ko99uD5pFxU192TNvMXXajLi+C9oO5VT2UfiizxiD93p0bgd3a9jqMd3MDEyYFir2lcxgzwwUqmRCVwP3DnaF5e9gEDm/vHcPWdGEUfNGMxRaAtpYYaXnsJRtnQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path. Time to retire follow_hugetlb_page(). Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal. Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 224 ---------------------------------------- 4 files changed, 1 insertion(+), 256 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..ae711f1d7a83 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -427,7 +427,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) * * We also don't do userfault handling during * coredumping. hugetlbfs has the special - * follow_hugetlb_page() to skip missing pages in the + * hugetlb_follow_page_mask() to skip missing pages in the * FOLL_DUMP case, anon memory also checks for FOLL_DUMP with * the no_page_table() helper in follow_page_mask(), but the * shmem_vm_ops->fault method is invoked even during diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 2e2d89e79d6c..bb5024718fc1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); -long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, - struct page **, unsigned long *, unsigned long *, - long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long, struct page *, zap_flags_t); @@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask( BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } -static inline long follow_hugetlb_page(struct mm_struct *mm, - struct vm_area_struct *vma, struct page **pages, - unsigned long *position, unsigned long *nr_pages, - long i, unsigned int flags, int *nonblocking) -{ - BUG(); - return 0; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 22e32cff9ac7..ac61244e6fca 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -775,9 +775,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. - * - * hugetlb_follow_page_mask is only for follow_page() handling here. - * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) return hugetlb_follow_page_mask(vma, address, flags, @@ -1138,22 +1135,6 @@ static long __get_user_pages(struct mm_struct *mm, ret = check_vma_flags(vma, gup_flags); if (ret) goto out; - - if (is_vm_hugetlb_page(vma)) { - i = follow_hugetlb_page(mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); - if (!*locked) { - /* - * We've got a VM_FAULT_RETRY - * and we've lost mmap_lock. - * We must stop here. - */ - BUG_ON(gup_flags & FOLL_NOWAIT); - goto out; - } - continue; - } } retry: /* diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b4973edef9f2..50a3579782a5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5721,7 +5721,6 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. - * Caller follow_hugetlb_page() holds page_table_lock so we cannot lock_page. */ static bool hugetlbfs_pagecache_present(struct hstate *h, struct vm_area_struct *vma, unsigned long address) @@ -6422,37 +6421,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -static void record_subpages(struct page *page, struct vm_area_struct *vma, - int refs, struct page **pages) -{ - int nr; - - for (nr = 0; nr < refs; nr++) { - if (likely(pages)) - pages[nr] = nth_page(page, nr); - } -} - -static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, - unsigned int flags, pte_t *pte, - bool *unshare) -{ - pte_t pteval = huge_ptep_get(pte); - - *unshare = false; - if (is_swap_pte(pteval)) - return true; - if (huge_pte_write(pteval)) - return false; - if (flags & FOLL_WRITE) - return true; - if (gup_must_unshare(vma, flags, pte_page(pteval))) { - *unshare = true; - return true; - } - return false; -} - struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask) @@ -6522,198 +6490,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return page; } -long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct page **pages, unsigned long *position, - unsigned long *nr_pages, long i, unsigned int flags, - int *locked) -{ - unsigned long pfn_offset; - unsigned long vaddr = *position; - unsigned long remainder = *nr_pages; - struct hstate *h = hstate_vma(vma); - int err = -EFAULT, refs; - - while (vaddr < vma->vm_end && remainder) { - pte_t *pte; - spinlock_t *ptl = NULL; - bool unshare = false; - int absent; - struct page *page; - - /* - * If we have a pending SIGKILL, don't keep faulting pages and - * potentially allocating memory. - */ - if (fatal_signal_pending(current)) { - remainder = 0; - break; - } - - hugetlb_vma_lock_read(vma); - /* - * Some archs (sparc64, sh*) have multiple pte_ts to - * each hugepage. We have to make sure we get the - * first, for the page indexing below to work. - * - * Note that page table lock is not held when pte is null. - */ - pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), - huge_page_size(h)); - if (pte) - ptl = huge_pte_lock(h, mm, pte); - absent = !pte || huge_pte_none(huge_ptep_get(pte)); - - /* - * When coredumping, it suits get_dump_page if we just return - * an error where there's an empty slot with no huge pagecache - * to back it. This way, we avoid allocating a hugepage, and - * the sparse dumpfile avoids allocating disk blocks, but its - * huge holes still show up with zeroes where they need to be. - */ - if (absent && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - break; - } - - /* - * We need call hugetlb_fault for both hugepages under migration - * (in which case hugetlb_fault waits for the migration,) and - * hwpoisoned hugepages (in which case we need to prevent the - * caller from accessing to them.) In order to do this, we use - * here is_swap_pte instead of is_hugetlb_entry_migration and - * is_hugetlb_entry_hwpoisoned. This is because it simply covers - * both cases, and because we can't follow correct pages - * directly from any kind of swap entries. - */ - if (absent || - __follow_hugetlb_must_fault(vma, flags, pte, &unshare)) { - vm_fault_t ret; - unsigned int fault_flags = 0; - - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - - if (flags & FOLL_WRITE) - fault_flags |= FAULT_FLAG_WRITE; - else if (unshare) - fault_flags |= FAULT_FLAG_UNSHARE; - if (locked) { - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_KILLABLE; - if (flags & FOLL_INTERRUPTIBLE) - fault_flags |= FAULT_FLAG_INTERRUPTIBLE; - } - if (flags & FOLL_NOWAIT) - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_RETRY_NOWAIT; - if (flags & FOLL_TRIED) { - /* - * Note: FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_TRIED can co-exist - */ - fault_flags |= FAULT_FLAG_TRIED; - } - ret = hugetlb_fault(mm, vma, vaddr, fault_flags); - if (ret & VM_FAULT_ERROR) { - err = vm_fault_to_errno(ret, flags); - remainder = 0; - break; - } - if (ret & VM_FAULT_RETRY) { - if (locked && - !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) - *locked = 0; - *nr_pages = 0; - /* - * VM_FAULT_RETRY must not return an - * error, it will return zero - * instead. - * - * No need to update "position" as the - * caller will not check it after - * *nr_pages is set to 0. - */ - return i; - } - continue; - } - - pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page = pte_page(huge_ptep_get(pte)); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - /* - * If subpage information not requested, update counters - * and skip the same_page loop below. - */ - if (!pages && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >= pages_per_huge_page(h))) { - vaddr += huge_page_size(h); - remainder -= pages_per_huge_page(h); - i += pages_per_huge_page(h); - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - continue; - } - - /* vaddr may not be aligned to PAGE_SIZE */ - refs = min3(pages_per_huge_page(h) - pfn_offset, remainder, - (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); - - if (pages) - record_subpages(nth_page(page, pfn_offset), - vma, refs, - likely(pages) ? pages + i : NULL); - - if (pages) { - /* - * try_grab_folio() should always succeed here, - * because: a) we hold the ptl lock, and b) we've just - * checked that the huge page is present in the page - * tables. If the huge page is present, then the tail - * pages must also be present. The ptl prevents the - * head page and tail pages from being rearranged in - * any way. As this is hugetlb, the pages will never - * be p2pdma or not longterm pinable. So this page - * must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, - flags))) { - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - err = -ENOMEM; - break; - } - } - - vaddr += (refs << PAGE_SHIFT); - remainder -= refs; - i += refs; - - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - } - *nr_pages = remainder; - /* - * setting position is actually required only if remainder is - * not zero but it's faster not to add a "if (remainder)" - * branch. - */ - *position = vaddr; - - return i ? i : err; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) From patchwork Fri Jun 23 14:29:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 785C7C0015E for ; Fri, 23 Jun 2023 14:29:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C3278D000A; Fri, 23 Jun 2023 10:29:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FD1B8D0001; Fri, 23 Jun 2023 10:29:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 064AC8D000A; Fri, 23 Jun 2023 10:29:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DF7228D0001 for ; Fri, 23 Jun 2023 10:29:54 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A90A81C8946 for ; Fri, 23 Jun 2023 14:29:54 +0000 (UTC) X-FDA: 80934246708.29.BA72628 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 19022180018 for ; Fri, 23 Jun 2023 14:29:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PotxnGN+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530592; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3EQ9G/WyCR5BzbeYKEOc/c4ex09bzeUnnwnMR75AIjI=; b=am5Wch9DV90PKqn5yY9VbpNWgZJkVYM7XrFrAZSiEQlAuMgZI6CcVHuJuBUZ6mK2iOQqPz IPfdB5cb6kEGPGquWoPk/qxTYdPKZXXa+JNAstOajJkthxPZ8nGD8DUikIy3S/uuzR5HkU jjjgw72Z+bQvWPgWh4JnTyZOdWXXJWQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PotxnGN+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530592; a=rsa-sha256; cv=none; b=ToK9kMLMr2m9EJDRU7ahJ/36FoFXre4byuCLDqnri7gGso3tG4HODoEtV2udKu37QjeoX8 o0Q52/HLpr7+7jK+0ES+v4Rn2+s9Y9mvMSLFCwoXRxL9o/K6NBTXxh30WADi5NswnsOqMT 9a7MuzQ4fWKV5AOE8e0tfYSrr1JIgUM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530591; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3EQ9G/WyCR5BzbeYKEOc/c4ex09bzeUnnwnMR75AIjI=; b=PotxnGN+Fplvn2hJA8RS+C1bCuikGKXvSJrRN9cfwms5EOfrjeV2DFQ9E887VuXa5dijOH vLkKSK7jwaWtlGDvxSwyP+Xpy3uT634FurP0ZTGMrin0xQ5RiTrhVstRzR+Xo3/BY1tyMZ aZWdS8hz9ysjLujqk2/6nbexPYgsnxg= Received: from mail-vk1-f197.google.com (mail-vk1-f197.google.com [209.85.221.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-44-IgbMGkMpPaGFxGlQme9ypg-1; Fri, 23 Jun 2023 10:29:50 -0400 X-MC-Unique: IgbMGkMpPaGFxGlQme9ypg-1 Received: by mail-vk1-f197.google.com with SMTP id 71dfb90a1353d-471cd52f621so48248e0c.0 for ; Fri, 23 Jun 2023 07:29:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530589; x=1690122589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3EQ9G/WyCR5BzbeYKEOc/c4ex09bzeUnnwnMR75AIjI=; b=XV8RteEJfq2fR54MQP0hpeo8i6T93kAIu1L1n1Wcd6U1KlMHWID36MEg8BoKYg1jUm uOrCfxoYcr9u7eDQWw77P0AfnSSMX+8mavir8Ozhu8vYXlwuYHfR+miNS7MtrAkfA6ba EnTifZGi0MSLEEDj3qayGbvct0LiPKmI1jOTwyBX1O5b3VvaIZi3YV3s5VyOXO0Ii+P2 DzKcEkoEvh06pC/kxVl4xXZsP/Xz+tCwBJhf3xdUZ4ZTuH0osZ9/ccZnHkbPHfhpRZ9p sRVPt4WwGOodOQJGfpLY6UZ64IvQoJBgk27tBycl8IbGUaYwWbvOa1VdJl+votlGbd8n OMVA== X-Gm-Message-State: AC+VfDxYNN8PpKxaV+ntNSOKHWBQd81ypnYGOx4hQxRbz2GsZKEDFZfy q8D9/PzAIwO0YymBgW6ScAqD/Pv1n8OHCBnzh8DKAaHegG/DH2tZg/P27mEjB8OIx2FnSDAcVoP yI9wIz2KKiXwqfzYrc3faR6vtKKzGMrtJ9VS7/qtQpALwPRpoy9PxR6kRJl3PzGih4jgc X-Received: by 2002:a05:6102:2158:b0:440:a365:ad50 with SMTP id h24-20020a056102215800b00440a365ad50mr6848587vsg.1.1687530589564; Fri, 23 Jun 2023 07:29:49 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5OV7Q+bQTZ8nHFXS9cjC6+LnDnU4rBFQijCL6Szo7CDKJQ/6LTVykZKitewlwT3KIiAhTFvw== X-Received: by 2002:a05:6102:2158:b0:440:a365:ad50 with SMTP id h24-20020a056102215800b00440a365ad50mr6848561vsg.1.1687530589175; Fri, 23 Jun 2023 07:29:49 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:48 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 7/8] selftests/mm: Add -a to run_vmtests.sh Date: Fri, 23 Jun 2023 10:29:35 -0400 Message-Id: <20230623142936.268456-8-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 19022180018 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: k3kd6sqe9fp59u3z6eco5f16wwq96jwi X-HE-Tag: 1687530591-495152 X-HE-Meta: U2FsdGVkX1/5SCtqRK7mzzqZi90Wg7xXY2PxQBPjmhHwRapvfXrX2JVgOle+sI/cyH0OgIh/l3HvcSIKUb71B3zZ97HAXg5lQgHzCmyL+a6uMVMGXSYe4MTR66AovArCg1sbClmjGwfVILDBS42eK1h/WNDmOvZilr7cWiFjGSvH7aUqgtuFpIFS7Ye9Tewb8P5ksyz7/EoWmF1V8dPJhssWkDAi6c3dBY/0MF4poaAczDfE2o+eKAu/uYY0h7Wqbf3T8l4tBaFYuD2h3gW/FNrAuyotfPos7X8toqfP4s0/VdQdYPUyAWdWeTBxih3k2mCzMpn6/WNzb474Tw8DaWBBukCLO8sgJTNGfpLbptQpejcSfx/6tdVW33DxTufjL//kzm/sY6e0ESTokiv4kHPk6d8Xs3BCTc3/qPyx4y2xxLCIVd4B5f/qJLAwIkRvsURie/aOkTNJy/yIZef6E/M0JsN0sSZfnHvsz6Q9u7iEcww5OTolHSyTy7+8aQ465sFCM8r8kDblobNJU3wEwSbLCKy8ssyzZZ4ntcZ6MQShvinD7BrbGTMpPDmSRHtR27T7/EPUg+txCYjyLyFSDr3aPTbPVRUQLH3fDWeUyV6Jte01qv2SgZIx38jVIVU5of9BjVZANROrY8zV4EkRvBxgPpr5aLcubY70y3aNzgn/NhZ+OhN8gu7Px4ojMthosWIlijYJ+BfFIgsmZL5lrz4Dn8ji72QXNyzgsJr2Ajo+vXu1V8Ka8X/T5Krge6z+IzgbVTNuOQpQV3C0RAPXjq4WqJGifR5AjrGWgkxBn7A5SoaziebUKsY7fNCZMBWsJUywq9hEz6mdf7XFDFMPprCPRRZ8V9+/bkkHqSS/YQhBhNOkS84rPj+Mv6QtcEGfZ1dpCgjQggWUF6lSwE+yAvpOrnktAwKH0pqfWSMVD+jAm28AxZCnS+mWYA3xxhGm4mnsmaqSpsM/sX+7gW7 vfZiCXcT HO/MEPpxBSOBxlMTV4EFpkxY5PWjombJfLGGFS8EUTgVtJ6fzzwQdSy9cYtFo1WQYWnlI0eZa2AZAZdipLz7DqXjX8dmgVt/+J3rMJHnG34PPsTZt6nAKr+Us2FP7g1z5HDLq2cZYfNqt+h6oHaf4ymuzZ2LaEWlUlU/QcbJly0xhlR9MiPBwvz5Rqcrl/xckYJCGDm2MkMkXaseHmrYDtm8Bi67dwKyFF+rw7XtKC7t6M2UhOuvZI9+0hNPIsizUvx72sbUyyHZg7EJo9gmqQvWbP4ZRfrbejom5j/Gfu6a+oSelr6wM8iTyJRaGPPcDccroYJp1O8ArsmZzQ7bfPG8usKEbEarHP/PZiCFdZnj5GyodKXVsrzLYCMe8XhQguJDQXxwirY9MAZZpO1H70YL70I8leOl+276plhBlzLiyEZS2eHOuRupL8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allows to specify optional tests in run_vmtests.sh, where we can run time consuming test matrix only when user specified "-a". Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- tools/testing/selftests/mm/run_vmtests.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 3f26f6e15b2a..824e651f62f4 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -12,11 +12,14 @@ exitcode=0 usage() { cat <"] +usage: ${BASH_SOURCE[0]:-$0} [ options ] + + -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message -The default behavior is to run all tests. +The default behavior is to run required tests only. If -a is specified, +will run all tests. Alternatively, specific groups tests can be run by passing a string to the -t argument containing one or more of the following categories @@ -60,9 +63,11 @@ EOF exit 0 } +RUN_ALL=false -while getopts "ht:" OPT; do +while getopts "aht:" OPT; do case ${OPT} in + "a") RUN_ALL=true ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=${OPTARG} ;; esac From patchwork Fri Jun 23 14:29:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79DBEEB64D7 for ; Fri, 23 Jun 2023 14:30:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 103D18D000B; Fri, 23 Jun 2023 10:29:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 018C08D0001; Fri, 23 Jun 2023 10:29:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C771A8D000B; Fri, 23 Jun 2023 10:29:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A693F8D0001 for ; Fri, 23 Jun 2023 10:29:56 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 10E311A0BFF for ; Fri, 23 Jun 2023 14:29:56 +0000 (UTC) X-FDA: 80934246792.12.80B3496 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 9D8D54001B for ; Fri, 23 Jun 2023 14:29:53 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RrMul7MA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lJajW3CLipjyQ640k+WW23diII/jzqCXLm/r+yoabGU=; b=wOM/Ll3wLWtWfZuX+JVQa0yPOiLGFbRjtw3MXyBsGV+k5dGKHJjOC6yEFDQSoJwBe7Sexf JeFUYJ5rKOSSVl9xGpC+O+dFwql70jp9HpKrEYWcI1IGt490yuSMm76KMJYM8elVMJI4N0 nRHEdDHYJKLc1lJWklV1CL0AWOCnllw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RrMul7MA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530593; a=rsa-sha256; cv=none; b=NX5OS8D/ViPMIki7W3p4ZlECU1gCEMV8GnCmgX+TW1qmK9qhMJA+zhj0BXmuhI5uR95v1y tb+tAkV0RX+Ef/uZoobky10rdB+Liup4neSlxE0/i/vMNqsIwm6p2JR3hXp4tmPbM8Cg0J q12VnR5cPOnH8JRz2n3VISBx16LkWCs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530592; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lJajW3CLipjyQ640k+WW23diII/jzqCXLm/r+yoabGU=; b=RrMul7MA6Rk34gsaegcXF6jSniHsHVgfn7EBSkPVgS5e6IHGH2bZYKS9UhPoe+XwB2nkAV DtfIypVKDSh++BB73QR9AVoAUfxyiXtl+wJHpPaWK1MGxUhl5i4Dp1kTW8JV+z1n6vnl9L nLezkB7b60af/YoAzXvatZWTXQaqr5Q= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-322-BP8-0__qNz2uDwI7Tsh2ig-1; Fri, 23 Jun 2023 10:29:51 -0400 X-MC-Unique: BP8-0__qNz2uDwI7Tsh2ig-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-62ffa1214edso1381726d6.0 for ; Fri, 23 Jun 2023 07:29:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530591; x=1690122591; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lJajW3CLipjyQ640k+WW23diII/jzqCXLm/r+yoabGU=; b=h22OljXEJpf9SQRlA4t/VJ6N5HodXin8uGEULWTDceNK7ewsjZrN8KYYcgCMEYLsQb aa4KiXjnOQ6L5cs7a2eN2M32F6NFXbzV2RAUAkbAa0SR2/ncUunUKErzEn3FZy96ZTUk C7iOIZIHKubSYQl4jIwj41JAQeHcE37ynTfuX4Qx+TkWLMAaxUG2ehtaABcmFpXqKoTx n6etJ1KSp85DpSDnrztJpuS0AoQsq5lyrDHGANBaH2qJdtvofF9XrRaBMa2xBqrhUJjA R7a0cccLlbiDlkEJjNOPCcLVW1BJddv23PLmayhmooKWd1baYvyjqzjHsz7y2l4BDVlv 4n7g== X-Gm-Message-State: AC+VfDw+NA1NO60srj3oKGFYIviqJ9wjHYbIxYHuSPB3vYnIvxdegJXH fIEfCT+F3q6YE5+1LXBQ/nyAkQ45ltvCMsjg3Syx4AqC98S9AnRk041XaDa45a1Q0HVDuQNB6t+ M8H21UjGycgwfUZOr7vIqAqZcArFxClZ5GVvnKMxFe0IvEF+Z8aEmRWX+ehSLSsCFqzdf X-Received: by 2002:ad4:5b8a:0:b0:62f:e4de:5bed with SMTP id 10-20020ad45b8a000000b0062fe4de5bedmr25209299qvp.5.1687530590877; Fri, 23 Jun 2023 07:29:50 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4LBdflQdmHl6rCFBv/aEN13DoGpkWXXDa3kzujb7sug+FgkYDvvz3ULmIE5oitFlMfpXnucg== X-Received: by 2002:ad4:5b8a:0:b0:62f:e4de:5bed with SMTP id 10-20020ad45b8a000000b0062fe4de5bedmr25209269qvp.5.1687530590562; Fri, 23 Jun 2023 07:29:50 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:50 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 8/8] selftests/mm: Add gup test matrix in run_vmtests.sh Date: Fri, 23 Jun 2023 10:29:36 -0400 Message-Id: <20230623142936.268456-9-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230623142936.268456-1-peterx@redhat.com> References: <20230623142936.268456-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: mwah5zckxorm7fbg88njbuf5jdsttife X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9D8D54001B X-HE-Tag: 1687530593-579711 X-HE-Meta: U2FsdGVkX19k2cwrL5qzkGgM1jAOCW4U4d8UTBHyD40mSdgIn8sh6bjksHt1Xx+gcaglrMOEJ8gRIapwqeglFihkJwanwr8rDGD+NqCASNnP7xAEcSsIbrZkzur2ly/Y5PZwaHjRbp8SXGBhM0ILvsnuzp5E1dTe9CAdD4lJvic0LQeDaGC6KsYzrdUMPvCaw2CDxYtuZChn/+x28grusq4ZaiRsP4BKrS9qP/4W32aZsJfm5Gz6GGjUQ6Yk3jtUMawxo2hDOh8UQHAealLkEH7vTpEc0G4YGiu0QGj9+EEXAXHN1Q2yvCm14gzb7c3sxpTPzNyVphez43e5QXV01hS5AtkjBEKNL91vkgiUj0fS0mX8bpxg2vFsnGZKHCPSBahISSMM32TURnV/uXQcYAE8aRl1aTUqScAeUic59ssjc+/Mc/szkvFyznJdfnj5MqGL9vgYZh+/yS8eSidpDxlUaJWOntj+8UpUpzO8mqf4RbXPPcWxdyYqJ1LkmYodhi53wOuMVtkFL29gu3AAyOMzH/mJUvMYSW9+fPPThvLGJc2aQ/gUa0tMjCd15R9r8Fmc5M/LUE3+Dxnse9ONhWPBOVOry1Tr3QO6YQrVHWWNo8WfcBC8ggpwdztvFgUFxE2vRrp7emmE8o+4GcEgcRQjWCfEm9dj7YL/CkZ6PcBBihXyzghormi/xqZfRDgqIcdj9EcYmOi3f9CbP+V5uSa0VJiRI08YiLyJgIImTx/YnUlm1mf5Vdr8SWWRrUtdEyCx5G4CdgGOeQVS+c4UwkxQ9pSEAlexCRhBBddqGTslQuRSb8UzQpjRObqtuYNSQ+kow5evOlr7nn5x084o4W6U13s6nTgNzHhrPhRXTKTuHLm9CoNlvFdshtol6pdL1F8LoAQ2i7SKESoR5xL359snl+jHLSCtAnZxptDFC9nLpwObY4hRs7XfE0z7hvdNRRih3KD6bBzBortRbIb Ca716fZc Odd4MExn/T42syjeBzDekl0yCcuvhelpIdrixzEv1L0mMDLt2rqym20NPIV1bvO3dlVD8Cp1ffyWC1YqMWbeJhoUjo0AsXugsNP7PUBpKbOhlTQ3DgwK9qm84ebsxfFGf+uBehB2fktY2sr7UF+591D7V9wzdUYWq4iDyER94dyffY7rE3c74k1sKF7LTGKW8jfeoOxGnbE7WljpWzq1khV+nFjQwdBJnViz/o8uwTcWxJmXfdyWWdOzSDrQ5fljDdgUBdHaPEnhnqz4hZ5OsNxtyvbJIvV6veCJM5xbtGaAjM0/0lTe19cYGt5O1Ll1WWWKygrfg9duVBAiNuQTzCKau263Ag8Y0YBT8bQKCvXGNpfNI8tjxHX6A4H6Z1V+YlGUWJD93EgnCvgkZ7uy3Bk9vual/n2xl0HRsljlvsfshyChooErWAjfIzA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a matrix for testing gup based on the current gup_test. Only run the matrix when -a is specified because it's a bit slow. It covers: - Different types of huge pages: thp, hugetlb, or no huge page - Permissions: Write / Read-only - Fast-gup, with/without - Types of the GUP: pin / gup / longterm pins - Shared / Private memories - GUP size: 1 / 512 / random page sizes Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- tools/testing/selftests/mm/run_vmtests.sh | 37 ++++++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 824e651f62f4..9666c0c171ab 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -90,6 +90,30 @@ test_selected() { fi } +run_gup_matrix() { + # -t: thp=on, -T: thp=off, -H: hugetlb=on + local hugetlb_mb=$(( needmem_KB / 1024 )) + + for huge in -t -T "-H -m $hugetlb_mb"; do + # -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pin-longterm + for test_cmd in -u -U -a -b -L; do + # -w: write=1, -W: write=0 + for write in -w -W; do + # -S: shared + for share in -S " "; do + # -n: How many pages to fetch together? 512 is special + # because it's default thp size (or 2M on x86), 123 to + # just test partial gup when hit a huge in whatever form + for num in "-n 1" "-n 512" "-n 123"; do + CATEGORY="gup_test" run_test ./gup_test \ + $huge $test_cmd $write $share $num + done + done + done + done + done +} + # get huge pagesize and freepages from /proc/meminfo while read -r name size unit; do if [ "$name" = "HugePages_Free:" ]; then @@ -194,13 +218,16 @@ fi CATEGORY="mmap" run_test ./map_fixed_noreplace -# get_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -u -# pin_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -a +if $RUN_ALL; then + run_gup_matrix +else + # get_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -u + # pin_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -a +fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 - CATEGORY="gup_test" run_test ./gup_longterm CATEGORY="userfaultfd" run_test ./uffd-unit-tests