From patchwork Tue Jun 13 21:53:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A3DFEB64D8 for ; Tue, 13 Jun 2023 21:53:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 103446B0074; Tue, 13 Jun 2023 17:53:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B3736B0075; Tue, 13 Jun 2023 17:53:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E97D98E0002; Tue, 13 Jun 2023 17:53:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DB3656B0074 for ; Tue, 13 Jun 2023 17:53:55 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 820971A0631 for ; Tue, 13 Jun 2023 21:53:55 +0000 (UTC) X-FDA: 80899077630.19.F620A14 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 63A1B1C0003 for ; Tue, 13 Jun 2023 21:53:53 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="AuW7i9/F"; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=HWwG9L2Hn01HieDsdKTsM+fW6Heseo+AVtJqKk3pma0=; b=gCAqTIjpGrSorx3x8ERugjGekn8iEgzmhQ7pxXAifrq7N+AzC6Eieq4ob32CPpTO4gUz63 mT+4WqYk2u5jMGfK7PQsq06isoCbf4zAjo2JJHAIRYvhwBGoi082S5Up5HvWSOVSd2OnMg I+2XVS93qa49CGsmSqpY4FOasgLHS/w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693233; a=rsa-sha256; cv=none; b=FpOTqikb1YO30vMxjLG/ZQjT87dP1MzEIKdKqP/kWxTvaYmvcTNyJltxj2U2VmhF258on8 8WXY0IOngcu1a+0pV8P/+dV8bMfRkpxa5O1l69ytI2oiaHII/5re/4+AcY8pTqdRQ+0HXe jEMfK8hzrt29qcQ+hGxo51/Vci409hA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="AuW7i9/F"; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693232; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HWwG9L2Hn01HieDsdKTsM+fW6Heseo+AVtJqKk3pma0=; b=AuW7i9/FKvjK0+Q6MdN+b1hmBVq5m9/4ONGwmdpZv/iMuep8qL88HI8Q9xO0kSbPM03hZ9 s8G1X/Q/PjsyGnYtp6nO8quE/OUNoJcYLKhW1JGKxXWecRhoBZ77QtDdCg7Tu5R4zoiGZq 5rHV3d15V6Jo2AZ6LGQFQX9onXAYw8U= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-611-YHYCmoh_NYSGNq8CfB3M9g-1; Tue, 13 Jun 2023 17:53:51 -0400 X-MC-Unique: YHYCmoh_NYSGNq8CfB3M9g-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-75ec325d255so77261785a.0 for ; Tue, 13 Jun 2023 14:53:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693229; x=1689285229; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HWwG9L2Hn01HieDsdKTsM+fW6Heseo+AVtJqKk3pma0=; b=B6OQ3ikgzwStvvulLTAzRl+gtuh3s7bm5OjUIF60O54FKkydr6PT1LGuBMXMxBqnxA dgp+ihj5922Xq77bqMRdaF8wuVbT1WD3IocUdZteyFcaTSo5Z8zlCL+YymsEhpq/BWcA 0Lwp9AGw+arYtRaPHnvKJn2JnktK1jVL8+srzvGmEPkfXSbte0YN3VmMs3yALmtGgMXq CPtnDGSCd2THVDHqoPoWFoJF5e+LS+O28nrXRm1m/lokW6FjRkHicnD+36we+f1SkN+D vRApeTbFMmrxBUkbWkRMn0C8uYDEq7b9jX//wayIxQgHVqeZ7HJ4lLMj3p0Z9/BcynGT wIjg== X-Gm-Message-State: AC+VfDxn/+ANlsg1VI8EMJE/VydhL1rUxus0l2/uf4lPF9Ir8tbBOAeB SeBtbPUX0EYM3kxzr1X1H33bih8Sj4GuevSKuI9Mj8XtFI2MxRAsZhrnbrmulRw1f6S8BZaHLVe g1DohW2P+S5g= X-Received: by 2002:a05:622a:19a3:b0:3f9:cb97:f15a with SMTP id u35-20020a05622a19a300b003f9cb97f15amr17412701qtc.0.1686693229683; Tue, 13 Jun 2023 14:53:49 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4LpuZZ8y6o+05nkbQUABsrbOlbnN9Rz4uLpZW4sN1JV6I6TVEVvx/zi1pYcdjTFS2ruush5A== X-Received: by 2002:a05:622a:19a3:b0:3f9:cb97:f15a with SMTP id u35-20020a05622a19a300b003f9cb97f15amr17412692qtc.0.1686693229394; Tue, 13 Jun 2023 14:53:49 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.53.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:53:48 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 0/7] mm/gup: Unify hugetlb, speed up thp Date: Tue, 13 Jun 2023 17:53:39 -0400 Message-Id: <20230613215346.1022773-1-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 63A1B1C0003 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: choorz3fmtoercy51t5j3xk9dma38z83 X-HE-Tag: 1686693233-111941 X-HE-Meta: U2FsdGVkX1908N0guW41w/EY1ksxKMGXRnDdnHWqf6qsclPbCC2L/HJPe14DHQSfY8xzVIIGsqwmuGbqpgGQWTpM5bLTljtFsQU/SPVbZjUFRMtA4oVkLBPMmL9Km91gObq2gEINiEnsNn7apEJJO/ElInmApintD2L36MmXCpKizovmyVzRkkCu7kXAGSlHTitxO8JHCy/8qLTQiOAh8Gn6Alknu3uZen0R/5riXYExvGgfVztXHCFnUwtT0gf6hTtUpyv9hlvwGiMhOs3TgxZLLNLzARMGlcamX6Gtppo7E656pn4I/knFwIQGNPdwEtpZU7QMubaVA+5c+Wi4FOrsEmRmOZheqI0mhvDKH8kqcUbAuPXADAfjoGLBJamO/fclmhWpjbyPDwvFkfLQkQ0eNkiaJa+rfXHCeWUkMkbfq7wSg+Q00wfKvA7lBTn4gaigsuNmUUkb+nROuLeBIRSnWX5+ZfmFvXE8tVAB80XB7HatXWZKoHpdmvLtzx4EnwZyhBwMd9K+AcMMTLDBUsMLI4U1ILb9goeh7DReB1RqzHTiNKwjdFuNx+NgId/N/bzvebqeCniYfdlbtS/qVcowjnoIz2XINF3f06W6kHGJQPfuNb3p1qTiDmPTbSmLw8mqm29XmScqey3bJo2YEQ9HJyIftcjSA4ZYkAW38RKize8UCS9/ZrILmO06Lp2p+wL0OmxE2ZvLHZ3UsEDFhykjZGC2gyhOmHBCnhlDnT5fsKVT9mS5t7uy+7iWKkiA0Kk7sF93KihxntPyfseIGHv7Z1R3ICrde8p+VVtOJLcca+dtmtXDjt7SAB6Z9cUkOAghZ+DGYHL+6jk8ZpZ7V1HKijZAt4ukRfuwgWy2JH8E1IOMQf5lLCKmDoXi0V+933yZWY2jijF96n/7N/QFdsvuKjtk26RVWFU7w7JIc2psE6ItfrSND8+NFGJ59Dbvvi36tA39Syvj46+1QOM 8ljvMWlV 03TKm9A0jOsCWXcdCJ9ayPG9RXzLDeh8K/qL3NwcupwtsXv/CQewUlsSygxiUIiZNLOV5k7j9ybgp2fBYklTLvzuQii7dXb1LOx/56KBpivj+HT1Mi/40xMs+bxFg4GqOTIHbGTiK7rBiIrDkqxItfMzVgswzxAXBnZP1pgr9KEtBKQvALaY+hSonKRRfzAybd2CpNrB32bOmSLjVyiOl9qtCTLa+Xf58YMGHpItNrubNDxPKnC6HyK/+81/GzM5+xSYNLoShP1S9XTxkBGnZdTug2EPdYavRlvL669yeFzBmYDcFVqdTsBWdQSgrJ9hbPHK4V1lHmTlBSE3WKdXDmF7DjBfiMs/4I1Eqk4tAUJRS3O0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb has a special path for slow gup that follow_page_mask() is actually skipped completely along with faultin_page(). It's not only confusing, but also duplicating a lot of logics that generic gup already has, making hugetlb slightly special. This patchset tries to dedup the logic, by first touching up the slow gup code to be able to handle hugetlb pages correctly with the current follow page and faultin routines (where we're mostly there.. due to 10 years ago we did try to optimize thp, but half way done; more below), then at the last patch drop the special path, then the hugetlb gup will always go the generic routine too via faultin_page(). Note that hugetlb is still special for gup, mostly due to the pgtable walking (hugetlb_walk()) that we rely on which is currently per-arch. But this is still one small step forward, and the diffstat might be a proof too that this might be worthwhile. Then for the "speed up thp" side: as a side effect, when I'm looking at the chunk of code, I found that thp support is actually partially done. It doesn't mean that thp won't work for gup, but as long as **pages pointer passed over, the optimization will be skipped too. Patch 6 should address that, so for thp we now get full speed gup. For a quick number, "chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10" gives me 13992.50us -> 378.50us. Gup_test is an extreme case, but just to show how it affects thp gups. James: I hope this won't affect too much of your HGM series as this might conflict with yours. Logically it should even make your series smaller, since you can drop the patch to support HGM for follow_hugetlb_page() now after this one, but you only need to worry that if this one is proven useful and merged first. Patch 1-6 prepares for the switch, while patch 7 does the switch over. I hope I didn't miss anything, and will be very happy to know. Please have a look, thanks. Peter Xu (7): mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() mm/hugetlb: Fix hugetlb_follow_page_mask() on permission checks mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN mm/gup: Cleanup next_page handling mm/gup: Accelerate thp gup even for "pages != NULL" mm/gup: Retire follow_hugetlb_page() include/linux/hugetlb.h | 20 +--- mm/gup.c | 68 +++++------ mm/hugetlb.c | 259 ++++------------------------------------ 3 files changed, 63 insertions(+), 284 deletions(-)