From patchwork Sat Aug 27 08:36:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 12956857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19FBBECAAD2 for ; Sat, 27 Aug 2022 08:36:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C986940007; Sat, 27 Aug 2022 04:36:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5774D6B0074; Sat, 27 Aug 2022 04:36:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 417FF940007; Sat, 27 Aug 2022 04:36:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3417F6B0073 for ; Sat, 27 Aug 2022 04:36:15 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E0A96140169 for ; Sat, 27 Aug 2022 08:36:14 +0000 (UTC) X-FDA: 79844715468.07.066E48B Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by imf31.hostedemail.com (Postfix) with ESMTP id 719082002F for ; Sat, 27 Aug 2022 08:36:14 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AAJdBdWOI3pDpYQqb7bGq8aMPrUPBUqrHObEO7HIuzte9iX+cDJiX6+ChHnthHuWlcZp4c46+YLhvMV3J1kuN1DAnFyDKGhnWqopRleHxdV1vJz3wjrtvkdiFGP4nbC2h/gcjsTQ2xZZYPEdWk/Ci0vBHzKNV0/fTN7IZe+tRTuKU35nAWgkCniz0Q3LzGP90eoV/iO+nckrVT3Xe2x0Gbn9i93reBvTvU+Eq0dNpQ5jym1ZLfrYYZdVoFS8wsYOuwdTvP7hBq/XqHnCge2Y/3Xru1ujwsGuSMOdoiAM/dmpHdX0z5ngeAMbrmtdTMTSXxzT1VvJChAFi8suBDA0UA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UO9IhewdmpPOqvg3QpSZ5kTO1SJPtnuFZRi+d4vMGI4=; b=O5+JbVTMnM3QYfSOa4LGo5c+9DLOyEg3GkOF5kPSeCogWj2ne0wWZ6LMkQADg3A3RM7IQ7SDxZwTUAVEd5HaKt2pfjo0/bJEC5CIXUsJR4VenzPuKLNTUH+tSjq+aoySaTXfpcw+YUgEIdrpHfBiYyg/NVS4S9w0RLmilc4jtqePcctFPRkqm/MpIlR8lM19e5eQqRFWtKqY9FmQggTVKqcReOMh7cU/TfHzbB617oEX10LzirdE+Iaur67vby1Z4XTuZWYsKQAakcKZD82RFcDT+6xWwPefXULgSc3fuLu1noXN/udzDVmg/BRwAXi1xDghOx+yl/ngCjuzCKq5Sg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UO9IhewdmpPOqvg3QpSZ5kTO1SJPtnuFZRi+d4vMGI4=; b=ePgpzUylfyGRiv8r47Kk55aMbrdcsdCdGrWr0bN2dNtm1rU3HU+WfB/1iKO6h+U1LuHyoKnYvUHPurCp6IXYfJOAVFJdC/yVPFhbLF6VZwgVTphMBAPnCd13MhKBkVKPhiPBSTCTuLZJPEkobcFefhRaDkJZyvvD17kGLd58pBFpyuce3KarbyS6j1+cppmzLHUWGNbUXEBkqBS56YA3WuJeZxgWcqNNC8WnRCMO2WTNQ3v1gXpABhmHrp4nrqlrwtk/asWJphTRIsZSvmhiO7CjTLoLjlO2bYzzPZ2XjpDG0awDTc67Ybp6a17USVR4VS2y/0cxH/Qg81tQFPYjJw== Received: from DM5PR07CA0092.namprd07.prod.outlook.com (2603:10b6:4:ae::21) by PH8PR12MB7027.namprd12.prod.outlook.com (2603:10b6:510:1be::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.16; Sat, 27 Aug 2022 08:36:12 +0000 Received: from DM6NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:4:ae:cafe::a9) by DM5PR07CA0092.outlook.office365.com (2603:10b6:4:ae::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by DM6NAM11FT047.mail.protection.outlook.com (10.13.172.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:11 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sat, 27 Aug 2022 08:36:11 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Sat, 27 Aug 2022 01:36:10 -0700 Received: from sandstorm.attlocal.net (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Sat, 27 Aug 2022 01:36:10 -0700 From: John Hubbard To: Andrew Morton CC: Jens Axboe , Alexander Viro , Miklos Szeredi , Christoph Hellwig , "Darrick J . Wong" , Trond Myklebust , Anna Schumaker , Jan Kara , Logan Gunthorpe , , , , , , LKML , John Hubbard Subject: [PATCH 1/6] mm/gup: introduce pin_user_page() Date: Sat, 27 Aug 2022 01:36:02 -0700 Message-ID: <20220827083607.2345453-2-jhubbard@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220827083607.2345453-1-jhubbard@nvidia.com> References: <20220827083607.2345453-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 64fbdea3-33e1-42f3-e854-08da8807326b X-MS-TrafficTypeDiagnostic: PH8PR12MB7027:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qnlb2BznKuv8HSR/mOMAdLL3epBJdze+9vbcf+5R1g1XlS+98sCJbl1S4qPk+1WdMa8i82GsCphaSdMgT+43MrcYVLxMY4LmSP6zn2AzLJa28esn7gqkx20aO+nRwTWj4+pD5raxucC0Xp7lIxtFtTiISaeGPnmS2cFK1qoLHqsU6rW5AYY0AWZV9Kqt5zS9ZAFHFjUaumIC99/dcjEyDjI6W3btlj2lbRaUXil/NseqpUgUBAxG7dR4pe0WwCTzfxOSPdeAbQnENJxc6rGj2GLnTKTXx56JZil30dezGUXxOSAJliQgna0OPaEOfkUWm51NlgphKLv1EWd0ODFqIIExYXgIpKmdDcHerJIEJOC1MkVt4P2m25/4viCsQyVAJH6nI9zBEGVeDNVEH4kefpBdLMwyj6AQwSpMC8X1+k8wD/T1sXrm2Z74nbaDqpCmXaxCVZv6CyVQA356qVrHXAGmgnQppWw5hhdnEx/e7R+Q9D5WmD+El0QAOgFEBPIQj9fFjh2V8I25EjSWtURVQBN1NOc9y2mnIZz/2HqXy1G2+qbj7vWUcauZ62RiMoCpCUVHF/iP3LYwvvy6Ho4C0F7s4MmEcvbZAX6Rg59GveOCyJ4CzmTFscuJfagklCSfH3Q9N92d1BvKTNOYg6RKI7Ou9B852gNDQVkDl2HqYgxxS3du9NZJrhHoKe6VJoK79kU5oGqN/Fe3mM1mMAKJ1mHJMqWT9zfUK8LjazyDzHGalG3U9LGMWnrsejcDLDj6KGok4NbLzoZzE/GYdQ8w/y/f795NSfRHBbJZiDhTvF6NZt/95KO7VfQIyGQBNiHm0o/l7zlEYGcDUcje3QI1iPsp2Pz/fh6FgqRYceUmbC4= X-Forefront-Antispam-Report: CIP:12.22.5.235;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230016)(4636009)(376002)(346002)(136003)(396003)(39860400002)(46966006)(36840700001)(40470700004)(6666004)(8936002)(81166007)(4326008)(36756003)(7416002)(8676002)(40460700003)(356005)(82740400003)(70586007)(82310400005)(40480700001)(5660300002)(186003)(2906002)(86362001)(70206006)(2616005)(1076003)(47076005)(426003)(336012)(26005)(107886003)(83380400001)(36860700001)(316002)(6916009)(54906003)(41300700001)(478600001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2022 08:36:11.7334 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64fbdea3-33e1-42f3-e854-08da8807326b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.235];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7027 ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1661589374; a=rsa-sha256; cv=pass; b=y46Sdb/gMWxvtKX/+LMt3vpYlosRER+ApSqNpJ+XE2GDaWDnCAWVF5efi+P9GOvOf0tGCP uuGeGzPiMAoj9g30zmVdABsk9thFRNl1WWHRB4GudzxuzBR9l9UomFzKwTzxkTgtqhK6Lp k08WHzQnVH2nX2O9K9E8oRjAR+D4Jbw= ARC-Authentication-Results: i=2; imf31.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ePgpzUyl; spf=pass (imf31.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661589374; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UO9IhewdmpPOqvg3QpSZ5kTO1SJPtnuFZRi+d4vMGI4=; b=NgKSHONwLN7NLLSc/BJ/uWTAD1SefeGXRGI8gXA6w6N4pJCxqZDMyyxiLkFvAitPDHRXNU q+PVzuJ6O2PgDMIVKXXoJkuwnGZ3L5MiNt3qRsBCRKiF4TtWx/xLXOPNCU4ljhFauWhdTy DB5F8URyZ8FWZJUmLrt+h4MbPWUjOCY= X-Stat-Signature: egmmqdtzz81bmrsmon4den7drw8ygsm3 X-Rspamd-Queue-Id: 719082002F Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=ePgpzUyl; spf=pass (imf31.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1661589374-972687 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pin_user_page() is an externally-usable version of try_grab_page(), but with semantics that match get_page(), so that it can act as a drop-in replacement for get_page(). Specifically, pin_user_page() has a void return type. pin_user_page() elevates a page's refcount using FOLL_PIN rules. This means that the caller must release the page via unpin_user_page(). Signed-off-by: John Hubbard --- include/linux/mm.h | 1 + mm/gup.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 982f2607180b..85a105157334 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1869,6 +1869,7 @@ long pin_user_pages_remote(struct mm_struct *mm, long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas); +void pin_user_page(struct page *page); long pin_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas); diff --git a/mm/gup.c b/mm/gup.c index 5abdaf487460..245ccb41ed8c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3213,6 +3213,39 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, } EXPORT_SYMBOL(pin_user_pages); +/** + * pin_user_page() - apply a FOLL_PIN reference to a page + * + * @page: the page to be pinned. + * + * This is similar to get_user_pages(), except that the page's refcount is + * elevated using FOLL_PIN, instead of FOLL_GET. + * + * IMPORTANT: The caller must release the page via unpin_user_page(). + * + */ +void pin_user_page(struct page *page) +{ + struct folio *folio = page_folio(page); + + WARN_ON_ONCE(folio_ref_count(folio) <= 0); + + /* + * Similar to try_grab_page(): be sure to *also* + * increment the normal page refcount field at least once, + * so that the page really is pinned. + */ + if (folio_test_large(folio)) { + folio_ref_add(folio, 1); + atomic_add(1, folio_pincount_ptr(folio)); + } else { + folio_ref_add(folio, GUP_PIN_COUNTING_BIAS); + } + + node_stat_mod_folio(folio, NR_FOLL_PIN_ACQUIRED, 1); +} +EXPORT_SYMBOL(pin_user_page); + /* * pin_user_pages_unlocked() is the FOLL_PIN variant of * get_user_pages_unlocked(). Behavior is the same, except that this one sets From patchwork Sat Aug 27 08:36:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 12956858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6922C0502C for ; Sat, 27 Aug 2022 08:36:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 182376B0073; Sat, 27 Aug 2022 04:36:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1335C940008; Sat, 27 Aug 2022 04:36:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E79106B0075; Sat, 27 Aug 2022 04:36:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DB7E36B0073 for ; Sat, 27 Aug 2022 04:36:15 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AC35F804B3 for ; Sat, 27 Aug 2022 08:36:15 +0000 (UTC) X-FDA: 79844715510.05.994C5DD Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2046.outbound.protection.outlook.com [40.107.212.46]) by imf28.hostedemail.com (Postfix) with ESMTP id 4E028C0023 for ; Sat, 27 Aug 2022 08:36:15 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A0QQz/FkFzjCALlCPcJTEURzDt2s/MLKPOgo0rOP5sZ0N1I5WxldqOIy7HZRaG2OIESxRL35viDpiG0sBg4JA9nRmy05N0c2URY9VFq8Tko3rWsX4a5enxg3+sTpaXTnKnhj1sXmFzwl1mcqDF6TwEHIC1+CEtRRO2deQZVNp72BqpdKiiSdD4l2VVyaq2gcuSkKhcMmLtIajhWw1WWUV4avNiEiyP0QVj/bOJgL+Q3I6IB4oB9rBFLH+irp0G7t3XrUOR83FJDnXyEKe281L4Fc53dDR1IaGp4RG8L7bZQPGi0Aq0zgWdxngGmcgUgPBIQkYxhOdfGmjM7C7bdbdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=m6juzxIN+dbVotM4FaLX0WTWAvPmSGnE+qYMrVKn5mE=; b=WlwNBc0zGqhVFyz3ork5Cha4bhOF6xPbulatdQ3k6FzCk10WMWdFGVk9a96+Kneo1vzYd7H1zya/qt0wgO0+rG0w//OMUiWER8+8FHSjtRcx9oQ+h0pUO3tzIL9C1WN4e3doxGM4nuWSzRCFIAlxTwMFNQUS7sLW4VNIZFZ0GKl/uIVr5WYNNuxFkpgvMbC67aj4/Xo/liSQhmIJx9Dk5GNM66S9UaUgzs43tihKpPGQHz8wHMLIdXwmsi+9uylvh93cURdtQnAmTYkSfwXoSAAyvxC63akzygBYJe2PMLZIFkICLYn6T/ymjxoCJUeOUu1DRvbyCpuUD98hRIZE2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m6juzxIN+dbVotM4FaLX0WTWAvPmSGnE+qYMrVKn5mE=; b=cGOp7gbrsET+75dtjHxMgRnH08IDxpuQbg5V1tCNMBL/kNwgJc0JYrT3LhhkfI0w2Lo/If9sIVBpyaevu/75IVQH+ZbcikZLB9X3Heoh2xqsFbqiEnbMFmPmSATda7jAJvz68Quk0E8FVPRV+y3TlyRRt8cOUNYvo9kdgzgZ4T802MjYql1wgESW3dyf859jwgiTniv5n4QfUacDNrxy1+HH+9yDIiB5xjy0R9XtuLu+M6G4hiFO1I407/uEn1avJr7NBDKQ95Uey2Y8bBPXH4vI5qIPx8tD00uxyY5h8ezTKtu4YxX5fSlP8nL8MphL4PYYf+QYnlPgcvSdvsKSzA== Received: from MW4P222CA0005.NAMP222.PROD.OUTLOOK.COM (2603:10b6:303:114::10) by PH7PR12MB6718.namprd12.prod.outlook.com (2603:10b6:510:1b1::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15; Sat, 27 Aug 2022 08:36:12 +0000 Received: from CO1NAM11FT057.eop-nam11.prod.protection.outlook.com (2603:10b6:303:114:cafe::c6) by MW4P222CA0005.outlook.office365.com (2603:10b6:303:114::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.14 via Frontend Transport; Sat, 27 Aug 2022 08:36:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT057.mail.protection.outlook.com (10.13.174.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:12 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sat, 27 Aug 2022 08:36:11 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Sat, 27 Aug 2022 01:36:11 -0700 Received: from sandstorm.attlocal.net (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Sat, 27 Aug 2022 01:36:10 -0700 From: John Hubbard To: Andrew Morton CC: Jens Axboe , Alexander Viro , Miklos Szeredi , Christoph Hellwig , "Darrick J . Wong" , Trond Myklebust , Anna Schumaker , Jan Kara , Logan Gunthorpe , , , , , , LKML , John Hubbard Subject: [PATCH 2/6] block: add dio_w_*() wrappers for pin, unpin user pages Date: Sat, 27 Aug 2022 01:36:03 -0700 Message-ID: <20220827083607.2345453-3-jhubbard@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220827083607.2345453-1-jhubbard@nvidia.com> References: <20220827083607.2345453-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8c80ba3d-e36b-4be8-f9a6-08da880732bf X-MS-TrafficTypeDiagnostic: PH7PR12MB6718:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Jx7dtFz53tTiOMq5bP8R6GG/n5j0MAjZq9JQWYUEIo4hKL0vrsAHVI/W10uxpuzSVMMq3Z7uZAc35bDAGua1gfhgvFQ8Q9v+QjXYEkwFYfRmjoVyIKmR6a4ywpw1ZpiRDRrMq+kDAPXmngZE9JS8CUmJTL+Sa/gpa+oK90f59M4PEDunl2Tu5NgIn2ZLSNVCmGtbFWFRt8r9/gKtffUhN3h5bHCA4OF754MWCfof3o28W9HvnV6v1LFDyV3kqE7eYkXVLdVu41m0XRUt8/rfYGkGcc2DpUUDT34+fMiplO554QCBozu8C+/YP/rE0TXPp2F8rD//oEGDV0qtO0XJfQHBeJpV2omUGqeveMfuSL+ImPUO8xnAtncDSI9tVv7/A4Ce6NwrQS41xmRmyOXZ3CZXf6zU6SoTY9rAtOXCoGQ3ochiUnAt+ejiC9FG1jh6xdodDA83C+zfAxWec7CU24n+wYYc1mp6HkGOdq9U3/QjL/6KmMQ8HUFYwbfuL4ed2FkWCoebIRp/uiIXahjBHK1BRF4TrdSvr0ETLlH/kp3zjmjKMWmJIC+mim/jZbOirEKl0Q13GS4Gcq+NH9BZo+rrcfuZtR2AK53ZLEtZ0d4T/ZCuVCQKnFptRX8pubAptyAY8QBS+sHJ7bDb7zXLopTmAC1Ns0BIwOWRFTQPndR1O3Wadg5yWQreaYEXJmK1Y1IxF78Nf6wciZqKKIcQkQlAcEPNNaY5d0zKyfBmRmO0aYcvN6D5C/TNmMhHrEf2MZ1op0ZfAeIi7JEjfLRUzLQaySv+vKp1kF72579POtKxMNtoxG/LSbmYqck3rfN/ X-Forefront-Antispam-Report: CIP:12.22.5.238;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230016)(4636009)(136003)(39860400002)(346002)(396003)(376002)(46966006)(36840700001)(40470700004)(82310400005)(5660300002)(26005)(86362001)(41300700001)(107886003)(6666004)(2616005)(82740400003)(81166007)(356005)(83380400001)(40460700003)(336012)(6916009)(426003)(47076005)(478600001)(40480700001)(36860700001)(2906002)(54906003)(4326008)(8676002)(316002)(70586007)(36756003)(70206006)(186003)(8936002)(7416002)(1076003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2022 08:36:12.2843 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8c80ba3d-e36b-4be8-f9a6-08da880732bf X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.238];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT057.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6718 ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1661589375; a=rsa-sha256; cv=pass; b=YEixMPxs7wCIhyWZAeS/Ge6kam0LdoZoyedD9iuskD38oby0Lm2GFMj68HNem3FP9u0Kbz Im+99iB5Y+pM0QJ0cy5O+cGq900kN+iDO/YkHC+wANgOuHyhFhHOWzF9Xm5mCJIngh1q58 lDMaoKyULtQNqWsCwEcrWWkmUCPmJJs= ARC-Authentication-Results: i=2; imf28.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=cGOp7gbr; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf28.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.212.46 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661589375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m6juzxIN+dbVotM4FaLX0WTWAvPmSGnE+qYMrVKn5mE=; b=kIiOlVHCAcNAYLisoFX4IlovO4HhWcOSuZBlhX64zeE9iwFT13wxFHmUK1cy39vyNPpoLa K8arDKux8ifSoVA3Oe4SfWxI2sWoqDqTJQNyoy+2TIC26tVeoaF3sGuCn01pZtlm+MlYSS 5+iaUbW+Di/rNXPOLZ/5bulnlWshyJ4= X-Rspamd-Queue-Id: 4E028C0023 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=cGOp7gbr; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf28.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.212.46 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Stat-Signature: yh644djm8s11g99tp68ao3s41dfkcu9x X-HE-Tag: 1661589375-191350 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Background: The Direct IO part of the block infrastructure is being changed to use pin_user_page*() and unpin_user_page*() calls, in place of a mix of get_user_pages_fast(), get_page(), and put_page(). These have to be changed over all at the same time, for block, bio, and all filesystems. However, most filesystems can be changed via iomap and core filesystem routines, so let's get that in place, and then continue on with converting the remaining filesystems (9P, CIFS) and anything else that feeds pages into bio that ultimately get released via bio_release_pages(). Add a new config parameter, CONFIG_BLK_USE_PIN_USER_PAGES_FOR_DIO, and dio_w_*() wrapper functions. The dio_w_ prefix was chosen for uniqueness, so as to ease a subsequent kernel-wide rename via search-and-replace. Together, these allow the developer to choose between these sets of routines, for Direct IO code paths: a) pin_user_pages_fast() pin_user_page() unpin_user_page() b) get_user_pages_fast() get_page() put_page() CONFIG_BLK_USE_PIN_USER_PAGES_FOR_DIO is a temporary setting, and may be deleted once the conversion is complete. In the meantime, developers can enable this in order to try out each filesystem. Please remember that these /proc/vmstat items (below) should normally contain the same values as each other, except during the middle of pin/unpin operations. As such, they can be helpful when monitoring test runs: nr_foll_pin_acquired nr_foll_pin_released Signed-off-by: John Hubbard --- block/Kconfig | 24 ++++++++++++++++++++++++ include/linux/bvec.h | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/block/Kconfig b/block/Kconfig index 444c5ab3b67e..d4fdd606d138 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -48,6 +48,30 @@ config BLK_DEV_BSG_COMMON config BLK_ICQ bool +config BLK_USE_PIN_USER_PAGES_FOR_DIO + bool "DEVELOPERS ONLY: Enable pin_user_pages() for Direct IO" if EXPERT + default n + + help + For Direct IO code, retain the pages via calls to + pin_user_pages_fast(), instead of via get_user_pages_fast(). + Likewise, use pin_user_page() instead of get_page(). And then + release such pages via unpin_user_page(), instead of + put_page(). + + This is a temporary setting, which will be deleted once the + conversion is completed, reviewed, and tested. In the meantime, + developers can enable this in order to try out each filesystem. + For that, it's best to monitor these /proc/vmstat items: + + nr_foll_pin_acquired + nr_foll_pin_released + + ...to ensure that they remain equal, when "at rest". + + Say yes here ONLY if are actively developing or testing the + block layer or filesystems with pin_user_pages_fast(). + config BLK_DEV_BSGLIB bool "Block layer SG support v4 helper lib" select BLK_DEV_BSG_COMMON diff --git a/include/linux/bvec.h b/include/linux/bvec.h index 35c25dff651a..33fc119da9fc 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -241,4 +241,44 @@ static inline void *bvec_virt(struct bio_vec *bvec) return page_address(bvec->bv_page) + bvec->bv_offset; } +#ifdef CONFIG_BLK_USE_PIN_USER_PAGES_FOR_DIO +#define dio_w_pin_user_pages_fast(s, n, p, f) pin_user_pages_fast(s, n, p, f) +#define dio_w_pin_user_page(p) pin_user_page(p) +#define dio_w_iov_iter_pin_pages(i, p, m, n, s) iov_iter_pin_pages(i, p, m, n, s) +#define dio_w_iov_iter_pin_pages_alloc(i, p, m, s) iov_iter_pin_pages_alloc(i, p, m, s) +#define dio_w_unpin_user_page(p) unpin_user_page(p) +#define dio_w_unpin_user_pages(p, n) unpin_user_pages(p, n) +#define dio_w_unpin_user_pages_dirty_lock(p, n, d) unpin_user_pages_dirty_lock(p, n, d) + +#else +#define dio_w_pin_user_pages_fast(s, n, p, f) get_user_pages_fast(s, n, p, f) +#define dio_w_pin_user_page(p) get_page(p) +#define dio_w_iov_iter_pin_pages(i, p, m, n, s) iov_iter_get_pages2(i, p, m, n, s) +#define dio_w_iov_iter_pin_pages_alloc(i, p, m, s) iov_iter_get_pages_alloc2(i, p, m, s) +#define dio_w_unpin_user_page(p) put_page(p) + +static inline void dio_w_unpin_user_pages(struct page **pages, + unsigned long npages) +{ + unsigned long i; + + for (i = 0; i < npages; i++) + put_page(pages[i]); +} + +static inline void dio_w_unpin_user_pages_dirty_lock(struct page **pages, + unsigned long npages, + bool make_dirty) +{ + unsigned long i; + + for (i = 0; i < npages; i++) { + if (make_dirty) + set_page_dirty_lock(pages[i]); + put_page(pages[i]); + } +} + +#endif + #endif /* __LINUX_BVEC_H */ From patchwork Sat Aug 27 08:36:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 12956859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70476ECAAD2 for ; Sat, 27 Aug 2022 08:36:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63D67940009; Sat, 27 Aug 2022 04:36:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5ED1F940008; Sat, 27 Aug 2022 04:36:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48D44940009; Sat, 27 Aug 2022 04:36:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 373C0940008 for ; Sat, 27 Aug 2022 04:36:16 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0CA7DC0796 for ; Sat, 27 Aug 2022 08:36:16 +0000 (UTC) X-FDA: 79844715552.22.5513ED7 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2080.outbound.protection.outlook.com [40.107.243.80]) by imf30.hostedemail.com (Postfix) with ESMTP id 81FA380036 for ; Sat, 27 Aug 2022 08:36:15 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TRn4JAVnSRE7dSLyjcLJ5Eb3xcKqQMMqMNmHhtMp7YOdAncnU0SubYx8gaJf0JYtE+gpnIYWa0UQ8nABIFfW49Sr2og6tmG7eLSgAyeRZ7J0aY1HKT2kny1T5N/cb69SjbKQep3ePUfF7Qr3mnI3a0v0AUsgoPF6yE8KGtJnIOR19zmkyzSC83dgdS/eJgb+6fyXo/7NkttFMm0zToUy1sfn5mVOjkEqm1OdUc8sopk8UlQbW1JCxNMO6E9lbFM7OJSpJl34W66s6s7Q8Lp6kbSWZLvXBL04LXQ7kHkf03Y9ODNKrPiPfYqpQsDsC9gAL92ms1SYzTJpPdXAZky+bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1se43bUMAuKkiRhgWx+GCO+PuzkFi6R5U8K9kr66PVU=; b=QXQeaoETZ/yLGtZwuHySGH7LFNFYNCXGVIJf7GnO1NgNwhZ5z8nRix/yOjKqAliIv6GuHJgJ7GRVgovPCx6OM91WI2if59gihCDULYYj9Rct6PzVREqpu5UJQw/+iGYwsej84kRzAK2lnXA280jrBBMq9cT+qTRQ+1JEQJ5wRoOhoeV/hvaMgVfG8TMHz0i3EOaYiH4nyFYcMGulXYmQ4MIulxj8tAELuqiQWtAxqIF3l9ZDuR67wVBFQrrwYm6Sfk+BV61nbQqERbSuau4a/l3TyFIbY8skmG1DAFa9mnXlG0di3kt5FcmTy9Jcu8wi7PgU1Gorio0dRyAjL81wJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1se43bUMAuKkiRhgWx+GCO+PuzkFi6R5U8K9kr66PVU=; b=MYgtZjwqzG6Ih6wekWj7eIwOKOc2sW+4TK13gy6L+Rbp8hJxnvgbLbaAlP3vSSnyfJBczdTNL1JLuKIwRrlY/4UyZFCbZXgdSyRGZxt620GFX2owQP119R6onJw0nQcS/p0HKpcWtlvSccyXfuS1E/BlruwzE/Y8V3CN44HF2g89/nqIRGwTC/YwlPBoKHuS5qOJPm96GyJp0k/NCiVKqOBGzFwivQSmSkALIgDjomJDH61SoC3jJCm43npqGwCr5Bdd6+iNMM9sn46kGRAY5uoIFN2BA82mfM3DK3rQHl/gRIJTXq6lnipcdynXiK1nBXuMozOuFzfRpKMzz9e00A== Received: from DM6PR03CA0045.namprd03.prod.outlook.com (2603:10b6:5:100::22) by CY4PR12MB1333.namprd12.prod.outlook.com (2603:10b6:903:41::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15; Sat, 27 Aug 2022 08:36:13 +0000 Received: from DM6NAM11FT085.eop-nam11.prod.protection.outlook.com (2603:10b6:5:100:cafe::bc) by DM6PR03CA0045.outlook.office365.com (2603:10b6:5:100::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.234) by DM6NAM11FT085.mail.protection.outlook.com (10.13.172.236) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:13 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sat, 27 Aug 2022 08:36:12 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Sat, 27 Aug 2022 01:36:12 -0700 Received: from sandstorm.attlocal.net (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Sat, 27 Aug 2022 01:36:11 -0700 From: John Hubbard To: Andrew Morton CC: Jens Axboe , Alexander Viro , Miklos Szeredi , Christoph Hellwig , "Darrick J . Wong" , Trond Myklebust , Anna Schumaker , Jan Kara , Logan Gunthorpe , , , , , , LKML , John Hubbard Subject: [PATCH 3/6] iov_iter: new iov_iter_pin_pages*() routines Date: Sat, 27 Aug 2022 01:36:04 -0700 Message-ID: <20220827083607.2345453-4-jhubbard@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220827083607.2345453-1-jhubbard@nvidia.com> References: <20220827083607.2345453-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 37c9f01e-66aa-4fcb-ef44-08da88073340 X-MS-TrafficTypeDiagnostic: CY4PR12MB1333:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SAMTNaoa7WNBuMuYYeIzOBA6mR/Vo8mXEqEXLG2d+h97217MgupLC6E1/cGhNFJGu9meNvULQohRuXkL9LOx7MhrzJ6l7TXv6eD1pWLknTfbT6poogyKNIVUPJDiMxliucXvpGG1GUnJR4B20hHo/7rvewHb5D6sefWURP0bVdCj46ThXtJ7DhLqidA+Qh1GkdEcrL3a30lom+gHcpv18h//IIOb6uco/4Rj4od4f+UUStTTl522U2k/CvnF3rM/qNDhjWB4ouCYvcjPdiKV5cLXnX6Sfm0YH+KGgzqBUHivRqYaFSeEb8uZu89gh1r9d+SU8n+LuldPSXXXi4lVg428qJcBXyYGPEOaPGr8ujqEjxev/tl6o0zzATuqcbfmrTqYv4sjagSO3LwlhStkJPm1DEWdxUZFzOYPO3I6mL5q/WspxxuX/QEXpHh9mBAxZGqJYjDoR+vokq5vwQfWQB0gMI3058zwyZdOghx1E0Z2++PqezCpxW1wK1EMcHuCsjU4QBuhzcp/P9hyB2qxUP3clstwlCj8443jRhymDZlN+Mb1bMpoXZV8hZEmw8J9pWhAub5djnkeZZeWjit0ud0nob8Lw8FCdZ/8/syt2fBuKjZ1aasNJKOKGJ99jizL2Q8HiUTpkZZ8OkCPCkOYai76Zvhu2ejwtDYd8RNKQM+IMF7NbYbC/vYmMkYYK3foS/jJRh2nPF9Z5ZrSc+/Um4MeegybttluXnbXnBY9i+hPOtDYGvnG+HBfCt1oz7eAx0IUH+S2b4WSRuT3opszcjAU1Ang9Tlxb3CEwHqq4Csh2KJ7uhn83uTgEbqlBqlO X-Forefront-Antispam-Report: CIP:12.22.5.234;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230016)(4636009)(39860400002)(396003)(346002)(136003)(376002)(40470700004)(46966006)(36840700001)(82740400003)(83380400001)(36860700001)(426003)(47076005)(2616005)(186003)(2906002)(1076003)(336012)(356005)(81166007)(40460700003)(26005)(107886003)(86362001)(6666004)(40480700001)(82310400005)(70586007)(316002)(41300700001)(478600001)(36756003)(7416002)(54906003)(5660300002)(6916009)(8936002)(8676002)(4326008)(70206006)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2022 08:36:13.1138 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 37c9f01e-66aa-4fcb-ef44-08da88073340 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.234];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT085.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1333 ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1661589375; a=rsa-sha256; cv=pass; b=C1708rEFPHsVTkLEiFiVhRGsP4Sc7XrGfZCT6vBpkkSfwfKutxsVqvCzJTDaMrw/JpeIxs ZX8R0z7mxTJrxwKX/y2IFawfZwIptkjs3Lu7b727iyZ5D6bdqjED771iWNW/1nIzIh1LnQ 42bb09s4Scdf/w2qWlsaW0pwkr6bFk8= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=MYgtZjwq; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.243.80 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661589375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1se43bUMAuKkiRhgWx+GCO+PuzkFi6R5U8K9kr66PVU=; b=HjGYylLVX/kDsVD0hZDFLJ888vLAvquKQ1KF6aMCRSQEZXTta3VsShhGv+zxUU0m1tpTrO aP/kkLuLMIH3VcvpgF4/r1fZ9ctsMaGRaetLxMNeJmSmUZ7Nloi+PvUD8OfcEDNtU+dqSC M+GfsviJU90DATr/hsoVGiteuX4u2Xg= X-Rspamd-Server: rspam11 X-Rspam-User: X-Rspamd-Queue-Id: 81FA380036 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=MYgtZjwq; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.243.80 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com X-Stat-Signature: fr17bnrff89shpsuqua6bhcj3w55mgom X-HE-Tag: 1661589375-698485 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide two new wrapper routines that are intended for user space pages only: iov_iter_pin_pages() iov_iter_pin_pages_alloc() Internally, these routines call pin_user_pages_fast(), instead of get_user_pages_fast(). As always, callers must use unpin_user_pages() or a suitable FOLL_PIN variant, to release the pages, if they actually were acquired via pin_user_pages_fast(). This is a prerequisite to converting bio/block layers over to use pin_user_pages_fast(). Signed-off-by: John Hubbard --- include/linux/uio.h | 4 +++ lib/iov_iter.c | 74 ++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 74 insertions(+), 4 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 5896af36199c..e26908e443d1 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -251,6 +251,10 @@ ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); +ssize_t iov_iter_pin_pages(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned int maxpages, size_t *start); +ssize_t iov_iter_pin_pages_alloc(struct iov_iter *i, struct page ***pages, + size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 4b7fce72e3e5..1c08014c8498 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1425,9 +1425,23 @@ static struct page *first_bvec_segment(const struct iov_iter *i, return page; } +enum pages_alloc_internal_flags { + USE_FOLL_GET, + USE_FOLL_PIN +}; + +/* + * TODO: get rid of the how_to_pin arg, and just call pin_user_pages_fast() + * unconditionally for the user_back_iter(i) case in this function. That can be + * done once all callers are ready to deal with FOLL_PIN pages for their + * user-space pages. (FOLL_PIN pages must be released via unpin_user_page(), + * rather than put_page().) + */ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, - unsigned int maxpages, size_t *start) + unsigned int maxpages, size_t *start, + enum pages_alloc_internal_flags how_to_pin) + { unsigned int n; @@ -1454,7 +1468,12 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, n = want_pages_array(pages, maxsize, *start, maxpages); if (!n) return -ENOMEM; - res = get_user_pages_fast(addr, n, gup_flags, *pages); + + if (how_to_pin == USE_FOLL_PIN) + res = pin_user_pages_fast(addr, n, gup_flags, *pages); + else + res = get_user_pages_fast(addr, n, gup_flags, *pages); + if (unlikely(res <= 0)) return res; maxsize = min_t(size_t, maxsize, res * PAGE_SIZE - *start); @@ -1497,10 +1516,31 @@ ssize_t iov_iter_get_pages2(struct iov_iter *i, return 0; BUG_ON(!pages); - return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, start); + return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, start, + USE_FOLL_GET); } EXPORT_SYMBOL(iov_iter_get_pages2); +/* + * A FOLL_PIN variant that calls pin_user_pages_fast() instead of + * get_user_pages_fast(). + */ +ssize_t iov_iter_pin_pages(struct iov_iter *i, + struct page **pages, size_t maxsize, unsigned int maxpages, + size_t *start) +{ + if (!maxpages) + return 0; + if (WARN_ON_ONCE(!pages)) + return -EINVAL; + if (WARN_ON_ONCE(!user_backed_iter(i))) + return -EINVAL; + + return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, start, + USE_FOLL_PIN); +} +EXPORT_SYMBOL(iov_iter_pin_pages); + ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) @@ -1509,7 +1549,8 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, *pages = NULL; - len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start); + len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, + USE_FOLL_GET); if (len <= 0) { kvfree(*pages); *pages = NULL; @@ -1518,6 +1559,31 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); +/* + * A FOLL_PIN variant that calls pin_user_pages_fast() instead of + * get_user_pages_fast(). + */ +ssize_t iov_iter_pin_pages_alloc(struct iov_iter *i, + struct page ***pages, size_t maxsize, + size_t *start) +{ + ssize_t len; + + *pages = NULL; + + if (WARN_ON_ONCE(!user_backed_iter(i))) + return -EINVAL; + + len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, + USE_FOLL_PIN); + if (len <= 0) { + kvfree(*pages); + *pages = NULL; + } + return len; +} +EXPORT_SYMBOL(iov_iter_pin_pages_alloc); + size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { From patchwork Sat Aug 27 08:36:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 12956861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D523C0502F for ; Sat, 27 Aug 2022 08:36:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF53E80007; Sat, 27 Aug 2022 04:36:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA492940008; Sat, 27 Aug 2022 04:36:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A818B80007; Sat, 27 Aug 2022 04:36:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 976BA940008 for ; Sat, 27 Aug 2022 04:36:18 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6FBDA407CD for ; Sat, 27 Aug 2022 08:36:18 +0000 (UTC) X-FDA: 79844715636.19.144E530 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2065.outbound.protection.outlook.com [40.107.237.65]) by imf06.hostedemail.com (Postfix) with ESMTP id DEF8F180020 for ; Sat, 27 Aug 2022 08:36:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Di9w4/4zZrdblMneJtuXTD416WiNIhM1x91YiK0eC+hqk0adLaA1+tyK2M9IvHCt5kqzBRX2Em5mQjAKD+mUMnxPzD/99QIPVXO58aChEcENOS6dcFjD0qhPjHC15+7b3G3yhrRTI1BBZh860PolkxVP+AWolTHspe6QQsvfEdLYPjxfnVg69iHvK0nPoJHKc8QJr8yDrKPiagA1ZuKstTX3B865gY0r38aUfyKT4bemeWaG1gAu7g1Lynjk1h3NWElCwkctjPwfk6YOB7n1aLUQtW37BJ14A4z+bGUkWpCQ4dETa/in611TQRLyZM0nRfTkNgkRxxEmWLGNjFUeYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gfeJQm0LK+R1tjTZehVFwsPGW1k7GJS7dUu3qTkFBT0=; b=gfDBDg1HSqlbtOz0TLnuoeQiLbmFGyD+gampdyl5FRA+YAcnK03YJNOMxYvM1C6qJxWJZDLguPsnu+VKGMHlL05WQxJ+OJB5RjfxG5e6h65HncpKvDESbngV89oIH0wFNP3/IiZ3/6NlagvX6//qTu0Sn0oXfYUtu9vPNBar5Yh4Zo7tGrptpkB8nWQqCXeIUSU65aFCGhOxOLCzLTAejJFXymthrrvUvS3eqJGlCiDUWSLd1MWq+enLaFXDpMBTvWSg4+H94HHjrkfGyivywVsUh2FFKpgV4OnX6TBr6kaqYJM+IB+YqPGn4c0FOW2o9ni/b8tR7Yf2xbiD8xLKTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gfeJQm0LK+R1tjTZehVFwsPGW1k7GJS7dUu3qTkFBT0=; b=KM3NwaQ2RwpVuLEbosVDHnRdrK4qs9JgI+ehoErF0dwRjdCn3WTSO+wOd9HzV/RsiWZYSX7EIm71844yQp30uLy4wKY0E6ogOa04WMNe1kFJg0Tn1lqjbxphtvLPT1oJJdTTuQk/6rwqGyRCWPfgDaJhZnwt/ZIjqDjAc60N8QgJvT2TIgtnCgi/moMlr7O1iA11j+/i21KNn0ewQNIVDjAq5iPkLxeHOqFICjOmSGQvuIcwQDBViKKRfBp9uzF7OdRNQC1TU6/CXeGWwBMBfgyOk7703fdZwcuAHcrSUCuOzUnGVgD82iC0UE/Vms5F9v4iTopYsdD5pH9sFwsukA== Received: from BN0PR04CA0061.namprd04.prod.outlook.com (2603:10b6:408:ea::6) by MWHPR1201MB0158.namprd12.prod.outlook.com (2603:10b6:301:56::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.14; Sat, 27 Aug 2022 08:36:15 +0000 Received: from BN8NAM11FT061.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ea:cafe::c) by BN0PR04CA0061.outlook.office365.com (2603:10b6:408:ea::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.236) by BN8NAM11FT061.mail.protection.outlook.com (10.13.177.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:14 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sat, 27 Aug 2022 08:36:13 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Sat, 27 Aug 2022 01:36:12 -0700 Received: from sandstorm.attlocal.net (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Sat, 27 Aug 2022 01:36:12 -0700 From: John Hubbard To: Andrew Morton CC: Jens Axboe , Alexander Viro , Miklos Szeredi , Christoph Hellwig , "Darrick J . Wong" , Trond Myklebust , Anna Schumaker , Jan Kara , Logan Gunthorpe , , , , , , LKML , John Hubbard Subject: [PATCH 4/6] block, bio, fs: convert most filesystems to pin_user_pages_fast() Date: Sat, 27 Aug 2022 01:36:05 -0700 Message-ID: <20220827083607.2345453-5-jhubbard@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220827083607.2345453-1-jhubbard@nvidia.com> References: <20220827083607.2345453-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 09c15a54-adf0-42fd-6f7c-08da880733f3 X-MS-TrafficTypeDiagnostic: MWHPR1201MB0158:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8UbbcK2DBVR1SHEGT2Wll47zeWPfFOjIu4YP10+EUoo9+n6wJ8O8ygqCjRNNbVht8jCwESPFiQYKQ+v+YC12KiHHe7/0fXpXzmfHJsPLlJDO/bObRRCQMiEfkorZ3w6EmVnBaCgwtTV6xf9/s6MHKG/XimeTxmFNGFjr+cO9JTaSaAKMsLIvt4PBmyopGTduOZr7tP00Bco/FJuOi3ZeYEiweaZMJRhKO5s9+oUINJaWIhfUGYGsfnnzTDbeQvE58slg+ad+r80SDmiz2eKob+rozIJ3VfvvJLNYphz1PqT3Fm3KzS9P60JQRRPjl55baBnlhFGmpWeiKZ16W9W0VWdTNy2Hsgwajysv+mJePwrZIUID/LjtJ9zv6teOWWzGvIBb7pnxNeMb7CGHYCVmD4tSPHlxG7D7CYmm8gYoXe+W9ilndGyM/mypxeNVprugGfleLcLib2fsBa4jQVwyxPhZGQsH/8sgoIGOgkgXajGD8sT96IQIePxsIKwhDfylJydmk9r8m1NcTPx+yru6xdCZ0+Su+q44uLPCvKfKfOFo45Zbhm2YotOuSWiYOkueQBd9LVLnf+1pDL/aseXz6lCXlqOJrq2avT/GzasXZgg1nd7sFKmeLf1ey4rRRhRhn53pYinTaWod7ytUnnWq32MTbud4q1jxDlVZK9LAtT9ZMQPSXDwPkzfP7heACba5SBCA5K8N/hHErYj6Diput/7sO2xMa+M1R3ulBljMKXR0eoVpDXgVPveX90t9nbIr4Sc/Rw++X+l8dgPomIptP1+uQmQ55VCisSyUSkbRxes= X-Forefront-Antispam-Report: CIP:12.22.5.236;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230016)(4636009)(39860400002)(396003)(376002)(136003)(346002)(40470700004)(36840700001)(46966006)(36860700001)(41300700001)(478600001)(2906002)(8936002)(36756003)(4326008)(8676002)(70586007)(30864003)(83380400001)(7416002)(5660300002)(40480700001)(82740400003)(70206006)(82310400005)(47076005)(26005)(1076003)(426003)(336012)(186003)(107886003)(54906003)(2616005)(6666004)(40460700003)(356005)(81166007)(316002)(86362001)(6916009)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2022 08:36:14.1587 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 09c15a54-adf0-42fd-6f7c-08da880733f3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.236];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT061.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0158 ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1661589378; a=rsa-sha256; cv=pass; b=GJzQhBYB58aNFTfxWcLrQnZiBw3886JJiIhU6/gn+OdyG6sPUPMyqBVRa/kXdfKKnXsQ7V dJ8JUmMOrHSl41DSm8Ff06lFQzVLVWCQ+VdylbGS9+d2cfcMU3q7VnCE4C9r2Y4g+gdy9D /lnVhhncGsCLSEH8ZcezMzEsl/zKWhw= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=KM3NwaQ2; spf=pass (imf06.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.237.65 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661589378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gfeJQm0LK+R1tjTZehVFwsPGW1k7GJS7dUu3qTkFBT0=; b=szpqEmZZEak9zwJm+aJvnyWSvZri+jOxyyUVqZKIqe69KCTjN66RGmFoKEIVhcFnrmLTPN WgIRJ44kGxyNot3OiX20QErUzZX4nmYskouWH5WzqbQAbQpwcWp60wgD1Dx7Zv9LSLbRUb 9r7EPAlQFXflp4d9lVkoMdRg8Vs6ehU= X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=KM3NwaQ2; spf=pass (imf06.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.237.65 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") X-Rspamd-Server: rspam08 X-Stat-Signature: dwftct45w1brr16b8humug9js5yx1xni X-Rspamd-Queue-Id: DEF8F180020 X-HE-Tag: 1661589377-153243 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use dio_w_*() wrapper calls, in place of get_user_pages_fast(), get_page() and put_page(). This converts the Direct IO parts of most filesystems over to using FOLL_PIN (pin_user_page*()) page pinning. Signed-off-by: John Hubbard --- block/bio.c | 27 ++++++++++++++------------- block/blk-map.c | 7 ++++--- fs/direct-io.c | 40 ++++++++++++++++++++-------------------- fs/iomap/direct-io.c | 2 +- 4 files changed, 39 insertions(+), 37 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3d3a2678fea2..6c6110f7054e 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1125,7 +1125,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + dio_w_unpin_user_page(bvec->bv_page); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1162,7 +1162,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, } if (same_page) - put_page(page); + dio_w_unpin_user_page(page); return 0; } @@ -1176,7 +1176,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len) return -EINVAL; if (same_page) - put_page(page); + dio_w_unpin_user_page(page); return 0; } @@ -1187,10 +1187,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, * @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. - * For multi-segment *iter, this function only adds pages from the - * next non-empty segment of the iov iterator. + * Pins pages from *iter and appends them to @bio's bvec array. The pages will + * have to be released using dio_w_unpin_user_page when done. For multi-segment + * *iter, this function only adds pages from the next non-empty segment of the + * iov iterator. */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1218,8 +1218,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages2(iter, pages, UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset); + size = dio_w_iov_iter_pin_pages(iter, pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1252,7 +1253,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) iov_iter_revert(iter, left); out: while (i < nr_pages) - put_page(pages[i++]); + dio_w_unpin_user_page(pages[i++]); return ret; } @@ -1444,9 +1445,9 @@ void bio_set_pages_dirty(struct bio *bio) * have been written out during the direct-IO read. So we take another ref on * the BIO and re-dirty the pages in process context. * - * It is expected that bio_check_pages_dirty() will wholly own the BIO from - * here on. It will run one put_page() against each page and will run one - * bio_put() against the BIO. + * It is expected that bio_check_pages_dirty() will wholly own the BIO from here + * on. It will run one dio_w_unpin_user_page() against each page and will run + * one bio_put() against the BIO. */ static void bio_dirty_fn(struct work_struct *work); diff --git a/block/blk-map.c b/block/blk-map.c index 7196a6b64c80..4e333ad9776d 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -254,7 +254,8 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, size_t offs, added = 0; int npages; - bytes = iov_iter_get_pages_alloc2(iter, &pages, LONG_MAX, &offs); + bytes = dio_w_iov_iter_pin_pages_alloc(iter, &pages, LONG_MAX, + &offs); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -276,7 +277,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (!bio_add_hw_page(rq->q, bio, page, n, offs, max_sectors, &same_page)) { if (same_page) - put_page(page); + dio_w_unpin_user_page(page); break; } @@ -289,7 +290,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, * release the pages we didn't map into the bio, if any */ while (j < npages) - put_page(pages[j++]); + dio_w_unpin_user_page(pages[j++]); kvfree(pages); /* couldn't stuff something into bio? */ if (bytes) { diff --git a/fs/direct-io.c b/fs/direct-io.c index f669163d5860..05c044c55374 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -169,8 +169,8 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) const enum req_op dio_op = dio->opf & REQ_OP_MASK; ssize_t ret; - ret = iov_iter_get_pages2(sdio->iter, dio->pages, LONG_MAX, DIO_PAGES, - &sdio->from); + ret = dio_w_iov_iter_pin_pages(sdio->iter, dio->pages, LONG_MAX, + DIO_PAGES, &sdio->from); if (ret < 0 && sdio->blocks_available && dio_op == REQ_OP_WRITE) { struct page *page = ZERO_PAGE(0); @@ -181,7 +181,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) */ if (dio->page_errors == 0) dio->page_errors = ret; - get_page(page); + dio_w_pin_user_page(page); dio->pages[0] = page; sdio->head = 0; sdio->tail = 1; @@ -197,7 +197,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) sdio->to = ((ret - 1) & (PAGE_SIZE - 1)) + 1; return 0; } - return ret; + return ret; } /* @@ -324,7 +324,7 @@ static void dio_aio_complete_work(struct work_struct *work) static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio); /* - * Asynchronous IO callback. + * Asynchronous IO callback. */ static void dio_bio_end_aio(struct bio *bio) { @@ -449,7 +449,7 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) static inline void dio_cleanup(struct dio *dio, struct dio_submit *sdio) { while (sdio->head < sdio->tail) - put_page(dio->pages[sdio->head++]); + dio_w_unpin_user_page(dio->pages[sdio->head++]); } /* @@ -716,7 +716,7 @@ static inline int dio_bio_add_page(struct dio_submit *sdio) */ if ((sdio->cur_page_len + sdio->cur_page_offset) == PAGE_SIZE) sdio->pages_in_io--; - get_page(sdio->cur_page); + dio_w_pin_user_page(sdio->cur_page); sdio->final_block_in_bio = sdio->cur_page_block + (sdio->cur_page_len >> sdio->blkbits); ret = 0; @@ -725,7 +725,7 @@ static inline int dio_bio_add_page(struct dio_submit *sdio) } return ret; } - + /* * Put cur_page under IO. The section of cur_page which is described by * cur_page_offset,cur_page_len is put into a BIO. The section of cur_page @@ -787,7 +787,7 @@ static inline int dio_send_cur_page(struct dio *dio, struct dio_submit *sdio, * An autonomous function to put a chunk of a page under deferred IO. * * The caller doesn't actually know (or care) whether this piece of page is in - * a BIO, or is under IO or whatever. We just take care of all possible + * a BIO, or is under IO or whatever. We just take care of all possible * situations here. The separation between the logic of do_direct_IO() and * that of submit_page_section() is important for clarity. Please don't break. * @@ -832,13 +832,13 @@ submit_page_section(struct dio *dio, struct dio_submit *sdio, struct page *page, */ if (sdio->cur_page) { ret = dio_send_cur_page(dio, sdio, map_bh); - put_page(sdio->cur_page); + dio_w_unpin_user_page(sdio->cur_page); sdio->cur_page = NULL; if (ret) return ret; } - get_page(page); /* It is in dio */ + dio_w_pin_user_page(page); /* It is in dio */ sdio->cur_page = page; sdio->cur_page_offset = offset; sdio->cur_page_len = len; @@ -853,7 +853,7 @@ submit_page_section(struct dio *dio, struct dio_submit *sdio, struct page *page, ret = dio_send_cur_page(dio, sdio, map_bh); if (sdio->bio) dio_bio_submit(dio, sdio); - put_page(sdio->cur_page); + dio_w_unpin_user_page(sdio->cur_page); sdio->cur_page = NULL; } return ret; @@ -890,7 +890,7 @@ static inline void dio_zero_block(struct dio *dio, struct dio_submit *sdio, * We need to zero out part of an fs block. It is either at the * beginning or the end of the fs block. */ - if (end) + if (end) this_chunk_blocks = dio_blocks_per_fs_block - this_chunk_blocks; this_chunk_bytes = this_chunk_blocks << sdio->blkbits; @@ -954,7 +954,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, ret = get_more_blocks(dio, sdio, map_bh); if (ret) { - put_page(page); + dio_w_unpin_user_page(page); goto out; } if (!buffer_mapped(map_bh)) @@ -999,7 +999,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, /* AKPM: eargh, -ENOTBLK is a hack */ if (dio_op == REQ_OP_WRITE) { - put_page(page); + dio_w_unpin_user_page(page); return -ENOTBLK; } @@ -1012,7 +1012,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, if (sdio->block_in_file >= i_size_aligned >> blkbits) { /* We hit eof */ - put_page(page); + dio_w_unpin_user_page(page); goto out; } zero_user(page, from, 1 << blkbits); @@ -1052,7 +1052,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, sdio->next_block_for_io, map_bh); if (ret) { - put_page(page); + dio_w_unpin_user_page(page); goto out; } sdio->next_block_for_io += this_chunk_blocks; @@ -1067,8 +1067,8 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, break; } - /* Drop the ref which was taken in get_user_pages() */ - put_page(page); + /* Drop the ref which was taken in [get|pin]_user_pages() */ + dio_w_unpin_user_page(page); } out: return ret; @@ -1288,7 +1288,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, ret2 = dio_send_cur_page(dio, &sdio, &map_bh); if (retval == 0) retval = ret2; - put_page(sdio.cur_page); + dio_w_unpin_user_page(sdio.cur_page); sdio.cur_page = NULL; } if (sdio.bio) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 4eb559a16c9e..fc7763c418d1 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - get_page(page); + dio_w_pin_user_page(page); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } From patchwork Sat Aug 27 08:36:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 12956862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00213C6FA81 for ; Sat, 27 Aug 2022 08:36:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 33563940008; Sat, 27 Aug 2022 04:36:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26A8880008; Sat, 27 Aug 2022 04:36:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10B5E94000B; Sat, 27 Aug 2022 04:36:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 04E44940008 for ; Sat, 27 Aug 2022 04:36:19 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CE0D8160830 for ; Sat, 27 Aug 2022 08:36:18 +0000 (UTC) X-FDA: 79844715636.07.940BC2C Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2055.outbound.protection.outlook.com [40.107.220.55]) by imf30.hostedemail.com (Postfix) with ESMTP id 63B7380037 for ; Sat, 27 Aug 2022 08:36:18 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M1imsOk9e7Q/mcgS2hXZ0IPOMIg8aHC+QYBAIghQfMIjNoMO6KSqM58Xcp1uCyVaUQUFTZ/eNN/fyPApQtOLP6s5R52Kz0zlTHehbZJnq72DpDRqBgWcYuOwsY0i/IepeLR0XBoUw2gOJnq8BAImK+WxNifJ6AUZ/6QwxYVFx2ViOy4OhzbTN4SGM4cqlzuZl2mjgKXf/jdXTkeXnL9K6qJwf/SFB0QPVBF/HeGScBMDuFyAF4y1ZYyITpg5ioxaX6C/dFiFT+Yzviwj/3jDTYmho2jFj2ykMAuYV8msGWtqUeRiCKGWAlPGsFa9y10ataopMyR1esI7nOv5mt/HZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dCJ4cBcOTlL0cm3WRQNtfx2LiFChT7DgHkr76NwhXwE=; b=jsnjOB1gOVJv+RXhADCgKN5Q+uF0gy4JO6SNk0f1oaR5uA2+Gzwvx/mtUKch+H0D0cPIavxZ7BsVBJHsaKHTJ5vcy+KVs8+IoZw3/tpO0v2obP7N/545yV1d2OIzvT5hOYBzlr6+vyJVAK9Tg0bYduao08roEjENWnmnF9PXAYJybMqnjTIvH+5xwo2nNvwrUjGDu9WChhzEwkUcEobTOxXtrfnl1w6WC+SJuiuEh4B7GkDV3tgagLz16KtrH3zEYrQofRZDwfBus08ahRUECJYuy8PdfAdX24A+Akt5fllSV9qg09AbUNL2zxrwHkv5AfZdlkyloji1tY3oaK16EQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dCJ4cBcOTlL0cm3WRQNtfx2LiFChT7DgHkr76NwhXwE=; b=HZbyc9jqVhwgkTWi0t/9KFSCao6TcLEV4E903wMuo2uUKPJ7bzPQAw01GfsxYrff3f5GHACulEcS6rLbQkrxI3/wuzzs72IIQfEr1Rt1jK++eqCFvWLp0PebsnyPGfywBhSp9Y/d9rdwf+zqOita3ZIK7Yg5PtIT+gDWsa+X0lLFjpWGmEPnNZVvl3PX7z7NMV6dTn9ArHeg7uPj82g6CqFn+z3Hnd4gLCeU79cHGsHvuenMH0Z3ugj/1cDY2k+BE7/6lG/7zluQUfD6mwCD3XKNedwg6OUtPZJYEM7HUorGwFoFIaxwrZC5R6dsOxifehw7ticWHjgzn8QqrFsF2w== Received: from DM6PR07CA0109.namprd07.prod.outlook.com (2603:10b6:5:330::17) by BYAPR12MB5704.namprd12.prod.outlook.com (2603:10b6:a03:9d::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5546.21; Sat, 27 Aug 2022 08:36:15 +0000 Received: from DM6NAM11FT003.eop-nam11.prod.protection.outlook.com (2603:10b6:5:330:cafe::d0) by DM6PR07CA0109.outlook.office365.com (2603:10b6:5:330::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by DM6NAM11FT003.mail.protection.outlook.com (10.13.173.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:14 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sat, 27 Aug 2022 08:36:14 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Sat, 27 Aug 2022 01:36:13 -0700 Received: from sandstorm.attlocal.net (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Sat, 27 Aug 2022 01:36:13 -0700 From: John Hubbard To: Andrew Morton CC: Jens Axboe , Alexander Viro , Miklos Szeredi , Christoph Hellwig , "Darrick J . Wong" , Trond Myklebust , Anna Schumaker , Jan Kara , Logan Gunthorpe , , , , , , LKML , John Hubbard Subject: [PATCH 5/6] NFS: direct-io: convert to FOLL_PIN pages Date: Sat, 27 Aug 2022 01:36:06 -0700 Message-ID: <20220827083607.2345453-6-jhubbard@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220827083607.2345453-1-jhubbard@nvidia.com> References: <20220827083607.2345453-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 67c097dc-a8cd-4984-4acb-08da88073446 X-MS-TrafficTypeDiagnostic: BYAPR12MB5704:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /T7dC52ZIAX2uTlrqTehDTCMXwSnF040OnIZpRiac8bgizTvt8qqB3N/0inFQ2bxMURI0EgvT7U9L77txe8JDN6MS8QjjidQSQpyd8RrYLzznsSKrzGLoyb3uyVulB1UxcmweYfz/1ts98UQNXzjGjkf6dUxY2wLEHAxFwcLpmkMfpkVo9uvqN56RDmeip0ctdOFgHLIhetslm8pyMv5VxYFz18hacdD01G0gZy+5mZWdygCC+xvBSotLybtGwD9D6QqOSXYGCODitg2x4J0klAiOgR5/T5KxdfkY7pSuhohyiFoLTLu2wTXGaweD6arsL4dzgqAz4UifCpnhPaaBFaH+hDq2AUJZcYfCXAYyaQKoyJfLrkVMhrZd4cCDEUnqKheigHKAMPJ8f/xtx0i5GOCXn76+qHc3zl4firm5I2oBfOpbOCoUOXihFNZgEl6EzzSsjyJefpszA8TMR1zKCIYLe399YJ+6W7xXrnkWO82RqGt/1SVRuaehMbkv854dXbyCSCn7kHrGwugFdkbsRPra036jz5DavbM4tBL5c9FcV/yuZf2SOqbGksYbquEJqEqxEKaWw3T9ivy0nF4FrJd7sdW2bTJEM2hk1HDYMFiinROMQQMS8PTajGX13oR9SieBL5/48QkLsosrnsvULBBYLT5ckQr/hz517KqgERrwt2kvuvSspWrNjKp/lju2cIJa+6KqmXpRC6GPrWLYV6JNtrruZqNBhXl3lgol/AS9ATLfRxdYN3gpGi5a7fJuLA5aLh5nfNujeysL3Th1nq7F68Nw7V8Bn1HnDIU8yo= X-Forefront-Antispam-Report: CIP:12.22.5.235;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230016)(4636009)(39860400002)(396003)(346002)(376002)(136003)(36840700001)(46966006)(40470700004)(83380400001)(82310400005)(81166007)(40460700003)(40480700001)(8676002)(82740400003)(70206006)(36860700001)(70586007)(4326008)(36756003)(5660300002)(7416002)(26005)(6916009)(8936002)(316002)(1076003)(6666004)(478600001)(41300700001)(54906003)(356005)(86362001)(107886003)(47076005)(336012)(186003)(426003)(2906002)(2616005)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2022 08:36:14.8423 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 67c097dc-a8cd-4984-4acb-08da88073446 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.235];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT003.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB5704 ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=HZbyc9jq; spf=pass (imf30.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.220.55 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1661589378; a=rsa-sha256; cv=pass; b=ZqCH1kxcGa90RpdW27+7z5mCY2RFZ6OPkVmOZPUkzyyK7wtPwIsq0qn1kXD0R+M/Aj4iIl 7dWG6MAoo6vJakBwfdoYNOT2ENOLA8cOnNnCLb4xx+0LSpLHdE6/MksKR+Z6/p7upWZSfb 8hcg5LDYqxYZRb+vJf8tLsIKVGAsO2A= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661589378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dCJ4cBcOTlL0cm3WRQNtfx2LiFChT7DgHkr76NwhXwE=; b=sxPbZFbmILpKpwhwmPgvkJ4ES2ci/X2J2a9xMKlplnUtdTtcbeUf8rWQ+MI14S7ESTLayk wAtuNApzkAPEeGnl22+1glwhQ1n3oSUXky2Iy0aGSSee8s0j9gwMCFg1bHH4TPOwsrlgu4 57MLCLBIxuM84FGfIlbRnONDhSROxp4= X-Rspam-User: X-Stat-Signature: 13trwir38fz5ndk9j7fa7sktdyj7sjs9 X-Rspamd-Queue-Id: 63B7380037 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=HZbyc9jq; spf=pass (imf30.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.220.55 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") X-Rspamd-Server: rspam09 X-HE-Tag: 1661589378-815863 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the NFS Direct IO layer to use pin_user_pages_fast() and unpin_user_page(), instead of get_user_pages_fast() and put_page(). Signed-off-by: John Hubbard --- fs/nfs/direct.c | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index 1707f46b1335..f6e47329e092 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -142,13 +142,6 @@ int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter) return 0; } -static void nfs_direct_release_pages(struct page **pages, unsigned int npages) -{ - unsigned int i; - for (i = 0; i < npages; i++) - put_page(pages[i]); -} - void nfs_init_cinfo_from_dreq(struct nfs_commit_info *cinfo, struct nfs_direct_req *dreq) { @@ -332,11 +325,11 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq, size_t pgbase; unsigned npages, i; - result = iov_iter_get_pages_alloc2(iter, &pagevec, + result = dio_w_iov_iter_pin_pages_alloc(iter, &pagevec, rsize, &pgbase); if (result < 0) break; - + bytes = result; npages = (result + pgbase + PAGE_SIZE - 1) / PAGE_SIZE; for (i = 0; i < npages; i++) { @@ -362,7 +355,7 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq, pos += req_len; dreq->bytes_left -= req_len; } - nfs_direct_release_pages(pagevec, npages); + dio_w_unpin_user_pages(pagevec, npages); kvfree(pagevec); if (result < 0) break; @@ -791,8 +784,8 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq, size_t pgbase; unsigned npages, i; - result = iov_iter_get_pages_alloc2(iter, &pagevec, - wsize, &pgbase); + result = dio_w_iov_iter_pin_pages_alloc(iter, &pagevec, + wsize, &pgbase); if (result < 0) break; @@ -829,7 +822,7 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq, pos += req_len; dreq->bytes_left -= req_len; } - nfs_direct_release_pages(pagevec, npages); + dio_w_unpin_user_pages(pagevec, npages); kvfree(pagevec); if (result < 0) break; From patchwork Sat Aug 27 08:36:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Hubbard X-Patchwork-Id: 12956863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B77FC6FA83 for ; Sat, 27 Aug 2022 08:36:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6F5580009; Sat, 27 Aug 2022 04:36:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF5DA80008; Sat, 27 Aug 2022 04:36:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FB2080009; Sat, 27 Aug 2022 04:36:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7EC1B80008 for ; Sat, 27 Aug 2022 04:36:24 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 61E8F140169 for ; Sat, 27 Aug 2022 08:36:24 +0000 (UTC) X-FDA: 79844715888.08.74BE8D7 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2051.outbound.protection.outlook.com [40.107.237.51]) by imf19.hostedemail.com (Postfix) with ESMTP id 0F7981A001D for ; Sat, 27 Aug 2022 08:36:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XJwnyuIDOEK+ZWlRAqsUfMuse45AwM0Ps2KDl3VXuAmoiiTLFLOHYEEgmiSmpDO8i/vc6n8B2n1/uvTfRjthk4/ghVplVoVBP30sTFBj0qPNU9+cs5roSVpPo5uE2RXaAT76QmvbZkrkfTJyUwuvDmH5bj9cdKjLRxvw3mZG+sHdfVendJavA/OJuXJOuHLB9e2ug1CtsUwLLRZoTrtWQ8hlhcVGGKveXNLAgd5fgsEi0uV0QTpSP3AibVJRDnknEaNf94qwil5YimEmjNysj0v/1aTA+7hUWY4ZXLEvtu1M/ajrf9ybKEL/j9DitVSVam4f8oMBKGsSysnCifWkrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=huPDqMjn3txLV0ohJIHpJLOiJMrfH0byrZT2L7g4oV0=; b=CkK3hhzCQy26Q8CeOfUMMfO6rxZCUMDA1vliHysDxn+zOLVqrkpOdxBvI0f+pl1dGQdzbr23vepUoyFcVb3ZTJi6E7pRld7euwB9gvWqAtxZdA3Lk9RCq39yo6S4jUQbN3w21YT6P88QybSkorgaZY0jisnZGRpKwcdk5vHS/6183TwEyMlh5WnqPRuSwwi5TITObED1aKZ/SWPOiV4bCWQgdUFgqe7ILYrl4eslR8ZehbhBQLZPCM9Myr6FE1QMh0oJdkInd78SPcG2CTW/0OGgcyy+bdlPqOegdGC2C88HwJXQKJZ1pDdk/BRxrx46uelZMaJ4K5yqDQlpypA8Bg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=huPDqMjn3txLV0ohJIHpJLOiJMrfH0byrZT2L7g4oV0=; b=TXqEdwaj+YC0sEWqbU6BBTmrh1IxSfXdjpxGmaiOCqgRhpSqy7IARL0edzcZl0wi/fLkiAYyQRzYHvxlcjFFiBJqY3J+OR7KmsnA2c0VZm3ylNBfXgeAKA20Ql2k5Ai7P7PrKZhrWjFYp/jAXq1woTzYZlkp5yB2E8YeuhdPUdVkpyqRUhA38Dw6rE98JrvueRh7tNDmWhYYUoZ1J/qFS5HgYjLQRZu8lMyq5Spm+m6LM/Yf9dk4lx/I/vrXup8t0D3xf8N+js+jXMIAjOiOR69BS2QZHm0b9zz7KaWPmgKRH9Qj3FDUyu5t/+iLk03SVC28NoCgkZdGaNeEc9Oa9w== Received: from MW4PR03CA0106.namprd03.prod.outlook.com (2603:10b6:303:b7::21) by BN6PR12MB1380.namprd12.prod.outlook.com (2603:10b6:404:1f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5566.15; Sat, 27 Aug 2022 08:36:15 +0000 Received: from CO1NAM11FT106.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b7:cafe::3f) by MW4PR03CA0106.outlook.office365.com (2603:10b6:303:b7::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5546.20 via Frontend Transport; Sat, 27 Aug 2022 08:36:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT106.mail.protection.outlook.com (10.13.175.44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5566.15 via Frontend Transport; Sat, 27 Aug 2022 08:36:15 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sat, 27 Aug 2022 08:36:15 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Sat, 27 Aug 2022 01:36:14 -0700 Received: from sandstorm.attlocal.net (10.127.8.9) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.986.29 via Frontend Transport; Sat, 27 Aug 2022 01:36:14 -0700 From: John Hubbard To: Andrew Morton CC: Jens Axboe , Alexander Viro , Miklos Szeredi , Christoph Hellwig , "Darrick J . Wong" , Trond Myklebust , Anna Schumaker , Jan Kara , Logan Gunthorpe , , , , , , LKML , John Hubbard Subject: [PATCH 6/6] fuse: convert direct IO paths to use FOLL_PIN Date: Sat, 27 Aug 2022 01:36:07 -0700 Message-ID: <20220827083607.2345453-7-jhubbard@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220827083607.2345453-1-jhubbard@nvidia.com> References: <20220827083607.2345453-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2dd39851-16d0-40a3-9b9e-08da880734a0 X-MS-TrafficTypeDiagnostic: BN6PR12MB1380:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lMKwUJFBqc5KWe8/0tClb0xMLOIU4eMcZKSJLudZXvm2XJtLI+5IhangcMQSaT31z8byiTJwSz/4T1QuHFVnNPpGbPfzzggHE1U1ukJVGC7rrwbMH/PRMlyMOh4br5DFd4oInSVAEuuQ7B13QhdMJR6sZrUCHtBtVw0iQgjXlxZaK7Ql61IF7YDwpnJQhYisZ/fppWPyjCi6Ti4aMfxz1Vhm5bL/xDcz9KKNH0ef/QeR1t9hs9bYLT16pvCFp7b96co9aC7kkt7iazKfiRMwL7m78PNJRF8Pmdd0p5/RllyJPGeqAYVjczeyFLPCvmAomkevbQkLKiy84VhEnjY1u5bKxKzCybpwPiiQqUj342CcG5aLtyXtTEbMtHoERhacL5Di3tly3c1Gw+Dzgj2ZGxDL5UIHJrUFwdKsFvlY22wc5UI6zQ0fMUspwy01vNmTQI3Xd3+UxIBxDy/vbwzJ/iYVmV4JRNswzHCLeP+uZUdMId5ETxEKJA4gaMsMXxLaBUwS5AADGfJQTSb+/+8GfJCJMIHnxQbMRF8LR8WPI6zGpL46AhuKQyKjf+oUA10mbTYzO1TtpKzsJXvsYaHnIebvG8+HAd40JvzP0gZ0MI6W2V370ApdROJunoqZ/Knleh1adkc8doRWo/DWCCRIOAIcUi8Kjp5iDUplx9Nf0Wkutat6X/JqmoanU8jMs08RbI8+RZ0H2Lo1iquqgnZyAjw3C3KD4J+YRy2p1wDcT7TdbIk/kSkHxrL22OQALIN0vynF42tsuIOCTpIGE7ZZRMHbQXctihTYVwk+7UKxo2Q= X-Forefront-Antispam-Report: CIP:12.22.5.238;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:InfoNoRecords;CAT:NONE;SFS:(13230016)(4636009)(39860400002)(396003)(376002)(346002)(136003)(46966006)(36840700001)(40470700004)(478600001)(47076005)(426003)(83380400001)(8676002)(4326008)(70586007)(70206006)(40460700003)(2906002)(5660300002)(8936002)(107886003)(6666004)(7416002)(41300700001)(2616005)(1076003)(86362001)(186003)(336012)(82740400003)(316002)(36756003)(81166007)(54906003)(36860700001)(6916009)(40480700001)(82310400005)(26005)(356005)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2022 08:36:15.4186 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2dd39851-16d0-40a3-9b9e-08da880734a0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[12.22.5.238];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT106.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1380 ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1661589384; a=rsa-sha256; cv=pass; b=MGDbk4BHHIYHCI77dqIl9WufMTycYvHsyBM4krq2vsRRYDxFA9A0eCjlTivLrZYTbHNHBN ANzPKHjBSBfgRtwpR7aSX2SUB/mtgsNiYG4Dwdi9kUjlwcAZ0MoSkjgFUz0UyeK7azR4qg 9GL64mGe+vsJaxHJYazhuOo7SC9e1mY= ARC-Authentication-Results: i=2; imf19.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=TXqEdwaj; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.237.51 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661589384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=huPDqMjn3txLV0ohJIHpJLOiJMrfH0byrZT2L7g4oV0=; b=JQwaQCLUOpXMmVftLGKGeyxPQIMipPxPJaGdkLU7fJgKtoR5BeMJ4R3zpGSxBmlofj+Wvf YJm4q5xNwiQlNHF40mdmN7xFXHEXfDgu3yfx1YhS9frWD/iqX2yAk2PBsdaPTkDRDAHMnX eqmQOK7asnSJh3QFNAqgOGw8352PLPs= X-Rspamd-Queue-Id: 0F7981A001D Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=TXqEdwaj; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.237.51 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: irb3ec5a63awk19npjixsjho6gfr36pk X-HE-Tag: 1661589383-943992 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the fuse filesystem to use pin_user_pages_fast() and unpin_user_page(), instead of get_user_pages_fast() and put_page(). Signed-off-by: John Hubbard --- fs/fuse/dev.c | 8 ++++++-- fs/fuse/file.c | 31 ++++++++++++++++++++----------- fs/fuse/fuse_i.h | 1 + 3 files changed, 27 insertions(+), 13 deletions(-) diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 51897427a534..eb841fc82bb9 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -675,7 +675,10 @@ static void fuse_copy_finish(struct fuse_copy_state *cs) flush_dcache_page(cs->pg); set_page_dirty_lock(cs->pg); } - put_page(cs->pg); + if (cs->pipebufs) + put_page(cs->pg); + else + dio_w_unpin_user_page(cs->pg); } cs->pg = NULL; } @@ -730,7 +733,8 @@ static int fuse_copy_fill(struct fuse_copy_state *cs) } } else { size_t off; - err = iov_iter_get_pages2(cs->iter, &page, PAGE_SIZE, 1, &off); + err = dio_w_iov_iter_pin_pages(cs->iter, &page, PAGE_SIZE, 1, + &off); if (err < 0) return err; BUG_ON(!err); diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 1a3afd469e3a..a79aa4fea937 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -625,14 +625,19 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos, } static void fuse_release_user_pages(struct fuse_args_pages *ap, - bool should_dirty) + bool should_dirty, bool is_kvec) { unsigned int i; - for (i = 0; i < ap->num_pages; i++) { - if (should_dirty) - set_page_dirty_lock(ap->pages[i]); - put_page(ap->pages[i]); + if (is_kvec) { + for (i = 0; i < ap->num_pages; i++) { + if (should_dirty) + set_page_dirty_lock(ap->pages[i]); + put_page(ap->pages[i]); + } + } else { + dio_w_unpin_user_pages_dirty_lock(ap->pages, ap->num_pages, + should_dirty); } } @@ -733,7 +738,7 @@ static void fuse_aio_complete_req(struct fuse_mount *fm, struct fuse_args *args, struct fuse_io_priv *io = ia->io; ssize_t pos = -1; - fuse_release_user_pages(&ia->ap, io->should_dirty); + fuse_release_user_pages(&ia->ap, io->should_dirty, io->is_kvec); if (err) { /* Nothing */ @@ -1414,10 +1419,10 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii, while (nbytes < *nbytesp && ap->num_pages < max_pages) { unsigned npages; size_t start; - ret = iov_iter_get_pages2(ii, &ap->pages[ap->num_pages], - *nbytesp - nbytes, - max_pages - ap->num_pages, - &start); + ret = dio_w_iov_iter_pin_pages(ii, &ap->pages[ap->num_pages], + *nbytesp - nbytes, + max_pages - ap->num_pages, + &start); if (ret < 0) break; @@ -1483,6 +1488,9 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, fl_owner_t owner = current->files; size_t nbytes = min(count, nmax); + /* For use in fuse_release_user_pages(): */ + io->is_kvec = iov_iter_is_kvec(iter); + err = fuse_get_user_pages(&ia->ap, iter, &nbytes, write, max_pages); if (err && !nbytes) @@ -1498,7 +1506,8 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, } if (!io->async || nres < 0) { - fuse_release_user_pages(&ia->ap, io->should_dirty); + fuse_release_user_pages(&ia->ap, io->should_dirty, + io->is_kvec); fuse_io_free(ia); } ia = NULL; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 488b460e046f..1d927e499395 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -290,6 +290,7 @@ struct fuse_io_priv { struct kiocb *iocb; struct completion *done; bool blocking; + bool is_kvec; }; #define FUSE_IO_PRIV_SYNC(i) \