From patchwork Thu Feb 4 20:24:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12068617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0895EC433E0 for ; Thu, 4 Feb 2021 20:25:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6645564F48 for ; Thu, 4 Feb 2021 20:25:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6645564F48 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3C57A6B0005; Thu, 4 Feb 2021 15:25:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 331796B0072; Thu, 4 Feb 2021 15:25:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 110696B006E; Thu, 4 Feb 2021 15:25:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0049.hostedemail.com [216.40.44.49]) by kanga.kvack.org (Postfix) with ESMTP id DFBA16B0070 for ; Thu, 4 Feb 2021 15:25:17 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A3BE9181AEF1E for ; Thu, 4 Feb 2021 20:25:17 +0000 (UTC) X-FDA: 77781715074.24.shame48_6217969275df Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 839541A4AC for ; Thu, 4 Feb 2021 20:25:17 +0000 (UTC) X-HE-Tag: shame48_6217969275df X-Filterd-Recvd-Size: 11330 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Thu, 4 Feb 2021 20:25:16 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KJeDM136245; Thu, 4 Feb 2021 20:25:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=d6jKva1VP2IX+9zJZLtv9om75cG79glVzUbC2f9Pw2s=; b=Ds6dr7NRZ7owDzKZghCOeWdigoqjUmYO1LZpvW9WYgiAW/KlFO9emJS2AJSEE5k437/2 KaJdi9tWto9Sft/nTNto//CYLTKGKmb/5Pmi/lqboonZtovGI0wyxpQhwTtoN9aOJLeD T0Gbv7L7/mfZcFljgrSo2DdmTrwESEdFgW9/1+uOnYVHOZA4ZorPhkFnLJwOoBivGN9W a2+bL4H6wBJ+H0QkLsUptyN9rVF3cTEOJ1mUjFmUuTJxADQGlVKThfYiIlGfdACKAuxu 5dU2ZclKDhFPLoJ1aVx6oFFbRbHmHWitKFf+OtW7IN3GQrvLRLZ3K0XlSDU23XsSG+sz Lw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by aserp2130.oracle.com with ESMTP id 36cvyb7821-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:14 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KFcf9115927; Thu, 4 Feb 2021 20:25:14 GMT Received: from nam04-sn1-obe.outbound.protection.outlook.com (mail-sn1nam04lp2053.outbound.protection.outlook.com [104.47.44.53]) by aserp3030.oracle.com with ESMTP id 36dh1sy19m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Uz8km7VeM5W1MAuuvxRyRVB+cdArqueK6YyEmsYR67XyJDh5zEMZTkKpI+/AioVkvc+lLbMT3+hubWkE+Qqb7q3cpjK8Rb9E9nMkL213hjCriBnvQIbIWQJq4y5fKV3E0CR0P78nO75nVybyaEW/H0h16iuB74S7DinPTxE22l8swZJXSo348kqxKK6xizrc6vNcGrH8KCImNeVGCKKVGgotjJOa/98M3dtXT86r4YEFzCYsu7XhGghBv5tVFM3ojHdHxF9WBCxwffv6FlWVLAvGb7gNikjWRDDujST0UNhCNoTBjtfBsQmry2O8+IvZPFFVkttApTgUeKHe4YhGhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d6jKva1VP2IX+9zJZLtv9om75cG79glVzUbC2f9Pw2s=; b=Ooeg7+uL+wqOcyoACF+byA1y6pyL8ALLhBjvrxZMi276ctjB1hdCABWGATKm6FwJ/U56Ws/gIri7qcDbYBo8xrvAa7c96OjLn8aCCDa6SfY6fyj1pm7fiigTl5/uscPJc+SGRopIZlXw+pqQ0ibfVekd5Xy/5b1tBQ1Ol8wAYsGadDCcpqFIoZvSkGbl/FirmGE8zH9aRKNdy8DO3oXToJtUP5/h3N3qJ46V0peYAkADLKQU5SUDfZs9Tw5CykXFkwH5ETNY9PCY/vF3padS+ZBeCeWyQgrHkVhB3YOYm7lXDvyrpW1mUH8k6brfhQMEgkworFEAznLim/KcyVtGIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=d6jKva1VP2IX+9zJZLtv9om75cG79glVzUbC2f9Pw2s=; b=uZP3Aq3KM4xD91DjYg+WF4NKdXTAYQQSwfyMLSMID7nP30UrqbLlaCq5thTms4Z6a5AHfTdMXrG3szXvVw6149spbFVVZS09HzsewnmMin8ZKVOc1ZTyLWkN6dFhXcal0hs2QiFpeyWqOJEMI0TT91s1gfyRcmpMGklCyDMzzj4= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4590.namprd10.prod.outlook.com (2603:10b6:a03:2d1::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.20; Thu, 4 Feb 2021 20:25:12 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Thu, 4 Feb 2021 20:25:12 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v2 1/4] mm/gup: add compound page list iterator Date: Thu, 4 Feb 2021 20:24:57 +0000 Message-Id: <20210204202500.26474-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210204202500.26474-1-joao.m.martins@oracle.com> References: <20210204202500.26474-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.21 via Frontend Transport; Thu, 4 Feb 2021 20:25:10 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5c6e678b-b311-4a76-c3db-08d8c94af96a X-MS-TrafficTypeDiagnostic: SJ0PR10MB4590: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:862; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VHwwmslulylkuH0xyOD+Q7kW6CN3nXb7o+dUVL9gkm6PjgTK5Z6snOkfHwkZnhLgbY4hMpjY0/uhMsfdziFiodDTddfBa6+rVReYtHO4p4a0LgMzo70ZBOE7G0gE48bcguXN3zr0N0ZRoQVoGqLhs8BgylJ46saQSk9hJxhnZb30w9w/HocNt55v5visb7VoE5zF1kHbOcIaqc9guH13Xb+wSoxbFMQEiZUIUX4kfL2oeiFLojcf/tlBYVRjGB8LkTlCrGL2EmravUnVRdzoS1UzWRuhn+v7Sghy8SHqqFuahoirLpHRphkJ/TEsIt636R1G+sTgZxbyT2dV9UWgz7TpRdc/9cvPKJMtOgkZ01zCJsr1Vxi0/1d6NVZhvcnY+ecCMakxy9qOcxzfG2BAkJ6hCgdtjc+cCR9n5H/4JP5Iz0SP+HLGzS+s29DIHJSQUBqwKoavw1LaeFQiBPtqTcAnJdZHtT4A4+sWMwn+Q2Bh7kFke8JiVhm+et+5zDxbyazt7exh9p8Hu2VUryQ4XQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(376002)(136003)(52116002)(7696005)(26005)(2906002)(16526019)(66476007)(186003)(66556008)(478600001)(103116003)(316002)(6486002)(66946007)(956004)(54906003)(86362001)(5660300002)(1076003)(8936002)(83380400001)(107886003)(36756003)(4326008)(6666004)(2616005)(6916009)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: m1wE8tAvtU6PpGTl1WR2rmV+5m3MVIgdNEqtFiP776Ars11EkhOwb9J1Fapd3hKET2wylNJFxr6VyjxFpX4ZwS/ahCmNadHOd8P773Cvq9lFnpHxqJRyzOM1UbYqmkv4d7INO8fuS9C+7bHDfAkl+jzIxoakAftBQ485erpdV8TZCndEYMRHVW/lAt85fmcG8Kp6T6BoEnkhu+EBeXOHY9Xez1QMCwJerY8tmfza9/58ESrZIeP5vLI+EBFfuIoGMdFuMCS5tT1W/Zh38yq+ZBjtgGCTPDxgkDfPRtFlVxH2PH51nVC4Q0qF3IeMoNlhScTdmVTapPv6vutUlqFuYFlRtdzrzzm/ENHN7yJ0hf9syy/yGBT3G0TLGqe74g3HT2EVI+o8GaRs9cuNtbShPFHgtS72OFkmU74P5ov6Y8E1pKWItOhxQDnDhsQ2SFBHX++7x6Ue8runmjCvsv6rmWR9f9EYXohw5Tb8hyHvyqTuv4pU4w8VpUoIDfjppOKrK2kmuX8ySCon562NMpEbYJjJsjSYhx2k3cXALQ0iff1HXaSOQvwcF7bRI9T+NUBnDE3mf6Owf+vpGLbFwTfK+LFQJMCmfa46x2JBOKhp1woPGez3I9FRv44h0m9IA92EnqXXMRxCAyiuAlGRu9xbtRq4mhm5cqsh5cetiAouDIbuwzfiQdqAo+T9T4SAV7FXz2Y7aq1MZv0VHThhPUCu6SJy7UpYqwFqhzE3p/pSnRXsMiL1FhzDOjdshIxE+IR7Ecemlxwdv6sVQ+G1KZbPHfWa3C+ly9bhV8AZlSpNivZB3Ue/G9lHVrbURykc7iDOLC7uzkQV0cNYvVPt6tEPybH2I0/vNPVqxjnpkx/v41uiv89GuJisRrkAD/Atornpy/lme3BGrCH1DMK20NvS19LJ8cPdQcD6TDPMzPxRpJDpcP1ZraiyFBkiw2KPkpb4GDNSLpO/7miH8JaupNtNJL5I238bHX5TuLqn7hnswIVCoJ7vV2rUd8JQoODC+zy6zWTL8O3AgkDBO1CrYhDJTA1IOGj6vLvXk/0yZxyVQ9nOFV1Pzi5x8+zILvTxe1CM X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5c6e678b-b311-4a76-c3db-08d8c94af96a X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2021 20:25:12.3561 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IH/VBeloAIpy0vPpf0z39JewxA/HX0KC+hVcu9NSM3mQ26zDaS+OuRrSDF03J17QRAumLrMp6hrqAXdZtlIbZ08aPCd001SCgpQU+XYAnek= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4590 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add an helper that iterates over head pages in a list of pages. It essentially counts the tails until the next page to process has a different head that the current. This is going to be used by unpin_user_pages() family of functions, to batch the head page refcount updates once for all passed consecutive tail pages. Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins Reviewed-by: John Hubbard --- mm/gup.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d68bcb482b11..d1549c61c2f6 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,35 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void compound_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *page; + unsigned int nr; + + if (i >= npages) + return; + + list += i; + npages -= i; + page = compound_head(*list); + + for (nr = 1; nr < npages; nr++) { + if (compound_head(list[nr]) != page) + break; + } + + *head = page; + *ntails = nr; +} + +#define for_each_compound_head(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + compound_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + compound_next(__i, __npages, __list, &(__head), &(__ntails))) + /** * unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages * @pages: array of pages to be maybe marked dirty, and definitely released. From patchwork Thu Feb 4 20:24:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12068621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CB89C433E6 for ; Thu, 4 Feb 2021 20:25:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9CD9964F45 for ; Thu, 4 Feb 2021 20:25:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9CD9964F45 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 272936B006E; Thu, 4 Feb 2021 15:25:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FBCB6B0071; Thu, 4 Feb 2021 15:25:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09F3D6B0072; Thu, 4 Feb 2021 15:25:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0219.hostedemail.com [216.40.44.219]) by kanga.kvack.org (Postfix) with ESMTP id E295E6B006E for ; Thu, 4 Feb 2021 15:25:21 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A7F77180AD81D for ; Thu, 4 Feb 2021 20:25:21 +0000 (UTC) X-FDA: 77781715242.09.doll95_3214ca4275df Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 8C8CB180AD82F for ; Thu, 4 Feb 2021 20:25:21 +0000 (UTC) X-HE-Tag: doll95_3214ca4275df X-Filterd-Recvd-Size: 12937 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Thu, 4 Feb 2021 20:25:20 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KJi0r094824; Thu, 4 Feb 2021 20:25:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=Mg0EG9jyZ1sq9/BmNNBYOGVTgJOCH8UybdaYtPghJi4=; b=uETIJsmVeP1ZhmOSx6f66dBJQn0GASXvhLqYnh44TkX06zDiw0EBRvU9wxVjRvvth49A flbLXk0Oa5uGCycx9zwl8YujVsuE0Wq4ec1PW/Zh2p2yrLkeISv3xlXgrmgl0SuXAomz Z/SeMmChLqssClrTCIEjiRCaOzUs4e6MS/mT1ZDG6HluFE2p7V6MfzNX6E6Fqs90zC5h MdttitkiacopP61mHCCRE34Y/79YCfi2mKPssEvQTWQaYRHjQdwMZmplPTH1LhmenGJT dNnZbGJHSC3lfNUA3WzFyiY8BSfN9hoelgndb/h4SHRzEv3ITwlaebDHmU34Q/wnCytn 6A== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 36cydm71xr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:17 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KF6Kn155762; Thu, 4 Feb 2021 20:25:17 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2176.outbound.protection.outlook.com [104.47.57.176]) by aserp3020.oracle.com with ESMTP id 36dhc369ty-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PwwWHI+DtPvF0hBkxhkwq1IOs8F52IxRnUtTw/0CjMh4GADjI0mgYHC7wdv4c9VKkPjAdntGT8yG01KXEHw/XwrJvd7oR7+7ZxQ3qXHDeRY6n4MlyuSWGrC0Va8ASMZKVb37GQdfSwcqh3eOFnYli/ogXqOWn4edJ0AXBmflOIUPixhkJwHrDuO4Oc41dejCgEMZGWhqJ7ivNsXptwUIH1aRkVENi3WpnXa+Ghum+3XzNF+xelh7O1qDvygqsAXf4hOR7o44EmV0LDZkmW2MPdBIYZCzGedHUWN1i5/4MYCVoOnY/NWwzFVY85qAG9Um2z4x0D9uCz6PzwBq3j73+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mg0EG9jyZ1sq9/BmNNBYOGVTgJOCH8UybdaYtPghJi4=; b=bVwTp7qqwXAMqR6fu0g44uWW6AQMbkf64TH9zcXm4Uw2l8otirlEHAtnX6AOvNnSXEFmqKDr4Uf3sMaxu+rYU4BMEa2sUNYvdDtIbHL2LTUrD0K21IC10xEgnDAk2m7hBfYJauZW795fOz+vqa1Ay4GM4nxGUpWsqESbooqkP1z+xAXhRU3J1fFKy0IRTMvhnrHBgJzHyKBvmjJU/P9Ar+W0kOeqYD5kFtrG0gKNlWGEgKPp7H4jsvmU8zWcN7guipQPonkgP6R0T2Hfd/5hZejLKRQqyv4ts1HyEVHMIfPD/+wRJdb1R6aVN80VNEDYkvoihuu8y2hrw5hk58jGlA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mg0EG9jyZ1sq9/BmNNBYOGVTgJOCH8UybdaYtPghJi4=; b=lhzX4tyz6elSWDbxAnnrqIsMVj1KKi4tcQuNq2RfZZYF3+ZQoOnWYCQX5oIrhOxHTKYKW7wdyGj2aN9ZviFUaJbeA/EgSTRojEn6sf1g2b5P/CHvTHIK7jtDxnodJq5qgi23HbPveXi/tbQY85NQchgrKOnv+G7dV1p6NcDluY0= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4590.namprd10.prod.outlook.com (2603:10b6:a03:2d1::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.20; Thu, 4 Feb 2021 20:25:15 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Thu, 4 Feb 2021 20:25:15 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v2 2/4] mm/gup: decrement head page once for group of subpages Date: Thu, 4 Feb 2021 20:24:58 +0000 Message-Id: <20210204202500.26474-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210204202500.26474-1-joao.m.martins@oracle.com> References: <20210204202500.26474-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.21 via Frontend Transport; Thu, 4 Feb 2021 20:25:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 591bc5d8-446a-4b84-381b-08d8c94afb1f X-MS-TrafficTypeDiagnostic: SJ0PR10MB4590: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VzZFmrYx8iaUxMHD8ZPjVkv4IO68S9G7x1v26dsKFCQBJYWRKUFmyRxnpbtq3wsfOwDTNZEXVNTYuvz3ozBREleDZ1XRfQ40aICF8+qAPfhnWFxIxGJdbpgQV4eOAz4Ks09Qzghda+HR3nL9bNZlpel+16wa5l/ugPBmCw2w124B7WCQZ1dg1cOcS9pT5KXDopUDUQ9tIk1P5xj+A/lKA2uzJMewFIPgdefGNF5zE7rvSXhZRRDLNJ+/dbyu9qqqQGzmv7+Arg1jcjOTdkp4Q0Jb5tzJGQUqUJZ6Ubt9FbiE8Bd1MlWRNEj8vuywsrhz5ydLOilb3MHjuZFi2UBtSjlHJ7TdYqXVBixizLE1xbKXRgVC/g78A/Z0E7u8oKVOTO06tAQM4vGdYAvn8xuo/+vY0Xsn/RV3WfaTP1nOF9uEitk8TSWcMvRTEY/PBtxmDOEqvo6YbRDsUUCqMzfOv7rrRcJlyElrJcp8eO7ex/DzG0+SYzEImaMkI0WSq/9L/d859g7nHvFRzJ02VoDOfg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(376002)(136003)(52116002)(7696005)(26005)(2906002)(16526019)(66476007)(186003)(66556008)(478600001)(103116003)(316002)(6486002)(66946007)(956004)(54906003)(86362001)(5660300002)(1076003)(8936002)(83380400001)(107886003)(36756003)(4326008)(6666004)(2616005)(6916009)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: svZl6ozDJnxKppplU2Xj+CdHRf1s9sVexj3nCSzgz1dV5cl37RMnjruCxcUc1UKaTSlxNDdy9+TF6TlMIPAG8RIOR3JKVoJb5kd3SnKtimrYMc30QOZ46OEypP03fVy8rbwc/jfUuiHZ6d6AMX7C83w06anGFHpMEaLe0ixd7ww9nZMhJdM52WNwH6pva2LYm7VWs1wYsasKaYJ2mDk5FgoCixqaxIn6gXc6RwW1sa6xGa4szlvk7hsg6alpOl3w5x4RKtJrbk4aQ4Zn7ZXO9THEQNptQQL7sVq5JRLjemPgWlantCU8fAa4FgyQsrMsH067+qhwj3eBFjrZfHWhZvTVYBijxsJ0hJzcMXcIaLxMeqw2MEK13fjShF4DPtse0FxApwCRS3dZEzw8JkYJQrlXiXuFw/8weMEJu5t4cJAFqAT+nwm9DTp7DRgGKH57FWnHkzib+ly/RO4smN4IM1cgSdbpp7lXjX0OoRJ412w0Z7Qg+MP2z/s1qKh2XUeKcpnHvNgfj5w9+0cCKCA0FeD8HSqq18y8ti5X1zq+YIEOc/+4VK1/i2+q6zBROZotEDgGlb/5i0e+wh2Ept2LUMdwKQppi6gp9iFh5vn9TnGD+tkgi3ysltRnkSCwhxbU/itTYuXb4/z9jw1UqjQguBYJzddb5c0yn+lTC88Vy855P0mj32cghOnNxax2GGW6iJUdm7HHw3rX0YNuI+Vrm9FerSEfmMy41NMeSfPf+yMQ4HBMSPlISuS5DT93a9vps3ji10cq8T+ljFEbHJbN5WyNoxRHFgjAOsa35cLp0Zbz4jMfj+OFmC7W+gkcRyQKpRrz1YzM6mO5pT292WBzGt0TteX0PsMpn+BXASjFmjfSROyDku+yR161JfRVoXsoeu/um0ltbekvAZbq6CjLf0BRlz144ZcXoNnyxc8fCyLdmM4I3qGNzmZxLn7DPmIscJqy7hJj4xpWeOkz52tzsz/8iyOSiMa5WyfoQq82tQsoDoJXpq/BnAh42s9v2i7PnPjr4kup8kWDHWLZ+yUAfpRyw8ZjpG3uZNnfG8MEiS/oaoJ0Ky+7wDhcsUhSX1Vs X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 591bc5d8-446a-4b84-381b-08d8c94afb1f X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2021 20:25:15.2354 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jxlLFD8qzSXD3HFipxxcKvc8aoQYXWpX438mWK7bsalqDckNrsF2U/2cINMCA9w4FJj4T4S0wbrhG0xEwj8EgCbneHi0LxxMSdpY/z+iobA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4590 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 spamscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than decrementing the head page refcount one by one, we walk the page array and checking which belong to the same compound_head. Later on we decrement the calculated amount of references in a single write to the head page. To that end switch to for_each_compound_head() does most of the work. set_page_dirty() needs no adjustment as it's a nop for non-dirty head pages and it doesn't operate on tail pages. This considerably improves unpinning of pages with THP and hugetlbfs: - THP gup_test -t -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK (put values): ~87.6k us -> ~23.2k us - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK: (put values): ~87.6k us -> ~27.5k us Signed-off-by: Joao Martins Reviewed-by: John Hubbard --- mm/gup.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index d1549c61c2f6..5a3dd235017a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -270,20 +270,15 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty) { unsigned long index; - - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ + struct page *head; + unsigned int ntails; if (!make_dirty) { unpin_user_pages(pages, npages); return; } - for (index = 0; index < npages; index++) { - struct page *page = compound_head(pages[index]); + for_each_compound_head(index, pages, npages, head, ntails) { /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -304,9 +299,9 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, * written back, so it gets written back again in the * next writeback cycle. This is harmless. */ - if (!PageDirty(page)) - set_page_dirty_lock(page); - unpin_user_page(page); + if (!PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); @@ -323,6 +318,8 @@ EXPORT_SYMBOL(unpin_user_pages_dirty_lock); void unpin_user_pages(struct page **pages, unsigned long npages) { unsigned long index; + struct page *head; + unsigned int ntails; /* * If this WARN_ON() fires, then the system *might* be leaking pages (by @@ -331,13 +328,9 @@ void unpin_user_pages(struct page **pages, unsigned long npages) */ if (WARN_ON(IS_ERR_VALUE(npages))) return; - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ - for (index = 0; index < npages; index++) - unpin_user_page(pages[index]); + + for_each_compound_head(index, pages, npages, head, ntails) + put_compound_head(head, ntails, FOLL_PIN); } EXPORT_SYMBOL(unpin_user_pages); From patchwork Thu Feb 4 20:24:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12068623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54577C433DB for ; Thu, 4 Feb 2021 20:25:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA05664F48 for ; Thu, 4 Feb 2021 20:25:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA05664F48 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3F6766B0071; Thu, 4 Feb 2021 15:25:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 380AE6B0072; Thu, 4 Feb 2021 15:25:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FAD86B0073; Thu, 4 Feb 2021 15:25:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0005.hostedemail.com [216.40.44.5]) by kanga.kvack.org (Postfix) with ESMTP id 0575E6B0071 for ; Thu, 4 Feb 2021 15:25:26 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BD976181AEF1A for ; Thu, 4 Feb 2021 20:25:25 +0000 (UTC) X-FDA: 77781715410.02.flesh40_570319f275df Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 973CC10097AA4 for ; Thu, 4 Feb 2021 20:25:25 +0000 (UTC) X-HE-Tag: flesh40_570319f275df X-Filterd-Recvd-Size: 14038 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Thu, 4 Feb 2021 20:25:24 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KJ5j1094495; Thu, 4 Feb 2021 20:25:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=L73QegqCirJmIv8GBpj0D/m45kMeBZWUda7V5m2UH3s=; b=vyrygB76xZbSzlbBxFpHaT8F8PsaDfE9CFagkqUU3LhuoVnd+Hf8CbMhHMw31JjwfIKc QKoWpybPg7s21uEtnXtIHEpgGMzZiuRQwY+y9IyqnR9dvr10Sm8AENTebraSizzuWIXl MsQSvNDwJcqCk7ybcYpfyghU5TrzJKlYIza57TK9bXw4A2K5SrJK2WIPjFOes+MQUbau 8wwHtFjVlhqZugCdbsYcARamkSNBVXcCpDK6qIKzdNqgUimkPD0y3hueJEIpGmLtmAAF 054QtLUAoFVINVl/TY/853McBYdbstkvsbceDGPdsCi7Z6Pv1ZJ5d20rUQw3yhbYhQq7 bA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 36cydm71y7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:21 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KGL7N070865; Thu, 4 Feb 2021 20:25:20 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2170.outbound.protection.outlook.com [104.47.57.170]) by userp3020.oracle.com with ESMTP id 36dh7vpbhv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jmi0vpnJ52HKtYL5sZJZU5VEEasO1DjejyVfl8GCyRIkTE3XQiwVr44X+M1xEeF6i88IiKnycvfmJvowhpZkl6/VKzPI7wCEz8qzXVOC+83lGeZmLEFOMD7gMADFjXj9zRZVmQI3QWAWjlaXlK52AdoxOqULrBhzAC7BzXgh9buqjFzfHHgKmixEkQmOeSbdwdr6/PrscWWenn5M2Irqv1B3abk5tiaVN/dCq/DByhWmAUWLE2c4LEdreqJ2efJQCvQuIploBuBbBRkXDMi3ub8QiZhWy8jFLMFablmfZLFsPkMZZzELGFxCwdo0yBzEYedeoGbgPvtjc/RGwxWPrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L73QegqCirJmIv8GBpj0D/m45kMeBZWUda7V5m2UH3s=; b=G6wI4gskMdaj21Jyxrwl8hyovb5Gp6rFkJIb42m6MOO0cvnNJ/yp5xEpHBGjhSJe2B27GylCPcCzU3vqSwmqQTeQ4DnenYgem+/oI/ayX9mPtTyPZnsv+C0EK4TFn0CT5naSjw+cdDF4+lrTxGnYRNsLWRvQFRKDdHl9vpdIrz9Rtj1deGhvN5dR+WPjvl7WJ7A1FqhqefZtYQ8Lt6yUernT39uLD/hk7GXsVuAvO39S7l8TXF70cqY3i09FuWmMKU9uj7AiijZAR592KwimBPUC1bvaaN3KidMGieMVuBJeUWqm8puY4ygn5bBPsFI4WmKqh2B/zjuDLcPa/o/t1g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L73QegqCirJmIv8GBpj0D/m45kMeBZWUda7V5m2UH3s=; b=rlcvQqP5i5mwY/52PLdjHAsMRZpR9yf0mzxZYJ8PcdBPlxz/m6Zccqnw7+ZhW7xuavwdgSlOk6DdxyT6Ue/bqC2MjwWQ0vMqrFwvviJrll/MyN9sz7m6o+a3ROznzeoU7IoZms2omuFInD3I7raZI2kRpjHYlbJ3tMw+GP8fqPI= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4590.namprd10.prod.outlook.com (2603:10b6:a03:2d1::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.20; Thu, 4 Feb 2021 20:25:18 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Thu, 4 Feb 2021 20:25:18 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v2 3/4] mm/gup: add a range variant of unpin_user_pages_dirty_lock() Date: Thu, 4 Feb 2021 20:24:59 +0000 Message-Id: <20210204202500.26474-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210204202500.26474-1-joao.m.martins@oracle.com> References: <20210204202500.26474-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.21 via Frontend Transport; Thu, 4 Feb 2021 20:25:15 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 885d7fdd-0ca6-4133-0e4e-08d8c94afcb1 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4590: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8ShhH/MNCHevEnDXSWg3LOdLowZ5VY0yd8supePvsYZfpldT6eYKGyi/8N0vl0VwfWmsjXngG0JAolXSeZGYfK1QdnP1l8o9Wbo5JT996bfSD0kUbcdAHSx+xvVIUH03jNyn6NUeKVuiJd66EpssH37GrWQ5haLjS9/gqASv92IAxN1EGs7rmcl5pYhXn9UEKdSqEkmKbWuon5NEIo70ZzdO1eNnp2SCJEWoSEHt4r2kdyBAg51lN/QATBEERsN4EoRDUBbMgh6hE6tNN7Q3cFjJtQouhUJpSWc+v0A0zL+lMz22wUF3Vinxb7OHO02poxjRGgXOO9bIa8e+F6ZxCVuJ/m4/pR6DC2NCx6G0qyapZZ2Rs119zunvdu4LTOVS7CLF/ZYyjLvcWw2TE11dU9ongdJzQmNSETHLmAI6JNionz27V0F1UzUkOBor8M92PgDK65EbzLfaHViN9PK2g47qObvWnsbGZ4vhCiWmHevNSdMXft6hJcULe+ZgfS9ByE9hyOz1Uee4oCW2NvXF420gtjwcq2ZBXRwp23Q0gldyB02XCmSG3pBq7U2QOcby X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(376002)(136003)(52116002)(7696005)(26005)(2906002)(16526019)(66476007)(186003)(66556008)(478600001)(103116003)(316002)(6486002)(66946007)(956004)(54906003)(86362001)(5660300002)(1076003)(8936002)(83380400001)(107886003)(36756003)(4326008)(6666004)(2616005)(6916009)(8676002)(14143004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 11kB88qnn7+T/jNZ0rm45s+J3e652MReUpzw1aCa4ZAfCVCua1WRgJdndNw0GVvHp7A4BFn7BHzgxOpHBZThGQcrMi9kXnkwS6321F8P32znpqzw64YkiE6b4UshQ0Og3iVslX2zE/tMEy3KgL4S+2PVcTfhn+Vd1PoEXezBPN2Xqgue1Raq5P1Ast+YVrKOUMf498jq8GRprzDqa+4jFeaZLb43A2vZbqACO36F9WJ6xLpuPbkTUY7bXNVCYC5IXaOIMT1i+jJLvmW3/74xxLU4HKzFVs7YUhcI5iMVXIcxvrj1zGtbfmsHJssigdsmGT5Blns/p+uh1EHSDqLHPbyvLOIDOsui31hrW7seL0J9vcljZNmiBPudnQSDeclepBRWZ6+9E5dHR6F6kLuW6GTWwPZPxyMRZ3nHZnKhS7ONbY8C8Dx47jYqmBnEzQDlZgWqDwEkn9rBdxs5ighBSxi/oA2qH05hhinZKNuupV1Y1WAymWH+mBal3F33/Q74GHm1SOH92dezKA62AZaNuHwaf4AbDeqqgIR5o0iTvMeblEsl91JeKWAWeyulpbH3CPdVYcaoAUwhJypxwuOW/BjBElcFQz/8GCJrx4YnyP8tAB4qzlqvlGQwcn6U3o7nFJ1O/+uOSgKzjInO01q5emzGzHNCMkv6saTn4yLELghjsjJpxBPUzWqt2AT8d1nqRKIYolw8KmhGlUfZQ6kBPQdZs9haSNAqJKA38EwSeNMVMMowi1wH7VqauyVLylwxWniVMURQ+Pe14nAxXvmF7BXoLZietWxUkGBiaEYGJaTW/RydfuMf1ad/2o761UoMDM73TvvZbBydD2YwDcJ9ZdURpcFhqyMBCaWPB2ACXSym24c3fSUm+s/FcclbI8XUkAAC38HAkoGiURMLyWprJ9kaNzQoNKXvSI+uEj8n/l6KAOdNbketsuxUisQCL+mrz8HID7GASAsWM+6oldAH2Bdz37hoJKJ/jNrHspMN0MsFBJVfD8qFv64yBGsGC9yp0g/cn+tRv8YbLZ8f/2N4H/YTL486Kj+rPdwFgLysAjKSzImwRhTPYm/1JVpQqoLE X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 885d7fdd-0ca6-4133-0e4e-08d8c94afcb1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2021 20:25:17.9968 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Nv+fOJ0ROcUO7INnPD5Q9PVLb0KphqtlrschaahOHUIzH/ZpM7WcKwORaVEu9N2H2jOCloIGPZUMg4tTQczDpgnwilVEvndwE0pMCRgVwd8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4590 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 spamscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a unpin_user_page_range_dirty_lock() API which takes a starting page and how many consecutive pages we want to unpin and optionally dirty. Given that we won't be iterating on a list of changes, change compound_next() to receive a bool, whether to calculate from the starting page, or walk the page array. Finally add a separate iterator, for_each_compound_range() that just operate in page ranges as opposed to page array. For users (like RDMA mr_dereg) where each sg represents a contiguous set of pages, we're able to more efficiently unpin pages without having to supply an array of pages much of what happens today with unpin_user_pages(). Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins --- include/linux/mm.h | 2 ++ mm/gup.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a608feb0d42e..b76063f7f18a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) void unpin_user_page(struct page *page); void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty); void unpin_user_pages(struct page **pages, unsigned long npages); /** diff --git a/mm/gup.c b/mm/gup.c index 5a3dd235017a..3426736a01b2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,34 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void range_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *next, *page; + unsigned int nr = 1; + + if (i >= npages) + return; + + npages -= i; + next = *list + i; + + page = compound_head(next); + if (PageCompound(page) && compound_order(page) > 1) + nr = min_t(unsigned int, + page + compound_nr(page) - next, npages); + + *head = page; + *ntails = nr; +} + +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + range_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + range_next(__i, __npages, __list, &(__head), &(__ntails))) + static inline void compound_next(unsigned long i, unsigned long npages, struct page **list, struct page **head, unsigned int *ntails) @@ -306,6 +334,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); +/** + * unpin_user_page_range_dirty_lock() - release and optionally dirty + * gup-pinned page range + * + * @page: the starting page of a range maybe marked dirty, and definitely released. + * @npages: number of consecutive pages to release. + * @make_dirty: whether to mark the pages dirty + * + * "gup-pinned page range" refers to a range of pages that has had one of the + * get_user_pages() variants called on that page. + * + * For the page ranges defined by [page .. page+npages], make that range (or + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the + * page range was previously listed as clean. + * + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is + * required, then the caller should a) verify that this is really correct, + * because _lock() is usually required, and b) hand code it: + * set_page_dirty_lock(), unpin_user_page(). + * + */ +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty) +{ + unsigned long index; + struct page *head; + unsigned int ntails; + + for_each_compound_range(index, &page, npages, head, ntails) { + if (make_dirty && !PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); + } +} +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); + /** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released. From patchwork Thu Feb 4 20:25:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12068625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 910D3C433E6 for ; Thu, 4 Feb 2021 20:25:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0494564F45 for ; Thu, 4 Feb 2021 20:25:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0494564F45 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1131A6B0072; Thu, 4 Feb 2021 15:25:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 09DA46B0073; Thu, 4 Feb 2021 15:25:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E09016B0074; Thu, 4 Feb 2021 15:25:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id BE01A6B0072 for ; Thu, 4 Feb 2021 15:25:26 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8CB631EF3 for ; Thu, 4 Feb 2021 20:25:26 +0000 (UTC) X-FDA: 77781715452.29.jelly58_2c05f40275df Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 67B9118086CD2 for ; Thu, 4 Feb 2021 20:25:26 +0000 (UTC) X-HE-Tag: jelly58_2c05f40275df X-Filterd-Recvd-Size: 11508 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Thu, 4 Feb 2021 20:25:25 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KJocn136311; Thu, 4 Feb 2021 20:25:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=0LTNyLhfcQU8WurGe4uk1JQtTl/ZkrEkTZA6jP+zRYs=; b=OewiCsqUbwT1WSPdt+KcuKs2qKRgwk68fdfxYnJ4fildLzTHOZhVRWJ2Sl8Z5JCZ4JMi dx4KVarWpH86Cc+SQ+i9L/kx09X0MXBxQ7JJs0BlxNCwghP+7sZ1ZVy7b+htVHtnWLut oKeOqbqNzyZrp4cPkUYev7EhCj8zrT3g8uUykgobo8tw4CMJOVd04RoZU5648wF1bUgu ff+x9rLZte0G2LmUuC1K3YZg0fXBG4pVKBSB7Ay0VUH0kRr5DvoZVmhXpfJweU+9HUzp EcZ1kv7z1edvXl+qVoBXPVFizA3JtUr+S6xEQORDUG+IkQGLuvBLIMBOqis76DMxJiIE QQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 36cvyb782m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:23 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 114KGLVq070843; Thu, 4 Feb 2021 20:25:22 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2170.outbound.protection.outlook.com [104.47.57.170]) by userp3020.oracle.com with ESMTP id 36dh7vpbjy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 20:25:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Oa2ujLm95mY8uMhZNGZjcZwIiVus2+z1uue7k6Zr0Ramk8LWThvSKi92HWSpZjIg2ySSWuAiMadlEsJLtbPlKRhIfNQrnYLAbvY2hDxyDC5nKHtgHTMcWT9FPfZAOVjuu2qe1M+NJSvFeMhAD0+Zd80cVyuBt19DUg5a9me4GZ/sDnWQhxbus+b3gobZQJPTs2wueDdWW1dDm1dvUerRHUM4+N4OPXZ3YjTSYe+PWmh3oVIqhzyZAO9GlAF0LblANnBuuH4WbyLofFUKNCBLJKKUx9s6iBlEqc15iPNgvkMzWzQBQ0+lBzGVlODvIwKMbpyE379sr4sPIBOcHg84WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0LTNyLhfcQU8WurGe4uk1JQtTl/ZkrEkTZA6jP+zRYs=; b=TZoHfc4C/wNgSvYJdCSmplv3ZvefCyBpT/X9Tt8gn8hLxd+oO3qdXBFuswZqdxoZKlQLJR1cX/i4fURernQ7swDouemcSOJvm1fiDt06cJ/cTevl1PRfRdzsiHmxw7JNmTwMvu68B1iwZKIGDpgrG4W/ZXunj7nXcMiVJv236huXeLAVg/AqU0z/ewXL7LwQ8/bzMxTOcvJza6iC07VBG/6oDBG6pmPqhNtidKBPx5/4e2PpHEqWndPXXKXccMYGV4Dt7QJtND902GVQhyOZ8LjwaS1k5vuUPH2xpXPD1T3161L5nwfdcaggBmEf2Y1udlpmBJ/1OLS9M3qLmXVK/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0LTNyLhfcQU8WurGe4uk1JQtTl/ZkrEkTZA6jP+zRYs=; b=W9fGzDBcgn1YyqjOlGPHP2CjzY+4leOK7g7JAsewlDgSeYrOyVQueJU2dyawuoFlBjo1jZBzdTpK+5ZC2FwkPOji7vufzjyh8Ig9gaBzGQhj1blo/8rV23ujMZtn7HzSAZZCTnoV/rCLIineK8yFPDtmjIpcVk+kCXlYx3u0vX4= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4590.namprd10.prod.outlook.com (2603:10b6:a03:2d1::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.20; Thu, 4 Feb 2021 20:25:20 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Thu, 4 Feb 2021 20:25:20 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v2 4/4] RDMA/umem: batch page unpin in __ib_umem_release() Date: Thu, 4 Feb 2021 20:25:00 +0000 Message-Id: <20210204202500.26474-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210204202500.26474-1-joao.m.martins@oracle.com> References: <20210204202500.26474-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P123CA0104.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:139::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.21 via Frontend Transport; Thu, 4 Feb 2021 20:25:18 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 462c7a86-d458-49f4-3996-08d8c94afe5a X-MS-TrafficTypeDiagnostic: SJ0PR10MB4590: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:370; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: H0yfwBVwQ9JChHEe5ZjArW5oLXhHwRUXnPG5O9ZLwhFCNc9S1T8Mn1cD1qI3Kdf75pgkRfeB7xydV5jYQurPGi27lBv5zSj/e4prKn+YRkig9itbrIVXtE5IGpgf7cmJl0Y3mzNnU1F5uvykEqasZCfKtZFy60yMwHJtQSru+BEf3n9edkQ+QT7+a11mPfmthNU9gAkQzX0OZtIZzQJimhSp6VQowMA2zE3nP7ea5faDJgRWwnuA6ldW+s/2i347tnAFp4UqhwIEhyd7ZUtUSqfyY+QWlUtHoWjnRGqhHPWwKeoHuaL/c32XlZi7ghDSqk0nj+4RT9fDQVETRqZHSrtHHkl7UFf5vkAZqCcY3b4QO3fswkLC/TDau8vPg3pqjrsNCCKrpA4J2DJjMwhnBXYq1Z3cuwglJ1fz6IoZl3TGBJrojzAMLbEY6y0mQE7or/ahm2WV+L/L1w+ApoZgzm4G/1vwEkoI8PhmMGpLhvubN6KpOPy5Bvgw8S5Igud8SgBH1JOQav1c3XxPektcBw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(376002)(136003)(52116002)(7696005)(26005)(2906002)(16526019)(66476007)(186003)(66556008)(478600001)(103116003)(316002)(6486002)(66946007)(956004)(54906003)(86362001)(5660300002)(1076003)(8936002)(83380400001)(107886003)(36756003)(4326008)(6666004)(2616005)(6916009)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: RXnn7J/Qya1eqKXG5A1j6wLBejq/ixKjme2DnAYE5vz09XqU/d4aOSRTDIstdAZ6v+NcvOJM55TB6iCzjbP47yaVYgQPGKoemPZ+I7ZuE4x8dRWMk/I2WMfjL1zk2YhMtBxhVXvD9myK0Oq2n55TszcvjtOIPBA9MVE0XGJ7GVgymvnx6yCilC/1w9EoeJfkJcnJQerW++l5vaGuHG+LDWfx8eVIf0fUxM82zna/2SCgFl0tPT5dFcEKp+Au+cS7pYLw8RmvWz+81Odz13pV0rblClhcWRYMA1OAMHFpdD9NznzR69fMOWnKsilmhBHGuWJ5rwTy2dWUTtplmr5tMPPfviJJPFlOVvIgM/eaSFiNfQIoaeqlhx9jVGIjgvieYnSsKGnvRKOoCiGxpXC97LHeMMUES7UnZRjZH1yTGhClaftdYJqQMoeh48I/nbbkYCg7i9MnZcRSPiMEh7UIA6PledBxhEQwiGozv+/0t3ccMZ21rUtmxUvWXT74rKHKkRvrIpUd6FhfwF7oywKhGd98RHJ30QYBeJ/Df8iLDzwE5ru44qAk4eii5jvqbqtPMDJmek4E8MmL7RyO/H+RBKY3aHaSt4ihkeCpPGOW2KWU2uP1PYeUpE7CNDhXgIirEUARKxrNg7ALSDt9by3d8FfRbQKexatyif3Cq3p3+0+40FhYr60XlFiBUrsiJoaYD/Df5QR5bM9jbHmjTgCCg7YhT2yxPI2zQ2W2d532plaHH/3/phP1r5/9lFe+uzAYMCga+iykjWqGFS5uVDmBYowsrxHNT0a/OMSPJnIWAcV955FLjrphzUFObtrLI4kafMxabT1a9DM85k58Zb+bVcEVPFdDgP/azqBGW3LdO4tpGaFj3h+GjBExVGnhE1rfkFeYy2qEPXeD79OskBr1bXqoL5XVgAQK9uZqx6q7FHXfmkmwTfz7cbK106JY6RERT+bHxMw0Cm+kZlOSGBnG9SpfglNRDwzjtAmkQ6yAmUthStr1hU22Ikt4faAKXBbJoaI1uTXJItnw6cjHgPqTy5wuVBZ7IUTK0hDcHOtuP7gA7ftnFRXrewniSEGqv6eX X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 462c7a86-d458-49f4-3996-08d8c94afe5a X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2021 20:25:20.6253 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DYROqw5k9ogOeoqWmwt7llfgIo7KYx+x3fn0TEkI0j/ZcR8+uAHbJtXHWIX4JoQ1K8NEsS0kFfn99jj/ATtuRIaMOxMXPNU8SP6xbl5yCd8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4590 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9885 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the newly added unpin_user_page_range_dirty_lock() for more quickly unpinning a consecutive range of pages represented as compound pages. This will also calculate number of pages to unpin (for the tail pages which matching head page) and thus batch the refcount update. Running a test program which calls mr reg/unreg on a 1G in size and measures cost of both operations together (in a guest using rxe) with THP and hugetlbfs: Before: 590 rounds in 5.003 sec: 8480.335 usec / round 6898 rounds in 60.001 sec: 8698.367 usec / round After: 2688 rounds in 5.002 sec: 1860.786 usec / round 32517 rounds in 60.001 sec: 1845.225 usec / round Signed-off-by: Joao Martins --- drivers/infiniband/core/umem.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 2dde99a9ba07..9b607013e2a2 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -47,17 +47,17 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) { - struct sg_page_iter sg_iter; - struct page *page; + bool make_dirty = umem->writable && dirty; + struct scatterlist *sg; + unsigned int i; if (umem->nmap > 0) ib_dma_unmap_sg(dev, umem->sg_head.sgl, umem->sg_nents, DMA_BIDIRECTIONAL); - for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->sg_nents, 0) { - page = sg_page_iter_page(&sg_iter); - unpin_user_pages_dirty_lock(&page, 1, umem->writable && dirty); - } + for_each_sg(umem->sg_head.sgl, sg, umem->sg_nents, i) + unpin_user_page_range_dirty_lock(sg_page(sg), + DIV_ROUND_UP(sg->length, PAGE_SIZE), make_dirty); sg_free_table(&umem->sg_head); }