From patchwork Fri Feb 5 20:41:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12070885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40FFCC433E6 for ; Fri, 5 Feb 2021 20:42:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B85F764FC0 for ; Fri, 5 Feb 2021 20:42:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B85F764FC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 72E8F6B0071; Fri, 5 Feb 2021 15:42:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 672C56B0074; Fri, 5 Feb 2021 15:42:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 228F26B0072; Fri, 5 Feb 2021 15:42:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0131.hostedemail.com [216.40.44.131]) by kanga.kvack.org (Postfix) with ESMTP id DAFBE6B0070 for ; Fri, 5 Feb 2021 15:42:06 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9CFC13621 for ; Fri, 5 Feb 2021 20:42:06 +0000 (UTC) X-FDA: 77785386252.23.fire65_5d0ffbc275e7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 65F6637606 for ; Fri, 5 Feb 2021 20:42:06 +0000 (UTC) X-HE-Tag: fire65_5d0ffbc275e7 X-Filterd-Recvd-Size: 11679 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Fri, 5 Feb 2021 20:42:05 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115Kcu95017951; Fri, 5 Feb 2021 20:41:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=i2BRVXmP5hvWQVGCMk07jhgUIaktkfcSnPtSBprIfl8=; b=oeSBeoC4LtUb5PV6THpsowGltd61VA7/FtpPurRpQILQh0irCIsdqbh+jtflxjCR0JWR aD1kcTWqkzcyZB672ziusqc0jRtSbUizw4cij9t4V9gRxQNNqdMO9C7zpjLq0vS0UMs2 LlSKIekPhpoOhiQCH185gTZG+gZlUgK9NTspy+f5U5aWcDPKS19xS5v7U3RJAV0HcG2S o9cayrPTY+AIpBSabb8JJmhIw0nxUmPW4o/aeru5IT206mH3A++6Z/PBBukC/cIMHRg1 3LeFmhvywLxOBSWTPijO2Fg6fNgXmyy07nDoAGu7pFt13Df9neIORk7MEMQeFMKmqYqy PQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 36cxvrdu4h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:41:55 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115KdfaE065160; Fri, 5 Feb 2021 20:41:54 GMT Received: from nam04-sn1-obe.outbound.protection.outlook.com (mail-sn1nam04lp2057.outbound.protection.outlook.com [104.47.44.57]) by aserp3030.oracle.com with ESMTP id 36dh1uej1c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:41:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JYqkrxj1SOIg6EQoA+iOXAF4qBfo00yTTVuIXKuZ8ykv5+otTqXKtfIXyEKjEMZQuZfKDM93u+GotL+zvSPEiRvk7zG7iuOpePuOPGNaC9svLs/7byosVvtKf2GYL49+VqIexvOTu7lfz2G29FaUMvy8k36FuZm1g5H2XOZfTz/+ncMDTMlQNCH0ctDQGzfTngPrQEBYy3nnMtPFTExpEAikn/tdKMurkykyLBc95vWi1/UTZPsAoMMc5a1Ad5DM+dLeyU/az9HAe+CAWDnQa1vLx1YXuGfOYdkzhBeyFdYVYbgCHQu8J2RhSglMNVI+dTkXXZdrzHeWF9EBJqC7GA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i2BRVXmP5hvWQVGCMk07jhgUIaktkfcSnPtSBprIfl8=; b=iYZq86m22jVqPmBQHeZej8WBGlw1h5SmKWd9Nu2k3CDR9hvPK0hKlSFtEktJdz1BNXF7nNCqx7JghCz0ZhjuLu8+5bMOJCLNXNG3Oj5jKe7nfFcHrpjQbvsSoy+/XGut6wbKMdLlMx9yQWcshNPpydenn1eK7Evuq22/nU2pu9yduzB2KqHZH+GStYUw0Cp6ZmG/PCevN8Tf5X3p7FyIqZyWj3HRcje1wJ1R0MDFWV9RliQvUdWiPrr6YQ4SuxgN7BreE16NZSAYvb4lwYiXbK2/CMf8+cDyBHV25c1G3O5mSpucifY6rQjIaA+A+Q6POcFV53r9+rsTHJfkwF/x6Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i2BRVXmP5hvWQVGCMk07jhgUIaktkfcSnPtSBprIfl8=; b=E7FwCRQlAs7mhdP382lSW9RMGaC7cc35KOuYYq809fUrgRECelIc9X+jJynPg0uVyQySNi8tq110dUNdL9ytFySnPr2uOeBudiFlWeItY4ko/JaeAjEQFv7bUc6A7qz6NxyvezcUWdRupjIjrXkD7tIXDNblicyRss0TEmPQdco= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4432.namprd10.prod.outlook.com (2603:10b6:a03:2df::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.21; Fri, 5 Feb 2021 20:41:52 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Fri, 5 Feb 2021 20:41:52 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v3 1/4] mm/gup: add compound page list iterator Date: Fri, 5 Feb 2021 20:41:24 +0000 Message-Id: <20210205204127.29441-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210205204127.29441-1-joao.m.martins@oracle.com> References: <20210205204127.29441-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.19 via Frontend Transport; Fri, 5 Feb 2021 20:41:50 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6ca7b1f1-435c-4a73-268a-08d8ca167834 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4432: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:862; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Se1kf2DXDtlD3PyR8udZlmXuKa4bK6KpEmg7nvjJ8xzdZIdRlVc935MfJJtNS+hU7HxYzHTos+h6KkEyWZ/cm99jqMC79tUdaA7SxwYCCa3m3cb7oX6W4lSW9m7Z2LsfgBU3P26fH5jyatgg5huLHRjYVfuEyNJ18/xRfdBn1falVhH4vcadtSvjlsCQtWCauzMIVtRuVwZTGaSesBYMVX5UkXEZeukObPDHBcyAaO4qsmAHwTNm8yvBeIv25laQAP6V7q4nSaLrVyM+VX1RuZgvW/fEr+9d2MVrP18YVdgkQMbAm1Vui9eOC0QYWAMOZGyK41Cj402zOQkXBJyetRUz5+kJCX+DP8fzLkfmkG1KZuHWVwvFkEKtSYQ16+FzpyQnvvuY2DLbl3M2dJp59+aEZxpBasuxD0wDDGLHsUs0O16hfEFFsLivFAtopQtKtcbYNwVu/xm0gKl2YRyICUgKifSzXBNC5y7H3vuhZegLI2SkDEdPathb0XCoYqlx9fs/mSccmmxS9SDvyXgdPQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(366004)(136003)(376002)(39860400002)(6916009)(1076003)(5660300002)(186003)(26005)(54906003)(52116002)(103116003)(4326008)(8676002)(83380400001)(66946007)(2906002)(36756003)(6486002)(478600001)(6666004)(66476007)(8936002)(107886003)(66556008)(16526019)(2616005)(7696005)(86362001)(316002)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: TVRHqMSp5W5dr9m36VEprk3i/+vTAg7Hwju+OVqKXOR4ztX+rQ3WYBWU5nEBunyIPqNg8R7aHuZ0OODiVFNcqW6JmgTKQtrPiQXyOi8Iw9LoF2QZr72028Z/pDqLLUrez6NL22nLx99OZk5wyfTF/LRgAIY3nJIm+GhblU1Etb3LtqgEKIHpLGnBS/cCi6MFqxStQ0g096/q19LaTdMnpuBsRu0C33z/zC6eL6QUIe3YZEdG6FViZ2jax2NmYqQYzW9eEYHkSb+zUNNzKYT5GachfaU4CjOi0+LXNv1Q5nFYzP3HK27la8gjibomMhCGG9X9xjtjeXZdT3U0J5eiUnJaGTohEmwOusyTBkW+P90vKAdyYFSI/Dnvrgx92sztD/H+ov+v1XfTrzpj24/jP4eyz+ulbO9+HPYAg1WS0Z+bodIdqcl009K8FUbmacjU0qINnSHlfVag8Gvh44ksYUQXCShH0hFqs6mlc0M+uho7nOMoZl+avAgEzMWcD7wOih1ZSMbwyFgjpKlsEpX0FocMwrqFYYpPpTashN6SWyA9aJxsClRb+iNHysEbZiHKF+NG1Ind6833Zm/iFsGz9LeODFgP9yVMdrgTi2eSLYT64VQ19JtJ4ILRblndzVJbJ0BJeQX6/yZMPh7eeP9/NEKil6dO0pskiS+Ia/uuvdgwpbHPuoC0UNx9m49jXiZLlSU5HefqRx0OMit+KyyESfpDEUfJS4TJgt+Z85knIHTCVj0Fb5UOD2mHd7vFdplRKMMamLDpyUilr5i9LarSHcIRvt0jHMsUzRpYYOspEkneZF/mYmMGpJZeN8sT+K00v5QZwA/XUo7N1PsybZZIJKdo4QvJbviq6VvclHbb+Ybn4GI1mJqFKzvgHFdMuhjC6jH65FT+L+bSsAHXUGdvcJERRKTYXrNuF86cQLXY4IkU3oIupFSsmd75E7JGR1YHchCXXaY+oFPRrru0W36cbP68uH4PQ9gl416wPMSgCLut8hs9YK7c5guwJCsp3vUwj2E2yViLQBfDaJNVu44Ba4oJNc4immHG7LeqZum1ooG9fUX8fE6vedmZMf00J7AhdnFmSdqQ19KwjBVHHynZ8kY3Yk3UtVx3urunkcabULLiqRydI6/Hog6lsKxb9nfptUVficWfYSzWRTAA7urmFt21UYHbz0oWCiDIbsObZ0INmde472KZKmRPByXTx4lw8fPXAiKwIfUbB7FygyG4NMgW2rNMK+vaI2khI2MAPMbh4UOGGBxuLe0Rf5g5VoCxtclqt40CLaHhDKj4PiLuHDLUkd/If34fFtklLR/UMw/yCea8xbcG/1BU1UWaTMyB X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6ca7b1f1-435c-4a73-268a-08d8ca167834 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2021 20:41:52.9009 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 20hFKJ/pOUcE85azmM7GDFczEU89xPU/HPM3xVy6/oj53GfUGVGZaOEGwWXj8M6ke1ESv9zQSDwx/M5tB+ZzifCFvBZk47ttYLD1LVIbgDA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4432 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 mlxscore=0 priorityscore=1501 spamscore=0 impostorscore=0 clxscore=1015 suspectscore=0 lowpriorityscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add an helper that iterates over head pages in a list of pages. It essentially counts the tails until the next page to process has a different head that the current. This is going to be used by unpin_user_pages() family of functions, to batch the head page refcount updates once for all passed consecutive tail pages. Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins Reviewed-by: John Hubbard Reviewed-by: Jason Gunthorpe --- mm/gup.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d68bcb482b11..8defe4f670d5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,32 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void compound_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *page; + unsigned int nr; + + if (i >= npages) + return; + + page = compound_head(list[i]); + for (nr = i + 1; nr < npages; nr++) { + if (compound_head(list[nr]) != page) + break; + } + + *head = page; + *ntails = nr - i; +} + +#define for_each_compound_head(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + compound_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + compound_next(__i, __npages, __list, &(__head), &(__ntails))) + /** * unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages * @pages: array of pages to be maybe marked dirty, and definitely released. From patchwork Fri Feb 5 20:41:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12070883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63708C433E0 for ; Fri, 5 Feb 2021 20:42:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C2C8A64FC8 for ; Fri, 5 Feb 2021 20:42:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C2C8A64FC8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4164C6B0070; Fri, 5 Feb 2021 15:42:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 361366B0071; Fri, 5 Feb 2021 15:42:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1637A6B0073; Fri, 5 Feb 2021 15:42:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id E808D6B0071 for ; Fri, 5 Feb 2021 15:42:06 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 9EB44180AD81F for ; Fri, 5 Feb 2021 20:42:06 +0000 (UTC) X-FDA: 77785386252.05.fact95_2517cec275e7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 6A11A18018714 for ; Fri, 5 Feb 2021 20:42:06 +0000 (UTC) X-HE-Tag: fact95_2517cec275e7 X-Filterd-Recvd-Size: 13261 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Fri, 5 Feb 2021 20:42:05 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115Kd1tA017993; Fri, 5 Feb 2021 20:41:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=nXQCScTnEmz9YpfAfESz0ZBlMiwzMlpLGxkAJpBxnVo=; b=QQtB5LdyOFniFaaDZUKwfzJj84PLhwPf/qhABEnHO1JfBcqj5Z6wCZQeW6fb+pgvpzIk SjguMYT/h50IQP/QFVAf5ZkmDpWSxGuwerukj145v+qwqvvSc49o8xwBa0tzqt6RRCXp Ks9grPaH8UDaMYvvLRWNyPrRjbZFXs4hbb7j3q6JF6671WBS/3QoSh0V2g4zxl5sTwLZ rnInH3rsKeB3iMkJesSzr1nDAQIFEgoH7GEEGTJ3jz9yeTl9gttT7vcGDaOgP0g1ebZv YORmnlprFdkuA6aidoBHPwwZU/o9lzGU8nz0ajaeaZlyY2ER3eudc/VwiQoEhzaQCMDS Yg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 36cxvrdu59-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:41:59 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115KelXG041067; Fri, 5 Feb 2021 20:41:59 GMT Received: from nam04-sn1-obe.outbound.protection.outlook.com (mail-sn1nam04lp2053.outbound.protection.outlook.com [104.47.44.53]) by userp3030.oracle.com with ESMTP id 36dhd3fqjt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:41:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eXxzYD7LK+Afh24bCjjKiSCbRoekJnDQG4Ii5Y/FKsf2s/hr2oTevVWeKdlHw3B7qAXatwRwBSuMWCkWhVEGkJvkzqg+es+hDjfIiSzGKPHKzY1PpVG3wEzCTLRQ1W9tslwtX4XDMs0G9XyrFP0bu3JjbckbeBHV0M4hjDDFfPkJ+lYVktuYM/RNxbQepIvU0sCEkA9a2wZhKa53yV1eiGvWKU7uB4MTsYgJ9OIuE+bwN5nu0zIPYhPKwZGIl0TGpnwx41gtkiCRiqHdkrybvVE30L32vaZCL5hgPQ1y7BJOsd8l+fxtkBNjKcpmpo3x7gOLPVnJO/BHBY6icEc2RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nXQCScTnEmz9YpfAfESz0ZBlMiwzMlpLGxkAJpBxnVo=; b=LjH1BygoFaWVwLkkOO3YU1XdNyWXSzJlp/4bUCOKmbyX7SKKLvyVpWJgkyRFzKXfDuvZp0bx0x94c9mXnaqIoV75+AHYNX+XQRkX6fJl6eiyQLlWtFXRWMm54GTY53RJBtY+z8BXqyKOnzCFM7mPUOjLt4XbsJf9bxg2CT0AKlLCUw63u15Yu9UbMnQLXbNOEIoB1SZT8iXtRbIdwMokdo9WDL9VgL0vIC6lGbXQ0OQ8ggYR/P4+TGM0G25/yBTe7HymrTQIW6itwmQbzRqgEsiTmThVIvT7filj67uh1VRY/7zXpfUg8M0xSZthFolkGjiQFcZFoUfdQh1N065qOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nXQCScTnEmz9YpfAfESz0ZBlMiwzMlpLGxkAJpBxnVo=; b=EclOh5Duf2mtaSXz35E6/1DO4QDtZUbQbSJP+WZ15AN1OOdWFugmSITxbyF9jVN4yha8slWs9AGLJgOr0Al7hoExbAhQaV/KWdPIT4O7pXKKNw8deI863HF2a4aYjfhL4EAbF+BDU17JOJyt8WJrrhe1VfTt4HAX+YtldhM6jNo= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4432.namprd10.prod.outlook.com (2603:10b6:a03:2df::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.21; Fri, 5 Feb 2021 20:41:56 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Fri, 5 Feb 2021 20:41:55 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v3 2/4] mm/gup: decrement head page once for group of subpages Date: Fri, 5 Feb 2021 20:41:25 +0000 Message-Id: <20210205204127.29441-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210205204127.29441-1-joao.m.martins@oracle.com> References: <20210205204127.29441-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.19 via Frontend Transport; Fri, 5 Feb 2021 20:41:53 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7d7084ce-126e-44cf-fe20-08d8ca1679f0 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4432: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hV0JdkC8m975A76sTARfBvHapW7oQjoTs7Let4b7nVHvCFuvRcfzqwO8n7cZDEvS+6qsnvlRozbhVs6P0i6r/LEuVmvbEKVT9Agnd4SuOFuFbMO4J9nMLdRcbpI3vLSTNWFZZFlcI6/wMTvlR2HCGJ7dXj0lEk13AaedcD/IoGrNa9LuN6HV2SjouvY3iat5qxF8GElypQlxf0fTxO3EVaXOVllZ+XRb2RaGYpLf22mLWz4XchOCWaQk7vdyKvDt4F8dY1yXSYBeWazGBLkRM3XV9VnjvfXR9FOJpO/PtRFyuu+QwKyzbxNjRHQ7X2V1sH33qLvT+KrzsPBvN8CvvLYSiToVnMY3zw+HnCOWMP/A2fJjmKW4I80YTJIrsdJqogYnfiDPX/WNdE6Gepm7b7rYH4ytaQZUSVBzpAaUZWVsl6QdrmySjohkKVsZVJnv1ukrF8mhV6EpTWn49hulCevWnljtjtKicWUe8dv2c4lJdyKs+/xImjY4UimeNPtGz9D953dmw+7GPegRbbEpIw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(366004)(136003)(376002)(39860400002)(6916009)(1076003)(5660300002)(186003)(26005)(54906003)(52116002)(103116003)(4326008)(8676002)(83380400001)(66946007)(2906002)(36756003)(6486002)(478600001)(6666004)(66476007)(8936002)(107886003)(66556008)(16526019)(2616005)(7696005)(86362001)(316002)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: RHB+wNW0N7tpUEVp9oj3rB5w1u70Z1VZTNhF0vWEEqYvw41h5Q73oBj9W2Mc3b9p1Q5neFVcuNLYqbcGMMDti5NRSG2PR+B5p0cIIABp9zLBT2314mYMy+HACQmU1nklRvkSzPCdePN2NT+QxYLteAE+nJdwwbHA67KeFyRE4A1fYUpoJXSZnPKsdhZ2OEUIzMZwoiiUmprxdDD/lSiQqu7LTz9W+Twi1oNJDnLkmuDiSIMDPwjSKmA/ZSLSgv/2wXHmOBx5HFYsSgP2uGlpDEAPvZ9hA2nYkEQsOrYeGZ7ziL3LlNGA11l486te5820PrD8Y9efbCseIk3VTDyg4S3sLVlixdn+rpZE7nwrHRIQIPitS2M4xU9KlELrP1x2tBoxynsrPXOnvBLU0WSw/A8jwxVCUsciUx6TtYNVvqW82eayljIkZomBbObwEXIuYnpDA2EN3gD3Hmu0fvMvNCTSSPaElIUKCndc62N3xN2/vbaB9CUZ9tM98DoSs/vudrHDk1/AssLFARp0JxrSJF4ZppcF4IpFUdWldnKoaBOKybn9bsdEt8+HNDUhl0pvmTLvVfbTSoSD44VaK6erSuQtW2tgUmJFr4/RsuGUwPABFPdM3bdojwAZPtesyeEqPJO1K+4ImcQzGWcKZfaKKBvpC4SsOkXxxWv/wbRBNRs9kmBIFIbiWg+tqiHIOsGgHIXy4EJKCT/8rWEBeW54BdamXh9KDoVLTERoe1Y5vWE+RBFcmOEkaXHDZ00pNz2HJwyO0VNwCVbcplAcuFBJaUJ6jInFN7ELuPNnVlpS6l7xuG/k+5iRfxH0z7fJnSnGQALMc6pGNjAeFITo2lQa4z0pLHPboepkSvEm2XQPb1lNiD1jufmw3mGWODXDym5YuP4kWPHT7qOYVktLNhqzEZ1fshiH/+AWRE8q7rHHxeg8l2reFG87xCLaIZ08s1UNNQoOOqufLu4uSWOOIsvp1Snh4OrbhOKMMCZav1pkeTD734V5Hx2SFO0R0r95amwSP5lcNAKOgENv5WCz0xu8bL1+K204b2Vzq75v8L2x8binailRuezEd9Mp+Jg+0wXPu6fh/shb7cn0uTeqXt/0WlMPJje9Lv3ASuZX7kJuEFTdJgPlKrzgwSpO5+VmUvRooFPHD/D4e88nXm74b3puGmywpOIhYuzdqAoLyl2pNk0IYx+R4w3XckPXUHX6Pna2EFkXfaXi+f4J5pjjHkctawUoQHIUGwx+W/Ihhj45Capm8pqX71H4HBjA/fsDbjKKDrpvYAfoG5VKdw4lTDdDUUwpaCiTQdRfYf6LIm+ODNEgILN+NkyEApORNY1uvxMC X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7d7084ce-126e-44cf-fe20-08d8ca1679f0 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2021 20:41:55.8532 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BGkrzbTDBEqgMJm9McZPtfjgjmP2Ho3sN2HN8j97951Hj+6LkimmvnlSFFrFzkwxFobQWVV9dzYV1gKyEsAN7+EKh2P2C8yLZywmK8qehfQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4432 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 mlxlogscore=999 phishscore=0 spamscore=0 suspectscore=0 malwarescore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 mlxscore=0 priorityscore=1501 spamscore=0 impostorscore=0 clxscore=1015 suspectscore=0 lowpriorityscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than decrementing the head page refcount one by one, we walk the page array and checking which belong to the same compound_head. Later on we decrement the calculated amount of references in a single write to the head page. To that end switch to for_each_compound_head() does most of the work. set_page_dirty() needs no adjustment as it's a nop for non-dirty head pages and it doesn't operate on tail pages. This considerably improves unpinning of pages with THP and hugetlbfs: - THP gup_test -t -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK (put values): ~87.6k us -> ~23.2k us - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK: (put values): ~87.6k us -> ~27.5k us Signed-off-by: Joao Martins Reviewed-by: John Hubbard Reviewed-by: Jason Gunthorpe --- mm/gup.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 8defe4f670d5..467a11df216d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -267,20 +267,15 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty) { unsigned long index; - - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ + struct page *head; + unsigned int ntails; if (!make_dirty) { unpin_user_pages(pages, npages); return; } - for (index = 0; index < npages; index++) { - struct page *page = compound_head(pages[index]); + for_each_compound_head(index, pages, npages, head, ntails) { /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -301,9 +296,9 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, * written back, so it gets written back again in the * next writeback cycle. This is harmless. */ - if (!PageDirty(page)) - set_page_dirty_lock(page); - unpin_user_page(page); + if (!PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); @@ -320,6 +315,8 @@ EXPORT_SYMBOL(unpin_user_pages_dirty_lock); void unpin_user_pages(struct page **pages, unsigned long npages) { unsigned long index; + struct page *head; + unsigned int ntails; /* * If this WARN_ON() fires, then the system *might* be leaking pages (by @@ -328,13 +325,9 @@ void unpin_user_pages(struct page **pages, unsigned long npages) */ if (WARN_ON(IS_ERR_VALUE(npages))) return; - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ - for (index = 0; index < npages; index++) - unpin_user_page(pages[index]); + + for_each_compound_head(index, pages, npages, head, ntails) + put_compound_head(head, ntails, FOLL_PIN); } EXPORT_SYMBOL(unpin_user_pages); From patchwork Fri Feb 5 20:41:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12070881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 370A8C433DB for ; Fri, 5 Feb 2021 20:42:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 99AF864FC0 for ; Fri, 5 Feb 2021 20:42:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99AF864FC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 24F5E6B006E; Fri, 5 Feb 2021 15:42:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18CF16B0070; Fri, 5 Feb 2021 15:42:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1F026B0071; Fri, 5 Feb 2021 15:42:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0065.hostedemail.com [216.40.44.65]) by kanga.kvack.org (Postfix) with ESMTP id D71F66B006E for ; Fri, 5 Feb 2021 15:42:04 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A296B1EE6 for ; Fri, 5 Feb 2021 20:42:04 +0000 (UTC) X-FDA: 77785386168.29.bell25_05101ab275e7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 7FC6B18086CDA for ; Fri, 5 Feb 2021 20:42:04 +0000 (UTC) X-HE-Tag: bell25_05101ab275e7 X-Filterd-Recvd-Size: 14213 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Fri, 5 Feb 2021 20:42:03 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115Kdh9D135318; Fri, 5 Feb 2021 20:42:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=a9tp19DS2rCWvHsMe2G8y7YH/MyxjrAldbxEbqGAl5Y=; b=PvbFuz2bHCpUU7dEbSNp82D2nqYJ6DY20YnlnZwsyGRIafF/RP7bf3w3wjWBgCam5DzZ Z0E9hbz7yxGuca0+xmQzfKnh0zLzJkBRk5apXvPLrcahE8r+lAQjaFK9hzhX1rrWuMCR QUV0t2wB6UvA1wxNqkPIw+Rj6JUI2t+4WwV+OaoBFmVaSosiyfF2gLxL7yvkAHnGh4s1 ywh84t9MnHO1xS1XKVHONGSTR9aNuEhyK0xn3jYrQwOA1H7CxpCxVsxExZD4f/84U3qn X+5NQl+26WjUFZT+XiebmxV9gV1DJ8W6KclJp6v4XgSwhJtl76FLEDO9yzKo2DukkSVq 8Q== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 36cvybb8ed-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:42:01 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115KfBFT034011; Fri, 5 Feb 2021 20:42:00 GMT Received: from nam04-sn1-obe.outbound.protection.outlook.com (mail-sn1nam04lp2057.outbound.protection.outlook.com [104.47.44.57]) by aserp3020.oracle.com with ESMTP id 36dhc4p4mx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:42:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HuWYtTHJOUTiiGKKbyrXPRbuFaJfTC7PYcjPxZ5A5Vk6PSz7OEaSuhwjlMkkiAet+l/2TPNfyMbrLm8DIt+21vsHPxOmMYVQg813Bxu9ACmn9I+dVZDC8KAQ+XX4Fc5fXu74bJPa4jraVEsiKASQVbIci4455/CFwB3yGjKVz0/9R6jbibDQFb0zDBZGav9B+lrK9J0malg/56UdXTr2/KAbgZNmzLyifKCT8X34Tu2vVDLMunuAY+QllCpzUxasDH6jYxbXfiN3pDpRKbAgr7CWKCFPzwq+A99y4j34onv+h8WeyrmmBpyTjJVJ04CmPijUkKUJtsKmqrNmwjQh+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a9tp19DS2rCWvHsMe2G8y7YH/MyxjrAldbxEbqGAl5Y=; b=AE1bkXmahpLQvLZSQ7CCRIz06TsrLGYaQvPbSbflDHmF8qOHS2cvzAXiel9f8shKsglnsveKkl2gYA46E7Cg96rzkd3yljQkIEAAFhDGnp0kpYfzZqSO2O/tzNuhPuUgvVhUr5p8wtOUSxjYJtdv8cQ2+vO5gVScNCcRDnHrIezkM0qgyZNCLMyqO1RG1f0/2iAL4vKHXXgndhoyQi7XzGiLhZ8O1dE3gg7ZeVvSH7hLr9/Z0YZOcx5pR3NFnSieaVii73luX15AjK7iypQyJMwVfi5usNUBrOO8GwDZgW7zz218dRrpNiJ6RluoT2I6o92+ZhBoV+0qJEsE2laReA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a9tp19DS2rCWvHsMe2G8y7YH/MyxjrAldbxEbqGAl5Y=; b=LK2OI58KNajJcqrOiGjuJouQAVmGMZWqB9MNrXqLYzoAS3zVetfl7SY7/ZXsbS70GY41e4FoQTETZ0o9bYKoHqZRhGtAFc3yDrvgHYI0g669cLJ7LEJNtq3nHDRgOof+nB0CcsuqbKv+4j1pyEyGTN/ZPP8BZdJA5BhNAvgEgW4= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4432.namprd10.prod.outlook.com (2603:10b6:a03:2df::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.21; Fri, 5 Feb 2021 20:41:58 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Fri, 5 Feb 2021 20:41:58 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v3 3/4] mm/gup: add a range variant of unpin_user_pages_dirty_lock() Date: Fri, 5 Feb 2021 20:41:26 +0000 Message-Id: <20210205204127.29441-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210205204127.29441-1-joao.m.martins@oracle.com> References: <20210205204127.29441-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.19 via Frontend Transport; Fri, 5 Feb 2021 20:41:56 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9cba35e6-5db7-49e6-4f46-08d8ca167bac X-MS-TrafficTypeDiagnostic: SJ0PR10MB4432: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BZUu4ZCwwvlmhjN2S2y0rFWjhMJtpAtPA+ih/YHSVRHfdPCdLpZm5MwR//+vkWQ9kDL97onlVxR/VUvamHgrokJ+fhiHQ1TPB9hVluTWgpi6e5dSfZhia3jef6ddDDlVxGazghFUZwp//Zd+CTWiP3QUPNIH2MYre52OX1QC1gxNyQmUPxycBZFb2Lx3qlq7ievfmfVwEc6LA2oS25NqgSVwkPDXyGXfcDROlco9PQx/DeSOT9fNPbIsNO09D4Bjg4FtQODyJeaz8gr+qneGXgrVbx9H7/Mtw71pU3smBA/1N3wr3acifO9tE/odcwW4oYLYXHDbbmDHdlAc6UwTYuLDQVJIzigX10UErWdDsIT7Oai4v90VyRJPFsv2CdI0AbPK5kXm10JFwFX47Gw520s7FOD1G0Vr+NBN7LjaE6NU55it6z57yMwSUfm0PIB/K9qxEThHSPwkZGnqd9njX5apRHtvWsU719L2hbdKh9KFhi0MEcwz+850WnaSs+tmfzz/PEljmnBdz8NDdlm7T2fkxI58sdgV5whcLfqlUoE8Y3pTXc4mohZ+QyKQV5xA X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(366004)(136003)(376002)(39860400002)(6916009)(1076003)(5660300002)(186003)(26005)(54906003)(52116002)(103116003)(4326008)(8676002)(83380400001)(66946007)(2906002)(36756003)(6486002)(478600001)(6666004)(66476007)(8936002)(107886003)(66556008)(16526019)(2616005)(7696005)(86362001)(316002)(956004)(14143004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: XBQAUJyu17fN9Hnz2ymhLZO+HCpnZOJoXjmm0JL2c1CW4abnsCs0tERZWhzpzYAgbBuQ2X7bfF4FjIxrynNEPL9wT12TE4otP/qepYhJN3A4Bh9l0KLCr99P3xwKjJ6+xsoCzZwaoch2qKmJP3JIWiY6fHkB3R670uvctYE9OH+pLqdseP8odrtpw1V0Q4ITxJKLS14nFqbdJb8n4dS0mHab+oZvZZcUwMxUICw45oZQ6UGpF/EV7SgYRyWQe58QzIfSbGldcSmW3xb4jFG16l9LPIjjSUq6TRgUlrf3triRGvDvj+OEgVI3tBt0MeAvYFaWJE1akaJZiNwa6kJrexK2ZBl1ffId5i80sx6UuFUNVWFvoSsH3W6Jou+BSGfAwARROjajPBsqtbxRSkla0Vg77AaTo9JjJ3wyNeQe9qGetLE/lUgEbcf/tnfPhUFyxlWkFqmX1oS7ZDb29lQBbIOeoQH3m9et3voGx2TRlIaqIS7vwl9MSNeAGy/uYh+O9rF+v+7t9kFhiy8SpEpA7jjD2X7TanRizLRB5sFlvK0SL+xwVfEhM2Q57H9j/4q/r1X1A4Bt/VUT7yNpU+y4yrwRY+ajMNP66osYYNehDqHsjx8MVvh/sMF3bX4PysOOgPluh2y4OnGrtcmrwOpLipYA9rn9q4p0fhCXcEHbIsUnnLw1NTZxHYenZocDdxfvurT8dBG95fhkG2L6LOW0lV5NaRu7gdXhN76SwwFe+6vCfG9htagYwo8ch80mk5gc4LIXR53tJw37X9Epiwp/fxWgbArpB8fnyATRhaCx4FyXLOxxD/G3i3W/cNjz7pCaAVJ9kYn326BY4dsGBSNFXbRGsAL4FwWQcS3fYu5XFhYkaJ6/tHfbu7I6uXewPdoduTZlsOl2rHs0Eb/Y6/xt6GR0ggTVLJW2XV9IKV+Zt/T/kDQByc9OEZzEn2ADVk7j79k4zMEXxXmek+5h5TIX543lKCI2t4jFR/1jjOWgGjy25fjSv1O388U0mTec0qN0ld+tgKg+t5IbXCKAh61fxxEGyYsHSTBuWvt9MEi7AKCBufPzPf4VJf6kDGX2RWXDtL7khDMozlU5622BDA+FAybXTT+WITwbP7rBiex0kskhGI8NTBfBypvjiyEHYdosGOkrZHw7uBRxHuOtBqhRvodwM9c7/Efe6ky66TQszsAVy/ruP8lS/TD5JqrSWRLbHoHkihl+zXYS/MTvAfQuktuzOVMwwt5cTBI58QgwgIsAR/mNHMbRWrGsfz7/5n63wB/+rS+//wZBFX30IJUTbCnP60++j3JTs4l17TVFJV2d259Idgzn0xpB+VmDJHGX X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9cba35e6-5db7-49e6-4f46-08d8ca167bac X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2021 20:41:58.7755 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mTC6KoNfMbfY2UEbQ5IPfSh+7TFzPEOclSgLcpVw9O0oLs1gzR5XnWfTJcgZy4jYPlVRwru0VnvsCHWGhtXYqm8yjJZ3/FYeVA8VoWRIViQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4432 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a unpin_user_page_range_dirty_lock() API which takes a starting page and how many consecutive pages we want to unpin and optionally dirty. To that end, define another iterator for_each_compound_range() that operates in page ranges as opposed to page array. For users (like RDMA mr_dereg) where each sg represents a contiguous set of pages, we're able to more efficiently unpin pages without having to supply an array of pages much of what happens today with unpin_user_pages(). Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard --- include/linux/mm.h | 2 ++ mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a608feb0d42e..b76063f7f18a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) void unpin_user_page(struct page *page); void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty); void unpin_user_pages(struct page **pages, unsigned long npages); /** diff --git a/mm/gup.c b/mm/gup.c index 467a11df216d..938964d31494 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,32 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void compound_range_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *next, *page; + unsigned int nr = 1; + + if (i >= npages) + return; + + next = *list + i; + page = compound_head(next); + if (PageCompound(page) && compound_order(page) >= 1) + nr = min_t(unsigned int, + page + compound_nr(page) - next, npages - i); + + *head = page; + *ntails = nr; +} + +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + compound_range_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + compound_range_next(__i, __npages, __list, &(__head), &(__ntails))) + static inline void compound_next(unsigned long i, unsigned long npages, struct page **list, struct page **head, unsigned int *ntails) @@ -303,6 +329,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); +/** + * unpin_user_page_range_dirty_lock() - release and optionally dirty + * gup-pinned page range + * + * @page: the starting page of a range maybe marked dirty, and definitely released. + * @npages: number of consecutive pages to release. + * @make_dirty: whether to mark the pages dirty + * + * "gup-pinned page range" refers to a range of pages that has had one of the + * get_user_pages() variants called on that page. + * + * For the page ranges defined by [page .. page+npages], make that range (or + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the + * page range was previously listed as clean. + * + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is + * required, then the caller should a) verify that this is really correct, + * because _lock() is usually required, and b) hand code it: + * set_page_dirty_lock(), unpin_user_page(). + * + */ +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty) +{ + unsigned long index; + struct page *head; + unsigned int ntails; + + for_each_compound_range(index, &page, npages, head, ntails) { + if (make_dirty && !PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); + } +} +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); + /** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released. From patchwork Fri Feb 5 20:41:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12070887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66408C43381 for ; Fri, 5 Feb 2021 20:42:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E0EB864FBB for ; Fri, 5 Feb 2021 20:42:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0EB864FBB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E5BA96B0072; Fri, 5 Feb 2021 15:42:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE9FC6B0073; Fri, 5 Feb 2021 15:42:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA1A66B0074; Fri, 5 Feb 2021 15:42:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0154.hostedemail.com [216.40.44.154]) by kanga.kvack.org (Postfix) with ESMTP id 957786B0072 for ; Fri, 5 Feb 2021 15:42:10 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 5DAAE2493 for ; Fri, 5 Feb 2021 20:42:10 +0000 (UTC) X-FDA: 77785386420.29.flame94_4c0165d275e7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 0F73618086CDA for ; Fri, 5 Feb 2021 20:42:10 +0000 (UTC) X-HE-Tag: flame94_4c0165d275e7 X-Filterd-Recvd-Size: 11835 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Fri, 5 Feb 2021 20:42:09 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115KdQZR195498; Fri, 5 Feb 2021 20:42:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=0LTNyLhfcQU8WurGe4uk1JQtTl/ZkrEkTZA6jP+zRYs=; b=k+w+qDHd8qSYcAXon/ucFHWY4VaerhKw/R4O5HqYGoG5c0xhw9HPTLeALNXtOeGMSDCf /yo65lmCFDfeNTCE/txbWKkkaBnTBr76bcJWz7IRwjkOQMQx3oywIlF2MxbVpiqqS8ZR WTn1LfWGVMiDVg+ooeey6e6UAboepZu1le7O+7qQK5KlSNtDSPxDNQVdcyWIWJgAjbcs ZCtyShVRY3DTdzQ15wPhdCJIZRMmZvBeJfYIxiVYOonWsgIWJiizpL/sv8zbRT3JLhuS XsSuZklT30W5tCOB/1CcNLgxXO0A/7vEIxKVtaDUf+PFraRcVgUzOQfcyNKVzIfv0bqf Ng== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 36cydmb22e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:42:03 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 115KfBmB034035; Fri, 5 Feb 2021 20:42:03 GMT Received: from nam04-sn1-obe.outbound.protection.outlook.com (mail-sn1nam04lp2053.outbound.protection.outlook.com [104.47.44.53]) by aserp3020.oracle.com with ESMTP id 36dhc4p4pd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 05 Feb 2021 20:42:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZAZkrkEf+z6BSGmYS2mfQ+6y+yDr6sG9UOeUY5TAVIRTP4D+9tWL9I0bsk227jBUOa2vm/oyPsSSOnZDgKS8ArGQd5siPLs1aB2bF4GXevAN+QrzZbe655al29o8eoTbvmoLNj5GYFYZO2AsVePvfNAgpEIEjfMUg4TXsC9N2ur3hF6UYjx9zUYK8/wSZM3iC8RHlM6Nhe5MHYBKtJd+vYeJ50E6HSCum/ZkIVVq4cAtteKmMecuFOXQkh562CLBKnwlvLpzWBHYm0J1julD+LfVqu6m+vMwIGq9R6NNx8Hoiv3OosT7eGyxGKXbywRjnPqBNaPvpSJl19gRhXEYNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0LTNyLhfcQU8WurGe4uk1JQtTl/ZkrEkTZA6jP+zRYs=; b=nXAEj/afdFeYAKz2De3O8VwPJ33nBOnWI4eJcoZU4GdwEVbpiF6+inyoBjSdTt/5r6a4rxhxeWsC5Co97kwUX6iLnvaDOpvHCKglIRliRl/e6wJVy4OoXwKAqK7YBn56an9eLHyaefniZgS2ZNKY6oLNZQgDlA+143/MW5bTZN7T5NdPATjkPIHRwrYbFGMqEvS22lDoTMwAuTdWT2JvsKvGJJLiCHIhR7Qz92lfYlCMPMspxCD8lzURKoTjmIHE2Vfu/Ne8uBo2VRqD+sJphqJ3EGTSqmFU2W6ns95nXBTcHDxi5bY9HhTCkK0EF+0Z38+1nS+F4IFxZ8PdlI9z9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0LTNyLhfcQU8WurGe4uk1JQtTl/ZkrEkTZA6jP+zRYs=; b=EOiYzQAV6LpDQNLRS6uUPoxIl54flngKnUpUu/LrbnueQh6/MNANQvspBNLleSUQmoLV8uLI8tm9fnGLmDjLQDcU07R7H4hcY64039RcsQT5pQD8IKMcJOn3CGZtdNNKwmOL1DVq3t8k82niBrKuojXYkylcQAGn2BWQSHYQONc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4432.namprd10.prod.outlook.com (2603:10b6:a03:2df::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.21; Fri, 5 Feb 2021 20:42:01 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Fri, 5 Feb 2021 20:42:01 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v3 4/4] RDMA/umem: batch page unpin in __ib_umem_release() Date: Fri, 5 Feb 2021 20:41:27 +0000 Message-Id: <20210205204127.29441-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210205204127.29441-1-joao.m.martins@oracle.com> References: <20210205204127.29441-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM3PR03CA0058.eurprd03.prod.outlook.com (2603:10a6:207:5::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.19 via Frontend Transport; Fri, 5 Feb 2021 20:41:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dc7463f9-eaa7-490d-877d-08d8ca167d58 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4432: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:370; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NVhnKcxoi+1YeQUlTXA/AghpDMvhCmuUaWzwKcebGrkbaVZl5nd7ZIMyEYlZaj3hgp8sI3BX4mQ2mV1rsLeCcldrZnCRnegOdQJjnmoV18UGYn9pJyxLBLU7ZfG+k0D9I4ztHJN4V2L3DzCPBews1dt3FkU9zqe4qU8YZc8XFV0hpe8CYVkhPHZnw+z/r0U0CGPChnuP+9lJ8Q5SDDPv1ZXFDCtLY65WjFzuhonRU7oAhOMzfyp2+pxDnBU8u16gBQPpNNxUkS3IzpzffM3SdL0EXwGGZEmjYwNBfKygLU5jzKrEJGckERy6oH8OCd1m+MSTgf/Tsv3txkPOsq1Y7bnRM2oOubhkBm/6B9JoRHkT6+pmUbMayeAmTmeg2b6ybWuRqETz0r2d9ghY+EGXlfPr9QV47NxUFQ/QZH9Ee3kThbirmHB9fZGt807QleAm5uuUNA0dsD8f13R2qXtFi9FRMqfpD/k+B+zqJMfhCTFdgqek5FkLuEDv/yS2snxjlBofc/0ND4t17YXbPMnFag== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(366004)(136003)(376002)(39860400002)(6916009)(1076003)(5660300002)(186003)(26005)(54906003)(52116002)(103116003)(4326008)(8676002)(83380400001)(66946007)(2906002)(36756003)(6486002)(478600001)(6666004)(66476007)(8936002)(107886003)(66556008)(16526019)(2616005)(7696005)(86362001)(316002)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: JRDoVXDvElOlBQAF/3+e+BtRTf7ya9ry3liTSNoCHQyfmJ/ZL1eRIxFSEjw04X6JADjKLmqPhVHLfDbggidYPbWJ3TMU2V1zUGVUUp2rocfo+m0OGqFw0tPfgjmQJV6qVF6o7lOR61ZGfhYUPSRcxSxqVeGzphGJBV/SWjl1SF4VHzorsTloKnfZHPa8i2rDaA/4RNiC6GhbaM0k5/DkyOn5aIigIRPgljyHzu+mUzwantmEm1vqvcn2J6e9lyuUf6RMZ5+MEYIOq+BPXTDHb33OjED1oCbSy40gF57BiOLJp0UitbFdzPwCbs5aPHHjqZF3sgeywlqfpzCpgzhm1ZuXuuGLTXQ1KhiZJGxfRIL5eBscNMG/cYgwUxfu41+jfZoFbXQEuXQZ2esg1UdeEoSMKyns+3fzknn15X29xSEvFP4el0G8mkG1c64dW4epwxce+xDaN+56I20Odsa2H7uvWJ7vHQFBS5yio6Cyuk/CqF4f0rSdF9WmZvqe4gTLDF9PAIR4kCgTtwtJeDcg0I94CsYFkV8JeJ9GfvLuMp0QMxS423qgZ5W8OUPlHZEjVGOfVZW7uk7jX/Awclpxij9Fw50YVw3ufxFB6VUaGOel+r3vSNWDCIy6F5I6avlDxGT4ub+IBLyCK+xPGMWTVOVrlFm663vjjib3OfUO2+3U9gP5plMa92lhXtHKyYKmm7eIp88L5t65vKp6n+B+H2RVm9FZDPOn6mdjzwMZxhrjP3CiRnZrMOaS1AXs6Teehs+1eVgqt4HHnBiIXhRHX4AXJ1I/u64Jjig2J1wgKbgO/bgSIDH9+SYJXd8nnQKvXv1Bin+81kKTS1SQjmnSzloTdfrVZZaDUInfW2DcEsJ2xC5Dqn5GvOnXBbstuxOLiJsLzOHcCW4SxosHi+8+FkrhTOHnQw9Mkbjn+PhI8KMetecOmgQr2aUNBtHzWJIRiyXqbaCoj1BzzQ0cQ0+2t3C44FBDpPaDbRWVGTfZ0W/U2VEIpBnRlKPCN8nJwikCc/KD/6V0ibp1OB63QFqCS+4Xmp3dGZdmdH5TlPXKCDOx732mfkAwh/sR/8hU9w4GZwmkFjeJ71r/Z1D6Y+ojObVJhhgGnMdwzbFGoisKnZ+GC8xwxVD1pKMSX90bYqVzsfmyo/jKvBJ+CeiU9oV3ol96bhde+i5Wj8KsZ6ICxQBtINzlEfkLw15DiznArIN60n1PO3A2gptyz3dlwrLIUd/l2x6OpyhOlPkRcYphi28N5fbVjax6rtmyFnQzc/XykCs/KlfrvYY7cC1UwHqNk37IWM3S8D+zAIM08OoowAD3moo4Y1EjjVEBIKcct1Cm X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: dc7463f9-eaa7-490d-877d-08d8ca167d58 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2021 20:42:01.5819 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: sWfPLzAY2chy3d8qstbtM/v0vQ+9ps1JlaVay5eudAKev0rvZx/z3P3MnuUQq8TZImI8wj6cSPT+DpHul2tXMWcyzjJ7CzD0fph+GUmB9S0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4432 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9886 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 spamscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102050129 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the newly added unpin_user_page_range_dirty_lock() for more quickly unpinning a consecutive range of pages represented as compound pages. This will also calculate number of pages to unpin (for the tail pages which matching head page) and thus batch the refcount update. Running a test program which calls mr reg/unreg on a 1G in size and measures cost of both operations together (in a guest using rxe) with THP and hugetlbfs: Before: 590 rounds in 5.003 sec: 8480.335 usec / round 6898 rounds in 60.001 sec: 8698.367 usec / round After: 2688 rounds in 5.002 sec: 1860.786 usec / round 32517 rounds in 60.001 sec: 1845.225 usec / round Signed-off-by: Joao Martins Acked-by: Jason Gunthorpe --- drivers/infiniband/core/umem.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 2dde99a9ba07..9b607013e2a2 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -47,17 +47,17 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) { - struct sg_page_iter sg_iter; - struct page *page; + bool make_dirty = umem->writable && dirty; + struct scatterlist *sg; + unsigned int i; if (umem->nmap > 0) ib_dma_unmap_sg(dev, umem->sg_head.sgl, umem->sg_nents, DMA_BIDIRECTIONAL); - for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->sg_nents, 0) { - page = sg_page_iter_page(&sg_iter); - unpin_user_pages_dirty_lock(&page, 1, umem->writable && dirty); - } + for_each_sg(umem->sg_head.sgl, sg, umem->sg_nents, i) + unpin_user_page_range_dirty_lock(sg_page(sg), + DIV_ROUND_UP(sg->length, PAGE_SIZE), make_dirty); sg_free_table(&umem->sg_head); }