From patchwork Fri Feb 12 13:08:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12085269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D73A5C433E0 for ; Fri, 12 Feb 2021 13:09:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6CD3F64E3D for ; Fri, 12 Feb 2021 13:09:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6CD3F64E3D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 119038D0058; Fri, 12 Feb 2021 08:09:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A25F8D0057; Fri, 12 Feb 2021 08:09:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE55A8D0059; Fri, 12 Feb 2021 08:09:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id C21AA8D0057 for ; Fri, 12 Feb 2021 08:09:17 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 90986180ACEEC for ; Fri, 12 Feb 2021 13:09:17 +0000 (UTC) X-FDA: 77809646754.22.coal18_48177ec27621 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 70868180742E6 for ; Fri, 12 Feb 2021 13:09:17 +0000 (UTC) X-HE-Tag: coal18_48177ec27621 X-Filterd-Recvd-Size: 11733 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Fri, 12 Feb 2021 13:09:16 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CD8hLb019060; Fri, 12 Feb 2021 13:09:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=gTiBt+CrqGfggJA3OH1RM7VHsAGGwncGhuIFXhwKGF4=; b=g40/U6e2jmUa+GprFcxuqQlqvJfkgCMpkLpsEUmqAUCMk1JqQoS70+yvVgAHFJxQ5jZ6 9rLG5znv7kqQNRuFSB7Btiyyi3OymkqV0XrjFEHWiC1Gs0jf9g/szUjKaiMA7g1247c9 7g3/9YB6Cd6As4huzdfPdKevZJXRjVjRxC7RHKv2p/y/MIDvgBIxaytdZboLDDmD4/Fc AlbV1w16NaYzf57RiQ9++tg7tRtAlwnjdEUhygVzQiXlQ4I//cqL4fyvUtdtax8C1rkR 8zmcOhUN2KgZOkRlnpqQudpD79lDHBUvSTxIo59jRv/N1bI8Kw93Uly2vJec2Wm6gKPs Ig== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 36mv9dwad9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:06 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CCo7BY055546; Fri, 12 Feb 2021 13:09:06 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2104.outbound.protection.outlook.com [104.47.55.104]) by aserp3030.oracle.com with ESMTP id 36j4psyf4u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MFBz7I/AAbdsIRRo1VmWjiyBYQbu2kF099Lj32qlaZvTq3VuveJY4Y+bduojbuSn9evXn9KSVaR3WaDMR4X8IZkra27FCTXkar/gjgShM3SCw32URjY0tIGrWckYpXDQhrrDP8j0NXDS15C9lOlQDpQmUbqyqmgDvbzS7s7DOQmkN4qEK7S6r3K4Sc5Je3yIzuXA2REPy4Ck0uB9q2z0BjokfKmHWJY18aXc6+rrP6d6UxKaMA8vh2e+Mmwdl+j/nvlMi28bPz3tDJilPjIKfL8rxyJcLqoQO9g8V4DrCCKZcIMuCeGzWzl+uTuqnHcDca/oJzijA7A38YfxWIp5Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gTiBt+CrqGfggJA3OH1RM7VHsAGGwncGhuIFXhwKGF4=; b=FJyeiuIGMiOC0TmdWIhR4QVvjXDiYgDeO20lGJ1Hdulyi70hrKf/3RU+sduQzkYRkxM4n+uU6vZ1bIRTm9/CTgdlhpr7uzDUKTQF1oztdk/YZiGdaO1rCpBF53g8pceD7CDPqate+EabkQ7OYehr5ubA6dCbzucbonPgSDgL+1UVScbBmTClUJS4eUJdw0Da+HjC8nAhNsLz+k0pzPMBCEfQk6xhZUYBSSPLvTB7owz66gIS59TscbVMuzzrv4cMIJntJW7PsrZjjKiOWjxrqWeuErLTVMlPiUg7BzVmR2jyBBj+oSmfye1y+p3jYi+Eg5oq1KhnTc3BWzMmPpNavQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gTiBt+CrqGfggJA3OH1RM7VHsAGGwncGhuIFXhwKGF4=; b=HYNP4zAyggo+XFNiK/sVVX2x7OMoggKSRBDaAYS43WpnueTOtI7hYyyXaGZz00oqHxTs7F/zdUw6j3k4JSlzVYC1Qu0GLPodFXUC2t28fGMZUWLNvuzu9NcQ+3bvjPzsbeHdtmGYW2Vj9ADH+N6Kaz3mhEdP+D8yQX4uA6sAuiM= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4509.namprd10.prod.outlook.com (2603:10b6:a03:2d9::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.17; Fri, 12 Feb 2021 13:09:04 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3846.034; Fri, 12 Feb 2021 13:09:04 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v4 1/4] mm/gup: add compound page list iterator Date: Fri, 12 Feb 2021 13:08:40 +0000 Message-Id: <20210212130843.13865-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210212130843.13865-1-joao.m.martins@oracle.com> References: <20210212130843.13865-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Fri, 12 Feb 2021 13:09:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d4bd07fa-8d66-4e3e-9f1e-08d8cf575f35 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4509: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:862; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UziUhbKscjivGO8FdEIYMSH6Wf2Y07vLXIWZJg1nX9bLf+6gLPlrJ/SgCUTN2W1v8HCF+AhVkELZ0NAczmZ0xBP3lIaSCzAP9YM097+F+2zdQFOeIcQ66hdqpB9qw7MJgauU5HLW5V1SySvJGRnQDPW8Jaa+x2xxrkJOqadg2CADCWZhk7nhSlw+QWtj+31iKr8weXnlZJupCjxQ6FT1ft/hP1ZfdVWPlEkdfyYe41BGSlduMCcHa1vwTjuZbEtTj5JDy++jE1i9b77NYO5UegS+q8zwjIol1PxWct8HfH4LquIb/jxcgmFaue2tbczMW+nAMwf+yzJyrDN3pVS+wRSSH8dx1xnGurHMFIp927XSsUvyMwle5vd7XVO2+vMQEXfa8226qsgsYfzEvKc0bqRcsmsy/3CGL98hkP2DfpZLe3c4Lonjpem+NOcup9B0cK0AicYQQpkOkzRPPMQg78DXPEADGhBmLOJ4qTYrfRVAw/ssWWKa61UbSmfVpG3EMPrPrrChZmXdAn2TdHF71w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(346002)(136003)(39860400002)(366004)(6486002)(2906002)(8676002)(4326008)(52116002)(186003)(26005)(2616005)(1076003)(316002)(8936002)(107886003)(6666004)(478600001)(6916009)(36756003)(66476007)(5660300002)(86362001)(66946007)(103116003)(66556008)(54906003)(956004)(83380400001)(16526019)(7696005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: D6jZcaCvbNWY/m5qCOfqBRGkkkANeVXIbE8Um3vC9e0eD0keYa3vg/DDTpCUdkT7dSqIFHNhkelP1UEYWi3ljTgdQXIkJeyWC2d1NsywrYyysa7kJ/AOrT5mzjjCsxgkdef3m4EX85gmsFDnm6mqi/n8PF/oNsSTjmljp4DAikk/H726jTXYpW8zawgcrCsAehkGvDFjUcl5wQsAYBAv5CTj3rDUXqLaWfPu3BqW3fFVCFuwnp9kqenOsIy+QUGzC21yhznGfEhdrebHl8Vn6sBCb/X8VTyAtv46hE8t2GXxHkqW74IlNdxQT9hoMmg/7fHOXrFso/LrNgIxdj4D379DEg2AzONOhMVA6SNVMG/fLZ7JM8JUoe9iq+GSy8x9+fBDEutjeXmpVZfrO3yrEb3m7c8yrOdku9r9UpqcV5HieBKcjxwtXhJkDmD6pGWCg4haSYhGdgxfOfC1JAyLa18Zc2aZwSkG+wxd+6pi3ZyIVb6r4q/oHUKZb3SF+svxKIssHt3deo7ErP7l69uv4rzwlWJ629pizbJ4YPs7S+aBxuRqbyC0Mogl9ugXd/bMs+QS6S8vc2ls+enGXt9lwucw8ayoH15iVfg2XrplmnaVc4UyfHkBpG/izIoG7+M6S3kCgFJolGPGByKK9vYKwlKwMxjg2KXMC4Vdf01Nu+j/G5PNRmm3bjJu2mpaVhAE/LHKaf/DZAQvFWZTkYpHKGDfUehPO0lWK87gdVT9dExlx121zUv7bfA66ltLFJ/zP8YtRSyOmLoQ5xLQMh20AkDgy5NExU89eyCeTVhPkbwvhbzvib8OlAjkWIMCoQ1yi54x66QyBuUI5ykuExGXnNwBe6TUP0B6K7z/LBKqKuc6b9ypp1DlPfDLF45yCvnaQFuVP1vuVyH7CKC5dbkmdtG4KeviC+Hx35oPZmfW5PG/TdvLiT2VSE7lwBuX+hKF0uBVE1OOSyVldhn/QucD93WM7FIwX0zbZFdgG2fZ/bFKO2I9skKs7DAYusz2vz5pAHSFpjec69xCB+Xe+yCuzhD90DbL52UfLaJL43pUH8OdUqBaoJXul9BXlRTRNHDwUIxuxvX1NhWbZHAHaDs0VZFNzmUATdsFDWcvogRIy+DcYAReyvjpDQkHkS36AVZFm1lOX3Ny0gBu3LcviJ4jvzA75J+NoEbS42tY4147shtcO5GQ2SDmIvtfZHdxtFmjjUkfe7/FyfGFnSOyR4jI0yXDurRCCPuCkCdvq4EzMJPt5sQ4x6OuxzkHK+c+icrZMoCTqdNV7nsWrUpjC7GzV0m4wTp8F1a5b0YxV0JPsZ26bDmtnqNfTxyNn2sc7sq6 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d4bd07fa-8d66-4e3e-9f1e-08d8cf575f35 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Feb 2021 13:09:04.1178 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SoNvr0ifO+Ur3VNItcVXh7DpnXREZiR52ryveHesmj379jKLoCal8oxpD2c4X3kYxracGzd8vjXWCoBqN68XDDSMdkJLf9pn0pzxfMuucPg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4509 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxlogscore=999 malwarescore=0 bulkscore=0 phishscore=0 spamscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 mlxlogscore=999 bulkscore=0 suspectscore=0 phishscore=0 adultscore=0 impostorscore=0 mlxscore=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120102 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add an helper that iterates over head pages in a list of pages. It essentially counts the tails until the next page to process has a different head that the current. This is going to be used by unpin_user_pages() family of functions, to batch the head page refcount updates once for all passed consecutive tail pages. Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins Reviewed-by: John Hubbard Reviewed-by: Jason Gunthorpe --- mm/gup.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index e40579624f10..1a709eae2bfd 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -213,6 +213,32 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void compound_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *page; + unsigned int nr; + + if (i >= npages) + return; + + page = compound_head(list[i]); + for (nr = i + 1; nr < npages; nr++) { + if (compound_head(list[nr]) != page) + break; + } + + *head = page; + *ntails = nr - i; +} + +#define for_each_compound_head(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + compound_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + compound_next(__i, __npages, __list, &(__head), &(__ntails))) + /** * unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages * @pages: array of pages to be maybe marked dirty, and definitely released. From patchwork Fri Feb 12 13:08:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12085267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A73DDC433E9 for ; Fri, 12 Feb 2021 13:09:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40C4764E70 for ; Fri, 12 Feb 2021 13:09:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40C4764E70 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B59D88D0056; Fri, 12 Feb 2021 08:09:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AFCE48D0058; Fri, 12 Feb 2021 08:09:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8408A8D0056; Fri, 12 Feb 2021 08:09:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id 6CA1A8D0053 for ; Fri, 12 Feb 2021 08:09:17 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3559A3643 for ; Fri, 12 Feb 2021 13:09:17 +0000 (UTC) X-FDA: 77809646754.03.copy30_291563327621 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 0F1C328A4E9 for ; Fri, 12 Feb 2021 13:09:17 +0000 (UTC) X-HE-Tag: copy30_291563327621 X-Filterd-Recvd-Size: 13316 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Fri, 12 Feb 2021 13:09:15 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CCoEBw167872; Fri, 12 Feb 2021 13:09:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=QBC+NEiAE4J7sT4BZ6CfLxXlcIQg8NpsGdBzSgXbwdY=; b=Wh2XoqNxgTS/nmwij6wDGOxxeP8pDhp4bzMv4N1bq24kj2NI2vju5dlDm3ANImU4QIne 9ItcIw62JtYHSY/Xcg+EAdKwAivCpDsT6WMlG2DXqXi8c2UmnYF+BPLtjrN+ei1ntFdF EAEeMkty2ByD8dV/UXHk5DwKK05oB6SP/jX/GfNzW6FL3l0D0HYvwZ4t4YF8awLDKkJ0 24jYNK55R7riNTOW4plL/gHvaIhj6u/QxHkQP+T6CO4G0XugMdDHAPOFSBSGrHIWzDrR SOlKf/jcIn+DY9A8dSf18HOC/cLZFbqpXKXCeG3jXnQgQTyXLBfKnkWOvsWFwOrUfuhE lw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2120.oracle.com with ESMTP id 36hkrnb2g1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:09 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CCnacs184188; Fri, 12 Feb 2021 13:09:09 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2101.outbound.protection.outlook.com [104.47.55.101]) by aserp3020.oracle.com with ESMTP id 36j515fn3u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P7nMAxmZmyPiqm2lvl4DO+n4bD9M+7dIvDCaIK7/n9JwmBzSUtwweRkKU9k/JXyO4nUdosp6Geu+IEqs+t8Ph6d6Fh3Y6ErGIwcChcSyfbyhtV5nR25HmQGisuPyJ/fYLNqQ6EzEUUIaOKCNoJcCxZzpIyaZhKOXipb/CLXKfESFuYIVfVsdsJgxhIAxc7KaozANSCkuIqJnldUnHDyNkUyjog6i3lnhFnN2FS1v3wpQSfJ7UI3UcTL/fmfOq9pkW7nm4bQs3J5dNCHebdBFSBIGUrbJrVR77Gyc170Uuthcq7AuOFYwD2/r9ukMy/mjRlEe0GgrHFMC7QKCfTo1Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QBC+NEiAE4J7sT4BZ6CfLxXlcIQg8NpsGdBzSgXbwdY=; b=dzZuuw+rQLVF1zec0l6r+fYh9mQRQxsUCCGgUpaUwhV4C5JpkgcaHnVwAjbEp2lzVk2yKg1VoGIpO4pZT6/lioQlygqX1OXZKINknNaCCFdGY2qKvdHN5UkM9q353/nkmQRmbxidV1pbVLBJLZSomUVH9XJc+CJ66hWgUrbxUQX+WOVOzw52yFyj08wSx60DMZoPk/h+LcMBJzcMPK5zOha+bFM3ZV9YeoNwluKr32ahPOFciJ/tAoZ+WA+tpnx+7m5PV7PRW5frlvQVbCYWkmjOKxTOHIafiOIeoz2u6i56B/kD4h0qpJg4jvVXQ4VAyvzsm2x4WdhaYlGvVJn5wg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QBC+NEiAE4J7sT4BZ6CfLxXlcIQg8NpsGdBzSgXbwdY=; b=dYURuKAg32RhEdQOE3yRETebZPhO+2nLRgKLft2G+H8c80F9ijXlkXkUhguVCdyIUuoGW8yzqT1cjxcdw85CQpYcpSVgQKSsmMtgLHSkVC1U1Yvq2jSdOM2YTJFqkyPUaZYrB3VqzaPvE40/9CsrZxC445Tiu8MbTLZfznlz8+k= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4509.namprd10.prod.outlook.com (2603:10b6:a03:2d9::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.17; Fri, 12 Feb 2021 13:09:07 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3846.034; Fri, 12 Feb 2021 13:09:06 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v4 2/4] mm/gup: decrement head page once for group of subpages Date: Fri, 12 Feb 2021 13:08:41 +0000 Message-Id: <20210212130843.13865-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210212130843.13865-1-joao.m.martins@oracle.com> References: <20210212130843.13865-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Fri, 12 Feb 2021 13:09:04 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b490f408-5d16-4f2b-3c81-08d8cf5760cf X-MS-TrafficTypeDiagnostic: SJ0PR10MB4509: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: amGFWmKdQ14t9r+EdHlYeqwXWGs7aYAqm4BvWNypZCtW5E4mOfSZCgC32gXu0dAVaTrEFzpQAJELo4oAzra8kyBynq/VG4cWevv2OxMubvQNTPrZBmw6tPs4UOMsRzs7Ii4CyvuPSH4/3SS4W0tMfAqHa1KtJv2FU1Z+W87D9BRgT8dIMIhTxfQJY3QnX1jMfkfVhmsbopdNFVSsTelvFj1sMM8O2SwPy+RjYQ9p4IBjGmX/BpqXiRkhP3yFJt1nH9cGWudrGcVOSjmRrnJZdtyTFdwyLYa0JI71BpkyKfs4x2QOgdAvK0nbbSh2295hUnEdfFT5uLcYFedoii2wgqbFFhWvMEcszFWoUwqxVHxMkVPMp1YI6UycsemlqIl0lPi77/3CrVcdS1KPC5KoLBt2gMY/3c7Szs2SOB0p39bf8RM3zQ4b8YzUnGqGp93/k0mzCthGriDKnm1/tz5GNQyCR20sCRilDaj7conAzDZZHcclTmQsRxnW3cPSwYNBjVmCFEgEUF+fyA6taHkTBA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(346002)(136003)(39860400002)(366004)(6486002)(2906002)(8676002)(4326008)(52116002)(186003)(26005)(2616005)(1076003)(316002)(8936002)(107886003)(6666004)(478600001)(6916009)(36756003)(66476007)(5660300002)(86362001)(66946007)(103116003)(66556008)(54906003)(956004)(83380400001)(16526019)(7696005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: cGfekFzzr44Je2OluGb/ps0JKO2F5iC+aI59txIbljtuNnoo4dWjUKQRdDP9P/oeyvPFsz2Y+eS8ob8FCVvfdTIM8AlUKUE66mdh1EuuwxTaU6ABU8CfmVpVY1hYrp/Z5Z86fdrMoDUL5QYH05uCM5QPK78U0yCzU+wQvFEwGQ0oBoTuF/Cife5713CWlahye81cUgz6xmXr7DoCgm0rnqT11N8ZRp84yskOCCBZuvKugPJBahOovQ2Cmoz/jaHbAjqSFJ/OFo+qn62nfBUO4APc4BmSjMD0OVqfeX4o7f6E1kPuEw2fff3abxU+Y7Qugd3oAsUS87JBeTQUV5gYD02ttJkjRXgy4GVNMrK1KworaeudKZBVWmVkQDesBVfTngfNpigx106Arv/nbSM46PmW0Q8oHg5oB2K349bvgXieldmQKzwaEkJgly/R2WvD40GbQdIBCN3A39L8FcYUm0DFoaN2cfi/L4pxUVDtjV8RxXnR477IeBxZFRWQPtVheByjZv9ZOm91B0CwyiNp/3VLYfBhhxX3huk+YUR+t6EiuKuHUEI753cl3JQbPJKIb1QYEFLxeUKIm+qypPNWIlU4xncb6RVwxMSqkU3LfI5FLANSVktvO7EUYk8mK5EeVUh2zapN3I8Cd9Og3qsa9wvFAstmFJshPXCXEDksx6Z6YAP+cNRpR/yLi7LSMJ5AQljmFU+OPcKSaxWc9xQEWPJIBgatMLwx+1IkwREfLzktGrtqpU65fnRFhSEzL8rdaIFEVcNmbkHp2TsSkm93lbhjKU0fiQq5feI9ina9rHOc03QR+oFiAmZSkvIJysiv/0XnHrjXSIWFl7xHkZayfSqeHFkHZgDfLyRqQtjaxoYF5Gb192u3GLOnv+q3mYSLPwTtjd3sj696JIsbHBIQ00eYImvEBFQUEzDC6mMzDF/gnZtyZuYI7ONt+3zIET27qotvd4lOQN8OzsDa7JmzAO3VqMRCuf/FeOoegGJmQp9lH/7ivo9q+zvRWOEmG3fQucBd34GAomDxMQnPAVA68xbqfMJgqDyv6608htxtfb2zkg+/yFgim/mjLAOFC61jSpqMNi4BjIHTZq7qsY/5uttnRjpFQulECya1cInui6jNPAAJUqioptofD9N8WVkRQssPhkpdX6sCBN+UNuJ7C5uzBL/2YedvS5rVdERsNtk3wD31q7UQbofmfqEiF0eiEJ8WQf5vouYQH3jVTeTmIu5mR/lHcuKLpFvI+iljh7sk2jEAGTKPByZsiyVmEfGpOqnWwNZF4e42XSC41VY5yDBqRxKPf9K3YQoCd6KUHI3CsH9ayZ4TxY+4iKmTAdQf X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b490f408-5d16-4f2b-3c81-08d8cf5760cf X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Feb 2021 13:09:06.8282 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CYcx/FKWnt32Qf4vy5VEiLUc31vGoL+tXNenmnRDFl0vvolhJbr1w520tHjWnXIZN0WKNBmtkZkYfmHM91xkJekNeQEngjIZUwHeP9f2qNc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4509 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 bulkscore=0 adultscore=0 mlxlogscore=999 phishscore=0 spamscore=0 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 priorityscore=1501 bulkscore=0 spamscore=0 impostorscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 clxscore=1015 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than decrementing the head page refcount one by one, we walk the page array and checking which belong to the same compound_head. Later on we decrement the calculated amount of references in a single write to the head page. To that end switch to for_each_compound_head() does most of the work. set_page_dirty() needs no adjustment as it's a nop for non-dirty head pages and it doesn't operate on tail pages. This considerably improves unpinning of pages with THP and hugetlbfs: - THP gup_test -t -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK (put values): ~87.6k us -> ~23.2k us - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK: (put values): ~87.6k us -> ~27.5k us Signed-off-by: Joao Martins Reviewed-by: John Hubbard Reviewed-by: Jason Gunthorpe --- mm/gup.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1a709eae2bfd..384571be2c66 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -265,20 +265,15 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty) { unsigned long index; - - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ + struct page *head; + unsigned int ntails; if (!make_dirty) { unpin_user_pages(pages, npages); return; } - for (index = 0; index < npages; index++) { - struct page *page = compound_head(pages[index]); + for_each_compound_head(index, pages, npages, head, ntails) { /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -299,9 +294,9 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, * written back, so it gets written back again in the * next writeback cycle. This is harmless. */ - if (!PageDirty(page)) - set_page_dirty_lock(page); - unpin_user_page(page); + if (!PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); @@ -318,6 +313,8 @@ EXPORT_SYMBOL(unpin_user_pages_dirty_lock); void unpin_user_pages(struct page **pages, unsigned long npages) { unsigned long index; + struct page *head; + unsigned int ntails; /* * If this WARN_ON() fires, then the system *might* be leaking pages (by @@ -326,13 +323,9 @@ void unpin_user_pages(struct page **pages, unsigned long npages) */ if (WARN_ON(IS_ERR_VALUE(npages))) return; - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ - for (index = 0; index < npages; index++) - unpin_user_page(pages[index]); + + for_each_compound_head(index, pages, npages, head, ntails) + put_compound_head(head, ntails, FOLL_PIN); } EXPORT_SYMBOL(unpin_user_pages); From patchwork Fri Feb 12 13:08:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12085271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F179CC433E0 for ; Fri, 12 Feb 2021 13:09:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8CA0964E57 for ; Fri, 12 Feb 2021 13:09:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8CA0964E57 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CB1DE8D0059; Fri, 12 Feb 2021 08:09:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C188D8D0057; Fri, 12 Feb 2021 08:09:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8CC58D0059; Fri, 12 Feb 2021 08:09:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0085.hostedemail.com [216.40.44.85]) by kanga.kvack.org (Postfix) with ESMTP id 85C448D0057 for ; Fri, 12 Feb 2021 08:09:21 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4BAE31E1B for ; Fri, 12 Feb 2021 13:09:21 +0000 (UTC) X-FDA: 77809646922.05.9AD4427 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf15.hostedemail.com (Postfix) with ESMTP id 5B7FFA0009CE for ; Fri, 12 Feb 2021 13:09:20 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CD8h39019057; Fri, 12 Feb 2021 13:09:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=pblMTxugz0sthwIAXE3b5+P7tukiVSJBSWzcnB9s24o=; b=NqKENkFVYalGUvP0sPkhFHX8EzjHBquWqOj/eiVDHCasJqItmsv2ZIZkqZcrLToY8vJg d4Ye3K/WnSPtklDaZKUZor/o/RQOKBCj3RiMup8sDcWzsDCYGEOVKv9khg+kWgHMoWAt AaxeIS+A3itIXq1JL6rBxOxU7oBSnjkYG3V4pUC9kUQM4ZE3NF+Hdf1HnM3R3xEG7O/+ ftyzOFGeyH388IPZSH5v/4zR7oEedGCHbuNra9nKX1MoifgOyMd2jfe4kcD2+VStVc4C 6WMyJ6GT8FbOXbTl6dLEmB1WSei/vgPB0Mrn3yP029aihCqVwEI5/PuhjJhdB8CNdNcb iw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 36mv9dwadn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:17 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CCopAr195735; Fri, 12 Feb 2021 13:09:17 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2175.outbound.protection.outlook.com [104.47.58.175]) by userp3020.oracle.com with ESMTP id 36j4vvq2mh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MO96dI9+qtmsq6PWfO5qY3G3VQH7RFqRGvdBLfmVUlHACixhl6/y5SvbMTr/Zsxw8P8GgTpenrEwC+WwReaQZ7wIgdUNyZS3n8Kh/SpxcuZ3cgnzbfph2wpN86pM80lFKlm6dLnReGFtDDF9iLSec/hoTmFz5ZrlkaBZ8vn4tBpR9Hs2nUDXR8W7wFmMJgxGympiQOpP2oBDUBb2sjELzLRetdgMtjcmf0XNJP9KDCJHoREx0hZimjOmsfH5gVFE+xv8IZI5V/4fIIBaKPuTUBK+YIkNxKAVZZrOk4TICSNgatG/oNESsu9YHURBj6cIazh1C0ajogMY+XqIPw+Law== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pblMTxugz0sthwIAXE3b5+P7tukiVSJBSWzcnB9s24o=; b=UJxUHHoCIdhsaDGK9aPdijGwPJckskF7uhsivwjhQ04NpMEAz0hl1SRLXrKrzdF7mirvXwmXiWRQ3e227Q15Bmw6fU3MnQO0xzSCAGxKYDxX0pcuS4TbfGVPJd3eslSemASeeu1gPW0N+aoZ8663BPnBtxjQ1aK7IcfuZ5qbB5DO/gl9RBPMiBxcdfePsbMuyeV6YQ6YTefAl6qyTNo6os/b7OGH+9Gm01O0NO4oeum3QX5z98AeMGqYaQo+IlMltliOd6Z/C85NwwqCtU78KYHBKUvwV+6sK0LvvkciucLdqetPpTSckL2xJ1XP740PyRYOb2dPPhlCDH4+PwseyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pblMTxugz0sthwIAXE3b5+P7tukiVSJBSWzcnB9s24o=; b=SJaS5q4Dpwd5n0d4H5xhtwBT2P3wlPfqr7ZFCtZlF/sXqbhog5eqzUgpQcXucI0qRNIv3WhgNi/tgVhvYx25wVJ66SGRneqQT/ZS0lGf0z1781tnMG0vEMRJLvDOJA5zDUEFXokcI4TAR6EWSf+HzKnp0FuyXOpROWTVZ+9jka0= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4509.namprd10.prod.outlook.com (2603:10b6:a03:2d9::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.17; Fri, 12 Feb 2021 13:09:09 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3846.034; Fri, 12 Feb 2021 13:09:09 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v4 3/4] mm/gup: add a range variant of unpin_user_pages_dirty_lock() Date: Fri, 12 Feb 2021 13:08:42 +0000 Message-Id: <20210212130843.13865-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210212130843.13865-1-joao.m.martins@oracle.com> References: <20210212130843.13865-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Fri, 12 Feb 2021 13:09:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dc65f803-6005-4089-b9e9-08d8cf57627f X-MS-TrafficTypeDiagnostic: SJ0PR10MB4509: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UjQn49ElAuyGDSrCEx7UYykwI3xZDRXTmJQ+Ttl7heqUA+vQmbRL7r34cLIkzMyI5DqO5ZdoHWktEzPlaOuOFM7w14aT7AWr83WIETILiNkT10fIZN0+e7Ft7H4YvSfrf5r5Magot8oJsGJ5z5AAIiZzRP31ed8uDADeF7LX+iFvYFYDTKa0MBy55g6gOiGuC4ztSpxy46fbquPtT+pZpGyiweRxbR/nccg2U2RWdubXIQRdTPnApwToRLuFQ8JkSBubl+24aHaKzOftdpYRk/1LORE2uZM+5OF+2j6iuxPfKzqDVEYPOjAIz4D71LXTksoG0wEY+vbul2oSA+7Q/KdXwBIPixuB759EKe8baMhZdwGPp/lHDujktzvY0wxjkG0xKRpiEaRXWvysyXM4KiDa2lZmxQYrEdN4HzIL5e4iw3M+B+NqvW1ap2eEN3PrUtDs7qrTOYDT+9T6g/IDokRYoxgkk7+0w1Yf+Juz5eklYabHPd1ef0CUBQkTaKc9k5bnfssvXpY1579Jw+gk2mlKfV66UwbUnfXEA4/aiOZ7ZJDHDkcFugOzaYyAWzUv X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(346002)(136003)(39860400002)(366004)(6486002)(2906002)(8676002)(4326008)(52116002)(186003)(26005)(2616005)(1076003)(316002)(8936002)(107886003)(6666004)(478600001)(6916009)(36756003)(66476007)(5660300002)(86362001)(66946007)(103116003)(66556008)(54906003)(956004)(83380400001)(16526019)(7696005)(14143004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: dtPWbyBZAgP6Iy1+EyF8kDi3dYBQPfi/8fw+ABexMWurzoZAeW71NOdIXnFMZ0JrBlq+J30UMb/SCySSqRQVyW9j83dvqYccLWI7JwoltbUVLiwdXVH4Z873P+to963w4mCiYOBILudeG1doRElw8bew/WHb+0NJgAFVQErrxSHkYiQTJbJuYXftcrL06FO3KQyCEgJB+nfP7ePQZ8W50jf+9SW+Bae2p0my1lmFi1wrMWAI520FRhE3BH0ms3Qc2kYxaa0qN5K7bdVBxqmEcuw3iTTWfVCNZfd2dB4KPpoRKy/4kJbtnMHmDdgB00nt8W9ujkM+D2z6TeRbQLoSR1sWW+ab3JG+2ZyE6lfdx46uMxsn7Okf+0SNmBGqIdzEV0uwN8dnepSYZqVJ5hMtlXhIr1byD9VtSv4gI7DJnujqK2XVsZIz/UzGjVe6QiCg4NCj/7trlOIbflW/xnj4xarbYRpceaQYiMMrRvW7fQysjgrwzSs6De6LcMMvy3jDspXQ8gLynx0P4wyreMKoUoSByKEK5YSXVq2tHthi0Pvh0aB8c3XildoDAi97SSVtlaFRv8k3oUkQoXecURlWhksuCAHV0NGIh7rejt4Dy1oMqJsN7euqwf34jnNq/Jt49luCxORkYPiw2LCRG/Ll5wFAbTQiuuXeETQcEE+b82WQYyBpyDniqp23DPA2FEZCJe0ebtE8yfJrVMx5mDbRG/MX7zJrOCJXnZYLpOPFK726BXOmf79R5AbMc7LXB7qtnvRmb10DAp1KCQNXnJVYUWGinKTd3TWmqAHZs7+SLzz8wCje+E12rIVym09V8hzdRWB5a+hS3IgC50qFhj9xmUtoK/P7etT3LQYGTZgYodMhHtoBHmsvc/+dvppeYcMCHOgEfn6QY55uO7e4o1SOcm5VAeSR40nzaIPsdpV44LvQtEb+gKeUD1QAC3Tkes9S85WXHkbnmGYDGQ5n9N8vU9dt5pQS6fEkyL6bgN1VyAR87n1iwG06KCotZFoghmk8mzoAR+jf/LAXUCxn81MlEY9WK0oI7s1tfIDKwFOmnTu580V18TYILTaMgZ6k9cPIjplNeXlAx+W12J3ltStHr+eGfvQtsDT2/0W5sz7Nxrd2HEvE+1t3YOX4e8HFgwpCfi3xJW5UMzTW6gwygJmajygat+gauapHuUWDsltstRbw6bBgIjYDtz9afrnzdRlFPch276cGtFgrMcp6fEzhSMEw4mEzvnc6nci/drsi0i+KzMVGyQUE8wk7drw02CwIkd/3VkK5wFgxUmkADGrPZFjk39XpQesw875NEGgaPhbIqyk0XS4JcICrNaSmTK84 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: dc65f803-6005-4089-b9e9-08d8cf57627f X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Feb 2021 13:09:09.6286 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PgJIwIhovVEDg1Ilr5xH9mFBDpMY673ljAs9svP57U3ziCtWNAX5Lq+tlosqMPM9Re5NGsav3NNSfEhjK5wwjCT2qBhUD703A2he+Idv5Zo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4509 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 mlxlogscore=999 bulkscore=0 suspectscore=0 phishscore=0 adultscore=0 impostorscore=0 mlxscore=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120102 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5B7FFA0009CE X-Stat-Signature: ip71qkjop881bjeikiy7kgud8f688zsk Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1613135360-479324 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a unpin_user_page_range_dirty_lock() API which takes a starting page and how many consecutive pages we want to unpin and optionally dirty. To that end, define another iterator for_each_compound_range() that operates in page ranges as opposed to page array. For users (like RDMA mr_dereg) where each sg represents a contiguous set of pages, we're able to more efficiently unpin pages without having to supply an array of pages much of what happens today with unpin_user_pages(). Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins Reviewed-by: Jason Gunthorpe Reviewed-by: John Hubbard --- include/linux/mm.h | 2 ++ mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 89fca443e6f1..43f59e4eb760 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1255,6 +1255,8 @@ static inline void put_page(struct page *page) void unpin_user_page(struct page *page); void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty); void unpin_user_pages(struct page **pages, unsigned long npages); /** diff --git a/mm/gup.c b/mm/gup.c index 384571be2c66..6b6d861727cf 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -213,6 +213,32 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline void compound_range_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + struct page *next, *page; + unsigned int nr = 1; + + if (i >= npages) + return; + + next = *list + i; + page = compound_head(next); + if (PageCompound(page) && compound_order(page) >= 1) + nr = min_t(unsigned int, + page + compound_nr(page) - next, npages - i); + + *head = page; + *ntails = nr; +} + +#define for_each_compound_range(__i, __list, __npages, __head, __ntails) \ + for (__i = 0, \ + compound_range_next(__i, __npages, __list, &(__head), &(__ntails)); \ + __i < __npages; __i += __ntails, \ + compound_range_next(__i, __npages, __list, &(__head), &(__ntails))) + static inline void compound_next(unsigned long i, unsigned long npages, struct page **list, struct page **head, unsigned int *ntails) @@ -301,6 +327,42 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); +/** + * unpin_user_page_range_dirty_lock() - release and optionally dirty + * gup-pinned page range + * + * @page: the starting page of a range maybe marked dirty, and definitely released. + * @npages: number of consecutive pages to release. + * @make_dirty: whether to mark the pages dirty + * + * "gup-pinned page range" refers to a range of pages that has had one of the + * pin_user_pages() variants called on that page. + * + * For the page ranges defined by [page .. page+npages], make that range (or + * its head pages, if a compound page) dirty, if @make_dirty is true, and if the + * page range was previously listed as clean. + * + * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is + * required, then the caller should a) verify that this is really correct, + * because _lock() is usually required, and b) hand code it: + * set_page_dirty_lock(), unpin_user_page(). + * + */ +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty) +{ + unsigned long index; + struct page *head; + unsigned int ntails; + + for_each_compound_range(index, &page, npages, head, ntails) { + if (make_dirty && !PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); + } +} +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); + /** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released. From patchwork Fri Feb 12 13:08:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12085273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 441FCC433DB for ; Fri, 12 Feb 2021 13:09:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DD4B464DE3 for ; Fri, 12 Feb 2021 13:09:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DD4B464DE3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 955208D005A; Fri, 12 Feb 2021 08:09:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 904618D0057; Fri, 12 Feb 2021 08:09:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A73B8D005A; Fri, 12 Feb 2021 08:09:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0035.hostedemail.com [216.40.44.35]) by kanga.kvack.org (Postfix) with ESMTP id 59B4A8D0057 for ; Fri, 12 Feb 2021 08:09:22 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 27E4E180ACEE6 for ; Fri, 12 Feb 2021 13:09:22 +0000 (UTC) X-FDA: 77809646964.09.602F32C Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf14.hostedemail.com (Postfix) with ESMTP id C9F83C0001FE for ; Fri, 12 Feb 2021 13:09:19 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CD8pwS019100; Fri, 12 Feb 2021 13:09:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=6oemqd8Diiulu5DhGb8pGtVKtq3abrahSXvKSZWQwI4=; b=iT81OQbSznwacNEdP8nIloLSv2VTaw17TKwH4MLO7wccNEUL9jNIXUHMTyOg9c6yu7z0 zwpyAlxGq/xMkv+tGNSWANiVWocoaiSUGG3g4kAjEQfHIgjA+13lC2IPHd3zk26UEM3W UWCMDCMJOfWDSYJS+2cH542R1ffeuWPeoCHRUJCTYtYC1tsVUsNI3XS3zDxvDSgwe7Hr FX0pkZVEoUMXfaqjLUz/gWsnttoF/WIPBdCuKSBCy4EuR9Txca+F1744wN7/boMx6gpu X1oBlLoealzix9GBshOaWKIdDRN7MsWEE1p0DqJHtEzP89+N+hXJDVU1UzFXIIcriZrY mA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 36mv9dwadp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:18 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11CCopAs195735; Fri, 12 Feb 2021 13:09:17 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2175.outbound.protection.outlook.com [104.47.58.175]) by userp3020.oracle.com with ESMTP id 36j4vvq2mh-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 13:09:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b6VGAlEDjWSFkEOg4la1+DQKlEch/0fMKVc8A42DyPOtljGGRPO2TBA98oj/Wx7hOuMgxKNDGNhyeo8Qj6kaXSpl24+LHyO1eWDrUAI9j3ctxyf3N+YdAN+ghNhX2XNWHsXSX9KbL5jaFKQb24qsbzx3Vqb6x5yfQAE0BlnOR5hfh/kwUjRrQQpD+A68+4JqFc4dEKOo0zZK4nZtISW7dH+OxGfSacN0nQrV09xnhuwjSTQQopbEDxSIXULLYHpmXSwmWFzQcEmKcltarx/B6hIKtrUt/LdBTusvM8lJV9snjaKCM5h3sRePIxQdUI/+5TgJ5VEUNyuXZAmGtaoxlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6oemqd8Diiulu5DhGb8pGtVKtq3abrahSXvKSZWQwI4=; b=of0Cuh1ArnrWpTNfvfH9ZCh8OCxxdxFRVstKEMtV2+8mKXwKL41n5Gv4XuZuhXWjcT96JzeBIUF0DBUlZdTSPLpSMUpG3t8I6QZUItT4cMM5psTFagyCNzkuyg1MTXLw1dHe3mdZXlMC1elWu/mEN1kIJknBbBiIo20e+KmjMkuvE4ataGop3S+4WuVQ8Wd4cOkWwSTes8Cy6gi7YQ3kkx62cgtLmZg2UFAD2M1SUOZc/Wk0gIHtunYybOwovcVEeth/AeG3x3KVsnSgGNXYnl2dH1+iLxhGT6JddMmvdMkAsPLWIQcUKS1EP4wqDfPMbgDBf2MO3Jj66BoLWPVM9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6oemqd8Diiulu5DhGb8pGtVKtq3abrahSXvKSZWQwI4=; b=OF0Z4KCslWt/8z1CRdoxbgL6qZVhrh6I/tXEZyNtQ/fSNUdL1BS8yj7tWwp7y7LmmVGikEjGeArmaqXL/DxOsMX4lMqyVWpRaeH+sOE5I6+Uw+bYFxoPFqA89uX0SvsfIUI4ztQyE6axt0aEVJFGhTrSf03NrXuBkQX3kA3jDXw= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by SJ0PR10MB4509.namprd10.prod.outlook.com (2603:10b6:a03:2d9::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.17; Fri, 12 Feb 2021 13:09:12 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3846.034; Fri, 12 Feb 2021 13:09:12 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH v4 4/4] RDMA/umem: batch page unpin in __ib_umem_release() Date: Fri, 12 Feb 2021 13:08:43 +0000 Message-Id: <20210212130843.13865-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210212130843.13865-1-joao.m.martins@oracle.com> References: <20210212130843.13865-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Fri, 12 Feb 2021 13:09:09 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4cbebe4d-b4a6-4dbf-118a-08d8cf57641e X-MS-TrafficTypeDiagnostic: SJ0PR10MB4509: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:370; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 37Vwgp8VV4VdwqkYjf6y/s/AxaXAhcPHMOq9oLDHrILgBa08TXuqQylUCkXOPfr+Qkjk7nwCtjRCQOpyLw0kQ0SBNnt5DmGzNeFQEAg2dFOl0WnDrENl84G3sw6qR/8C4e+uJpycH7CuLhavP4ZzdXWIJ7kbQfSctzAMTPVH2SHYPwtEbvd3jZccce0KYoMMdQzpeG9ksjhoy2SXe2yOHptp5OtwMMzNsVN38AP9X8+G1WkLfOoviBygPAo2Wk7hGhGQe8szaaiHKFZWDlINKXa5vvDdHTWihkb3JAdU2JsLJ+QGNOnfquiNnCo5tFgTVbtC0fzYWgOgkIhYRNFEJoGOTJQY94HLCDELzu6XUueHf72SaHlY3oivXUSK69W0dfmDbhd+SydMdLoDofqcbJJAzxJUyrxTbS8HtBtSXY3Zk6BZlPUQ9FTDPUXk2GvTnhI7VDTjNdQ4Mex8CFCUU8BP4xEeSlqwpHqJ6JWebDcu+YDM0VzMQv5ezP7xeHGGvf3UGRztRjuMTvnwlYiseA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(346002)(136003)(39860400002)(366004)(6486002)(2906002)(8676002)(4326008)(52116002)(186003)(26005)(2616005)(1076003)(316002)(8936002)(107886003)(6666004)(478600001)(6916009)(36756003)(66476007)(5660300002)(86362001)(66946007)(103116003)(66556008)(54906003)(956004)(83380400001)(16526019)(7696005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: nccj6ngQJgArfH/nqd/JuQ24PLwsNdKXg6T5KF/bqumXqL4O8M2X6ZcK36N48vOkM3rjlpOvS6yWw+NU1UVMDV/E3IYEpu/oHPx6c6p5PmjY4HFI78HVOXmCAe2LtwbPd0twNACG1N8PTf1oSNph+KfV7sRHUXsyyaJQbDf0GdA7xIrEl16DqUE2tNfbafNgH+Gg7LseSqh2K+zmv+CWwN5To4TcPzwgz7BT2m4wx80KI+AOOA3UfI1I+uEA/SV082hHTUecYscLzS48qjIlk1QWt80y56G8kimttmPCl4sKVVAxoySabjJivcMOO8ySfLSvuFFuHwsuwWsfb/QDPYvkG1CTBRLKlXWY3QoZZqeBF3hLFq7DKfOkUhKHkjo0d/fBu89a/dsv79cLwGTHvexGhD02XITh1w4PIybWOwZu1FZB8aBDxGM9JCgwSPKEyvMYJennK+DSbwPyD8JqIbFDG1Jzqe0asQF2a1+Faa+nU62zjEe9/nnB/zCmdH1FiyiDHB4PR5U9HXoRQPt0AVt/KTpj+8lDRJK7cLEnv+0o98NeXKxqlHmlVL0H52QGgHjYr2GJ1JevqLQ7D3zqqlulMY1AxB60SzEeuwoApYax3uaJTsvGlG/gEMpOBHUGJi9l6X9iM14RWx0l6UkckroD7Ig+/ocpLRQhuqSXyL6IvMAZP+QGzLIrMOha9ToqWPPXkYMU7hJi4aGgZQRO2UC+flM00grrKUf87yEfbh7hoC6SKDXw++uFeO2c1AZmoSlGr7OuIlIR3ufCH1r6LPNy+FLdV0FhtgHv+WvWnbWA/p/mW8IEZJ6qcnwsNLCBgZ2O6qD5lJo7fG99cRDL7XhW7qkx1aAqLSw9oJnRw1r9p/xAck/Evt7fUWh2M+bQ+GfVbIu2GcG7NHHqV4AtNnMewoJC5Is3OuP3U6/OMzqvkk1w7bDUy0hXeGJGQUOF4RscsjINZeG2c8yNM4gFTX+rqcy1YuJBmbyWQol1QNvkHkY9iMsAmzl6izznHftC+0VCHNCR3DNcgOeNsFXCrYVPKDDVBh40ycwa+NPqKqJFUHhxgF810K8/i4deM+RMAMmzqfoNMrVpF9J+RvO5OCYt2IsqBoghukzUDStCOz0pBiEkzTwaID2pEdyxhxhQo0BWAZK+gTohF1WpzOaB95LXMKkpIGS2X4RRvJlO8EP9I/LMAai1EB8YKEe9GXxMv80K0A+GIzBL4gVEBHEeSKD7BUbEkVVZzdyhguSjJadKDQ6AlENmwBRCPGEKo1Dpkxc4f9ujx7O5PKhUfRBu5FP1/OMtab8e2K2ANpCGlU6Gqj5OQAHWPs7OK8edL9gy X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4cbebe4d-b4a6-4dbf-118a-08d8cf57641e X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Feb 2021 13:09:12.4510 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wEKN4pxQynWnvWfLlyDdW1WH8E202yWloEoi/C+srOcJuf7FQdaBTdYDH3ryPRDgFKUeXkQusmMIHZNriXpSdCXsU9/cUqJok/ZIu2RAIww= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4509 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120101 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9892 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 mlxlogscore=999 bulkscore=0 suspectscore=0 phishscore=0 adultscore=0 impostorscore=0 mlxscore=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102120102 X-Stat-Signature: y8118d4emy7q43e6r3uo68euwd9cgans X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C9F83C0001FE Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1613135359-798654 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the newly added unpin_user_page_range_dirty_lock() for more quickly unpinning a consecutive range of pages represented as compound pages. This will also calculate number of pages to unpin (for the tail pages which matching head page) and thus batch the refcount update. Running a test program which calls mr reg/unreg on a 1G in size and measures cost of both operations together (in a guest using rxe) with THP and hugetlbfs: Before: 590 rounds in 5.003 sec: 8480.335 usec / round 6898 rounds in 60.001 sec: 8698.367 usec / round After: 2688 rounds in 5.002 sec: 1860.786 usec / round 32517 rounds in 60.001 sec: 1845.225 usec / round Signed-off-by: Joao Martins Acked-by: Jason Gunthorpe --- drivers/infiniband/core/umem.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 2dde99a9ba07..9b607013e2a2 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -47,17 +47,17 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) { - struct sg_page_iter sg_iter; - struct page *page; + bool make_dirty = umem->writable && dirty; + struct scatterlist *sg; + unsigned int i; if (umem->nmap > 0) ib_dma_unmap_sg(dev, umem->sg_head.sgl, umem->sg_nents, DMA_BIDIRECTIONAL); - for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->sg_nents, 0) { - page = sg_page_iter_page(&sg_iter); - unpin_user_pages_dirty_lock(&page, 1, umem->writable && dirty); - } + for_each_sg(umem->sg_head.sgl, sg, umem->sg_nents, i) + unpin_user_page_range_dirty_lock(sg_page(sg), + DIV_ROUND_UP(sg->length, PAGE_SIZE), make_dirty); sg_free_table(&umem->sg_head); }