From patchwork Wed Feb 3 22:00:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12065531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BDB4C433E6 for ; Wed, 3 Feb 2021 22:01:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CE8F064F6B for ; Wed, 3 Feb 2021 22:01:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CE8F064F6B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 518DB6B0075; Wed, 3 Feb 2021 17:01:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F0F96B007B; Wed, 3 Feb 2021 17:01:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34D056B0075; Wed, 3 Feb 2021 17:01:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 1670C6B0075 for ; Wed, 3 Feb 2021 17:01:11 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B77233630 for ; Wed, 3 Feb 2021 22:01:10 +0000 (UTC) X-FDA: 77778327900.11.noise83_2018127275d6 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 92F41180F8B80 for ; Wed, 3 Feb 2021 22:01:10 +0000 (UTC) X-HE-Tag: noise83_2018127275d6 X-Filterd-Recvd-Size: 11413 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 22:01:09 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113LxkTu136679; Wed, 3 Feb 2021 22:01:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=PoCciuS2kupNF4/nFmW0PvArn3uE1gW//0DfhSrI6Go=; b=jwmhOvn7kvdBZGUl1r0cKHDXE0V/Ot+z7YMHsNBKHtCwBCaw4W/BTiV+68TnONBNj42L G55QyvIdCilT/6YbQqDoudvgrfa2Rzd+hfRkdlFW+894e5+5wERcvHEOOF1Hasq392Gv pfsLCI53Pwyee9tmedz9F/t54ZY6GGwj9lbAzOYKl1pXesSol1zB+CGUI2DcpiXabm/K gn5OwaeMI0eqLDozrqDfd3PjDHfeBqZYouRXtqjwirdXxgIfEDiAe2hBn+Jp0nrsalHA zyEFTtP+/K0b2yxpMM+iRntMFMRkdNZtZInEdYr7ek55WakwIVY0gz/N3F3T3x5qupwt jw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by aserp2130.oracle.com with ESMTP id 36cvyb2mfw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:03 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113LxaVH097865; Wed, 3 Feb 2021 22:01:03 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2107.outbound.protection.outlook.com [104.47.70.107]) by aserp3030.oracle.com with ESMTP id 36dh1rjjp3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mea4cQ8//ObM/rzBMQe8G42ywOMZnCeq+IgnJ6LIP7HOrEg3i7/hKkbEBfZex6Gm9lveTZexVJIrJ7YgAAUQsO5DSXtwtFvLC0RATP27aMFL9M0YKxFlZ97RQk58kWZRs6buolVhVfCKaIskU3iIope1q7NNSW7h9wa8ZEcTF6a3xaOLozSUbI82qFdaczSVzDCDRoa/ilUtDl64rV8tBb2C5CHqXEs5Nym95IsD6b2ew8aUO0XsYOWM8/W4NE5W+IY6lVZzD1nkOrfZSqP/amnAWJk8FJtQF8a+M9JY2HYBjjLum8F63cO62U858ThY/13r4DVFA0ev5j4UHFj4Iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PoCciuS2kupNF4/nFmW0PvArn3uE1gW//0DfhSrI6Go=; b=TQgdaP8ECOlusZtyhPdVQIylYt02ekXd+S68z/P3uaoDn/+25LOk74CeAP7QEZinzogg33l3tYoBRVmpqle6bC7bMzD0/OlJO+TgMDSaXWH4xldo7U4hj+Gdg+wS6xmzdwSPMFuboQBohXQdTuXfzrSanoDTYINvqWdoDbbZj7h5ra+1TTfxMkQSpsOdO8PUtugqbZ4iiLkJ9HLBPnYaSyDIQNXnoqJ9NnBHSszggNnWbHmHNXRgPWNKvOLv7syWn66M9xdhgP9uVi82evZ3MTXICNmEIVfLnCdOgt3o2QUp9uuN6yMs8NR21krIa+vyY5qy8QMGns9Bhpf/LH9kUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PoCciuS2kupNF4/nFmW0PvArn3uE1gW//0DfhSrI6Go=; b=w2U6CrPmdoOnqc42EIeinsHPx7RQWPROVAKnE7e1XUjuNr4ky2jw1rlCy7XPvZ7SRYFK9dbInanwCpdEUfqXGg1yIr0lK9/lPeiruyAztZ5AhisaZIMSnP8EF5sr4a1lESS6xWNpByH7kEN0fSOiZkgYvr788nxZuiScUuW9ATo= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.19; Wed, 3 Feb 2021 22:01:00 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Wed, 3 Feb 2021 22:01:00 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH 1/4] mm/gup: add compound page list iterator Date: Wed, 3 Feb 2021 22:00:22 +0000 Message-Id: <20210203220025.8568-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210203220025.8568-1-joao.m.martins@oracle.com> References: <20210203220025.8568-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Wed, 3 Feb 2021 22:00:57 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e4bdbfab-c560-4844-1eb6-08d8c88f30f8 X-MS-TrafficTypeDiagnostic: BYAPR10MB3077: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:862; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WJrZ7uKGQ10Zmba3t9DlCpddQ22guFqyaB6BX3YiN4vODdyNyKK3F7GKjb/KjQBQlABcsdMPfQEbkJhyV/xQRpK+pPxbqideIqd3grl01bLPv2k8P5UCrIUMIFkj88tuf1qCn+Mo019evX7IafDevF16CS5UTSKjXHpX2R4RIcCbnIz+OedjaYdMAv5EJ/xPacOogCoiB1McXScbv/CynAK6bzAm1HGMMTd1KGtsqLw3NQwIJfl49sXfPN6TL6fwFo1NEUOU/zTyjm3yMAtxNLRG69EY8A/LKzoj6jq6JaQNVI4L9zziA5cMlOj+B2Lg+AvWHisvp/e3SG+qBeogathSjyE0Z/Xfg+UoB1+DV7mxEtgGkOy7kdrA0JQ4kFwg1ItH7nEGA58NkA3UMg7iHA3s4XOBiPsOG5CANkXlQOEab80gukGr677Rp3iui6LpvMVa9DXOr9HbonrGRoAMjrj1DyLSB4Ao+Iw7fhij9J8I8VKPAvSz4J+7R5Xj2N9REQo5OZ/MK7UJp60Qi2OPKQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(366004)(376002)(396003)(136003)(6486002)(83380400001)(86362001)(52116002)(5660300002)(1076003)(478600001)(8676002)(107886003)(4326008)(26005)(36756003)(2906002)(103116003)(16526019)(66946007)(316002)(54906003)(186003)(66476007)(8936002)(66556008)(6916009)(7696005)(956004)(6666004)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 5x4exipdbRoT8Lluf0N0SIDn8L63Burxf95n3RUmBC6bzD4/LKSKmT2Ao4Dr8SQPp47+zvLHvyBhxa3PA1lrUpyLK0GDCaAvFf7BFXUGqvkg9gDGzSHFYOcqxka72EC/Qxu0OM63M0Keb/QeMfobWRoYk7/rLNNGQ1okDTG7zqSDytupMtgOu8lN7psdZKuPlqLYWdtgEOkfebMQ0I1tufpCrN0on2n9koULek/bwJQpco4F7EpPY6BRv5SOssw69l3iMedGqSYThJCO6K5ub5fidw8WJZZguaVGMKxAmY+JCLeNfYkSBHP/xSzApc9KkfOfNUI3JplAp9qm+16AR9eSxNZ1idD8WQpQDeytEEUUN1s7CuCQFkYSarUGJ7kC2PxlYuWGOLjHdyJpBEq7RNu3cxuBZ5t91bJU5dVYwpS4+3oNJ4ih/njxz/t9kqR1NiMaJ6GPTLmphHEapOt+ve3JObgFarZcB9zRElqOeo2gjbJ4MboOPLiQlHhccsr+YYkqv+IgNw/PDF6O1qmHDEQOFgymU7ADNp4DIBOwp3K0lta7Ry7hdIzBpDDSxgmX5B9NcNjJiza3tMm0CMd/P+h3cDktDeyGa7bNlPF68+BXJFUsv8ifSU6mMqfGGe34oZofkZOREE40kD3mFMNcRI7pLiMhCI8IFQBqO0WMeCqw1QCbY2fREJE6Bg+3UgYQw9TQ9qe3AL0F9KCApHQTWPRqHirJXsJvHmO0k+eNoEtXyENKr3U05Vy7vTnDvW/D6lphssH+MGWHSa/iP6sA3S6C8lHrUwKhU8LyfQOfkuvCEz3X/cfTVg7UBc/RqLSynHh7m0pQJu7bU0MRKeNoCQcsCBN3CwUtA6eajqcYcDGjtVpxOQgzqRCWcE1eeWNOiUj8xx+1UaQ9ReNgnJz2HOe5tg0p8xo1DGiukI/fve1wBddZr0KLqITrhrAtaNDZFWVMzh9RKMmMzbZgATTkhqjuuWvwgt1oAeTRCsCuatOQvS+yecnxvSigVhFd8L41R+2UOAv3+A/0nfV69uHsvStSfFcBEpKwYG0f0JPggnK/yXHCH+ND7OZbg2HmMDjQ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e4bdbfab-c560-4844-1eb6-08d8c88f30f8 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2021 22:01:00.4548 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Zfu2QDF7fcp27B1oouKiVkqoviOQP3pQBScleY06nn5+f3sPzb7Rvqdn9WA/YT7/i0H8WNg9kpQpEEhUqKsoTJ+yPnQplFmI9tlHe4/VTAQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3077 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add an helper that iterates over head pages in a list of pages. It essentially counts the tails until the next page to process has a different head that the current. This is going to be used by unpin_user_pages() family of functions, to batch the head page refcount updates once for all passed consecutive tail pages. Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins --- mm/gup.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index d68bcb482b11..4f88dcef39f2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,6 +215,35 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +static inline unsigned int count_ntails(struct page **pages, unsigned long npages) +{ + struct page *head = compound_head(pages[0]); + unsigned int ntails; + + for (ntails = 1; ntails < npages; ntails++) { + if (compound_head(pages[ntails]) != head) + break; + } + + return ntails; +} + +static inline void compound_next(unsigned long i, unsigned long npages, + struct page **list, struct page **head, + unsigned int *ntails) +{ + if (i >= npages) + return; + + *ntails = count_ntails(list + i, npages - i); + *head = compound_head(list[i]); +} + +#define for_each_compound_head(i, list, npages, head, ntails) \ + for (i = 0, compound_next(i, npages, list, &head, &ntails); \ + i < npages; i += ntails, \ + compound_next(i, npages, list, &head, &ntails)) + /** * unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages * @pages: array of pages to be maybe marked dirty, and definitely released. From patchwork Wed Feb 3 22:00:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12065533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D8EFC433DB for ; Wed, 3 Feb 2021 22:01:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE9DF64E4E for ; Wed, 3 Feb 2021 22:01:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE9DF64E4E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0F9206B0078; Wed, 3 Feb 2021 17:01:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AAAA8D0001; Wed, 3 Feb 2021 17:01:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7A306B007D; Wed, 3 Feb 2021 17:01:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id C069A6B0078 for ; Wed, 3 Feb 2021 17:01:11 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 85F0A181AEF1F for ; Wed, 3 Feb 2021 22:01:11 +0000 (UTC) X-FDA: 77778327942.18.farm91_5500410275d6 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id EBEAE100ED9C4 for ; Wed, 3 Feb 2021 22:01:10 +0000 (UTC) X-HE-Tag: farm91_5500410275d6 X-Filterd-Recvd-Size: 12882 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 22:01:09 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113LxkTv136679; Wed, 3 Feb 2021 22:01:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=vzB97g62e/yEZSWcUFQa5IU9s8oLQsgB8kq9NlEZFvQ=; b=HTwTbtXS1k1/mfdr473Is2ZLsSbpw/LW0mEedRvVryuhTCCFGziEH4Qp3NwdnFYo51Jf ymu554pdXMK7n6hQ9ILnWxhdd83/+554E5Cutztx8bpP2SA7VG8zMrzR7bNLxEk8wxt/ r+eyAIVLYnLsUywv9N8LbSNnCRPQ0AlQ1myPmPdjI8n9XRE6gUVT18YtLn6VScoClIjN 8k8p7mkO4NVtGMwcw6pwRG6MkNuYpYlnd0XQCtdrcQ6duqRkpRsjxUJdzY6LOTN8nqGM QQoELNJRbJqM9zLmZpAYVe4h/HYiaGpUVlFbnN0IHG9Px7AM1yuzT2sMnbAJZurx3M1s LA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 36cvyb2mgj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:07 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113M14Fa042767; Wed, 3 Feb 2021 22:01:07 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2108.outbound.protection.outlook.com [104.47.70.108]) by aserp3020.oracle.com with ESMTP id 36dhc1t3at-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KiDOZUqstscJyHD8aol7jbL4c+vLI7Bm8FVkbxjzHPCC74frNMgj8QPVvs4zyoAvl10SK04/yLeZmb/wwewYSMqgl9RDSOMsc/Eo6dO1ZqQZgPK4AGQc23SJ2Yo2bPcJyMVQt+iIR4nHCVh5BQq3sWFO7o/enpqs6hT4RLtHOSziUz6m7HmksVskkv424HnwgkUQRy3kBZ9QJlG9NEVRiF2dQ/BbZfDxa7MDer5IRwfMClX0ANq8z+Z+F3tLAUeN+KNQKv1afNHHhVKSCPFyOTRFhtgqJtLWyC+yuwUJV4sxUXcs/g2E2mcpztQ5XnrW7hFIchEnIfcjTPQuOqvIgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vzB97g62e/yEZSWcUFQa5IU9s8oLQsgB8kq9NlEZFvQ=; b=m/2srNaU2hZZ3wbaUTVkhsW36vFNnCOBk3l+MTxPZazGu1rMayNKDuznz70O8v/iIQ3Ua87iCtgfqK2Qb6bTtDlXd+bHEZqhJjZOHDTnn0YEiVPEcUwTjwRoFYQNVRU0ZDOYNFtljxhVtEnvuGCibW1Y7JiFcFL+IcONXGZT1V5tk3MWucExXKRS0MR5i6BX1nBFyQjmEi/bkvzMnnVpDBVrtjteaDJTRifiGwpd/6qM6l/iSCb9a9aujDkMJiDKrP652rZ0g7DRk05sDodJzYca/IVqVuF2XLemKbR+XeUmfsSGFrxoABDXzMGL0pdyTlH+RgB6igmA6Qai1lkBVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vzB97g62e/yEZSWcUFQa5IU9s8oLQsgB8kq9NlEZFvQ=; b=ghh+uq6rACMp7VOHVnoQfbDndrUkS1D5VTO/X5Pxx7S8AHkjIEVuoHLr0h/heq16TbDNzJ1ne1yCB1M5yRr5rvucN1dfvPR7nWm+k8hABHvJZSu13xkjyACKoFLwJu8IXchaA81ZHwjllS3sKXEvqEb3KhVLMs1j/i77jV2o82M= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.19; Wed, 3 Feb 2021 22:01:03 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Wed, 3 Feb 2021 22:01:03 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH 2/4] mm/gup: decrement head page once for group of subpages Date: Wed, 3 Feb 2021 22:00:23 +0000 Message-Id: <20210203220025.8568-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210203220025.8568-1-joao.m.martins@oracle.com> References: <20210203220025.8568-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Wed, 3 Feb 2021 22:01:00 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 526eac20-7b3b-42a2-49fb-08d8c88f32a1 X-MS-TrafficTypeDiagnostic: BYAPR10MB3077: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eX2iuzmXIXVRUJFbsdpMidDqiatd4hW3Bfahrnni+d1zWcDfKJnjUf+Ix4XW9lIoYl5EIKSr6GA0IPSr3ixEsasf1k4oHeGIs94Lj5EiGrwQsiu9d+MTVF+uzdwEUtuGWtdBDefa8zWPGHIBVYRQrwm4ADI4rWfaU6QKLXFoTYE8rp6Tw4tEjS73syrJ2MjAnWtwgNqOAXWjMAHts3YA3PNAf5JGLVcvI6BA9NO1YKajSEIc57EEWnW4OYO7wk1RbyT92bERl1DkikagbxoTCn7nzWawxJw6vSD0LyBIyeS7hI4Pnf9o9zn6G64vYRuH1ogsPN7g49b/h4nBRUf8hh08cGZ1ZnyJ7JhMGQPe6WvVcuYgctrsnfnBTz3nGW8vAkxdpwDTUv/ex7D1UEF32Uw3J2JxTQgTPfpaS6Z2M/VO8Ixd5dI1ps+1doT9KbL4H+LJ0Nj7E3130AeZJ9ceo4Zwh59BLKnLtmIL1OWKA82XP3yaMAYX0YMaqgIKrsR1vyOjMePWtA8OQO85NivwFw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(366004)(376002)(396003)(136003)(6486002)(83380400001)(86362001)(52116002)(5660300002)(1076003)(478600001)(8676002)(107886003)(4326008)(26005)(36756003)(2906002)(103116003)(16526019)(66946007)(316002)(54906003)(186003)(66476007)(8936002)(66556008)(6916009)(7696005)(956004)(6666004)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: q3pdFOFTrzCakjbsQBZV6BEaENo+bkB8eob3CLGSk2Ay+ONqhVYfVkWG5ffR0KNsDK78LP4V9ibSLvJku3ZWW6K69/hd3zAJZDAJUQ9lanTfSW7Ez0JX0BnA3h9/clBQQ4JleDtfPiHDUpGJy1ciJHkEmUgRW6U9AxpTqbaQlH4nLXSGN6+4kYEAFHKWuE7YTSYhK+PQI6lkRmkMtpSbgbDrxe0ylqHPZJaEiyt3iSSNCuOhvouogQp78nOhPRpi0pfDWLlvvlneBeK/phnkeFVXf0eRAjZodHp1DY+T3RdB0atpBvRHzvNlDT8kFwhc+wsGOqd6aP3vdak+cyGyWoVD1U9DhDxL0ARRd07ECXr4Le4q7J0oPY50SC0Gt4ZgfAf07dhwPmzTVEUaMzFty1E4/5fQ0QTZnxnQl0reCWym9/6w2tYVR6Fkz2FRvGUtg0nY4rBZsxHOckpki9rWK9v0wn4sulksLDJcL6Fueo4IDwLoIpm4bMyjxxV4P635yFUWzuZLzEfQPqtIXA4XmMvhKpXGEvoRODxsjESzwLtvJGv8LIAU4SVl/ZN1C1iq3zM+beitEO7lomNrW1HuApMa2sTyQJAQmvG5aCg7ro8yzgmmB6ZkXJAmwfR1lyMnK7pU4oJc8FDvtt3cGowKGlrnX5E/3P+vOH3y/acP0m7uIuRhyN1ACMwOEjtsE+PinjZZc75v/LzVJcpnjYeRqlQBwuubN5B6tSUg99IBHCBsNzKfT3e6wZHb3ALOtTiE25A0408keFZ8x35ThNLR23bar0HzmFZXimS426Ju0gygDv0zazwLhVGd0/dZ3XocrwPSE9EDRwbAvNi19xJmo7Md0pk80BKZlvoVZqbIGObEtnCzOl3V1heAvOLMTFTxwOj4WHS4kpWW6brCZjBOg/LAqRpLRN30vB7jkVKfSz5TBiPleEt4AZ9f55NsKlLyK0cQWHGYaDc7S4T7RWQb4P4jY660OTPhxm8m6Qo2TFQ+WBHj/rqkhBwSjGD2xSdzEG8NYvH75YmlA77uci7uOoGJrP6f6WvwKqbLxqhqL9fR/5dHcvkcCdS/WCRQkkGl X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 526eac20-7b3b-42a2-49fb-08d8c88f32a1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2021 22:01:03.0243 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wBbsQCCQ8vfDktq1Myp/0uCAH4ke0nnFw8qGouDpOzsmVFfgjgU+1wC9Fb9VEGDGsQpB+xBK827ORt2aJoWaCEvhXZJdvoUFueq3wmaFy4U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3077 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than decrementing the head page refcount one by one, we walk the page array and checking which belong to the same compound_head. Later on we decrement the calculated amount of references in a single write to the head page. To that end switch to for_each_compound_head() does most of the work. set_page_dirty() needs no adjustment as it's a nop for non-dirty head pages and it doesn't operate on tail pages. This considerably improves unpinning of pages with THP and hugetlbfs: - THP gup_test -t -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK (put values): ~87.6k us -> ~23.2k us - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK: (put values): ~87.6k us -> ~27.5k us Signed-off-by: Joao Martins Reviewed-by: John Hubbard --- mm/gup.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 4f88dcef39f2..971a24b4b73f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -270,20 +270,15 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty) { unsigned long index; - - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ + struct page *head; + unsigned int ntails; if (!make_dirty) { unpin_user_pages(pages, npages); return; } - for (index = 0; index < npages; index++) { - struct page *page = compound_head(pages[index]); + for_each_compound_head(index, pages, npages, head, ntails) { /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -304,9 +299,9 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, * written back, so it gets written back again in the * next writeback cycle. This is harmless. */ - if (!PageDirty(page)) - set_page_dirty_lock(page); - unpin_user_page(page); + if (!PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); @@ -323,6 +318,8 @@ EXPORT_SYMBOL(unpin_user_pages_dirty_lock); void unpin_user_pages(struct page **pages, unsigned long npages) { unsigned long index; + struct page *head; + unsigned int ntails; /* * If this WARN_ON() fires, then the system *might* be leaking pages (by @@ -331,13 +328,9 @@ void unpin_user_pages(struct page **pages, unsigned long npages) */ if (WARN_ON(IS_ERR_VALUE(npages))) return; - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ - for (index = 0; index < npages; index++) - unpin_user_page(pages[index]); + + for_each_compound_head(index, pages, npages, head, ntails) + put_compound_head(head, ntails, FOLL_PIN); } EXPORT_SYMBOL(unpin_user_pages); From patchwork Wed Feb 3 22:00:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12065535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEFE1C433DB for ; Wed, 3 Feb 2021 22:01:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 42E7464E0D for ; Wed, 3 Feb 2021 22:01:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 42E7464E0D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9D2328D0002; Wed, 3 Feb 2021 17:01:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AE998D0001; Wed, 3 Feb 2021 17:01:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84AEE8D0002; Wed, 3 Feb 2021 17:01:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id 67C9B8D0001 for ; Wed, 3 Feb 2021 17:01:15 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2E919180AD817 for ; Wed, 3 Feb 2021 22:01:15 +0000 (UTC) X-FDA: 77778328110.09.badge00_270a4b3275d6 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 14F99180AD815 for ; Wed, 3 Feb 2021 22:01:15 +0000 (UTC) X-HE-Tag: badge00_270a4b3275d6 X-Filterd-Recvd-Size: 14107 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 22:01:14 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113LxhRY193663; Wed, 3 Feb 2021 22:01:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=X5ASRJoQeYxm6UfOnHd3bOScEfbBCULqmJMp0gLXSiI=; b=xfAxIolVIUxsqbdZBg1iljOtrYxFYje1H+fjcMSnJKnXVAkuUcm16H7pAhgmDOetvVzE YyZorZmq0PFe1oqdc3phv+DYDeYvJ/M81nlhcXcY9jYH5UX4XCQBpK/NJJQr4XHTsgMZ 7fbNLx9CwF5QxvRlR1ZVupsUNa7iHe1oRzXjHJFF01GutN5dLW8bXsPin52aIfzgo4kE 2+yXSOdzFaA596qIzCNNUkRrMghcr1lw4oAwCDmkILTko6FJbpsjW5f3iM0JKsVeUQxw 9MHaGMFIJVAr6TgMg5ca+6ItfBc3PIZCcp7QF1xDgdlmID4CkrE8qBEFtwWd6uPd9gVZ WQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 36cydm2e44-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:12 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113M157f042797; Wed, 3 Feb 2021 22:01:11 GMT Received: from nam02-bl2-obe.outbound.protection.outlook.com (mail-bl2nam02lp2058.outbound.protection.outlook.com [104.47.38.58]) by aserp3020.oracle.com with ESMTP id 36dhc1t3e9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BKz2QBtGVA5agCvQ48QQqDoUbbm03nK9pveYiXzsW6R4yMdyPGnCoV1PKVxNvmLaH31UcLgNQpf+pYfTXRNNdVVPLrOodYDFXigOmntdo5Quw22XGYJLPWF53d8ja/mirgwr9qPdK+L6bFkOpXuTZkVq+QpQFLXvoOydkJ8gfaW0LMFr1xVqU6Cu49bVKiLMiD6B/QXxBnQ+W4rhPiAkKni3H1u1kGm5T3oV//qEqe/6s+pu9u8Op7uxV8xiQMezbbbYigp2ZXKTFQrARw0y4hzlHEFtkifcNWIs/KchTPv3MHzTHjjZ1jaSryEdj3aKmDR5bp/QYOZLJuy9yE6dvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X5ASRJoQeYxm6UfOnHd3bOScEfbBCULqmJMp0gLXSiI=; b=DFaisX429Qy+hc1MT20H6JRa3CTiZsMlXs12QC3D7RcK7Ym4yCBIK6YReLL5S990POzM4jkQ/rBO7cmjBMguHIICencwQwNeYiqQkBU3AMyUU/RI9b/+oICDMvcHysbcuVxuo3D6ZS7ftfJ5tMVG0HnCw5UtorXE1DtC8qOIGKsMLHT4w156qje+vy0atZSX7H5UjC0z00IKA5DsybAfXCjRxXA/GpjSrp1fUotyzv4sBr5D86xlGiO+Ab4HAf+FV3nLbAay10VLZegadBgzxrdjvF2wa1t1YRFJGS63xlmnPkw1nI53kbQNUlIb0BjX7omkk7ce9TT6iegc9cgonQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X5ASRJoQeYxm6UfOnHd3bOScEfbBCULqmJMp0gLXSiI=; b=nu5OPkJlM6NCjSzOrDoacmAFmHcSR9bz5rx0V5yi3z4drs5yzs3OUXHWqWyMznCt6N5EFK3fxIDC/JMDV0lwBYPna1jAOvblhpw2ZiHw1FElTTkqyDn2sfbBCbeQ53zkQuyNw6uXgr/p6H3967WaiCrvUYLO712n7eXSsQ2wGyU= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.23; Wed, 3 Feb 2021 22:01:06 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Wed, 3 Feb 2021 22:01:06 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH 3/4] mm/gup: add a range variant of unpin_user_pages_dirty_lock() Date: Wed, 3 Feb 2021 22:00:24 +0000 Message-Id: <20210203220025.8568-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210203220025.8568-1-joao.m.martins@oracle.com> References: <20210203220025.8568-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Wed, 3 Feb 2021 22:01:03 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 03284ce4-822e-4c91-5521-08d8c88f349e X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2089; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: l9yoZyJklCyqv3WH+QQmF04veckKe083WF6WAFZqFjniOFYzRsQd7r+DbN7Dzd1IWOYJTNIwTV1dQOpnIEAmwLune7WBZruLYe950TWJq8Jz+/X/12/jrz8m2QsRkaIaHfVoA6qvlIuCTe5TR7tBxjpAPQgG0XCs3K5XzzveHzM5EvoIQ1KsD3AUMgEBc0EBBvLFH889ubivut3MuDaJHmKXE42YORiQ7/y8ICsmiausXoIc7U4ZLLFre5/Bjwa5DZPE/IpFTp7b6JSUV/R3uq35CbCFb+eDBeDtbih3hIH23VJr2fk/639xzxQe0DwG7RhC980qeLpObFb0g8tj4Rmca5Ex9QfPYRLsbdIxqhditRwUB8qC4Zj+HFyP+P5MceXzbQM0BImLD/LVQxA9OHzOUZaKwKTUjjCccsA2fP9Zk6kf3F4GiJSc9BTFXQiOx+dXd7SjoW72QSadAycvKGbQrsirJCrjiJys4QGsEta4GKwwMl5tKxFF/1uxEL7jPbBMeRo7qoTaBeHUJ8i3kuDR9Mk6na0CfnuyVFUm+H+Y4CsXBVKZUnYEYsZwjlPE X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(39860400002)(366004)(136003)(346002)(396003)(66556008)(54906003)(6486002)(316002)(186003)(2616005)(956004)(16526019)(107886003)(4326008)(26005)(66476007)(66946007)(1076003)(103116003)(52116002)(7696005)(6666004)(86362001)(83380400001)(36756003)(8676002)(2906002)(478600001)(8936002)(5660300002)(6916009)(14143004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: tmYQZRB6+eJBZtAF1LH2gtG7aiR3Lkh1Q8hQWhrApI2xXzY8DHy2oVxHuz4oMqD1td7rWZBu2LVFBbXH8K7PlOikq4w2FoT0bjdkF8O+sNHmZ42rTZRBtMnGD1W9khXDOjH5GgO6/uU5rN1372wSdbtACStcmdxnMCJgnGAqL9PcSLgSVUM9kllfrW3aVD5f9kSRehqP/nk9KgFphtoXFBfSm9JUKKY6kcFtKJnujTlHVr/fMgEjgcXRiXPBy+xcwyhOoeArC/zYAiTu9TE9daJ6QzCqV8DOQYo0eNEa5l/rk+posK3l+wWLQplFKCtLOnOk3dd3GW/wnLiFg2e2vHvKkiySJ3/X8hOF0WUDcNgE8JltaV2eKHKydjHyt01yJ4iSW5YDD8DhUvvlmRGJQwtpC9B0IgjErotzSkxpfo1PCBx61wEu2GBl5GqoYR64XDUlDwYn5fpJ+Ry1BNWPQL77uXWnEU3V7pCysvn2jgie0vTgagBmcd+fZJcHb5lAHUWWgTxlt9SJ24Q3yJ+15SXdCbXpjL69SK8A9LqUOaZi9ywJOYXRfsCT54YlUKkweFzC4dYFzqY0A3uj3x81orISYGsOY3c1i7NOIxt0SPFMm4zXwqrIqjxuqWPBlrklJBeaQ0U/7YJYaepOIJuFwS5wRw0rX8Fnqc3/a2CauuMS8+yjiR3bHTwZ11qubOkcHCf7MiyVc7+DDYrgeTLPajXEWTFBUx/umYVLFvRbkM7Nmms8uYe8WI+o8vsPSgLnU8N/3/83NLWnYoqKuZWF3e9e8WFNMRsmVpz7ChTWuFe30uwGTjolxISX14IEYeF8OQyo6DbFpXPJUpnx8eXYcuIbzMjtBkOcsUYkH5HgR3zFJsI+f+q1hID1t4kWulbGAml5/Vw3W6yHlxpx+K0xTS87mzI+3mBSPAk74cbtSa046WOlt3qKG1OY8mHi1jCVKrOEplXpOj9FUF6jjowcXwvbvpVevCDiNcDr9gG9D+pDPw6uZkmvkAiv9ji/JU0HIj/Ji9j35UD41VJXDKgCLxtwKp9hhl18oM24AiPfFxvVVabOOdPRCTvdsMyUdykI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 03284ce4-822e-4c91-5521-08d8c88f349e X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2021 22:01:06.3344 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 78mq/widpJGjBsTpGtAQZc6ec5bkp61BbOLxKK/hU73eO43V51+XvTt0oySKYXjJi3djGzUXVZGqpF+xiaCHc7IPPHeasA4lc95VoW35Tts= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 spamscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a unpin_user_page_range() API which takes a starting page and how many consecutive pages we want to dirty. Given that we won't be iterating on a list of changes, change compound_next() to receive a bool, whether to calculate from the starting page, or walk the page array. Finally add a separate iterator, for_each_compound_range() that just operate in page ranges as opposed to page array. For users (like RDMA mr_dereg) where each sg represents a contiguous set of pages, we're able to more efficiently unpin pages without having to supply an array of pages much of what happens today with unpin_user_pages(). Suggested-by: Jason Gunthorpe Signed-off-by: Joao Martins --- include/linux/mm.h | 2 ++ mm/gup.c | 48 ++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a608feb0d42e..b76063f7f18a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1265,6 +1265,8 @@ static inline void put_page(struct page *page) void unpin_user_page(struct page *page); void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty); void unpin_user_pages(struct page **pages, unsigned long npages); /** diff --git a/mm/gup.c b/mm/gup.c index 971a24b4b73f..1b57355d5033 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,11 +215,16 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); -static inline unsigned int count_ntails(struct page **pages, unsigned long npages) +static inline unsigned int count_ntails(struct page **pages, + unsigned long npages, bool range) { - struct page *head = compound_head(pages[0]); + struct page *page = pages[0], *head = compound_head(page); unsigned int ntails; + if (range) + return (!PageCompound(head) || compound_order(head) <= 1) ? 1 : + min_t(unsigned int, (head + compound_nr(head) - page), npages); + for (ntails = 1; ntails < npages; ntails++) { if (compound_head(pages[ntails]) != head) break; @@ -229,20 +234,32 @@ static inline unsigned int count_ntails(struct page **pages, unsigned long npage } static inline void compound_next(unsigned long i, unsigned long npages, - struct page **list, struct page **head, - unsigned int *ntails) + struct page **list, bool range, + struct page **head, unsigned int *ntails) { + struct page *p, **next = &p; + if (i >= npages) return; - *ntails = count_ntails(list + i, npages - i); - *head = compound_head(list[i]); + if (range) + *next = *list + i; + else + next = list + i; + + *ntails = count_ntails(next, npages - i, range); + *head = compound_head(*next); } +#define for_each_compound_range(i, list, npages, head, ntails) \ + for (i = 0, compound_next(i, npages, list, true, &head, &ntails); \ + i < npages; i += ntails, \ + compound_next(i, npages, list, true, &head, &ntails)) + #define for_each_compound_head(i, list, npages, head, ntails) \ - for (i = 0, compound_next(i, npages, list, &head, &ntails); \ + for (i = 0, compound_next(i, npages, list, false, &head, &ntails); \ i < npages; i += ntails, \ - compound_next(i, npages, list, &head, &ntails)) + compound_next(i, npages, list, false, &head, &ntails)) /** * unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages @@ -306,6 +323,21 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); +void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, + bool make_dirty) +{ + unsigned long index; + struct page *head; + unsigned int ntails; + + for_each_compound_range(index, &page, npages, head, ntails) { + if (make_dirty && !PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); + } +} +EXPORT_SYMBOL(unpin_user_page_range_dirty_lock); + /** * unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released. From patchwork Wed Feb 3 22:00:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12065537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46D4FC433E0 for ; Wed, 3 Feb 2021 22:01:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C48ED64DBD for ; Wed, 3 Feb 2021 22:01:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C48ED64DBD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A21F88D0003; Wed, 3 Feb 2021 17:01:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D5948D0001; Wed, 3 Feb 2021 17:01:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D6E08D0003; Wed, 3 Feb 2021 17:01:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id 64E1C8D0001 for ; Wed, 3 Feb 2021 17:01:16 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 33C9C181AC9B6 for ; Wed, 3 Feb 2021 22:01:16 +0000 (UTC) X-FDA: 77778328152.08.09E12B6 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf03.hostedemail.com (Postfix) with ESMTP id 449F9C0042F7 for ; Wed, 3 Feb 2021 22:01:15 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113LxaZh136605; Wed, 3 Feb 2021 22:01:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=HnBuEyXtPyPJCNeig1nrZDABBup9AWpTwYMW7sV8nAg=; b=EE46T2tgF64r2ehJvLmdhXrGHo+Kp5b8bRfbVckqy84el+Cvd5Ux2uhsNW+dNcvO9WVS WfYJcGolTvtvZYUnj7unUX1IVPsBh8rUuREghpF9bXjss4J0NbHytLDrV3vaqOZT32xq 8TE36oqQe6pI0ogSXhy5XgOSgw4FTlCleO1TUc4fe1HsJWdYAEtpGEc/XEDp+fTPVcYP fDDIMG0t7PUhnGRgYqS+OVYJUWXijeRIC+9nCylXnafe7e6YugPXrnpls2vrTL8vaueC lrzdbOxGoVSbFiH1rrdf7vN+pZ3jD/Vpeo0ZWPm/t+0GkeT0uwptb9rckGghnbTVMjsW aw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 36cvyb2mhm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:13 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113M157i042797; Wed, 3 Feb 2021 22:01:13 GMT Received: from nam02-bl2-obe.outbound.protection.outlook.com (mail-bl2nam02lp2058.outbound.protection.outlook.com [104.47.38.58]) by aserp3020.oracle.com with ESMTP id 36dhc1t3e9-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DqpxcQpVH5+mKrs68nx3odt7KO8N7nKqGUaUkNOmhUzOWE91ZizwLhdPjXV57guQofCPgVtF+CAdsNiNw8rlsFv3gLN23huG7v9MYP2iiY/8uYwOc/+2HZtgjd/v2MIgfPEUi244cVXED7nVWU9aTu7d1tq7/lrkgc7qyK4f39TEFrc5NsfqGBWSYrF6SpFFm+DZxeb0CrAD96sa9q/N1CoOwJSxBqEj1LKWvQiW3E3Fg2h6hqRCZMa6EHoNjr1hQHKdWcRiPm95XZIVkFHDF9hfSbT3ckZDzr3Y9s7Cod1xlLDW80CRcJE5pPcwY4AMQq4NJtQrsciIddG8KpjQgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HnBuEyXtPyPJCNeig1nrZDABBup9AWpTwYMW7sV8nAg=; b=BYSxqA0kQDwu6JXGqOn6wzyhAGpF6suHXm/y64kc0KUpAyKTejBOn07kYeMRVyZP3CeAsF4MzPSl5qgdjXja/C1RUWHif/g20Ynaze1GDhaIAmCxGWnQkHkIHI/y/BMeaN7x2hcYxbEGrbSD60FQ9IGfffOCv6hDhJ9Tq0STFPMMgtITgPqxlqOhga1NSptw+dsuvDxmoBNa5sMwNU8iFetbSiDjw7ya40GIlmN6MdeALKBEeQxk7KB6Bl2xaevHZ+Rq5jMKNat1BRfdr+7CMgyTbDgVKo5gDHPWwHeuhPLkWegw0WLl3Hz2NASKiLZzbsdX9gFpycRwdN41RfeNqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HnBuEyXtPyPJCNeig1nrZDABBup9AWpTwYMW7sV8nAg=; b=wPIFWPpKdBt9oDpHiuOJ9XxQKkVFSbWLm3jDtBL2/aUyPuTNe3GxOJk0+avjcAGc9GnywVWEzF8EwUhi4Ia5ZFTFRPM/+espsrqeo+pHvP3pOBoCdt5SpU3JQK4QuXDaZeMCKUMmWn8dhF7tG4WVWqsSEu30x1VYbbMKriASUHU= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.23; Wed, 3 Feb 2021 22:01:09 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Wed, 3 Feb 2021 22:01:09 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH 4/4] RDMA/umem: batch page unpin in __ib_mem_release() Date: Wed, 3 Feb 2021 22:00:25 +0000 Message-Id: <20210203220025.8568-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210203220025.8568-1-joao.m.martins@oracle.com> References: <20210203220025.8568-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Wed, 3 Feb 2021 22:01:06 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 596c279f-0dae-41a9-2772-08d8c88f3629 X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:370; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Oxhnprz0Iz8p7+RZxD+qjuYQJ5ID34jZAGxd8edMFYhYDrP+frtnGbwEQDvfCtOfyTNesp5grxBHKtcttF5jJZ3drpoeHmIQPOfqagf9MKijQs9WpNME5Fvb38TIZiXeH5qT+w+/vnuPM5HZUwwTf5JcBNXbxJir/lLc3Pm/l0H32MBwgGQahaUPof27RPPpyFOFyPP5tlw+Ib3ULizm/Afr8YtHX08CXs8zw+prM46x9pGmfm5Q7xQJXe8nXwHQfzFmdIl9Lse54MawO5nS8MZRNfq0uApQNecRGQwNOwPorrIBfrdCH14SBqS4gvaVP4S8XkS0OSRDahckZwjd2Lvg1cCVFkHXkiqQLHNrsjB9D+wkjgRi75erqxAOAoldBZNIbRkY6d8MB0S0avTJRyBkB4cB4YEoNCspz1sLyBn4utnhM+zso2FRTJHNhcI6ZtouOULp8N+k3T8pkTzm7dNIIPtlNWh5el5m4f1QSoV+YOBUfYMpal0QOR83wy5y0X8+tAiUs2h2BgoZ5l0QuQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(39860400002)(366004)(136003)(346002)(396003)(66556008)(54906003)(6486002)(316002)(186003)(2616005)(956004)(16526019)(107886003)(4326008)(26005)(66476007)(66946007)(1076003)(103116003)(52116002)(7696005)(6666004)(86362001)(83380400001)(36756003)(8676002)(2906002)(478600001)(8936002)(5660300002)(6916009);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: sYrS0bvt9OyLa1HA/i7fV3WqIWdD348Pgf9FmXm4WsfR2atpQtcJy+/iT8NLGSUr7aDboI4IxGnGGHbqtRIEut1FnWvdr4hEeYu+cahrBofknzabrxpayONRDUycvg731MSwW8BJenvwxdw7aW3r0x/8Ocu1N6DShAwEjF90M1fWrOP0b3SDmM9Vo6uCAORL27+fC8o9AxkWO3Zz4FxhfdlZxSFRVSnO1jgJOg0sh/Wh3j2wtICU9uc6idNem7kH+fs3rPAxLhuel0UjIFF1HI2evjzDHFN3+efDT+tmGgkep48/TMYlALhEj0ZXStk5nMoOty6GjEsuB7bqLeJjtV9NtHvjayFo1gKJNwFSBdmktPBlgCjMG2eSyOfQjeJPJBpSLNDrClY0LVmaY7UQbB9J43f60QjGTUxYcRhNNqorJc7LfdjJGONYmthsJ0dDYm/WPSE0U+ZXM6uNSrC93yruj/kCDFtlQ07G7eBFlImCPbvBrdqXcJZ7K6WpnqLTqAuR4zBuhRaxC4//jSl3VgXqXD1k9e5ar6QRrZkEYKgK6QvBtUHl3QxtEQJoIIBGI1rDcevyNTTX1gv98UZrDYf4o3erIKx1jODSDZ2DHkUWTKeHiD9qEnm9AUr7LPFOBejPyR4ekebeHxVInyayasRSzRWyE9AR54k+MWmTekj2A146sqy/rCVw0gbVfoDthiTwIp4voyoluJHqXqYxs3QRI/DiLfxoRLF37p3gewBzdMPE7tk/g+wqpIWdUG6dLXrXdqBcFcP6ho8P4ul7AY1fCOws0HgPmd/6iK+k9eVjEb5fjH6duHdhoUfz4Jgr7rLCe72FOoyvCR3JV0vGENB0xRMJcdrHhuvOP2kKFEt7pQGiYuSVR+NdvJfLgGsfoj7w5gd3DY2ZBt2OziyQjIO1OTjB/r/54WzZZJS/B4VyA66WvbfTTriQmqPVZwj63332ZpKf0dHac/WTU8tkUwfD5LDEI89ili+zjzBfHqE3aqblrWuibkydYk1a3UvAHfSSHmYtbVi1qsdP2IOAPBT2BmwWXc9mljREcVBkXcVBQ7TTqiymFu4iZWpWeq93 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 596c279f-0dae-41a9-2772-08d8c88f3629 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2021 22:01:08.9309 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: M95JymCUkBn6cyeU0MajZr3FKVV9dy81Xqm9OlbGsBHihiTPDgr/dgkaSJGgypJmD3Qkd8odngZzzMBGbT5zsErDcsm4ENwYEDlOGoS4kmw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 449F9C0042F7 X-Stat-Signature: xzwwoz1xn8t4wn6sq9wafm794tzpkzib Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1612389675-824273 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the newly added unpin_user_page_range_dirty_lock() for more quickly unpinning a consecutive range of pages represented as compound pages. This will also calculate number of pages to unpin (for the tail pages which matching head page) and thus batch the refcount update. Running a test program which calls mr reg/unreg on a 1G in size and measures cost of both operations together (in a guest using rxe) with THP and hugetlbfs: Before: 590 rounds in 5.003 sec: 8480.335 usec / round 6898 rounds in 60.001 sec: 8698.367 usec / round After: 2631 rounds in 5.001 sec: 1900.618 usec / round 31625 rounds in 60.001 sec: 1897.267 usec / round Signed-off-by: Joao Martins --- drivers/infiniband/core/umem.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 2dde99a9ba07..ea4ebb3261d9 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -47,17 +47,17 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) { - struct sg_page_iter sg_iter; - struct page *page; + bool make_dirty = umem->writable && dirty; + struct scatterlist *sg; + int i; if (umem->nmap > 0) ib_dma_unmap_sg(dev, umem->sg_head.sgl, umem->sg_nents, DMA_BIDIRECTIONAL); - for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->sg_nents, 0) { - page = sg_page_iter_page(&sg_iter); - unpin_user_pages_dirty_lock(&page, 1, umem->writable && dirty); - } + for_each_sg(umem->sg_head.sgl, sg, umem->nmap, i) + unpin_user_page_range_dirty_lock(sg_page(sg), + DIV_ROUND_UP(sg->length, PAGE_SIZE), make_dirty); sg_free_table(&umem->sg_head); }