From patchwork Mon Jan 25 20:57:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12044359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF3B6C433E0 for ; Mon, 25 Jan 2021 21:00:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E801207B1 for ; Mon, 25 Jan 2021 21:00:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E801207B1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 937BC8D0048; Mon, 25 Jan 2021 16:00:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C1E48D0023; Mon, 25 Jan 2021 16:00:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73B4F8D0048; Mon, 25 Jan 2021 16:00:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0214.hostedemail.com [216.40.44.214]) by kanga.kvack.org (Postfix) with ESMTP id 572238D0023 for ; Mon, 25 Jan 2021 16:00:06 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 115ED1EE6 for ; Mon, 25 Jan 2021 21:00:06 +0000 (UTC) X-FDA: 77745514812.13.cows05_37089e927588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id DA81718140B89 for ; Mon, 25 Jan 2021 21:00:05 +0000 (UTC) X-HE-Tag: cows05_37089e927588 X-Filterd-Recvd-Size: 14052 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 21:00:04 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10PKsnVv175388; Mon, 25 Jan 2021 21:00:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=QUtN9ijQd10g1b30SZ6Ui3bEJJlMszFpH4tGaJekeWQ=; b=NEm/MulRbysyiQva8vXIZ2LLmcVG9WlT4WAs2NwZMJDXzHOEyCFkWhSRHE1GRAMSkQ2b ORlULmxLF3pKbTp9F1UUFTcYi+vvhRBIoRIdYVLxfHTANcB6YrAStN8ght9pTKRd4OMg MAeXfI6gUSocGpzcVUjw4iB1a+hZh1lSHj+lewtQT/sE6gG33JpfnSaCl8nbPKLlC8MX 6cduIE+Uik1etbqWR2/CKplwJOxc1/c+cwD4QafcOuryfsNrKJ3Q8gKteDNMKMtAde7Z Dje3nvlglLNGPXk2mfWYXwVjSHbt4nF4M7Tbk8XVNgzLNl1E7lC11Jsx6W+uDZGLKiyD Ww== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 368brkf9bx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Jan 2021 21:00:03 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10PKuTv3136398; Mon, 25 Jan 2021 20:58:02 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2176.outbound.protection.outlook.com [104.47.58.176]) by aserp3020.oracle.com with ESMTP id 368wpx4rwn-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Jan 2021 20:58:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=im1clGZk1L6I/LAgzInaPPdWqhgUKt/O1hKA7DNesksrkFV62V/GtLqZhixpztDmx9vCKFh6t05qzSCB1tzHiFFtBUUltgeubL5SdQQmyqlX3aLCNkETOky67tqG91lZxmve/r7uZKy+4hiOM5EG1yBXmptMtm6gnjidU4WyXODycaEtuKGAp7IToPQ08u+Y6UJi8pLzSOOmdoAvT44ylTmnkpdUQwpNmBghYzXvDvjz7QVtnCB0OKpKVIcxqBCYmAS2+nJ4uZziOJteiFlewPaZGwXamtKBc66HtM/iji8zt/ElIUyvTRa6udawMpq1YPHqZCax54fa6HQgIb9j4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QUtN9ijQd10g1b30SZ6Ui3bEJJlMszFpH4tGaJekeWQ=; b=oQk8NKA4weG3Qp7pzr1yo1tUBX1SDrOvgTuDuFYuyliENdTArU822GE/BIT9miQ2ocNujmYjoqsXH2hhv48XFxuxriWPCCjgOEf5eEicB0DZGYuH3hf0qL3SdJ+Vdb0jYnVrxT9I+7uf2A4wghYPNlhzxwmxLfBDEjVR5bUd04Gwzd9MyKlkL7Gi3AGzgEwmDrZl2kUNMYPe44OWu8JBgXx2ele2GfZ/MjCeJ/TMU+Ca4GGEtUgh/uMjuJdKqZBmo4e9CyPMLR+TRunloPeL+OguWl+GVJd0wdXaNcYc/VukOjSf5NCRh61NF6SBHYWn/lzbQ8+4m74j1kOtBxQ1zg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QUtN9ijQd10g1b30SZ6Ui3bEJJlMszFpH4tGaJekeWQ=; b=D1lCnbu/xSKKt4z428Xje+fFC7cihUNNhN8DgOL6L1jpicVbo0rvGBAPP61w9oeqR71MYPftinYYGm5uf8Wrj4dg22ANJTD6NGjZlNHQ7y2tOtkHYLuSko9KYS2fthona+C5ma2h+9bTVjx0uUbpOrSOWo3ou95SGDxt5HXYxtE= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB2776.namprd10.prod.outlook.com (2603:10b6:a03:87::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.13; Mon, 25 Jan 2021 20:58:01 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%6]) with mapi id 15.20.3784.019; Mon, 25 Jan 2021 20:58:01 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH 1/2] mm/hugetlb: grab head page refcount once per group of subpages Date: Mon, 25 Jan 2021 20:57:43 +0000 Message-Id: <20210125205744.10203-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210125205744.10203-1-joao.m.martins@oracle.com> References: <20210125205744.10203-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0293.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:196::10) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0293.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:196::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3784.11 via Frontend Transport; Mon, 25 Jan 2021 20:57:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2ce13914-9410-4ba9-2cd6-08d8c173e6c6 X-MS-TrafficTypeDiagnostic: BYAPR10MB2776: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4941; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uMMpDidIHlJrv9bwl+QsvIvTnSPCzRzo/bCOjTD0K6OOb5cl0HpnzX9kwUaWLuIUdqZpzxqv2BEeNh0CWYWVH9tWGFRxHhKDgUljqBQlF4UCDyjSObb5OMA48+etMmwEi7ayQTz4uL3yT8Mu86NqzZ0ORC+ZIvmLlmIeG1E8ycppdo+U189ETQd3iY19dRLXGzz/mjd7gdDyLWWvx/yB5IAz3elbI/WUZ3nRElPEq0pXmAnlYkFHjG3Xc7YGGm31lqaUtDKslxxsbMRNyKeeKy5xqo0mJ4aiUEtk5ZugdwMwbHTJRVOIpu/d9WYiyggnydO/xg4JjQjyOqSWrm74YRlTkLlaqnpsSZH+xbbhNBEGFFJ/bkDIpOgBgr1amorWTE02sMANFwqhWU/owtnkeC6wRspPLoK0k1jIJdG+znGCGMWKmiooTTBIAtMHYxhwt80kQh+1bLmfRghxjCAJNs00L8uymKtVf2PJIlshk6X6HMuvRblNQm9w+3bm7HndoZjhAy05augdmHxYItourg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(366004)(346002)(39860400002)(376002)(136003)(6916009)(478600001)(107886003)(2906002)(1076003)(103116003)(4326008)(52116002)(956004)(83380400001)(6486002)(8676002)(7696005)(316002)(54906003)(86362001)(66946007)(6666004)(186003)(16526019)(66556008)(8936002)(36756003)(66476007)(26005)(5660300002)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: +vPEEBkclWbkVD2z8L/afr7BeqDjwFgDBDDNDULsUnkU8tsA5n3ZQq/TzVs5Cb3U7c2xp71dCQ9VYFTyRoDcYURHbLJiaYHwBlx6Rw0S8cF88glf4wEXdCbaMkJ6tGhDB7HD8ZRwZfWvHZliXatfFIB0glQu/RqAq7nxkQWHxlsBmoT38xHey5xJXHqNpru5sWgvoJPtX2/pDxus4t+R4QDFh4QgSV6MmgMjvyQHLByVn4Fw586mv58ndhESRmp3iNn4Dr1eGn7bNio2ZrAyNYdTALzGhcTvmkJztHr5sYQkLJBi7xtLYi8KzM/Qrq4QD4nVS8rrxVg0HwIyYcOSm+50wBDM4ivJ1LI1kFgj9ExZIsANiO+vRDftm7YXbhF8jDkDnC5Zdf+zmvfgOMNnojyumTFXvGXa5RgmunBYy9aeSjY2a71TuewFFaDPFBalxVg7eCeg5TMXmokQnxEgXTR85kPwfGDfahQmtgLHrI3VB3Mg22RzU1OzixuE90u9rXilvZcbjamyyXhG7yga2jGRH6y3r2bsTxoC0//w/QW86hAvPkdrJiuN0HSFVnG+Wy3H0yuagyW8XMIuco1SMIbgnzoGbIPKzfrBf9958D2T/QSnF/MZzf5L/b18ZESLUouAsX4p0y1EKs2fAKbLdzoo0OpPdDghv0IfdBvDfJDZwzbda774oWjRkxGL9D4TqcOzeiyH1Quh1UrhRR9YQ+UEMiXQS4NJfXH5geVl8Xeu8a02G0Fabo57MFbXZq4A2T5n0YwcEcphmNilHLx8M9XbDKxD6UPPci+jYb9XWAGYyGSDh0fEBTiS1zVgFftaJHqFDOxGEzeEL4NReJy2biHK+9Qsc+QI9FHZUKpa4rmfMp0tGY2uVYBH9FEK/AvTOOC0eLv273ed5QPqD8eoxGbxU5ESw+YAppFi5fxBD5l+kaQQXEph9piBgBor9aBpaHsiqsQeoq5yHdZpmxwVifjvBRnqcYAh2pCDqyQjENmubwP0menfCp29r1yBRCpgliOLPJcPO/cK/A/rAWj2n99L7ej5CpKnPuifvF+4IunwBNIOHTJ98s1b3eJ+7HDx X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2ce13914-9410-4ba9-2cd6-08d8c173e6c6 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2021 20:58:01.1911 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mH2wzUK73TUCnGiF4ilqFCSIwXXqVb0dLS3/LKEOzHPh66njCQAd/kfjNWKWn+/F4Zob/nCKTuG62pdg5pt90M+ilRqKFeqO8BGLc1/oAHQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2776 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9875 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 phishscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101250107 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9875 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 impostorscore=0 phishscore=0 bulkscore=0 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 spamscore=0 mlxscore=0 suspectscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101250107 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_hugetlb_page() once it locks the pmd/pud, it checks all the subpages in a huge page and grabs a reference for each one, depending on how many pages we can store or the size of va range. Similar to gup-fast, have follow_hugetlb_page() grab the head page refcount only after counting all its subpages that are part of the just faulted huge page. Consequently we reduce the number of atomics necessary to pin said huge page, which improves non-fast gup() considerably: - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 -L -S -n 512 -w PIN_LONGTERM_BENCHMARK: ~87.6k us -> ~11k us Signed-off-by: Joao Martins Reviewed-by: Mike Kravetz --- include/linux/mm.h | 3 +++ mm/gup.c | 5 ++--- mm/hugetlb.c | 43 ++++++++++++++++++++++++------------------- 3 files changed, 29 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a5d618d08506..0d793486822b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1182,6 +1182,9 @@ static inline void get_page(struct page *page) } bool __must_check try_grab_page(struct page *page, unsigned int flags); +__maybe_unused struct page *try_grab_compound_head(struct page *page, int refs, + unsigned int flags); + static inline __must_check bool try_get_page(struct page *page) { diff --git a/mm/gup.c b/mm/gup.c index 3e086b073624..ecadc80934b2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -79,9 +79,8 @@ static inline struct page *try_get_compound_head(struct page *page, int refs) * considered failure, and furthermore, a likely bug in the caller, so a warning * is also emitted. */ -static __maybe_unused struct page *try_grab_compound_head(struct page *page, - int refs, - unsigned int flags) +__maybe_unused struct page *try_grab_compound_head(struct page *page, + int refs, unsigned int flags) { if (flags & FOLL_GET) return try_get_compound_head(page, refs); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a6bad1f686c5..016addc8e413 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4798,7 +4798,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long vaddr = *position; unsigned long remainder = *nr_pages; struct hstate *h = hstate_vma(vma); - int err = -EFAULT; + int err = -EFAULT, refs; while (vaddr < vma->vm_end && remainder) { pte_t *pte; @@ -4918,26 +4918,11 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, continue; } + refs = 0; + same_page: - if (pages) { + if (pages) pages[i] = mem_map_offset(page, pfn_offset); - /* - * try_grab_page() should always succeed here, because: - * a) we hold the ptl lock, and b) we've just checked - * that the huge page is present in the page tables. If - * the huge page is present, then the tail pages must - * also be present. The ptl prevents the head page and - * tail pages from being rearranged in any way. So this - * page must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_page(pages[i], flags))) { - spin_unlock(ptl); - remainder = 0; - err = -ENOMEM; - break; - } - } if (vmas) vmas[i] = vma; @@ -4946,6 +4931,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, ++pfn_offset; --remainder; ++i; + refs++; if (vaddr < vma->vm_end && remainder && pfn_offset < pages_per_huge_page(h)) { /* @@ -4953,6 +4939,25 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, * of this compound page. */ goto same_page; + } else if (pages) { + /* + * try_grab_compound_head() should always succeed here, + * because: a) we hold the ptl lock, and b) we've just + * checked that the huge page is present in the page + * tables. If the huge page is present, then the tail + * pages must also be present. The ptl prevents the + * head page and tail pages from being rearranged in + * any way. So this page must be available at this + * point, unless the page refcount overflowed: + */ + if (WARN_ON_ONCE(!try_grab_compound_head(pages[i-1], + refs, + flags))) { + spin_unlock(ptl); + remainder = 0; + err = -ENOMEM; + break; + } } spin_unlock(ptl); } From patchwork Mon Jan 25 20:57:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12044357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB979C433E0 for ; Mon, 25 Jan 2021 20:58:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 82DC022513 for ; Mon, 25 Jan 2021 20:58:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 82DC022513 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0124C8D0047; Mon, 25 Jan 2021 15:58:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F07658D0023; Mon, 25 Jan 2021 15:58:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCEEF8D0047; Mon, 25 Jan 2021 15:58:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id BE1928D0023 for ; Mon, 25 Jan 2021 15:58:08 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 68C143645 for ; Mon, 25 Jan 2021 20:58:08 +0000 (UTC) X-FDA: 77745509856.04.whip23_180e87527588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 48885800CDA9 for ; Mon, 25 Jan 2021 20:58:08 +0000 (UTC) X-HE-Tag: whip23_180e87527588 X-Filterd-Recvd-Size: 12966 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 20:58:07 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10PKt6Pq037855; Mon, 25 Jan 2021 20:58:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=pOZcO6nSLJxfVXnj7nABSasxHyB+2kW7ArzI5P5TE1U=; b=vJDWlvYPVYQXhy/JZcgE3WHyXjhDSJw9GZFEkBqGdlWBsx9N3h3aZa1PxnjdaRBmU2M0 XlSwk7Rj74NOD9Ppnescy2cB7IOuERbs8lzKFhe29rgMbVwpLM3YaWtGY2WjrQ4u2Pes rs3wVJLulhMj4+XbDRptps03Ol5ahv+H83JYtIc2Tcusun7CIoCnFC5GVsihXIgg2lSt CS81MyE7KHAKqvVfNGoY9oM6dq4/Sq1uMAq3a5AGWSdFH+ahqqoDXgCkSu6Z2ifAGiER +13XjAAxtIc++OX9LRROM3JmRWr/EoAV1ImfiJl5z340S9CvQnyQGRk4/dbkl5lZqOo8 CQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 3689aafg4h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Jan 2021 20:58:06 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10PKtVhB138383; Mon, 25 Jan 2021 20:58:05 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2171.outbound.protection.outlook.com [104.47.58.171]) by userp3020.oracle.com with ESMTP id 368wjqa3ux-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Jan 2021 20:58:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ai9O7+1jaxvw88sCVatHuqG9fMMmTnPJ5ctVz4Wop+AxQh+05cTvXaUomyIpF6pktbvftCDVD5s3Sl+rcbqDkCrlVt0bkut4kDdT4bisHptRIkVtBRpFNvXHYb9nmKDWpPsCreeBJ/Gl7eCT0RQRPUfo9z/md+xOVKqzgGvUQ7Su5g4WeYlH+i6rIpDuzPUGCjYIIier/coWvOwU4xeDEp1W6Y8aV7Xbf2kIMSGLLuSfypdJ73s13xKkrl5+87eNMo9v06syEIBdIOvr9DJcBeEWe0QK0fwvEMGI5chB9YECAL3ilRhEldM/8KBrHdP/ZPIuNqhOQrOK+6+2CNOIfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pOZcO6nSLJxfVXnj7nABSasxHyB+2kW7ArzI5P5TE1U=; b=e7bpJxEUfKA4lgzCtRRWAyErIyk8Y+8Qx3p49s5YpqVZ6J88xUEG0po+xvsDc4Oq55z8Mu3VaM7qpf6rf0tAB+nMmbmwotwdyntoksQBk/ofM3f3odaechlLEM7kmo8bl+m9msljTi1S3xR1nOpymWjQdGubAc3yUnOfB0NvEC2+ZbLrL1H6ykm7QUvOL97WJUAz4iuFGV8C4bHO+IqnIsJDriHSpOOGp5CNRgAZ5t8B+7d8QDIxVIxFuEvHGd6inq5/wmLOypBR451BWU+fds9ztFjN6lzhcmnZqQY7Kp8mkoJiJkKp7gSb7kU0dBFyvRM9xAVxOppH2tIww2r5hw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pOZcO6nSLJxfVXnj7nABSasxHyB+2kW7ArzI5P5TE1U=; b=qgj1Ez/rOK8JOXGrstkGrILnW5KlqmcJfJ2TeiLrAr6Lhi2LD8x3j0FWzuKNZsJ262Riv86q6DfSQYumnv4Zv7DpdDin2XRkx2GMpp5MCtXpN4SXWEyA6sHCIA6P5tOrlxKJLK942vGy0zNdqgfK0XjvYW82o3CwgHn03KMbFpU= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB2776.namprd10.prod.outlook.com (2603:10b6:a03:87::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.13; Mon, 25 Jan 2021 20:58:03 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%6]) with mapi id 15.20.3784.019; Mon, 25 Jan 2021 20:58:03 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH 2/2] mm/hugetlb: refactor subpage recording Date: Mon, 25 Jan 2021 20:57:44 +0000 Message-Id: <20210125205744.10203-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210125205744.10203-1-joao.m.martins@oracle.com> References: <20210125205744.10203-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0293.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:196::10) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0293.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:196::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3784.11 via Frontend Transport; Mon, 25 Jan 2021 20:58:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d92f44b2-3a17-46bb-3ae3-08d8c173e7e8 X-MS-TrafficTypeDiagnostic: BYAPR10MB2776: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SVmRdsHXAzUhlEsNZoT3df4FvmZHbBUWP1l/JsuR7RPsLSK0VTFU9nb8ZcGS6lMWmrFzc+K4HEH/byuiNC4lpT2YwSvVj+x8btwt3PLzcTDnOv+6OIJnDfK7pSTP5EcSk3nNxBCRC7EErKjv+MTMirUBR1Wu7FL63ZuN4ZlzzpdEgTNMaD6GF7xol0lAqIL8k29D6giXp8GjOsh+nu6CDCSNRQ/GlRUiMnWmiFm1An+8VJvyj5ZzV1cey6CqdwOMUjpjBWkuIBGVNZUOZ3S0s6DHD9QmGHLcwjAXCq+LXQTL3o96aWImN9H+bQqSTZWGQC6G7Pwyhac6vdRuOoKpTRJu9CcRULjDXTLVq6Zj8zQU0Y/D5W4Tlh8TH9/lLQasd9A3UoSu+q6ur6WLXpi+n8N9kZz2mEdMRlpGi0xaXnozU4UEatCBaYCR42rGhrxcgcOSZRyZTHzRV1ta+VA8lxx9sRJAHu45PsMghRmzA96fBJDVGdz+aaHfyKog3Gnqhgk9sTkymjUw1lLyBxOeeg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(366004)(346002)(39860400002)(376002)(136003)(6916009)(478600001)(107886003)(2906002)(1076003)(103116003)(4326008)(52116002)(956004)(83380400001)(6486002)(8676002)(7696005)(316002)(54906003)(86362001)(66946007)(6666004)(186003)(16526019)(66556008)(8936002)(36756003)(66476007)(26005)(5660300002)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: ssZVv5xHIdPCv46E9NH1ljE06EzHMTVKENFMZs6tzZpfC47u1EnHXxgU3h23sMyxVuNQwQJIHg/y0KxmcM7C9M3FO0KY+RAu3PjGujjoECic1+TB8TTgZA94tCsBqItD4ju4vOR+1bZOxtHNTf0Y4SO65W4TACQBUG4IRd/8LGxbQctDR/yQ4czZGu7p/BaM46imM80rMlOvZggduhq0zkNTfrCm4aL2X4t4ppbVII9xfeW8k8cMa6Sfk1upP98HH2qur3NXpDz9+p3fxveN9DaJjE+0xd+u3fkH1YjpZ0sKjipb6z7YTltOzrmUhGVGDVG9W7rQl6XPiq8melD3ExXRSpad4s5V4OMncDwbfkjVVCb+wg+sSgyWgSy1uskzP+gYtsUbXvaKJBZY2CPaHjA52VLc7mvVX1hnOuccrMe8LDobJZLsYWVFLSr3UhDbhuCHZuBd2jwMTeELkRnV7jdrfUndWWAK9a3jolCzr0mZUTmAEg1qV66IvEUG3bkmgyGUaLid1Cg1sSteeKEeWc1o5uB2zyhQo3Ys7byqik3pUzt6d6Knzn96FAwHEkqFl7JpI3nFsTzeynEh0glJSRHQ8yfQEWETKS6vcZ9hYEYwzs4oGjj+7Hn+VjpMh7NZXu647DO7aG4OWug9gxBv8Y/8FNwZ4C7ThFYv12Nd30Oxf3caQJ+4Kv8KjVsduoKEIxHch0sJXToWTs8iJNJwgMDyLkvh5m2XP22TRRgcsj4RdeCXoKKbUAL17ObMeqgwZYN9lnp/WQy3xtdpKmt/ve3B30AdSUeYOaDLyI7aqox8Wtjf2bVCKM/TCCopIqqh121hpFC1yduFyojXrFq/Mii6XrTJc22VcNIRAz+xn6ImV6xpfivc8HicFBqGJJ80I5zEvaMHcfoiNNbSLg8QOy1F3MAFyQnHE11f1lDGdfA4m6fpsMtxZm6vSGX5EI8QquDrmKu/01FkUswOwr+8/LTJ3+fXPpqS4duyjdsr0EpI5IWjcntTT8MxgDxkmTRGJQRXI9zH488sCu6YX6E7V0vHX+13vLPyg6P//+6Rz/DZSYMG4bBc/toHVb1tsJeI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d92f44b2-3a17-46bb-3ae3-08d8c173e7e8 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2021 20:58:03.1040 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: D++TXeNV0php7BYIOKxvbi7OtIbnYdZisE5t75vHF5IpPe7WwF72MbjOosjo2c1Dn3jaR/WISOn8NRk2dmcSbJVIG2V+zI13fM6elOGAFNA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2776 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9875 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101250107 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9875 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1015 phishscore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 suspectscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101250107 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For a given hugepage backing a VA, there's a rather ineficient loop which is solely responsible for storing subpages in the passed pages/vmas array. For each subpage we check whether it's within range or size of @pages and keep incrementing @pfn_offset and a couple other variables per subpage iteration. Simplify this logic and minimize ops per iteration to just store the output page/vma. Instead of incrementing number of @refs iteratively, we do it through a precalculation of @refs and having only a tight loop for storing pinned subpages/vmas. pinning consequently improves considerably, bringing us close to {pin,get}_user_pages_fast: - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 -L -S -n 512 -w PIN_LONGTERM_BENCHMARK: ~11k us -> ~4400 us PIN_FAST_BENCHMARK: ~3700 us Signed-off-by: Joao Martins --- mm/hugetlb.c | 49 ++++++++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 016addc8e413..1f7a95bc7c87 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4789,6 +4789,20 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, goto out; } +static void record_subpages_vmas(struct page *page, struct vm_area_struct *vma, + int refs, struct page **pages, + struct vm_area_struct **vmas) +{ + int nr; + + for (nr = 0; nr < refs; nr++) { + if (likely(pages)) + pages[nr] = page++; + if (vmas) + vmas[nr] = vma; + } +} + long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, struct page **pages, struct vm_area_struct **vmas, unsigned long *position, unsigned long *nr_pages, @@ -4918,28 +4932,16 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, continue; } - refs = 0; + refs = min3(pages_per_huge_page(h) - pfn_offset, + (vma->vm_end - vaddr) >> PAGE_SHIFT, remainder); -same_page: - if (pages) - pages[i] = mem_map_offset(page, pfn_offset); + if (pages || vmas) + record_subpages_vmas(mem_map_offset(page, pfn_offset), + vma, refs, + likely(pages) ? pages + i : NULL, + vmas ? vmas + i : NULL); - if (vmas) - vmas[i] = vma; - - vaddr += PAGE_SIZE; - ++pfn_offset; - --remainder; - ++i; - refs++; - if (vaddr < vma->vm_end && remainder && - pfn_offset < pages_per_huge_page(h)) { - /* - * We use pfn_offset to avoid touching the pageframes - * of this compound page. - */ - goto same_page; - } else if (pages) { + if (pages) { /* * try_grab_compound_head() should always succeed here, * because: a) we hold the ptl lock, and b) we've just @@ -4950,7 +4952,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, * any way. So this page must be available at this * point, unless the page refcount overflowed: */ - if (WARN_ON_ONCE(!try_grab_compound_head(pages[i-1], + if (WARN_ON_ONCE(!try_grab_compound_head(pages[i], refs, flags))) { spin_unlock(ptl); @@ -4959,6 +4961,11 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, break; } } + + vaddr += (refs << PAGE_SHIFT); + remainder -= refs; + i += refs; + spin_unlock(ptl); } *nr_pages = remainder;