From patchwork Sat Jan 16 00:31:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12024429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1BFEC433DB for ; Sat, 16 Jan 2021 00:33:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 55245229C9 for ; Sat, 16 Jan 2021 00:33:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 55245229C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DE1D88D01F8; Fri, 15 Jan 2021 19:33:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D93E18D01F6; Fri, 15 Jan 2021 19:33:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C34848D01F8; Fri, 15 Jan 2021 19:33:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id A8E638D01F6 for ; Fri, 15 Jan 2021 19:33:48 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 75E88181AF5C6 for ; Sat, 16 Jan 2021 00:33:48 +0000 (UTC) X-FDA: 77709765336.07.eyes35_451734527533 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 58DA01803F9B6 for ; Sat, 16 Jan 2021 00:33:48 +0000 (UTC) X-HE-Tag: eyes35_451734527533 X-Filterd-Recvd-Size: 20542 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Sat, 16 Jan 2021 00:33:47 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0XVeJ029233; Sat, 16 Jan 2021 00:33:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=wV6ngGm8Bte5h7bTxXDmG6bO4Bbj07mbez4jVE4asdI=; b=Yc5Sbn2WxMO+MlQHUNLXz84Vq5yFlMZpcoN0HKyDVI/S6CDLzc1Yr5oJhJ9H3toqPaTh Ic8TRUne5yxcWfnB19aSUrUbqFlHjt1yAk2wZTXm4ECW5+qQKVq+pdt7357QbsHRB1tS UsezcxD6lCqNd+C/vQvidfI2auCcbANOEHNXe/E93/PAvdnf9OILnbgXgDV2cSMozGDC J9qBucrxCwxZ3PkAuyB7WcEvMwoxirHXyYtIdZB5MqJ+s5Bl7vr9HlN1TKC6dCQehLnU PlXg0r5KxVOrouuZN2vOPm5Dt/eU5taVjkkqUdEUilZfXmO8ZpSvvFpGE/tl9iAKbT// yQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2120.oracle.com with ESMTP id 360kd077pw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:33:37 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0V38M035166; Sat, 16 Jan 2021 00:31:37 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3030.oracle.com with ESMTP id 363n8qs8r3-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q3kE54KquXzqytOTOPzJSL1JgUFHti5Pr6GxENdGIuQ3hm/QvmkNwWGnC0HgzDLSzQlFR7kWFRIvcnyaqJY4o/vVOew3Z+FhFwRFTlVu0Q3lZBmxMfV5raXSEbLLsvkJsK1iBbYPzmGdLnLp6kiPFKmq2zE3oTCEa3WnUmkGkR9vGT4lZ+6wX7Qezp8GLla4vr77j30aaNvGN2QDvIr38+S42Q1sFgsMJVWePk/iI90Xe7thbR5sZYHBNTzD1DBBzCK5F1F6JupaEKgj/dVN/kYeKKLvdkJkD80+xAyQ/ZI+EVv8Ln81cfDIDEp5SydknjDhOBmFYIVhKIlSZEvoJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wV6ngGm8Bte5h7bTxXDmG6bO4Bbj07mbez4jVE4asdI=; b=CNTEpkGlcj58zOLNVrq0G45PGuB8WnjUiA9ja0Msk7YXvU3q924btkrbADyIyL9NgZY4NBN80dCJgmGqLsXAzfx7rGpBnWwPMtp/tBU7XcTFVgh50A1/2CZ98MAh6bTPYQXiJlcJjYijS8C/jojU01+AtccgFt/UQ5DTxkP+DNUH57u/r3y36TQYMPkC9Bu5zwyekGWFFbDHaE8qNkz4fHFHQIXY3IbQzgmZHAJ/mxQ5bOETUsodqMEvVsWCxSttHUWt3T2oqUfSfXxHlmplzrPXChkNQ1m+Ko52MNqKQ+IMiYrr43P5AqsKmD4l56JfTk6/9Wk2bi1FRsmZW49YVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wV6ngGm8Bte5h7bTxXDmG6bO4Bbj07mbez4jVE4asdI=; b=HHBsktYZ6AYcmAdUeK84GM2l+43iJv93CgWsQOPvAe4y7f8aQdP/DH30k8PFNfM2G1lS6rhjmuJxE0RSo7Xl9bckLDJ1y/gR9zjJzBeL6GWK58GCuW7q0QecxUqbC8Urw2x7T41hF+G0xtjSlV4Rp9XKngPpDDoMXjgerSzJ83g= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4450.namprd10.prod.outlook.com (2603:10b6:303:93::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sat, 16 Jan 2021 00:31:26 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.012; Sat, 16 Jan 2021 00:31:26 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH 1/5] hugetlb: use page.private for hugetlb specific page flags Date: Fri, 15 Jan 2021 16:31:01 -0800 Message-Id: <20210116003105.182918-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210116003105.182918-1-mike.kravetz@oracle.com> References: <20210116003105.182918-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.11 via Frontend Transport; Sat, 16 Jan 2021 00:31:26 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b56a337f-3391-40c9-c66b-08d8b9b60f20 X-MS-TrafficTypeDiagnostic: CO1PR10MB4450: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kU0zFwKu3mseaJMKG72USYR6YoUDnOH4k5ilz6FjJmxTrcpoMiKz/Y2EmumxU1o5qV3PCaavVL0aQQ+qFJJJZf24XbPLREwGoQUM3SadoIUE14TXokfbXVhijzeeEkjRMfZa7QasPRZsnfIONmdPePthbCdAz1jloz7SaZU2Zzz4gsMvJGm+jyiqgmQJgwxxt7fO8QBS9FWu41RRghbzyrlotneYvfZRNbNU/la6113l+UGzhv0c8RC9DO9XSl/48PUFSvfeDeZbV6OYmTweToBmFDF9qRDU4JS2gJTv+JjyZfPphukC3I8z4jq4cXYqNJAEmtaN1fORlY8CKCfLAazmB8mbDkqQgL571avy5Zpft/IsKyy3AHYUNqGAxYsds/FOk1S9NxVMkRZw3QTPAA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(39850400004)(136003)(346002)(366004)(6486002)(478600001)(1076003)(30864003)(6666004)(86362001)(36756003)(2906002)(44832011)(316002)(5660300002)(956004)(8676002)(107886003)(16526019)(26005)(52116002)(4326008)(54906003)(8936002)(66946007)(66556008)(66476007)(2616005)(186003)(7696005)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: fIZp6t+ZPtqPuc07403jENoiOlt5jcHZI23utCeQ8FAN8bZ54gzAUaiDzcCzF6OEY1K7QASGFkt0W0h36Cgrr8O+YIIhbaFhvAzfXIl2ixW7+aJpLoUxHL4IF/CqhgjhbVGPFQnpUir5QNJAid9njkDd2AI/OJQiKeKCTOd72iGlWzVl64jNEAUpcNO2THQ2XqXaGfdrIt7PfgAOHSVXcP0ZofnWyZN/F5jZO7I3UYoNDtG+5T8xREPDqnGyaATkqUaMHeUqyGxxPFG+r6c4uoT5QtznwX8lWPMBJZ2Tykuc/IBPmoOys7/DpLv5xZjPRLGWCIJx4749ib09OneLqYMdJwMotnuq6gWYAPQFmoE0N9Pkn4fdbR55FscmfuUXG0pgT7cd8sWGLKjBvbkSY90KLsSeoSkxX3t2koP3xYotBiyEZLtoVTYyYsjcb1h5QJPLRA56T2T6f7zPADq0dbv7v0bXNrxF2p02DRktt3mxsFRtYeoPs7uY9aIpMWo9rayW7oKONjtZ39+Nmq42GkR8brP3uiU2Gnow1TB+EIhQFDeKr/rdcQfaA9IGX5D9q5M8NgJxJGjevCbNmce0EJ19oVwcOfNJqtiEgdXYaIAVTh5SI2NnCQbkvDEylvUhIfvt4hKaNZ6r6X7uxmrhVplec5ov/Q+Hhq4RkoEcGzb6Tj3ZZVI5/5Po/foEgjeLqIubhk0TvE4w/l7V4YlXTMJd+39xqA+AfozfsBrqsigsOF4heQuWqOtoDsGZ2+EdyZiUquen5HhBXwsMdoC66Sm65SUrX7IeQi2Fg3BVD4Sq35ZO05GKpJKkQ45e/P11mP4HSiMC8HX8DcMyxudh82zT4QGxYBzMtmwLiEBbqqww6GVpmmXdQ3DPIYgbOYPNDK4vBuNeOZdkKUsW9UWMcuOIQIF3ljZMAAYxObrKFU0yxmgsTQTPRn5Wmhzzbf4qcAoF0gzdwseo6lJDt2d911TXarsagNxNLIJ7eJhh+ao= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b56a337f-3391-40c9-c66b-08d8b9b60f20 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2021 00:31:26.7425 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lDdZg9O50JFBUSmZznXvvbPdqLkawtPMwt/M8/fBpuybKm+9FJUHuaXdhbA3TftRxqnNMsFvBP8b/1HFMHoqvA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4450 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 phishscore=0 impostorscore=0 bulkscore=0 adultscore=0 suspectscore=0 malwarescore=0 lowpriorityscore=0 clxscore=1015 mlxlogscore=999 mlxscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As hugetlbfs evolved, state information about hugetlb pages was added. One 'convenient' way of doing this was to use available fields in tail pages. Over time, it has become difficult to know the meaning or contents of fields simply by looking at a small bit of code. Sometimes, the naming is just confusing. For example: The PagePrivate flag indicates a huge page reservation was consumed and needs to be restored if an error is encountered and the page is freed before it is instantiated. The page.private field contains the pointer to a subpool if the page is associated with one. In an effort to make the code more readable, use page.private to contain hugetlb specific flags. A set of hugetlb_*_page_flag() routines are created for flag manipulation. More importantly, an enum of flag values will be created with names that actually reflect their purpose. In this patch, - Create infrastructure for hugetlb_*_page_flag functions - Move subpool pointer to page[1].private to make way for flags Create routines with meaningful names to modify subpool field - Use new HP_Restore_Reserve flag instead of PagePrivate Conversion of other state information will happen in subsequent patches. Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 12 ++------ include/linux/hugetlb.h | 61 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 46 +++++++++++++++---------------- 3 files changed, 87 insertions(+), 32 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 740693d7f255..b8a661780c4a 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -955,15 +955,9 @@ static int hugetlbfs_migrate_page(struct address_space *mapping, if (rc != MIGRATEPAGE_SUCCESS) return rc; - /* - * page_private is subpool pointer in hugetlb pages. Transfer to - * new page. PagePrivate is not associated with page_private for - * hugetlb pages and can not be set here as only page_huge_active - * pages can be migrated. - */ - if (page_private(page)) { - set_page_private(newpage, page_private(page)); - set_page_private(page, 0); + if (hugetlb_page_subpool(page)) { + hugetlb_set_page_subpool(newpage, hugetlb_page_subpool(page)); + hugetlb_set_page_subpool(page, NULL); } if (mode != MIGRATE_SYNC_NO_COPY) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ef5b144b8aac..64f8c7a64186 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -472,6 +472,19 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, unsigned long flags); #endif /* HAVE_ARCH_HUGETLB_UNMAPPED_AREA */ +/* + * huegtlb page specific state flags. These flags are located in page.private + * of the hugetlb head page. The hugetlb_*_page_flag() routines should be used + * to manipulate these flags. + * + * HP_Restore_Reserve - Set when a hugetlb page consumes a reservation at + * allocation time. Cleared when page is fully instantiated. Free + * routine checks flag to restore a reservation on error paths. + */ +enum hugetlb_page_flags { + HP_Restore_Reserve = 0, +}; + #ifdef CONFIG_HUGETLB_PAGE #define HSTATE_NAME_LEN 32 @@ -531,6 +544,38 @@ extern unsigned int default_hstate_idx; #define default_hstate (hstates[default_hstate_idx]) +static inline int hugetlb_test_page_flag(struct page *page, + enum hugetlb_page_flags hp_flag) +{ + return test_bit(hp_flag, &page->private); +} + +static inline void hugetlb_set_page_flag(struct page *page, + enum hugetlb_page_flags hp_flag) +{ + return set_bit(hp_flag, &page->private); +} + +static inline void hugetlb_clear_page_flag(struct page *page, + enum hugetlb_page_flags hp_flag) +{ + return clear_bit(hp_flag, &page->private); +} + +/* + * hugetlb page subpool pointer located in hpage[1].private + */ +static inline struct hugepage_subpool *hugetlb_page_subpool(struct page *hpage) +{ + return (struct hugepage_subpool *)(hpage+1)->private; +} + +static inline void hugetlb_set_page_subpool(struct page *hpage, + struct hugepage_subpool *subpool) +{ + set_page_private(hpage+1, (unsigned long)subpool); +} + static inline struct hstate *hstate_file(struct file *f) { return hstate_inode(file_inode(f)); @@ -775,6 +820,22 @@ void set_page_huge_active(struct page *page); #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; +static inline int hugetlb_test_page_flag(struct page *page, + enum hugetlb_page_flags hp_flag) +{ + return 0; +} + +static inline void hugetlb_set_page_flag(struct page *page, + enum hugetlb_page_flags hp_flag) +{ +} + +static inline void hugetlb_clear_page_flag(struct page *page, + enum hugetlb_page_flags hp_flag) +{ +} + static inline struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 737b2dce19e6..b01002d8fc2b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1133,7 +1133,7 @@ static struct page *dequeue_huge_page_vma(struct hstate *h, nid = huge_node(vma, address, gfp_mask, &mpol, &nodemask); page = dequeue_huge_page_nodemask(h, gfp_mask, nid, nodemask); if (page && !avoid_reserve && vma_has_reserves(vma, chg)) { - SetPagePrivate(page); + hugetlb_set_page_flag(page, HP_Restore_Reserve); h->resv_huge_pages--; } @@ -1407,20 +1407,19 @@ static void __free_huge_page(struct page *page) */ struct hstate *h = page_hstate(page); int nid = page_to_nid(page); - struct hugepage_subpool *spool = - (struct hugepage_subpool *)page_private(page); + struct hugepage_subpool *spool = hugetlb_page_subpool(page); bool restore_reserve; VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(page_mapcount(page), page); - set_page_private(page, 0); + hugetlb_set_page_subpool(page, NULL); page->mapping = NULL; - restore_reserve = PagePrivate(page); - ClearPagePrivate(page); + restore_reserve = hugetlb_test_page_flag(page, HP_Restore_Reserve); + hugetlb_clear_page_flag(page, HP_Restore_Reserve); /* - * If PagePrivate() was set on page, page allocation consumed a + * If HP_Restore_Reserve was set on page, page allocation consumed a * reservation. If the page was associated with a subpool, there * would have been a page reserved in the subpool before allocation * via hugepage_subpool_get_pages(). Since we are 'restoring' the @@ -2254,24 +2253,24 @@ static long vma_add_reservation(struct hstate *h, * This routine is called to restore a reservation on error paths. In the * specific error paths, a huge page was allocated (via alloc_huge_page) * and is about to be freed. If a reservation for the page existed, - * alloc_huge_page would have consumed the reservation and set PagePrivate - * in the newly allocated page. When the page is freed via free_huge_page, - * the global reservation count will be incremented if PagePrivate is set. - * However, free_huge_page can not adjust the reserve map. Adjust the - * reserve map here to be consistent with global reserve count adjustments - * to be made by free_huge_page. + * alloc_huge_page would have consumed the reservation and set + * HP_Restore_Reserve in the newly allocated page. When the page is freed + * via free_huge_page, the global reservation count will be incremented if + * HP_Restore_Reserve is set. However, free_huge_page can not adjust the + * reserve map. Adjust the reserve map here to be consistent with global + * reserve count adjustments to be made by free_huge_page. */ static void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, unsigned long address, struct page *page) { - if (unlikely(PagePrivate(page))) { + if (unlikely(hugetlb_test_page_flag(page, HP_Restore_Reserve))) { long rc = vma_needs_reservation(h, vma, address); if (unlikely(rc < 0)) { /* * Rare out of memory condition in reserve map - * manipulation. Clear PagePrivate so that + * manipulation. Clear HP_Restore_Reserve so that * global reserve count will not be incremented * by free_huge_page. This will make it appear * as though the reservation for this page was @@ -2280,7 +2279,7 @@ static void restore_reserve_on_error(struct hstate *h, * is better than inconsistent global huge page * accounting of reserve counts. */ - ClearPagePrivate(page); + hugetlb_clear_page_flag(page, HP_Restore_Reserve); } else if (rc) { rc = vma_add_reservation(h, vma, address); if (unlikely(rc < 0)) @@ -2288,7 +2287,8 @@ static void restore_reserve_on_error(struct hstate *h, * See above comment about rare out of * memory condition. */ - ClearPagePrivate(page); + hugetlb_clear_page_flag(page, + HP_Restore_Reserve); } else vma_end_reservation(h, vma, address); } @@ -2369,7 +2369,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, if (!page) goto out_uncharge_cgroup; if (!avoid_reserve && vma_has_reserves(vma, gbl_chg)) { - SetPagePrivate(page); + hugetlb_set_page_flag(page, HP_Restore_Reserve); h->resv_huge_pages--; } spin_lock(&hugetlb_lock); @@ -2387,7 +2387,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, spin_unlock(&hugetlb_lock); - set_page_private(page, (unsigned long)spool); + hugetlb_set_page_subpool(page, spool); map_commit = vma_commit_reservation(h, vma, addr); if (unlikely(map_chg > map_commit)) { @@ -4212,7 +4212,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, spin_lock(ptl); ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { - ClearPagePrivate(new_page); + hugetlb_clear_page_flag(new_page, HP_Restore_Reserve); /* Break COW */ huge_ptep_clear_flush(vma, haddr, ptep); @@ -4279,7 +4279,7 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping, if (err) return err; - ClearPagePrivate(page); + hugetlb_clear_page_flag(page, HP_Restore_Reserve); /* * set page dirty so that it will not be removed from cache/file @@ -4441,7 +4441,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) { - ClearPagePrivate(page); + hugetlb_clear_page_flag(page, HP_Restore_Reserve); hugepage_add_new_anon_rmap(page, vma, haddr); } else page_dup_rmap(page, true); @@ -4755,7 +4755,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, if (vm_shared) { page_dup_rmap(page, true); } else { - ClearPagePrivate(page); + hugetlb_clear_page_flag(page, HP_Restore_Reserve); hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); } From patchwork Sat Jan 16 00:31:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12024431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B802C433E6 for ; Sat, 16 Jan 2021 00:33:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9D6BF22AAC for ; Sat, 16 Jan 2021 00:33:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9D6BF22AAC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 668CA8D01F9; Fri, 15 Jan 2021 19:33:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 617928D01F6; Fri, 15 Jan 2021 19:33:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E0738D01F9; Fri, 15 Jan 2021 19:33:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id 3353D8D01F6 for ; Fri, 15 Jan 2021 19:33:49 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F377F8249980 for ; Sat, 16 Jan 2021 00:33:48 +0000 (UTC) X-FDA: 77709765378.24.jelly21_3616ef927533 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id D89AA1A4A5 for ; Sat, 16 Jan 2021 00:33:48 +0000 (UTC) X-HE-Tag: jelly21_3616ef927533 X-Filterd-Recvd-Size: 17165 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Sat, 16 Jan 2021 00:33:47 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0TLKc139387; Sat, 16 Jan 2021 00:33:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=1+UJZoZJUHtRu356Yu5J7VTmRAWtMXyT4hDL9j2ixC8=; b=wRV9YPTP10Q0MhoZLzT8JYPELYqLCaXbT7ZGOu4ugH1nGFMpt6z2i+W6Jw3JpNzhOxrI CgBdb7R+GQ+8Nvp8vBwaw8ojDNFwAXfZd0iIK36e+M4ZiHoVow5Oo0M2ggPADUW4vhwT qTM9/4UhSBQcy02O3f0RXHfbPF0xu/6PCn3dFRGz9agFdDhBss4CHLF6APDxhBZGHXPX rKHfVzg6xG8gYKGi5XCQYSiEsfBUocHnXzwCWKuab1FtU26MAizrPKccVhxj5+qKOOwK e7ba3DYKU42T0XhQdF2o3QiaH58dWktpGLkFnESskqtTRBhCTZrluUZ3CeHYk3Y0Rzp0 fw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 360kvkf62d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:33:38 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0V38P035166; Sat, 16 Jan 2021 00:31:38 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3030.oracle.com with ESMTP id 363n8qs8r3-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H1MaY3uoM8rf9Lr+6++0zu8FA7DozYLC8aIRRdhq9SHYuCzdIemqbxFherWGhQjGLJZIv3bhyccuOrXjr1+TifPuK7/psXHMkV9JlZt9vxqkORVwYsMJFMm4K+jTtWuk65XREsxSszFSaawsIYf4yUS8HiNYMfLinLTM5daA6cRqWZWTC9ZEtqHteqXBj6d/bEwSJzesRTXDQ14ASIz1F0BN1pgAe5FTrtGBJcZyKJjtr5C8HhUvYTq2tRmy9GkQcHAF1tMuBDP/o/V45LBHj6HLp6RpjQ0ofZeyxKerjf/ZMd7zybvj7Mpwsh2mMiISRpc6oj6cEjnh1XuBoxejxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1+UJZoZJUHtRu356Yu5J7VTmRAWtMXyT4hDL9j2ixC8=; b=MxxXVxsFO7jvClwBIWZ2//Uyb7vPVlF2QkruxA5xS1ccAJcJzlfA690crl3eRg37mEi8YbRORIIFeCUQvFyJsPBvzNzgA1yZQ/0xFwJjuXmRYq07kb5Us4hLJu0oQUHrJ+f/XNdWpeOwSyR6JHKh4/tbHrOzOCavpv3wKoXaD4t63O0OMdPKEW/fkZILMCdoTExv/ACoqSaV+NcClk0L1XuAVikQhrTUMHk+/uXxN5ePcElGySRNJwvxYvGWM1j0v1M0/CqUAlBJ1CLeMkBdaFWtky6EsAjVtZ85IZV/ewKFRMvSh4yudNzB35YQujAsePB1XMpLEo51GadMfP/BFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1+UJZoZJUHtRu356Yu5J7VTmRAWtMXyT4hDL9j2ixC8=; b=AK/HkJIAcAcmBbctt9MRVoIJ/1vHCuluIHaBsJl33DvbjVp8Sb0ZcxmsjgvLFrOkq5w0+mjxJCwUb6vEF/V6JjWV9vWzfqUt/69rPN5LOkeGb1VdsGO9tp7h07avtyFdxledqGnIXRJFSLC8PXq1tMcxnP2O4TModcbGmikmMWI= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4450.namprd10.prod.outlook.com (2603:10b6:303:93::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sat, 16 Jan 2021 00:31:28 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.012; Sat, 16 Jan 2021 00:31:28 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH 2/5] hugetlb: convert page_huge_active() to HP_Migratable flag Date: Fri, 15 Jan 2021 16:31:02 -0800 Message-Id: <20210116003105.182918-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210116003105.182918-1-mike.kravetz@oracle.com> References: <20210116003105.182918-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.11 via Frontend Transport; Sat, 16 Jan 2021 00:31:28 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e99cfc14-c953-4e21-13ba-08d8b9b6108c X-MS-TrafficTypeDiagnostic: CO1PR10MB4450: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RU8YNEVH7wsJ/3X6RoHYQkJoTyOXZzJ3AQhRJlV2TSeG2peJK70Ukpk1VP2ROnCW9qskiM8aPEIsVviYOV+KKocqgBib+oSeqMHKTYmuXTEOWkXec/4MDLYDLcirtGcDo72aXlx+UgIYXAnndegkkz9n5xspNLFZ6ygU6IoQnji7uvqUVFOzpClvzVHN1txfEGVfQIgsXUZVLGrRqD9K/LZb2+cTtTyabHVv8ELr2EJ8jZTNSa7IKbH4xvrY27vhZDDTK3hiK+QD3ckm9qq5erbuOROpIXzWmvTkm3NUy5PVnztnceBw4FUOUDtncPcQpEoJFx9oRgIVN7/PEBGMyJbAngb22Vr9bk9Lhah3i4esqG5usR1Xt5gZqROrSI8lmMrsqNkT57bCcfJETjeLAFB00rCmksA6DnNqyunlCVqXfBOiOSTdwZvu1gaMtKJr X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(39850400004)(136003)(346002)(366004)(6486002)(478600001)(1076003)(6666004)(86362001)(36756003)(2906002)(44832011)(316002)(5660300002)(956004)(8676002)(107886003)(16526019)(26005)(52116002)(4326008)(54906003)(8936002)(66946007)(66556008)(66476007)(2616005)(186003)(7696005)(83380400001)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: syhWH4JXS9dd+8QhX0WvEWNBEOyY276SP528yGKurG9LVvxPqcaE18PkNx6nosz/ZhesnspZZSyfKZKccj0kUQANFUbbFfOv9wLDaR/1yUm6tupNoW7VDaMmKrzEgPzsj6I4dLfflkBa1GDO3dvu0aCz0zok//mWlcBOId7Bo3WgY3BOfv9QqSmoohyK9Jd5oKeCMQILeJno/4HbXl/nOkdgJVZq238ZafFTy4QloxvTY/qG86l2PHPqzfpEpdTZ94ylu7LZ/CnoHV9ZVM2HdkgOdetRpXZvetr+hWH9cCAEy1QDvOShqkK5S5hr3AtWGpsCrSVorn5BPibQr1e8M6AzCdS+zZ0rHTDpIXVT0jyuSXhrlIA2FAKy6nS8NKnzO5g5stcWAeXElEqboDTCUx4D0Gzsl+4MEUXDejE0Bd+V7oGZ+an0mW9mpU0IIv+nqtRkwz6J6hc0SwRuafdjGVBi2j4df3O9zqqyP1/JoFo1K+TdjDkwmNZ8CyhdLINQ56MZy0S9uB+mMwqNqKsa0mZsjuTh5pHDAfiIaHGW7ja7WLGoIv9ndupAFIAU9hfpZcO8oSfleD/CbXX+03uTypTOsyppROJStZWC56VHc68GwYKH7zVyqSMOLNtDR87hkB1G+G0a23eK4momTaoprAPIpx9LLCJYOpUU+79wnfHOQoBMLi8/WRUFL5/PgqjIJSDHrPTgR4FWy7806+WH6pD/98Ukt31bIn+tUYclD8xX72x0AqW2u7Vue1M+K5jgDgRkhUPKdV50hD63CepIvKuy5LHWUAp/2c7iQGxEohrlavWFOAJ7pBdH8rrVKPrhSN4uRm5dRU0XF3igBPIF3rPCZz26d9iNpnxa6ZAr7+gYeSbfNwZ74UQ3UIAPfLjOmMKuTjLPg3bT+j3LW4lL0eNb4WLzOmErnfwm+YldQVNRBoKabS0KiATvT8M2TbOD2ZxzL7VagqkOFeCDyruqS9eCamwvjNN3z0GcSFGsz28= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e99cfc14-c953-4e21-13ba-08d8b9b6108c X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2021 00:31:28.7806 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uYWqgfvFxUUTopfzFF40y+yXcgV6Wuk5Hvfkv4y2jZH2mJ78imBy9zM4XACT+7/J3yGuVZK8wzFoMY5K9fqefA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4450 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 phishscore=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 impostorscore=0 spamscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the new hugetlb page specific flag HP_Migratable to replace the page_huge_active interfaces. By it's name, page_huge_active implied that a huge page was on the active list. However, that is not really what code checking the flag wanted to know. It really wanted to determine if the huge page could be migrated. This happens when the page is actually added the page cache and/or task page table. This is the reasoning behind the name change. The VM_BUG_ON_PAGE() calls in the *_huge_active() interfaces are not really necessary as we KNOW the page is a hugetlb page. Therefore, they are removed. The routine page_huge_active checked for PageHeadHuge before testing the active bit. This is unnecessary in the case where we hold a reference or lock and know it is a hugetlb head page. page_huge_active is also called without holding a reference or lock (scan_movable_pages), and can race with code freeing the page. The extra check in page_huge_active shortened the race window, but did not prevent the race. Offline code calling scan_movable_pages already deals with these races, so removing the check is acceptable. Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 4 ++++ include/linux/page-flags.h | 6 ----- mm/hugetlb.c | 45 ++++++++++---------------------------- mm/memory_hotplug.c | 8 ++++++- 5 files changed, 23 insertions(+), 42 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b8a661780c4a..89bc9062b4f6 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 64f8c7a64186..353d81913cc7 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -480,9 +480,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HP_Restore_Reserve - Set when a hugetlb page consumes a reservation at * allocation time. Cleared when page is fully instantiated. Free * routine checks flag to restore a reservation on error paths. + * HP_Migratable - Set after a newly allocated page is added to the page + * cache and/or page tables. Indicates the page is a candidate for + * migration. */ enum hugetlb_page_flags { HP_Restore_Reserve = 0, + HP_Migratable, }; #ifdef CONFIG_HUGETLB_PAGE diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index bc6fd1ee7dd6..04a34c08e0a6 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -592,15 +592,9 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); -bool page_huge_active(struct page *page); #else TESTPAGEFLAG_FALSE(Huge) TESTPAGEFLAG_FALSE(HeadHuge) - -static inline bool page_huge_active(struct page *page) -{ - return 0; -} #endif diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b01002d8fc2b..c43cebf2f278 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,30 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Test to determine whether the hugepage is "active/in-use" (i.e. being linked - * to hstate->hugepage_activelist.) - * - * This function can be called for tail pages, but never returns true for them. - */ -bool page_huge_active(struct page *page) -{ - return PageHeadHuge(page) && PagePrivate(&page[1]); -} - -/* never called for tail page */ -void set_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - SetPagePrivate(&page[1]); -} - -static void clear_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - ClearPagePrivate(&page[1]); -} - /* * Internal hugetlb specific page flag. Do not use outside of the hugetlb * code @@ -1438,7 +1414,7 @@ static void __free_huge_page(struct page *page) } spin_lock(&hugetlb_lock); - clear_page_huge_active(page); + hugetlb_clear_page_flag(page, HP_Migratable); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); hugetlb_cgroup_uncharge_page_rsvd(hstate_index(h), @@ -4221,7 +4197,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - set_page_huge_active(new_page); + hugetlb_set_page_flag(new_page, HP_Migratable); /* Make the old page be freed below */ new_page = old_page; } @@ -4458,12 +4434,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spin_unlock(ptl); /* - * Only make newly allocated pages active. Existing pages found - * in the pagecache could be !page_huge_active() if they have been - * isolated for migration. + * Only set HP_Migratable in newly allocated pages. Existing pages + * found in the pagecache may not have HP_Migratable set if they have + * been isolated for migration. */ if (new_page) - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); unlock_page(page); out: @@ -4774,7 +4750,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); if (vm_shared) unlock_page(page); ret = 0; @@ -5592,12 +5568,13 @@ bool isolate_huge_page(struct page *page, struct list_head *list) bool ret = true; spin_lock(&hugetlb_lock); - if (!PageHeadHuge(page) || !page_huge_active(page) || + if (!PageHeadHuge(page) || + !hugetlb_test_page_flag(page, HP_Migratable) || !get_page_unless_zero(page)) { ret = false; goto unlock; } - clear_page_huge_active(page); + hugetlb_clear_page_flag(page, HP_Migratable); list_move_tail(&page->lru, list); unlock: spin_unlock(&hugetlb_lock); @@ -5608,7 +5585,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - set_page_huge_active(page); + hugetlb_set_page_flag(page, HP_Migratable); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index f9d57b9be8c7..10cdd281dd29 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1260,7 +1260,13 @@ static int scan_movable_pages(unsigned long start, unsigned long end, if (!PageHuge(page)) continue; head = compound_head(page); - if (page_huge_active(head)) + /* + * This test is racy as we hold no reference or lock. The + * hugetlb page could have been free'ed and head is no longer + * a hugetlb page before the following check. In such unlikely + * cases false positives and negatives are possible. + */ + if (hugetlb_test_page_flag(head, HP_Migratable)) goto found; skip = compound_nr(head) - (page - head); pfn += skip - 1; From patchwork Sat Jan 16 00:31:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12024425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35844C433E0 for ; Sat, 16 Jan 2021 00:32:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E849229C9 for ; Sat, 16 Jan 2021 00:32:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E849229C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2F20E8D01F7; Fri, 15 Jan 2021 19:31:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 22C508D01F6; Fri, 15 Jan 2021 19:31:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA93E8D01F7; Fri, 15 Jan 2021 19:31:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id C81E28D01EC for ; Fri, 15 Jan 2021 19:31:58 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 96D78180AD801 for ; Sat, 16 Jan 2021 00:31:58 +0000 (UTC) X-FDA: 77709760716.05.show48_06163fa27533 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 7DDDB1802ACD8 for ; Sat, 16 Jan 2021 00:31:58 +0000 (UTC) X-HE-Tag: show48_06163fa27533 X-Filterd-Recvd-Size: 13798 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sat, 16 Jan 2021 00:31:57 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0TIwe139352; Sat, 16 Jan 2021 00:31:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=Pex/uPAMbaGLWr+GH0t7IsuQ8V0vZVhyQQ9TNY71Cgw=; b=YrZGAdpMpqYY2Q9wPrHZfIIV+kZv+kTUOtIuuiSanaOdOeXO7ycwgCXfrNwOdZ7Jk8iv SnsDw8YpJa/FbHAw9vxZvzqwmVPb0m/wlPoFuNaD6jTEAP+mKf9jPHnl28tqlfhG7zQP I3KEPN+fDgz9IWzgaLVX2qZZjE+SybQD0mrcU8DPT4Wh7i/aQRPAkF1Ja6B+qm7+qaqJ ESh+SYg3Gy/1pR5bFO+ZQQcikcAK4VDZHgQlkO49Qbvp+TsyhvS2IHoPz+DMV9jErKJQ qIqWvXtTsdGsn1Dw2ultAirz26UbM8sXx1Ug7wzTYJB1zLYAXe8D92O6uDXqjYg8iIMl 3w== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 360kvkf5yd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:40 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0V38R035166; Sat, 16 Jan 2021 00:31:39 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3030.oracle.com with ESMTP id 363n8qs8r3-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X+sbQs7lrnR6EVpWblXSLt6RWJg43m8HFWOZtFT0rN2q/HZuYgIfX+HIoGPQGvjAkuGJco0+a4BO4gHOuDq/+HWCDmD4BrDwDFkahzGPVLIlRfazd/EdJ5BPF9EFkrCFWj2gEBn3poeJols8Y7hRrgVDxJvNE3Sq9irHbblY8MJzNwZo28Se9JAVrqglisBx1SEB0TbMq1f+ImoiDa5JijYi6TeSXdO5oW1Yt6mvCiThv9lgBMQ5CFPY6e6FIusGusO7DCpReI8FLlrYxv0cMg6hAkhgR9tcjF1hSLomRtNETSfcVJYNFAES1e8B6k3XCqn7Iv2K6IMEMlpKmGQUsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pex/uPAMbaGLWr+GH0t7IsuQ8V0vZVhyQQ9TNY71Cgw=; b=C5ypaY52CfR5KSyphW48AZvSshIl2jix+ilj682fF3HZPSmJjPrQuBUfOJNzhZ8vWotKpsYI/eiNGdsXJLgVzezt8485Sw5++p75v384htWelw9GPRPPDJfLQVmi97fHsG0OYx4aK2fqDE+xW5AuLrd+nNHtUMdqLs2EwdiLoOYRzIf5H9DdEMmr7xwKsjh097gv2uF8h/zOo661YjO0twfQJYprViBDsJoMsj0LEiONL6cxoUYApInUP/E1SDJ/srinnNnPT9fsgaNuT0lMO8ddS1+VyOICjgRm4QtRU5OzKrKyY3wKRE1SuwMGgLbQ5vL+C/sFBPVG2m7lNq4e3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pex/uPAMbaGLWr+GH0t7IsuQ8V0vZVhyQQ9TNY71Cgw=; b=MRXq34/cYm1KK/eiCFFB5uHstMbtuUquA+2XnlfhDW2rB9pAPXkZQUTSIhklaQAzEvkZ3scvBLIoYzRflKc2XvfEmOfYKjVCuTZDjdHDm9CpljoEWBgSCAilpjXaUs2vMq/PWmqEM4Ets5EdQbueL0ZRhmuUQDdIehFFpVfhyaM= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4450.namprd10.prod.outlook.com (2603:10b6:303:93::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sat, 16 Jan 2021 00:31:30 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.012; Sat, 16 Jan 2021 00:31:30 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH 3/5] hugetlb: only set HP_Migratable for migratable hstates Date: Fri, 15 Jan 2021 16:31:03 -0800 Message-Id: <20210116003105.182918-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210116003105.182918-1-mike.kravetz@oracle.com> References: <20210116003105.182918-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.11 via Frontend Transport; Sat, 16 Jan 2021 00:31:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 441bc66f-792e-450c-a244-08d8b9b61181 X-MS-TrafficTypeDiagnostic: CO1PR10MB4450: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bW/PymAOwrWk4F3wHOMuu3ZmewO9tmYTZLVN8TqeRrES1skeJGMM4ka0fpw1iLwfzbJsvIor5JeTlOK2MO3yMc17ZtyQLHNXcHwVNVr4dJfDiGiZlGgXrwuQoZ5VixBaaak4Nb2byy6LTt6bgSIKEkPiev/eHBkNv5eS4LV6lEohVzZp1jpm2Mi3KWbNg4S6VxhGiVx1tLC7bGbTtCuYhq80eu5TeYcHtmDxF7W54qeTnkLvcT4cxdFzFSeEdJgHoUHtXRFXBOq7b+fG5UdUF6BdKlGVzCb0SSg2LhrX/4UA2+BljI75gUSYLNgkhQv9ubWfAf0QRMdHmAwjuB5GDC05HfrFQDe6/Sj0zB1etmm72AxrpqztU2NYi1q9gI+v X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(39850400004)(136003)(346002)(366004)(6486002)(478600001)(1076003)(6666004)(86362001)(36756003)(2906002)(44832011)(316002)(5660300002)(956004)(8676002)(107886003)(16526019)(26005)(52116002)(4326008)(54906003)(8936002)(66946007)(66556008)(66476007)(2616005)(186003)(7696005)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 7x891Zl9Gv1pWtI17FlfCTcMGR1q+q7doHaB+f1fLZNyqLSxQrHaHY7BzJX+lcM4HYhBLB2qoAbC3WyhTWKdLD8ehFKfOpMj1UN1TqVgmB3ZyFO2EqqiWq+/jxIuyB0uQF3m22OwRCFwSaiIG7Rn/rFHN1FAY0kFu4d9pMzuY+mG+DXR6I64msKPBX62hR3J8B1RqFiKvCyTmAYjUb+3uM/+Rv8BBnwmYSC/FtD9/RTr2C+9I4glWzG/ogVeL2qZVK3f95DWe8BSJqCLo6yQSa4T07o0NlS3KH4etpwtfYl59HQdKVPYm8qpNDmARbtgAfPL9iy2b9054ZFhxxekvXd2+utXbRblCc5IYLfELxDf+mdo3HRJeM/1gMj46SNzn+c0nU75NbFMUmivcTrXJsrT47ikIgXGtV4OiIVrkxsGIbuCiuDBKvAkW1C+whdl5gCSlpSd4ieaCUefdkaQptrRi19tPlS9y7liDf51AtMuMKcboHQYVDAG0/U2ikVZ7ugGaoP+2YovTfHJ5fOMXhAPH5ccb1ct8VDK5nkBdk1HwY7dSCFtBPM4HJm/o+mTIQRtjZ3HVkPzkI3LbXOZBIosm5tgZDk/tdTv/ZLT1Q4kFC8q6NokPwGFNNRE209FiBkGy2KIL5wW3WkN6vxKfHOKB6Q1spdwLE2TM9Fdv0rDrviv30BndcwpHTHDe7tLnXWidvfN8SfGSqLEub3WHzrnSjnHeSqIZm+kLbmUdcGqPy+okEN6tlrxGbhKzW+Abkm5CTYn3q3LN04Y6TlVNZOmq3Pupfdml+NWAZAh61bQUx3pW2D/T71QJ6sn0PBxCKQMxvyDPEPDc1NGtPZZCSXLS0tlq3o+WWejzLPm37b0ZwCWUBLqRMV2DAsyGOjfEjJN11AV8xJBvipabmOTy4f1MuFfhynynmWZyARArlwfvVchFoJAS7n4ntf/UMYzxwsnXKsohWdx5B+Orw38dEFDqU6aOkjuLT6TFp4XSh4= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 441bc66f-792e-450c-a244-08d8b9b61181 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2021 00:31:30.3759 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: smAfUlJDqVrozUXKW0uCUmQou099ZEB62dl6+yGDHdLICO+ScAREL+kWscVpPe0D2aNvqJ8fXvoSjPTC2t/86w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4450 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=633 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=683 phishscore=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 impostorscore=0 spamscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The HP_Migratable flag indicates a page is a candidate for migration. Only set the flag if the page's hstate supports migration. This allows the migration paths to detect non-migratable pages earlier. The check in unmap_and_move_huge_page for migration support can be removed as it is no longer necessary. If migration is not supported for the hstate, HP_Migratable will not be set, the page will not be isolated and no attempt will be made to migrate. Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 9 +++++++++ mm/hugetlb.c | 8 ++++---- mm/migrate.c | 12 ------------ 4 files changed, 14 insertions(+), 17 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 89bc9062b4f6..14d77d01e38d 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - hugetlb_set_page_flag(page, HP_Migratable); + hugetlb_set_HP_Migratable(page); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 353d81913cc7..e7157cf9967f 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -716,6 +716,15 @@ static inline bool hugepage_migration_supported(struct hstate *h) return arch_hugetlb_migration_supported(h); } +/* + * Only set flag if hstate supports migration + */ +static inline void hugetlb_set_HP_Migratable(struct page *page) +{ + if (hugepage_migration_supported(page_hstate(page))) + hugetlb_set_page_flag(page, HP_Migratable); +} + /* * Movability check is different as compared to migration check. * It determines whether or not a huge page should be placed on diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c43cebf2f278..31e896c70ba0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4197,7 +4197,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - hugetlb_set_page_flag(new_page, HP_Migratable); + hugetlb_set_HP_Migratable(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -4439,7 +4439,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * been isolated for migration. */ if (new_page) - hugetlb_set_page_flag(page, HP_Migratable); + hugetlb_set_HP_Migratable(page); unlock_page(page); out: @@ -4750,7 +4750,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - hugetlb_set_page_flag(page, HP_Migratable); + hugetlb_set_HP_Migratable(page); if (vm_shared) unlock_page(page); ret = 0; @@ -5585,7 +5585,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - hugetlb_set_page_flag(page, HP_Migratable); + hugetlb_set_HP_Migratable(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/migrate.c b/mm/migrate.c index 0339f3874d7c..296d61613abc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1272,18 +1272,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; - /* - * Migratability of hugepages depends on architectures and their size. - * This check is necessary because some callers of hugepage migration - * like soft offline and memory hotremove don't walk through page - * tables or check whether the hugepage is pmd-based or not before - * kicking migration. - */ - if (!hugepage_migration_supported(page_hstate(hpage))) { - list_move_tail(&hpage->lru, ret); - return -ENOSYS; - } - if (page_count(hpage) == 1) { /* page was freed from under us. So we are done. */ putback_active_hugepage(hpage); From patchwork Sat Jan 16 00:31:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12024427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 527EDC433E6 for ; Sat, 16 Jan 2021 00:32:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D2F0B229C9 for ; Sat, 16 Jan 2021 00:32:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D2F0B229C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 84CEB8D01EC; Fri, 15 Jan 2021 19:31:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 425BD8D01F8; Fri, 15 Jan 2021 19:31:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A0738D01EC; Fri, 15 Jan 2021 19:31:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id D74988D01F6 for ; Fri, 15 Jan 2021 19:31:58 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 947B92478 for ; Sat, 16 Jan 2021 00:31:58 +0000 (UTC) X-FDA: 77709760716.20.care09_101552127533 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 74345180C07A3 for ; Sat, 16 Jan 2021 00:31:58 +0000 (UTC) X-HE-Tag: care09_101552127533 X-Filterd-Recvd-Size: 13275 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Sat, 16 Jan 2021 00:31:57 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0UOwt140194; Sat, 16 Jan 2021 00:31:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=fLFZlJKB73ggWSUtIHsj4EWpsN6eIRM3PQlQM0oiwBE=; b=uQzAOlRvqJEN/qUSSvEFst96W0XlEHUsN/wem9tGYEbhxO5olxbhgm9Gcu+XcOUmyPIG /v4L9p8Z/t0459WQ4EjaG6ARag7aojg6eHTopeEp5kpKD14PQwjiKI5XazJUyz1uqU5b G5fWRuVp5/CNXSY/LHcfqXizIX25MrF8PV0AznhBcvcPm57qMrWAe+3TvS41V22Nd+Ay G3mJLBtEH06fIyurW0vzkmNT4g4XeFWRChSAzlKgrMlIXDXx/zuGv8wIz9lslhssayA9 tJ74SDT7m4rtTYdJ+w2bX3/QzR01xVFgsjIZwWL3shsl+wbUC4g+JuM3QmoxhYxVB5t4 xA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 360kvkf5ye-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:41 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0V38T035166; Sat, 16 Jan 2021 00:31:41 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3030.oracle.com with ESMTP id 363n8qs8r3-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mmXSGfa2ceDjuojeQ1un+xTnFO3ByZWiCsk6v7PGJfsfckgolMTeyr3WAqQtM6RatIRmMyId1LmZz8azhwUA5eNSc/KiZIF4ybi3nOyOR0OnA0FHCG9uilTYDMRLSlfT0SVReIZIsdNBaLEYIA5SP4uWSyzckoiUIZYMgawPLwjT5NgddJxwVjS5z6PHV81kSKsc6TQvAQQALSTgtdAG4Z/5VHV+M4dfgcenMHKrclY/J8+Zk/BMRaeYodOO7Dry3hHs68suqoErvNt4fWDN5PXR6xdvRPcbjHB+H42vVfUfrCKQ8C1p6I0+vUUxufHwruvMXJW1jxwnUAytrIuXjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fLFZlJKB73ggWSUtIHsj4EWpsN6eIRM3PQlQM0oiwBE=; b=CZ1+uDLw0FScOkRxlKE1+aDYMpm8rHHBxiIGChK3NnLXBLOlNCSGSaW46Bm1wDCpfp19HpwPkfEnCtqE7nYnCPDPfBqPciejaF/BGJtVhqY2TlZGJ37TZs7AVW+IqhhgQ9SGMn3vYphX2cNUBjA91rWXvhB8zUglxNS/5DU7JBpuSp+wH/FYMo5/WKPhMTHdS8pEjGr3D4JuwVojMQGUa9Rt6SPZYGNjyzCBNeKO2dTVJSBCryk1ZaoxpEUrePUWXW+oDtfURZJFqr2EiAXCUiSqLaGlQIWwmaw28X1T/anfgMhvzdi8RyGR8WxmWsTZO9NPzuepPA50Hx0YN8sB/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fLFZlJKB73ggWSUtIHsj4EWpsN6eIRM3PQlQM0oiwBE=; b=gF8Q7155zucrPUzbR7SefULVxOgUtgbM4ArbpABRzKhf+9jISVlBS0DujbyY5JOFF1Sc3ufkW4NmItY2eHSoIPslgIPHp8kBxQn8GPYcDAXQS4CEwmaci4/rox1cd+3YxpA/ehZSfvOr1eMlizbWnlB68RxnSpZyvQHBAy1UQUc= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4450.namprd10.prod.outlook.com (2603:10b6:303:93::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sat, 16 Jan 2021 00:31:31 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.012; Sat, 16 Jan 2021 00:31:31 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH 4/5] hugetlb: convert PageHugeTemporary() to HP_Temporary flag Date: Fri, 15 Jan 2021 16:31:04 -0800 Message-Id: <20210116003105.182918-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210116003105.182918-1-mike.kravetz@oracle.com> References: <20210116003105.182918-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.11 via Frontend Transport; Sat, 16 Jan 2021 00:31:31 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 642954ef-001d-4a3d-d2c5-08d8b9b61251 X-MS-TrafficTypeDiagnostic: CO1PR10MB4450: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XZNBgdkLhYbeFq5O3TAiG3SrO9SMF0EF3isYxTI8PhtwRgL9NEVxmxuMgtkha25vYTUgOOb+Pqqs6iQI1BZh1uIXYXQPQXZEOKqkxDrMYfKA5yVmDw6s2FZ5s/Fz1kpwK+TWX8vHyrc75ZPxJ+e5KwiBqJMXdd8Fr13w0E8ZSPE/jRvSHvyPNSFZDSp/vVfW2A4yX+mz68T+2/0ssjMfv9smC7zSd54ukqnUTlZ+kKZVsqn6nw4rpGK2z7C4wm1u9wBiQ1bwdQS6AcYyuInBK9v1aSSv/ZNGLymglIRV8wvBok9u0nvg5WvOM+18T5F6YzwDmtzEnzF4pOz8f0ga+i3YTGb51j47wP0Ty04tTpwJcc1ks78aEGppDd7haJuGY3A9Z5Vha+mkrtD0ZKF+bw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(39850400004)(136003)(346002)(366004)(6486002)(478600001)(1076003)(6666004)(86362001)(36756003)(2906002)(44832011)(316002)(5660300002)(956004)(8676002)(107886003)(16526019)(26005)(52116002)(4326008)(54906003)(8936002)(66946007)(66556008)(66476007)(2616005)(186003)(7696005)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: aXcq/MvvmSWgVL4slxHfUOJ98KLiqqAUythmDi55as3d4MLIHQUmZpmbyzd8/YMPrq6Et/wasuWFlP+vjItDe+G/0MNZap5/Ae5QCUUKISCwht2je2/EgVlPyf/QHReYPlmhFZHVZRo3WUvMhlJBC/GkETKlqUVoSkaD+J/haPU2ZFqdFa4ppqnSD7fIVyX2zHNeQ/m5hYU3XoHq1WFcYaKiKjNrFRk8reQ+RVwMBKzguSLQYPD2/vL5Q9B9aRQl5Ca2lfJgQy0ECfQpsTXnE7V8l8LXfOK41inSVHb6Ge0JIntOifiqburDo+Fi0/MtiQ9e5HN4SM+EfOk71JW10rWIb17ay6+YU86aqFeg9DAwnp2GlZ6CBjAYtpox+Mn0FUFMyicDdSFTsnRXNMKCotRpXEBA1QPl/ABy2CGe5PpZDYr9i8CUyzkx/eb/9gFE6iXdR+fo7PBZwgg/jUEedhLWiR8zyZi28IWmCO3IsHAGI50x3AjGfMksFdCq12+Kend2AIoKaIvjjuTtkPfikmH1rr0OZ5AxQIPwmmC0LUX8tW/CxiEEDNuRhNGVeGy92uPmMrUrs+X6a3wIviijBr2MdXm3zMK0G8i1M+T2aBK/3UnIOdm75+Sb6NhWmO+5HuyeP8VwHnxEpI8+B3L8nLk4jNHjRX+HN+KJlragNW2fSKd1lWaZIwoR1EdGUHlni6LNbhJCNV4CYEaTycUOWevx0rncVf/q74VLNghprVWfABMlg7myIPZi53ijTdEIcyR5DXa6cfWPKOiU88LUGa6YYJmealC5H9Id+klEYewXhAmaOeCENjx3jTo82G+yy87nFxyPjvBGm+C0QOkvhai8sjfpZuUlkFC/nhQcMDLpApK3o3RAlG4t1nLTe0CM2EBTcSAhZrt8HcNAb6cL54oRTQ7auvUY/dBUlZyCePh6K6YwQIrh5IpJddKnh+Ek7Zoky2PnloP7mCLOwW/ZyFL8UEhZUlhQ3yxV9BDqZ5k= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 642954ef-001d-4a3d-d2c5-08d8b9b61251 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2021 00:31:31.6843 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WRV/jIxW8JJk1l+pIe6G+Bk0uL+0A7i138TTNYT9zGvpFfA8sfWcmld0MXCtlpq/wKIctJqOh6FW4QvbaD0E5w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4450 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 phishscore=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 impostorscore=0 spamscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new hugetlb specific flag HP_Temporary flag to replace the PageHugeTemporary() interfaces. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 5 +++++ mm/hugetlb.c | 36 +++++++----------------------------- 2 files changed, 12 insertions(+), 29 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e7157cf9967f..166825c85875 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -483,10 +483,15 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HP_Migratable - Set after a newly allocated page is added to the page * cache and/or page tables. Indicates the page is a candidate for * migration. + * HP_Temporary - Set on a page that is temporarily allocated from the buddy + * allocator. Typically used for migration target pages when no pages + * are available in the pool. The hugetlb free page path will + * immediately free pages with this flag set to the buddy allocator. */ enum hugetlb_page_flags { HP_Restore_Reserve = 0, HP_Migratable, + HP_Temporary, }; #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 31e896c70ba0..53e9168a97bd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,28 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Internal hugetlb specific page flag. Do not use outside of the hugetlb - * code - */ -static inline bool PageHugeTemporary(struct page *page) -{ - if (!PageHuge(page)) - return false; - - return (unsigned long)page[2].mapping == -1U; -} - -static inline void SetPageHugeTemporary(struct page *page) -{ - page[2].mapping = (void *)-1U; -} - -static inline void ClearPageHugeTemporary(struct page *page) -{ - page[2].mapping = NULL; -} - static void __free_huge_page(struct page *page) { /* @@ -1422,9 +1400,9 @@ static void __free_huge_page(struct page *page) if (restore_reserve) h->resv_huge_pages++; - if (PageHugeTemporary(page)) { + if (hugetlb_test_page_flag(page, HP_Temporary)) { list_del(&page->lru); - ClearPageHugeTemporary(page); + hugetlb_clear_page_flag(page, HP_Temporary); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ @@ -1863,7 +1841,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, * codeflow */ if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { - SetPageHugeTemporary(page); + hugetlb_set_page_flag(page, HP_Temporary); spin_unlock(&hugetlb_lock); put_page(page); return NULL; @@ -1894,7 +1872,7 @@ static struct page *alloc_migrate_huge_page(struct hstate *h, gfp_t gfp_mask, * We do not account these pages as surplus because they are only * temporary and will be released properly on the last reference */ - SetPageHugeTemporary(page); + hugetlb_set_page_flag(page, HP_Temporary); return page; } @@ -5608,12 +5586,12 @@ void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) * here as well otherwise the global surplus count will not match * the per-node's. */ - if (PageHugeTemporary(newpage)) { + if (hugetlb_test_page_flag(newpage, HP_Temporary)) { int old_nid = page_to_nid(oldpage); int new_nid = page_to_nid(newpage); - SetPageHugeTemporary(oldpage); - ClearPageHugeTemporary(newpage); + hugetlb_set_page_flag(oldpage, HP_Temporary); + hugetlb_clear_page_flag(newpage, HP_Temporary); spin_lock(&hugetlb_lock); if (h->surplus_huge_pages_node[old_nid]) { From patchwork Sat Jan 16 00:31:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12024423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AECD7C433DB for ; Sat, 16 Jan 2021 00:31:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2DCD322AAC for ; Sat, 16 Jan 2021 00:31:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2DCD322AAC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 473708D01F5; Fri, 15 Jan 2021 19:31:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D4898D01EC; Fri, 15 Jan 2021 19:31:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 275EF8D01F5; Fri, 15 Jan 2021 19:31:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0166.hostedemail.com [216.40.44.166]) by kanga.kvack.org (Postfix) with ESMTP id 0A3468D01EC for ; Fri, 15 Jan 2021 19:31:58 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B9C54180AD802 for ; Sat, 16 Jan 2021 00:31:57 +0000 (UTC) X-FDA: 77709760674.06.paper91_290eec027533 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 8C6AD1003A871 for ; Sat, 16 Jan 2021 00:31:57 +0000 (UTC) X-HE-Tag: paper91_290eec027533 X-Filterd-Recvd-Size: 12564 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Sat, 16 Jan 2021 00:31:56 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0UnKL160205; Sat, 16 Jan 2021 00:31:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=U9MCdQB1uUR6Oa5upkJkKkOf9bLofSyZk6D883uy1PM=; b=I10kiruZ9yE2nyOs28CYurBJInIHNB2ZoHSS8HdIyPoauXkw3w8UFljk/Kgr9mtdRIJ3 7i8PJMwrRvXoxRWFtpeIAsWavnnDHaOMP16adxu3TFlUAI8b277v21Hv1CA/qhSGzma8 JK7veEiVtaMsFSPZmppSZ07GV3mwBXjcLmiR51w422wzreL/GqTq6vA5iQsa96oamLS7 rc8jqY75WeKpaCnJOaSzoBXzxzjvvGyu4JcsIjXAMyOUp4DMK4WuLLilG+yxZaasd1CS G1P1npnunGN0+N+j5ukDoSFgaVYbMgsCHZH228dx4iqk48col4jproDpb2x5yZkBtyGm 7A== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2130.oracle.com with ESMTP id 363nna801w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:42 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10G0V38U035166; Sat, 16 Jan 2021 00:31:42 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3030.oracle.com with ESMTP id 363n8qs8r3-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 16 Jan 2021 00:31:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dkeXN4QWFdbq5nW+YNHH8upktAWMw4fRS0N9Z/+OZugnV6PcVyQMEXddouNq7xo2r6wB3QuUN7K69TiSKAS4C4PDBr4oPUR+rncUJear/loGmolzDWnbrxiK0hhEeO80uAAKjsivZ74D6GhyHTQLspwZGuwbAW23Zzgl0Y7wWV526rqx0xs04Btuwf4PaK5Roh5dH5bmu6+PQ8Yfjxmvl65Kz91nlJ++J5TXZYnmLbA0F/4jCzZcjZe+FIVrNgP7GU6KxgJyEL0xfvRMwLeGTo82vQm7+tvmQVXNtG8dv/zRua7vO2gyAOXen6CAYzjt8KVzF07BM5E1hKfjZKz70A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U9MCdQB1uUR6Oa5upkJkKkOf9bLofSyZk6D883uy1PM=; b=G8oivlko+Ot5wQ+xqOfcYtQlrO/npOaxYBL56Cr4XgdfP83KMeWB44Ci8inivy9D5IVQwP2u3t6rfP/43XIjJJWWvogi9F3zrcygP0KsBG1OHLjcsUahwHACa1qwA59tpksqqwiOTbfbLib8mwa/QaVKR4OLHxRgZteszM31R1+Nac+pWcoRoMJ0m6JHJ+GoC9B0avD4Cd81k3otr9vbEBTmjVohS7iYS1TdABfLuNPMNfKbebL8bvWjoZe/d78z37WUIKTgbzaI6AyQp7brvBiQ2WHit+kfMyrvRT2XCfL5rFdzB3/uD9puDKLtCYZffelEskAacHM+gOV/pCRbcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U9MCdQB1uUR6Oa5upkJkKkOf9bLofSyZk6D883uy1PM=; b=M1nF9CzCQ/ylR5LXd2S+GfL+kJrwfbFuyr4jWB+LvDS4ymCmrcJpvQ3MRfaTAe0hIpZyZg5FT6KT7eoeHp1IqaM6ZP3k0QSCMgp+SHfvawGUagMCRp/exSmqIc9AqbLO54Muzs2ldrIxyRSbQb516wnn9fCuSJ/Kr+DyiSdxz2g= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by CO1PR10MB4450.namprd10.prod.outlook.com (2603:10b6:303:93::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Sat, 16 Jan 2021 00:31:33 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.012; Sat, 16 Jan 2021 00:31:33 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH 5/5] hugetlb: convert PageHugeFreed to HP_Freed flag Date: Fri, 15 Jan 2021 16:31:05 -0800 Message-Id: <20210116003105.182918-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210116003105.182918-1-mike.kravetz@oracle.com> References: <20210116003105.182918-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.11 via Frontend Transport; Sat, 16 Jan 2021 00:31:33 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b4212882-aafc-4aa8-4fa4-08d8b9b6134f X-MS-TrafficTypeDiagnostic: CO1PR10MB4450: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wBE9pFVoKiSD1osLNgJLqDvSzcZ+RnKbz7uHH6H0iydK00UbUPMGV+oUFEeffjjcbpNbQUpsb/ClR5KcdHH0qA/uyKLkYBZTL/skRu1XKkfxcEb7l1pFdaaKv8JGRd4L+x7u5Gs5cgsg+9WE0WXsHnlNbz1Bgvs1UAsZbccnKc3zkVYIaaLoY7wpYlD+CnzXs6vkkeNFHKnbu4KZOMabQq2lWV2Y8zyuYRvXUENnXd4JTzUOsbSt6Aa1CZXs+Wmq112uZpzUXXX9dCGCrmvUxM9hOriVDygDJwIZv1RhlSdEgvC4X+YjhmW6vJmJi1IpXGS31UbJdJerXJDld3ydiOh8Yx4cbLUd0jE5lpeZy/RJz/XGelCHEYAJDoLPZQLCPj9RcSshkmboU+DJ0D72EQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(39850400004)(136003)(346002)(366004)(6486002)(478600001)(1076003)(6666004)(86362001)(36756003)(2906002)(44832011)(316002)(5660300002)(956004)(8676002)(107886003)(16526019)(26005)(52116002)(4326008)(54906003)(8936002)(66946007)(66556008)(66476007)(2616005)(186003)(7696005)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: u3BW92mD/G0zI/AcRzKTiHGPFE0clgCCmUJ6S77qejCMsQS9H1EKdq/+8C4HhDj4aXP/SAVSPMlWNqlRucGx2hcP+TJ2gLSgMzbsBtNs2TtC4DDo6lEv9Z5kN7cn0bHVKermYb+S7m3BwroVkpY+prEkCVlCFwTm8JYqrnCDoN+rt0+OSNM1/sW+J7SQ7h3VhWBBfGgTfyfwQEO6ai8Nx4z+nr7NNfnHCqoxwyiwGmEVe79WsBZzhf7JQY30r80ZxpY5k7C5ciPI9ztf777Gb/TkWsJvnOFE/e5uBnAAQeJOfJxe90L72mCJxRLlp8d1668YhAxp0GnS0IslbC5ri1+4T92bxoHkxD6BzUr2kwoMehoJFWKAevyCQya5iW1SnEMUIePn/gAwwe2KHHTVzwy9O1z042Bh0CODAMhMCYdjdowETG0sZLTfcJCixTCHtgzlFpBDEZ/zk5JcVP6r8B62v96YW5BC+k0ADCQzzJadx995cXvA/QGwOq4NfNtHT8JMdbyJCvv0GnmgtMDedeVlL+dpceJV9IoGVFlmd0awibsS8sxk/e3rjWece9pJRbrUr4rTssCFyqacDHPVV86H6THYTEHkvyGDha/f8IAcb5GJdyfG/CaqSsmrfu/+U/LQ3l9JTNQ7OguVYVJrVhIiSfXWL7bTewrYmvFwpUVPfXLory4VmiDLs7IlWUNxUk9adxAJTME3C7wM13JIK8tKiobi5rA06PqsT9EYnOS1Cvajos3+QGTdWz83wSh+Uw0L/I4Q81Ii+xodxd4FXOmtlcV+xxoT7Uc/fPQ7gEBtw6N/TbEwcl9wEUT8fCRuJbj91n3c3DsXL9W6B0knNfDDJjY4iGzsHIXfo0JF+KWySY3J7NaWuWra/weyzLZjZgCA/78w6Ql39n9zLKMMgZtGA3nJZvi4iQWVZsjhmeEEQZnm0jpPzy248WLFcquWapb16F27uTsJWneIxepZgtfVph6745RQ85rl20OnKUE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b4212882-aafc-4aa8-4fa4-08d8b9b6134f X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2021 00:31:33.4975 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qX2KMIusMfvIifzJkBPWplUPBSBqycREPfkQh050TYlutt8gw5zJnyb2oIwTRyjMGAxVR6kizUIBdzss1AnebQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4450 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9865 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 impostorscore=0 spamscore=0 mlxlogscore=999 clxscore=1015 bulkscore=0 adultscore=0 lowpriorityscore=0 suspectscore=0 phishscore=0 mlxscore=0 malwarescore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101160002 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new hugetlb specific flag HP_Freed flag to replace the PageHugeFreed interfaces. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 2 ++ mm/hugetlb.c | 23 ++++------------------- 2 files changed, 6 insertions(+), 19 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 166825c85875..5c99969fbbd6 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -487,11 +487,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * allocator. Typically used for migration target pages when no pages * are available in the pool. The hugetlb free page path will * immediately free pages with this flag set to the buddy allocator. + * HP_Freed - Set when page is on the free lists. */ enum hugetlb_page_flags { HP_Restore_Reserve = 0, HP_Migratable, HP_Temporary, + HP_Freed, }; #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 53e9168a97bd..073137b32657 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -79,21 +79,6 @@ DEFINE_SPINLOCK(hugetlb_lock); static int num_fault_mutexes; struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; -static inline bool PageHugeFreed(struct page *head) -{ - return page_private(head + 4) == -1UL; -} - -static inline void SetPageHugeFreed(struct page *head) -{ - set_page_private(head + 4, -1UL); -} - -static inline void ClearPageHugeFreed(struct page *head) -{ - set_page_private(head + 4, 0); -} - /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); @@ -1043,7 +1028,7 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; - SetPageHugeFreed(page); + hugetlb_set_page_flag(page, HP_Freed); } static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1060,7 +1045,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); - ClearPageHugeFreed(page); + hugetlb_clear_page_flag(page, HP_Freed); h->free_huge_pages--; h->free_huge_pages_node[nid]--; return page; @@ -1474,7 +1459,7 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_lock(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; - ClearPageHugeFreed(page); + hugetlb_clear_page_flag(page, HP_Freed); spin_unlock(&hugetlb_lock); } @@ -1747,7 +1732,7 @@ int dissolve_free_huge_page(struct page *page) * We should make sure that the page is already on the free list * when it is dissolved. */ - if (unlikely(!PageHugeFreed(head))) { + if (unlikely(!hugetlb_test_page_flag(head, HP_Freed))) { rc = -EAGAIN; goto out; }