From patchwork Wed Jan 20 01:30:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12031189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A622C433E0 for ; Wed, 20 Jan 2021 01:31:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4030A20449 for ; Wed, 20 Jan 2021 01:31:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4030A20449 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 774F66B0006; Tue, 19 Jan 2021 20:31:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7001C6B0007; Tue, 19 Jan 2021 20:31:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DB726B0008; Tue, 19 Jan 2021 20:31:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 32FE26B0006 for ; Tue, 19 Jan 2021 20:31:30 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 039AA180AD80F for ; Wed, 20 Jan 2021 01:31:30 +0000 (UTC) X-FDA: 77724425940.06.judge04_201141327556 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id D2ED3100437F2 for ; Wed, 20 Jan 2021 01:31:29 +0000 (UTC) X-HE-Tag: judge04_201141327556 X-Filterd-Recvd-Size: 20982 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 Jan 2021 01:31:29 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UDQV117216; Wed, 20 Jan 2021 01:31:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=FhV/HeyL8L1JINdhS6sXklpxJlBsiWWrvT5wm8KT1oc=; b=QkB509wyHX11bj/yM11gOz7a8Zb9iKTAzkUrZulx2GcTSWDmAasVB01s31w+T69MjVOt cQrbWB79Qsc8qwekOl1vJ5BOPuhIM5H9Hb6QE1SCHriJ1TIagYxZuRzcxDR98TwcuF4G wR2k4eGVjQaemxo1ptjfoO8x3CyEoxRwGm2vLz/UpJc/EJiNiTelsxIP2C4HleFs3f39 w4YOg9heZe04nUFaYo2PIQnHdTt45I57i/rEPmUMLmTTYZcqmkEWqLPsbyQXOUW0S2bp Kp3gUzYcpmzHaGmz15EL/AmwTJZNineC4KuL1w2V2VrRsKxtLpIPB6cV2MQhp0hSUPRg Xw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 3668qmracj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:15 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UfG4016928; Wed, 20 Jan 2021 01:31:14 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2105.outbound.protection.outlook.com [104.47.58.105]) by userp3020.oracle.com with ESMTP id 3668qv5kn7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GvdIzUV17dlztPjRTbEOmSD9pHbG5/iSQbzMo0sPYFQ5W+9atZnPtH6Q037mXR4672aRZHbmolEJZeibfPSH4EHZ9r0Y4BROmM+fcECDToSaHe9wCkU3E10x7OLAUoHTKYwSWp9t1Fsm1exipxURhCEEBmXwzPJF9FIfSnTHWwoQpoYuCtR/AcK+AhR9EK6fjqHUI0QuK8FDSnoGvR1vcMdvN0p1bJQJwHybco4DTb8kjnfvviEEb3cGbw6Bxw2EA1/YmhE0aK26154Sf44w3o5BoECElOZ0OQWhBXmMU7raAnVrdd6De4IZU109YFI3fI8D5s7m6KZ5A+zE043r/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FhV/HeyL8L1JINdhS6sXklpxJlBsiWWrvT5wm8KT1oc=; b=J5FHK6wOmImm9Rddi7tMw2kdwqYEwB1XwaOToFQRkTwPHRBKPH8UEEufK/jyQiVDRWJT70vLuFpc1snVVF6Dw81TZq4KlupFx9Sssx39NfQA6MqSygaUPgQcXumpMrspI6oME+xU0P1kcLR3u6rn+93Do6xekTHztoM6nQVerhgpbQNrb4fMfqwu8XrXIG5EdGuE08GE5lV6iYZJ2wje9LINBUxl+kloRVHvmTymWiBOX4dwerHcWGVC7Lh1bhcl9hO4zEzflCCn85xrZmLlRAXCXC1omYNDSjt74ukWOstvh8eNn6xUgVDnL7jl9G42pW7mQzf42tO3wJy4ueK1Gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FhV/HeyL8L1JINdhS6sXklpxJlBsiWWrvT5wm8KT1oc=; b=Oww6JFe7xK536rfF8KkbW8+SwGX/vcyInRttn1H3xWwraiyPNHGRSIy4HxgDHtFOd20V5rEx/W2NGj2+AmKe7BxErk7MqIUVIoQd0rx7gjsyOSNbTjdiL8mjzA2I9gbxLUYT04WBu4e5OSqbeTgePD/GA+Om5+O72c0osJnkHlg= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by MWHPR1001MB2384.namprd10.prod.outlook.com (2603:10b6:301:2d::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Wed, 20 Jan 2021 01:31:12 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.014; Wed, 20 Jan 2021 01:31:12 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH v2 1/5] hugetlb: use page.private for hugetlb specific page flags Date: Tue, 19 Jan 2021 17:30:45 -0800 Message-Id: <20210120013049.311822-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210120013049.311822-1-mike.kravetz@oracle.com> References: <20210120013049.311822-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.11 via Frontend Transport; Wed, 20 Jan 2021 01:31:11 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 40d9d0b3-f785-4889-e0a0-08d8bce311f4 X-MS-TrafficTypeDiagnostic: MWHPR1001MB2384: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2d8s9UBIY6cU8mE55MaAOORYFZ12tw/8rAwD1hSt4xv7qzDvlE6MSVOBuEs6p6a7E+xh+0INhYxBbbEWfg5LNQfp7k6XpLEeRK2k6kXxv4FifqVg84foRy1tjwHijg6Qa3pDUUae2L1kCIRJOCR6rC6WO/NfaXWf5lXEQN+/TM1/sK4lGWmOZwWO5/2OzS02uK01cEEIhMi1w33UJa4cocBqC4sfUyyrJxVVGV1iWj/DbPphN8D13jlUJj62kD05nmD3hZxp3US0Mqi1DAAl1hzL24bj0f1uLEo+JBtBcRpyo0AlSqGgn1EYxAameG8bisixVNT4vuBsklbe+T5zfJbdSYksBSSTkhfmE1zIRneyU3/skg2udC8cFq9pXTOniFC/EBrKhmkyF6WfMlX7I8clk45hTFCZwye+fHTbH8jfwKkRBUU3ypbXfKZqGrkna6MRia1p1Otk4k0ZVOQaW/vOZNQbhcxoJ2YkOAu4aLfFuOykpwOdxiju7iJ2n40cEOzsXZTd5MMoi3iFzm0gnA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(136003)(396003)(1076003)(8676002)(186003)(36756003)(107886003)(8936002)(478600001)(16526019)(4326008)(956004)(316002)(86362001)(5660300002)(54906003)(6486002)(2616005)(7696005)(30864003)(66946007)(66476007)(66556008)(83380400001)(44832011)(2906002)(52116002)(6666004)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 0/8YNTc0IHHp5biU0fknqSIol3XhKVGH0gyHbRbWWoc09NtbCxusBD5sTEf/qwEjzGGB9hngVUoabB8ou2Ih/OalJj+w3vhVGQqe/sVpYbd/V280VVsDoH7sAtgWZ7DZ4ki21ZACsoij0ypfVdAVcknJaC+MIH++KEBCCKwazhJu0Ra+ZPF8L5flAu4FPVTJG5PpHPF/h+3+Cu+FguiShx+3Gqt8Fx1ap0mycgqIldrcxs/OGtqce/IyXI6l2twn/fsI1om6sUIEnh1Z1a4MWT8wu9yHA1W4G8yGk32xgsKfdxlDBLxbs1hFuivY0ZontVmYnQx/XDtYVwpuEgqCMSx04AkLZZ52q5ri5QbQ+ojzKBtiYfMvbFFG4WgmeEAGodX9hu/s1C0KOnIAy8vrH1/WAUl+hGf88K9Lg29Jwdlpbo52voD1eqTQ+xoXjrGM0/TsJ4hpI/p0byKtVdeghoB7dMUrDlR9T5XDXbAqcORYUFFBr0Ai6kDkL+JRc1volVNYBZdL+vnwGhvlGYqsk6n5rlhcsdTLwES8ocMRNX2YolA8HPB3BQ/f21H7fHAidpboE2asEvrxgRVus4D+mMFEnmOUwP3mkIgqvib+gcGx4k0HGfCfD84v87vvJ1lZcwxz+5V+BdWiVf9ihSOF9PTNx6cKNZlS4ZVgnshyxAQzlm6GdwknOGl93P9xWEokvMQ1UnoQ4TKxEe/d8txsQR/i82NYvy6H+Rzlr6R/XJlXzSxcvg8lLv2RFgd6GC8t4fWoQkNR9ckPEa98yTz5ihw9VEI8Z1bIiRQubQV0gr/pxnAEAVgs3Q9GnZ1Bd3iDWvmjEMPmoUlzrPMe0IL+i6sfiOhpYtw+fF/vKVTtshRAqsIyomcEkEeWk+G2O/9o7T1AMJ0iA0ec3tQw8L769wsp2BNE6O2Lhy3YWoWkoXyQktk09VgjHw8buUs+bg/lGniSA2Q3bpJVUE0rnsMbc/tIhXombIUhQ+XvNdDas5UVT7sp9MKXQSgTbqrHWXi0D3Ip1U9eFB8ad/ICLy2OF3LoEYixBYnoDvewJxt1h7j8YL4XmLynSjT7Zy+wemwL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 40d9d0b3-f785-4889-e0a0-08d8bce311f4 X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2021 01:31:12.0213 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mE85Hs0SCrT8iCBS0mxdfsKkJT5MFpCDSuaBDJfEzztPY4Bpogc2qc2aCII8Ig0pU/G4dRU9lMC5jhDFTbUNcg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2384 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 adultscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 priorityscore=1501 adultscore=0 impostorscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 phishscore=0 clxscore=1015 bulkscore=0 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As hugetlbfs evolved, state information about hugetlb pages was added. One 'convenient' way of doing this was to use available fields in tail pages. Over time, it has become difficult to know the meaning or contents of fields simply by looking at a small bit of code. Sometimes, the naming is just confusing. For example: The PagePrivate flag indicates a huge page reservation was consumed and needs to be restored if an error is encountered and the page is freed before it is instantiated. The page.private field contains the pointer to a subpool if the page is associated with one. In an effort to make the code more readable, use page.private to contain hugetlb specific page flags. These flags will have test, set and clear functions similar to those used for 'normal' page flags. More importantly, an enum of flag values will be created with names that actually reflect their purpose. In this patch, - Create infrastructure for hugetlb specific page flag functions - Move subpool pointer to page[1].private to make way for flags Create routines with meaningful names to modify subpool field - Use new HPageRestoreReserve flag instead of PagePrivate Conversion of other state information will happen in subsequent patches. Signed-off-by: Mike Kravetz Reviewed-by: Miaohe Lin Reported-by: kernel test robot --- fs/hugetlbfs/inode.c | 12 ++----- include/linux/hugetlb.h | 72 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 45 +++++++++++++------------- 3 files changed, 97 insertions(+), 32 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 740693d7f255..b8a661780c4a 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -955,15 +955,9 @@ static int hugetlbfs_migrate_page(struct address_space *mapping, if (rc != MIGRATEPAGE_SUCCESS) return rc; - /* - * page_private is subpool pointer in hugetlb pages. Transfer to - * new page. PagePrivate is not associated with page_private for - * hugetlb pages and can not be set here as only page_huge_active - * pages can be migrated. - */ - if (page_private(page)) { - set_page_private(newpage, page_private(page)); - set_page_private(page, 0); + if (hugetlb_page_subpool(page)) { + hugetlb_set_page_subpool(newpage, hugetlb_page_subpool(page)); + hugetlb_set_page_subpool(page, NULL); } if (mode != MIGRATE_SYNC_NO_COPY) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ef5b144b8aac..be71a00ee2a0 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -472,6 +472,64 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, unsigned long flags); #endif /* HAVE_ARCH_HUGETLB_UNMAPPED_AREA */ +/* + * huegtlb page specific state flags. These flags are located in page.private + * of the hugetlb head page. Functions created via the below macros should be + * used to manipulate these flags. + * + * HPG_restore_reserve - Set when a hugetlb page consumes a reservation at + * allocation time. Cleared when page is fully instantiated. Free + * routine checks flag to restore a reservation on error paths. + */ +enum hugetlb_page_flags { + HPG_restore_reserve = 0, + __NR_HPAGEFLAGS, +}; + +/* + * Macros to create test, set and clear function definitions for + * hugetlb specific page flags. + */ +#ifdef CONFIG_HUGETLB_PAGE +#define TESTHPAGEFLAG(uname, flname) \ +static inline int HPage##uname(struct page *page) \ + { BUILD_BUG_ON(sizeof_field(struct page, private) * \ + BITS_PER_BYTE < __NR_HPAGEFLAGS); \ + return test_bit(HPG_##flname, &(page->private)); } + +#define SETHPAGEFLAG(uname, flname) \ +static inline void SetHPage##uname(struct page *page) \ + { set_bit(HPG_##flname, &(page->private)); } + +#define CLEARHPAGEFLAG(uname, flname) \ +static inline void ClearHPage##uname(struct page *page) \ + { clear_bit(HPG_##flname, &(page->private)); } +#else +#define TESTHPAGEFLAG(uname, flname) \ +static inline int HPage##uname(struct page *page) \ + { BUILD_BUG_ON(sizeof_field(struct page, private) * \ + BITS_PER_BYTE < __NR_HPAGEFLAGS); \ + return 0 } + +#define SETHPAGEFLAG(uname, flname) \ +static inline void SetHPage##uname(struct page *page) \ + { } + +#define CLEARHPAGEFLAG(uname, flname) \ +static inline void ClearHPage##uname(struct page *page) \ + { } +#endif + +#define HPAGEFLAG(uname, flname) \ + TESTHPAGEFLAG(uname, flname) \ + SETHPAGEFLAG(uname, flname) \ + CLEARHPAGEFLAG(uname, flname) \ + +/* + * Create functions associated with hugetlb page flags + */ +HPAGEFLAG(RestoreReserve, restore_reserve) + #ifdef CONFIG_HUGETLB_PAGE #define HSTATE_NAME_LEN 32 @@ -531,6 +589,20 @@ extern unsigned int default_hstate_idx; #define default_hstate (hstates[default_hstate_idx]) +/* + * hugetlb page subpool pointer located in hpage[1].private + */ +static inline struct hugepage_subpool *hugetlb_page_subpool(struct page *hpage) +{ + return (struct hugepage_subpool *)(hpage+1)->private; +} + +static inline void hugetlb_set_page_subpool(struct page *hpage, + struct hugepage_subpool *subpool) +{ + set_page_private(hpage+1, (unsigned long)subpool); +} + static inline struct hstate *hstate_file(struct file *f) { return hstate_inode(file_inode(f)); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 737b2dce19e6..8bed6b5202d2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1133,7 +1133,7 @@ static struct page *dequeue_huge_page_vma(struct hstate *h, nid = huge_node(vma, address, gfp_mask, &mpol, &nodemask); page = dequeue_huge_page_nodemask(h, gfp_mask, nid, nodemask); if (page && !avoid_reserve && vma_has_reserves(vma, chg)) { - SetPagePrivate(page); + SetHPageRestoreReserve(page); h->resv_huge_pages--; } @@ -1407,20 +1407,19 @@ static void __free_huge_page(struct page *page) */ struct hstate *h = page_hstate(page); int nid = page_to_nid(page); - struct hugepage_subpool *spool = - (struct hugepage_subpool *)page_private(page); + struct hugepage_subpool *spool = hugetlb_page_subpool(page); bool restore_reserve; VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(page_mapcount(page), page); - set_page_private(page, 0); + hugetlb_set_page_subpool(page, NULL); page->mapping = NULL; - restore_reserve = PagePrivate(page); - ClearPagePrivate(page); + restore_reserve = HPageRestoreReserve(page); + ClearHPageRestoreReserve(page); /* - * If PagePrivate() was set on page, page allocation consumed a + * If HPageRestoreReserve was set on page, page allocation consumed a * reservation. If the page was associated with a subpool, there * would have been a page reserved in the subpool before allocation * via hugepage_subpool_get_pages(). Since we are 'restoring' the @@ -2254,24 +2253,24 @@ static long vma_add_reservation(struct hstate *h, * This routine is called to restore a reservation on error paths. In the * specific error paths, a huge page was allocated (via alloc_huge_page) * and is about to be freed. If a reservation for the page existed, - * alloc_huge_page would have consumed the reservation and set PagePrivate - * in the newly allocated page. When the page is freed via free_huge_page, - * the global reservation count will be incremented if PagePrivate is set. - * However, free_huge_page can not adjust the reserve map. Adjust the - * reserve map here to be consistent with global reserve count adjustments - * to be made by free_huge_page. + * alloc_huge_page would have consumed the reservation and set + * HPageRestoreReserve in the newly allocated page. When the page is freed + * via free_huge_page, the global reservation count will be incremented if + * HPageRestoreReserve is set. However, free_huge_page can not adjust the + * reserve map. Adjust the reserve map here to be consistent with global + * reserve count adjustments to be made by free_huge_page. */ static void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, unsigned long address, struct page *page) { - if (unlikely(PagePrivate(page))) { + if (unlikely(HPageRestoreReserve(page))) { long rc = vma_needs_reservation(h, vma, address); if (unlikely(rc < 0)) { /* * Rare out of memory condition in reserve map - * manipulation. Clear PagePrivate so that + * manipulation. Clear HPageRestoreReserve so that * global reserve count will not be incremented * by free_huge_page. This will make it appear * as though the reservation for this page was @@ -2280,7 +2279,7 @@ static void restore_reserve_on_error(struct hstate *h, * is better than inconsistent global huge page * accounting of reserve counts. */ - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); } else if (rc) { rc = vma_add_reservation(h, vma, address); if (unlikely(rc < 0)) @@ -2288,7 +2287,7 @@ static void restore_reserve_on_error(struct hstate *h, * See above comment about rare out of * memory condition. */ - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); } else vma_end_reservation(h, vma, address); } @@ -2369,7 +2368,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, if (!page) goto out_uncharge_cgroup; if (!avoid_reserve && vma_has_reserves(vma, gbl_chg)) { - SetPagePrivate(page); + SetHPageRestoreReserve(page); h->resv_huge_pages--; } spin_lock(&hugetlb_lock); @@ -2387,7 +2386,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, spin_unlock(&hugetlb_lock); - set_page_private(page, (unsigned long)spool); + hugetlb_set_page_subpool(page, spool); map_commit = vma_commit_reservation(h, vma, addr); if (unlikely(map_chg > map_commit)) { @@ -4212,7 +4211,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, spin_lock(ptl); ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { - ClearPagePrivate(new_page); + ClearHPageRestoreReserve(new_page); /* Break COW */ huge_ptep_clear_flush(vma, haddr, ptep); @@ -4279,7 +4278,7 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping, if (err) return err; - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); /* * set page dirty so that it will not be removed from cache/file @@ -4441,7 +4440,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) { - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); hugepage_add_new_anon_rmap(page, vma, haddr); } else page_dup_rmap(page, true); @@ -4755,7 +4754,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, if (vm_shared) { page_dup_rmap(page, true); } else { - ClearPagePrivate(page); + ClearHPageRestoreReserve(page); hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); } From patchwork Wed Jan 20 01:30:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12031195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B421EC433E0 for ; Wed, 20 Jan 2021 01:33:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2268922241 for ; Wed, 20 Jan 2021 01:33:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2268922241 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AF2BF6B0005; Tue, 19 Jan 2021 20:33:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AA0E86B0006; Tue, 19 Jan 2021 20:33:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91BA36B0007; Tue, 19 Jan 2021 20:33:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0166.hostedemail.com [216.40.44.166]) by kanga.kvack.org (Postfix) with ESMTP id 7BD0E6B0005 for ; Tue, 19 Jan 2021 20:33:32 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3C38933CD for ; Wed, 20 Jan 2021 01:33:32 +0000 (UTC) X-FDA: 77724431064.08.knife87_510914827556 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 21D451819E62A for ; Wed, 20 Jan 2021 01:33:32 +0000 (UTC) X-HE-Tag: knife87_510914827556 X-Filterd-Recvd-Size: 17541 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 Jan 2021 01:33:30 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1SeBL121604; Wed, 20 Jan 2021 01:33:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=GkEtRCoCPbH2VxS8quwLysFsPwNTBQ+D3u0/k/wZa04=; b=EjkESOLVEQmZuFB1go6Z21hwwLt4H20skqKWxsgeFblBAUjHLGYc4QCEAaeO24sKuP64 F8V8eGKUguAWgjUJgUQ1qlcAAmD66YJ+wJ4P8K1/ND+GupQHQ+FW/sI4bdYnM0tgwspI UGiE/xlQOpXCY5ckzIOTYlLXw2dQfbGtDkEfJmzKvJh08m+P8FMaRJ4NWX1QZCP3TW6m O8m7MBWzaNmzjWjNSXp9wbT9m+Yk9XM/ueQcDYABecKd514c4FrVuCoFTYNAM9bDk2f1 LHmOhrvfxQEeigMf1AjIJwnAzFyqs2KEXhG3CJlkT7akINmY74DFOyquAXk5Moh/fo6q MQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 3668qr8ah7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:33:19 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1Tjrb072254; Wed, 20 Jan 2021 01:31:19 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2107.outbound.protection.outlook.com [104.47.58.107]) by aserp3020.oracle.com with ESMTP id 3668rd56g3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=be5EN3cfQIWaHsqzfX+GpMiTu/SfNFEVFe5e2BMaPCR0qFmwetttn30IP4M+MilhiX4y1PLkL/WI6md/dN3Nt0puX3rKfvy9AjlJA6q1wixQaSyCI6iDB0JvDQF56/CVB+w1Z267Wxg8Xwc95BWcQUboulYmwidb79QWOuHZwx9rkQa3Sm/bJ0p3imVftbijnBcG1k57wRovkJbKJuhyfBbpxCPqU+b3YBN1IkVw0fzn2JSJhT3xPwbhEiTJ0PL58SLDzp9yYlKFLGUZTzGXy7CszJQKQ4NCimnxIqjlHrk56258RMWCmZXr0At4i27BNxb/6pNtWpr1DfSW6HrOTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GkEtRCoCPbH2VxS8quwLysFsPwNTBQ+D3u0/k/wZa04=; b=Q2kFcJa0tw533b109iO8edn61Lmcv+xWel+9xV1Rk7Oc/uOlg8DwUuxHNixWsALdCR0mTLWt+/8hvZo6B3KUsqKlf8gN/pDhKF4Oj5JNLYGXBus8Y4u/I8v5+r61Hh3a10qaaRrDDuyjfBGqaJH8TjJk2/2sCWyD6lVaogL7DgbRCf7k7xOcaPQpWRbmogEYKd9OSnFVpRBCQ/+zg+9W+GbdkTy6YPhZpG6xtSHccWw+/0j3a82s7Mb8dTVPT+t/N/v6w72rVpoVGIhfKzf2fb79pgs//MbLJu88iOuGoPyCYC9E5NVCXJVHtwIe39gdCkMHT3rJfHbcDlFO+HVV5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GkEtRCoCPbH2VxS8quwLysFsPwNTBQ+D3u0/k/wZa04=; b=ERsZm35kspP/WEUB8vgnS8bq6sXYShBvJ5EjK6YHEKKXd8YwX9yNPlYT8wIKVdBnEa1cGDDsPpDHsFCLxBDxzoaeADkDN8X8LCzh4vt73jXUnhSXtqvOqqRktPPazqM3/EyUOj+46BnC90xBVhpHslHkjfH7WXDdbX7zxPlVlfk= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by MWHPR1001MB2384.namprd10.prod.outlook.com (2603:10b6:301:2d::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Wed, 20 Jan 2021 01:31:17 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.014; Wed, 20 Jan 2021 01:31:17 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH v2 2/5] hugetlb: convert page_huge_active() HPageMigratable flag Date: Tue, 19 Jan 2021 17:30:46 -0800 Message-Id: <20210120013049.311822-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210120013049.311822-1-mike.kravetz@oracle.com> References: <20210120013049.311822-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.11 via Frontend Transport; Wed, 20 Jan 2021 01:31:17 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a0d9e085-0b14-4b72-0682-08d8bce3153e X-MS-TrafficTypeDiagnostic: MWHPR1001MB2384: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Uky+p+qNNKCnR6uR6r+iGbkrv5ND83yhINZGrp2XEdEju2LuuU5sBravHVVuXEPiDoHn8EvB86J4G6iImT9tKmO1GbYmx5eWbhszxkn5zp50Omcz1WdjcNkMmYJxT7aDH/nIgffJxwbaW+tGtUs8ZqG8V6fTyZ+BdYjKjCKX+6iPKy1aoCKc1+78OPeKsQYFUok3OdsfNOttssTq8vaDujrle4p3dlFnxIVzZfnRSbjtfqHXYpnt9I+egeAb00S/DkZOVMSHJxA2gGL1UbucSw2uDFwpJnAtjDIxD2+6BA9IoimIls1A80fOMBys/yp/35NrAKwK7bHSXBZ1AjhuTMOAUEYZXzWrlyknvNJtnxoNuFVHoehAbCXGPRWPkjE6K3VLjOaxnZqssws28tqBdWRgfO3hmHaBuRvi72cj6RGk4VEa8YHxbwPo+zkyMsuu7sibGPEOA6nxW0g49lz/LQWhfbi82BNL0Ka6/Gc/+7vHkylWngrhipxbwqZanlBnTg1UdJoQdsIU/pNj47wT2jGLva+Yb/2qUduDzoO4/BlNvtaaVAs8onDFaSSRlBP4 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(136003)(396003)(1076003)(8676002)(186003)(36756003)(107886003)(8936002)(478600001)(16526019)(4326008)(956004)(316002)(86362001)(5660300002)(54906003)(6486002)(2616005)(7696005)(66946007)(66476007)(66556008)(83380400001)(44832011)(2906002)(52116002)(6666004)(26005)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: QQprUyw28FEtVCEmXd/q+R3oONHAbCUgC2AvwIjr+/g3fO+WH9wdiWHN9sq8EwAyA9utIcquz/U9iOnCVJpaFpeqVRCnAkEbFEE6F7PIToSC0Ie1iSJBdYpkhCZ89YxYunMhmqAJAwu6Q9Dzz1Yql/Ijh7CLk0vUb4SWtJdUkcPnN2YctowA6ec2TgvC21Ybk8LGt79kVlb/EgTESCsaZJza/p8XH+YHckrlS2FyELAO3XsvJ5J0r8hOfUFmKCOgRp5FIb8ys6zU72p1+aTx+LbmVSH7OfKIxtr2LD9f1XuYtm3RbyAJ2C26WzwFRdDVDwH0kI1nBNwv+fqHxG3HzkljJVV1PPn8Z9DIgDE6M+s4FWNT7zB2YcxAZ8KV/I/lf5wgcNcvOe7qFMdyx7Lvwp4BDqMDbaFOKMDNlLDQRVA4lDVGaLb7+2H4/eW025RSiCpiPXlA6gVrDrk3bpE3IgWYdIirHkwSJUKPFu/Dl4MNIXZu5W/OPflzOd/Kn1zG0YdC5kEHPGUDxSYsdOSgsPMPZSannYaOCKdl79SR6RFp02y/EnXo4oXvyS4TU+kGCm+7ucoLbS2gmQ60CrrtnpQuhC9f8jwvqP4qjpGBVCneSCgrei7hyvHIPB0r7pCY+HCCt6wrOMGzLIcC/NwJkWF8FI218iIbYkDUhE/uNc+gOxtljO49qKcIABPyRzFHzy1fCKIWp9/E8q+/8klLqcp3X5luxbd5S40JpT59PrmXgPYjO3dZglL9Upy0zwVOkjCqKC0CXmf0+ifHTOpecEMWKydA2C39M0/6c6DldPWMviBeKUpe5vh/YbHdeK5rveEvykm+uLysymPPr0pcPiC5y1c1IoVWeeZ302IYSn42Y6JLR8+/hi4YkMWWTby+O25JlOAGs35kqFIAU5OKe4hWgu8KJrFrzk/ii2+EKOFtWvJlyr3pY0LNj8KDrZSa3TX+BC41ZpymskokQViIXtYKLnpxizLX53260GgdAY3YsLOgXdXVsCV7tyg49iXgdk819hvMjmLCx5L6cfzEA3jBlApRj8/I/Kgc97f0XZ5n98E/qee3tGMqjFtoaC4M X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a0d9e085-0b14-4b72-0682-08d8bce3153e X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2021 01:31:17.4588 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: du3cj/HSWycxuFgPgeGjypPPRnzrr+FqyGrlx88ChOFHJEDvf2Smgf+ye5/95i53eAt0JxQq6nErLxkDCdChgg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2384 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 adultscore=0 spamscore=0 phishscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the new hugetlb page specific flag HPageMigratable to replace the page_huge_active interfaces. By it's name, page_huge_active implied that a huge page was on the active list. However, that is not really what code checking the flag wanted to know. It really wanted to determine if the huge page could be migrated. This happens when the page is actually added the page cache and/or task page table. This is the reasoning behind the name change. The VM_BUG_ON_PAGE() calls in the *_huge_active() interfaces are not really necessary as we KNOW the page is a hugetlb page. Therefore, they are removed. The routine page_huge_active checked for PageHeadHuge before testing the active bit. This is unnecessary in the case where we hold a reference or lock and know it is a hugetlb head page. page_huge_active is also called without holding a reference or lock (scan_movable_pages), and can race with code freeing the page. The extra check in page_huge_active shortened the race window, but did not prevent the race. Offline code calling scan_movable_pages already deals with these races, so removing the check is acceptable. Add comment to racy code. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Reviewed-by: Muchun Song Reviewed-by: Miaohe Lin --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 5 +++++ include/linux/page-flags.h | 6 ----- mm/hugetlb.c | 45 ++++++++++---------------------------- mm/memory_hotplug.c | 8 ++++++- 5 files changed, 24 insertions(+), 42 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b8a661780c4a..ec9f03aa2738 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - set_page_huge_active(page); + SetHPageMigratable(page); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index be71a00ee2a0..ce3d03da0133 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -480,9 +480,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HPG_restore_reserve - Set when a hugetlb page consumes a reservation at * allocation time. Cleared when page is fully instantiated. Free * routine checks flag to restore a reservation on error paths. + * HPG_migratable - Set after a newly allocated page is added to the page + * cache and/or page tables. Indicates the page is a candidate for + * migration. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, + HPG_migratable, __NR_HPAGEFLAGS, }; @@ -529,6 +533,7 @@ static inline void ClearHPage##uname(struct page *page) \ * Create functions associated with hugetlb page flags */ HPAGEFLAG(RestoreReserve, restore_reserve) +HPAGEFLAG(Migratable, migratable) #ifdef CONFIG_HUGETLB_PAGE diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index bc6fd1ee7dd6..04a34c08e0a6 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -592,15 +592,9 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); -bool page_huge_active(struct page *page); #else TESTPAGEFLAG_FALSE(Huge) TESTPAGEFLAG_FALSE(HeadHuge) - -static inline bool page_huge_active(struct page *page) -{ - return 0; -} #endif diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8bed6b5202d2..c24da40626d3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,30 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Test to determine whether the hugepage is "active/in-use" (i.e. being linked - * to hstate->hugepage_activelist.) - * - * This function can be called for tail pages, but never returns true for them. - */ -bool page_huge_active(struct page *page) -{ - return PageHeadHuge(page) && PagePrivate(&page[1]); -} - -/* never called for tail page */ -void set_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - SetPagePrivate(&page[1]); -} - -static void clear_page_huge_active(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHeadHuge(page), page); - ClearPagePrivate(&page[1]); -} - /* * Internal hugetlb specific page flag. Do not use outside of the hugetlb * code @@ -1438,7 +1414,7 @@ static void __free_huge_page(struct page *page) } spin_lock(&hugetlb_lock); - clear_page_huge_active(page); + ClearHPageMigratable(page); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); hugetlb_cgroup_uncharge_page_rsvd(hstate_index(h), @@ -4220,7 +4196,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - set_page_huge_active(new_page); + SetHPageMigratable(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -4457,12 +4433,12 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spin_unlock(ptl); /* - * Only make newly allocated pages active. Existing pages found - * in the pagecache could be !page_huge_active() if they have been - * isolated for migration. + * Only set HPageMigratable in newly allocated pages. Existing pages + * found in the pagecache may not have HPageMigratableset if they have + * been isolated for migration. */ if (new_page) - set_page_huge_active(page); + SetHPageMigratable(page); unlock_page(page); out: @@ -4773,7 +4749,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - set_page_huge_active(page); + SetHPageMigratable(page); if (vm_shared) unlock_page(page); ret = 0; @@ -5591,12 +5567,13 @@ bool isolate_huge_page(struct page *page, struct list_head *list) bool ret = true; spin_lock(&hugetlb_lock); - if (!PageHeadHuge(page) || !page_huge_active(page) || + if (!PageHeadHuge(page) || + !HPageMigratable(page) || !get_page_unless_zero(page)) { ret = false; goto unlock; } - clear_page_huge_active(page); + ClearHPageMigratable(page); list_move_tail(&page->lru, list); unlock: spin_unlock(&hugetlb_lock); @@ -5607,7 +5584,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - set_page_huge_active(page); + SetHPageMigratable(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index f9d57b9be8c7..563da803e0e0 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1260,7 +1260,13 @@ static int scan_movable_pages(unsigned long start, unsigned long end, if (!PageHuge(page)) continue; head = compound_head(page); - if (page_huge_active(head)) + /* + * This test is racy as we hold no reference or lock. The + * hugetlb page could have been free'ed and head is no longer + * a hugetlb page before the following check. In such unlikely + * cases false positives and negatives are possible. + */ + if (HPageMigratable(head)) goto found; skip = compound_nr(head) - (page - head); pfn += skip - 1; From patchwork Wed Jan 20 01:30:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12031197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C691CC433E0 for ; Wed, 20 Jan 2021 01:33:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4CB1E2222F for ; Wed, 20 Jan 2021 01:33:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4CB1E2222F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BDD9E6B0006; Tue, 19 Jan 2021 20:33:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B8D1F6B0007; Tue, 19 Jan 2021 20:33:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98FFD6B0008; Tue, 19 Jan 2021 20:33:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 857F16B0006 for ; Tue, 19 Jan 2021 20:33:35 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 557AD181AF5C3 for ; Wed, 20 Jan 2021 01:33:35 +0000 (UTC) X-FDA: 77724431190.21.yard53_3b0a0b527556 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 3D273180442EA for ; Wed, 20 Jan 2021 01:33:35 +0000 (UTC) X-HE-Tag: yard53_3b0a0b527556 X-Filterd-Recvd-Size: 14034 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 Jan 2021 01:33:34 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UDBc117229; Wed, 20 Jan 2021 01:33:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=1Td/TX8Ho0q33dZW3yzF+EiHcp4lTewFB9iBKRO8LT4=; b=LAQzsI2IZ0zRD+Ftb24xQdsn/IgtSw0FTj63LvSRSNNtZqWbhhzJ7nrNJlX9QnYPKg7m X04w81xRcOMLF0TEs80SmfxDzSRBWUhFktr2Gpevj/xtB1mp+pL2cnEukIZ+SIGITNwW L8wgl40CRijHvA2gqmdjBwT9uJkSuwLAkDvDUMRiMwVgjTNLfD+icU0gMsoNVJENS6MF mnT0VyNvUJ7Ke4XCqtXICWA3m4tHRqqum2MHKW/49ugC03iqBPUtJsDZVyWqWHjyGNVZ RQgrLlrfs5Lbsizp8DI7TS72wEryIHA7aBdUtP9U199IV48cQ/mNLCpH0TgrhbfrUY0B og== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 3668qmragc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:33:22 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UeBc016838; Wed, 20 Jan 2021 01:31:21 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2109.outbound.protection.outlook.com [104.47.58.109]) by userp3020.oracle.com with ESMTP id 3668qv5kr7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SH1LAcexUiNUt43xyHDVd4r34MzNMr3uMKjlI6rnQoy5RGvvlWSN1DFz1AAj4h6S0VkDqL2lyRNdIEmyhn/roovqPokhhtRqtDVAI3XJhYKAbcFrJd1yx+VFS9HGLOZ7utm10bcK14/BWCfOzQ9+3EVP+7s9Pyl7COX1StE8bfatXIfoKFmRT2lgN/YU3JVjCgQ/Xm9wp+y2UhEYXjkh2y1KIRhLyxJ67M0vqn/URhzYs0nLT7OHaPwkgr/VfID8vH4OxV/pMc/BXI30IQbbYi4xY9HyJst6yM1nVhU8AjEUR+z3KqLhmRM+jkp1X5IChTC7aU2Xb9zwKAwBaUiq3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1Td/TX8Ho0q33dZW3yzF+EiHcp4lTewFB9iBKRO8LT4=; b=ZJDzap4pbrDuRlaqtkhYQelaQ8feSHfM5Wh95NF926us+rz2GTkYmkrpUMdEtd5+JuHxlCXCztFBoJPZY0oMLWu2AfUuYBfJKqC2+Xekmd+y1LDT6zq2jeERQvHIlv4ZyumE2yousqBF16WpnCAIEP54xF6I2K0s7VuRGn+pE6T6FrAFkVJu+P6fOTiAcn90C+SQoaJxkLo4xOaVrxlNY63C9iYkqwk0dt5/oRdDtikx3+isFLcJoaDNe2js/tDzgHwRGSUAf8meTJInWjGkgZ2xDfmHA21+7WebNHDFuGVua0/5wM3CkKiOeHKkG6gB87UNnz1MbIzKQh92xcBR5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1Td/TX8Ho0q33dZW3yzF+EiHcp4lTewFB9iBKRO8LT4=; b=RrdkytVuPnp4k0K8jI3xLja5POP1zmvHSBRv8i+aKmJHmfeLNwYpGQ3BRk1NMVez2FfvNCXvgVLlwTEs3HhfmiZlUO958zwwqyampw0Kt6vI6b+IykC8+TcTXI2xkptRxdn9VDNKAbmmZJhCctYUfDktsbRs+2zNBdj6gV8keMc= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by MWHPR1001MB2384.namprd10.prod.outlook.com (2603:10b6:301:2d::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Wed, 20 Jan 2021 01:31:19 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.014; Wed, 20 Jan 2021 01:31:19 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH v2 3/5] hugetlb: only set HPageMigratable for migratable hstates Date: Tue, 19 Jan 2021 17:30:47 -0800 Message-Id: <20210120013049.311822-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210120013049.311822-1-mike.kravetz@oracle.com> References: <20210120013049.311822-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.11 via Frontend Transport; Wed, 20 Jan 2021 01:31:19 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9f785e2a-fefe-482e-2b9b-08d8bce3165a X-MS-TrafficTypeDiagnostic: MWHPR1001MB2384: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XRhN3C8WVJyTjlcLXDerVXg06u0U/fpsF/Rxbw7X9zvAnWlCd1aV31WaaJOG+3WtFoRynWcc4XowAPqfLtN8r285Zg2QBw14S8UyJEJvowZ3jOJXlAioAZGcqn0hWEUUJ9kJiwnS1lv+kCECuTsL60egDHL4lnd6eczVwxFOXaL46crc7YZ+Nb0DgFIy1AxQ1B5Yism7Tlhl+w6NbU0wEXQ3LKw567vTlgk2/tol0meigZyeDx10C5UU5glwBOC7TABXxd2jR2iwflM0i2iFEPi1JGrVvg0yTzA3e064iyok+OR392NGzPUDJ4TT+C015g0M6D4C4J/vz7zus+OuZJ9deOuOqeraA6ZgbPZJ+5skeHugdExwVBnv87K0cFcUAEHEhCt+/uVh5ChwdomA8u4x+Q4bK3KWvQF+cs8FRtYoAs8wZ+7gIdL2M00YsAKECeoSN7h6HdZ+SqTCLONufsESDWdYoWyXflgyXeHrOSmcw1bzHoIJrDjHgjsfgPYrxEEC3PqqtU6QYLpI0XJ21A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(136003)(396003)(1076003)(8676002)(186003)(36756003)(107886003)(8936002)(478600001)(16526019)(4326008)(956004)(316002)(86362001)(5660300002)(54906003)(6486002)(2616005)(7696005)(66946007)(66476007)(66556008)(83380400001)(44832011)(2906002)(52116002)(6666004)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: nu0FIzCdiTXHTgSCAjcyYPhq2E9m25KtX77VsWFLNyFWH9z7OCRnQkmFO4T3RI+3t/gJvwxALmrC/UcdA3tDIytFoeOCVSD+nQg7ADrBMD0S0FttYRiP2qcEN9az6nglrz+FD+HXGWqmH8bmNHNh/HPln+CxAO/2VvzVGJ5+R5dSxXsaCKarx/QwAJZU8Sa2JZ6YQfngc53tZUVe0jV3cx+hZpqahgr4YtMv9iQmAJIaUSqQl1lv2s7dt7HphJyU3OthMzjm60cgQjrd875t/wblRlPaEhpBtpbBLbuvr9CeHYusAJSLldaXFx2fV7PWxMx3rw483jXNvS4yjNaN3nIptOrW5IG5vgFPfp1sLEwvHCWjRFJNzWXDsVEOTrW32LYOYoyAjVGjBXXka4EzXUXu44fyc/h5lt5EhzoxZX5fvDGF0mNvxuDHqtgI18Izpz6WH4ABtLZFbE+ndf4CIFOrL3zx7XZLYdVOyGjQa2fiiFsmRUmjEW3DWMHxzxv1cF/kcneAEgU0KTxWCeMzaNQv3RzSuQaR9f3z3spsvF1mM7Vnmvkhxpz2us43VkGZQfAbq2f9TpjuCfQwUJh+hCboIgi9KkFhQeTBe1CVjf4OKGu1n29eJkdns5ohx5apU4KvWbrzg4/AVJ3GlSrA8vB//wT3aZWtQIlFa8BTho+7ZGbwlcCaG2zvNjUuXp/3DXxUgW6Afypggxp5Adwv5QOoCD5FegNT1G/7grP8M8yIPBgvnK4szN2r4cRzYV9mbwoKYu96tBV4hdvgsEj9+0Od+/b02lGVXNcPD2BPBNONKjevWzCD0HQ+yQt8KIzIcQgh/wv2BeX9AxT/nhLJEaBee7rw1D8YgdTU+g8SKwuZjSUCf7bRcnr2aenmM/Fmm5VYmIkw1Mxfq8sJdBEtgOPTlsY0ibOtfeaJk2pvKW2sGB0djxcWEM5vhshrfAT5xxz+ewCzvYQcZ3jL1oUrUZwsj99QVOkVMN87JKNkI3ZY5UNuzsGMZSmrNcsmdswrzJNp+uYCohzjkfN4hDeqH+tZqIPAhSNKondWANyGxMj8jdjgA55YEKKA6HlnIiwp X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9f785e2a-fefe-482e-2b9b-08d8bce3165a X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2021 01:31:19.3900 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7dN+0g+IOXPACFd6gaf6x0H4ODFVq29ip6xHeSz1u4x6+5mI47GnmXriEnAVEO0r5w7bpKIT9XYKR0IYxXyvIw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2384 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 adultscore=0 malwarescore=0 mlxscore=0 mlxlogscore=710 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 priorityscore=1501 adultscore=0 impostorscore=0 mlxlogscore=760 spamscore=0 suspectscore=0 phishscore=0 clxscore=1015 bulkscore=0 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The HP_Migratable flag indicates a page is a candidate for migration. Only set the flag if the page's hstate supports migration. This allows the migration paths to detect non-migratable pages earlier. If migration is not supported for the hstate, HP_Migratable will not be set, the page will not be isolated and no attempt will be made to migrate. We should never get to unmap_and_move_huge_page for a page where migration is not supported, so throw a warning if we do. Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 9 +++++++++ mm/hugetlb.c | 8 ++++---- mm/migrate.c | 9 ++++----- 4 files changed, 18 insertions(+), 10 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index ec9f03aa2738..8b8acdafd0be 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,7 +735,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); /* * unlock_page because locked by add_to_page_cache() * put_page() due to reference from alloc_huge_page() diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ce3d03da0133..1e17529c8b81 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -744,6 +744,15 @@ static inline bool hugepage_migration_supported(struct hstate *h) return arch_hugetlb_migration_supported(h); } +/* + * Only set HPageMigratable if migration supported for page + */ +static inline void SetHPageMigratableIfSupported(struct page *page) +{ + if (hugepage_migration_supported(page_hstate(page))) + SetHPageMigratable(page); +} + /* * Movability check is different as compared to migration check. * It determines whether or not a huge page should be placed on diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c24da40626d3..6e32751489e8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4196,7 +4196,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, haddr); - SetHPageMigratable(new_page); + SetHPageMigratableIfSupported(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -4438,7 +4438,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * been isolated for migration. */ if (new_page) - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); unlock_page(page); out: @@ -4749,7 +4749,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, update_mmu_cache(dst_vma, dst_addr, dst_pte); spin_unlock(ptl); - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); if (vm_shared) unlock_page(page); ret = 0; @@ -5584,7 +5584,7 @@ void putback_active_hugepage(struct page *page) { VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - SetHPageMigratable(page); + SetHPageMigratableIfSupported(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); spin_unlock(&hugetlb_lock); put_page(page); diff --git a/mm/migrate.c b/mm/migrate.c index 0339f3874d7c..943391cd1a7c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1273,13 +1273,12 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, struct address_space *mapping = NULL; /* - * Migratability of hugepages depends on architectures and their size. - * This check is necessary because some callers of hugepage migration - * like soft offline and memory hotremove don't walk through page - * tables or check whether the hugepage is pmd-based or not before - * kicking migration. + * Support for migration should be checked at isolation time. + * Therefore, we should never get here if migration is not supported + * for the page. */ if (!hugepage_migration_supported(page_hstate(hpage))) { + VM_WARN_ON(1); list_move_tail(&hpage->lru, ret); return -ENOSYS; } From patchwork Wed Jan 20 01:30:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12031191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6443C433DB for ; Wed, 20 Jan 2021 01:31:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 550F220449 for ; Wed, 20 Jan 2021 01:31:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 550F220449 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CA3086B0007; Tue, 19 Jan 2021 20:31:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C2E9E6B0008; Tue, 19 Jan 2021 20:31:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF3C16B000A; Tue, 19 Jan 2021 20:31:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id 98AA66B0007 for ; Tue, 19 Jan 2021 20:31:35 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 67E5B1EE6 for ; Wed, 20 Jan 2021 01:31:35 +0000 (UTC) X-FDA: 77724426150.09.bite89_331367027556 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 5044A180AD80F for ; Wed, 20 Jan 2021 01:31:35 +0000 (UTC) X-HE-Tag: bite89_331367027556 X-Filterd-Recvd-Size: 13634 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 Jan 2021 01:31:34 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UqPG117344; Wed, 20 Jan 2021 01:31:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=T/y0C8KGg6VbaNBVwE5V18yFykdnOaMirXBUIgX4rWM=; b=MuOo3R0CaTEz3Q3Bet5rn4y9NfLC5EVfV8jglPj7gBV4rd5UgKw48mf2XnPRhyEX5mxe jTTWfdDySUa/z1m0iQgZP2xlw7MNRD5/gauwx0UK0dfA61oo+vJ4KhWGet5tZG5Np1r9 yNuR6C1CxRAqTHbQWxwQV/gjAtCVbro5O4B9Y+dBbY9kj5lofWHtt8TetTYPqbPYzkS9 2015jUa2pGaJM300v5jEdWBrUfPN+lgRKA0hT2VXWTFb5a/OwNH5sssAb4L0DxX+9b+Y t93LJN1peuKgMX7uhkNre7IkLqwj2fhxS2hYxlMe9S+I1384dYOm/F0PVIOO+kHqIAN9 Hg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 3668qmracr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:23 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UeBd016838; Wed, 20 Jan 2021 01:31:22 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2109.outbound.protection.outlook.com [104.47.58.109]) by userp3020.oracle.com with ESMTP id 3668qv5kr7-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Di6goIXyjWjl9ZTbvso7DpcLuBi9Gdk4JRogDJGBZZB3DCvScWztaVE50rgIwaZSkHdKCTD4f6qbxB6pQT/Ku8HuuzYSDfCDfKp043AZ3TTrSnOOaoG0BYlpR8827wpv84/KyPz1OCp2vOijR5h2pZjDDzYsTOFjSo5ubh5bmavgwADw1rkmUL0jd8zXEKFBbObB2+aaYVRmplOiFZl+VY8V+pwi4Sjozn1W2EFykTU/9Dkc85jgafpBE2JVrtM4G/s+47KNIYFPw3zave0NUhwbMbVq/5MaCDfM5zbnLNLMo4MCKFwbi09RqxJ9kMQFpBFxADdR0yl75mp+bsUwoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T/y0C8KGg6VbaNBVwE5V18yFykdnOaMirXBUIgX4rWM=; b=KBZN/4ivv/m/YWb/fxEPjEVdyd9ecB4jiZCouwO8VPClXZ45G4Np8+tlcwvomGZ62j2kKjzGe/IP0WotSPAWD9FIZlBnQtd5tCac2dxKttxjDY6PbqRw2G6/J/FH/EMNHEUN2YGTQurthVFFYMCjVJdOYjlPQCTG76JMPO4zRDs4RfcECKbWZqf16a6lCUfLJIyTx5XmbASN6KAVAR70pVb7HLC3nJjbBE+gAlmGKocWmbPULlTbkT3zUqktmbjhoqduSlltGdGdw0pZCI+RYGis/SWMYJ8B+3rMksHAvmi1ogXk9dEiNtyi4H65xK7osEvzqKXEti45QBol2bBcow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T/y0C8KGg6VbaNBVwE5V18yFykdnOaMirXBUIgX4rWM=; b=GG+WCqkfkmNf/MfnI5JQKVYxqz6d0gS94o2Ceqg/8GUBCAuH7mzrUm7qEYXEJop8LE7tt5+HQMuobLHjqHgF8kdQoizWg4Nhn/Nhdf/WD5xMEkc9Q4tQulYBRdFUr2a7tZDG3YVeEK9H9VUPcLafVFDQuKJUP+ICxUJKU9kMlhQ= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by MWHPR1001MB2384.namprd10.prod.outlook.com (2603:10b6:301:2d::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Wed, 20 Jan 2021 01:31:21 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.014; Wed, 20 Jan 2021 01:31:21 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH v2 4/5] hugetlb: convert PageHugeTemporary() to HPageTemporary flag Date: Tue, 19 Jan 2021 17:30:48 -0800 Message-Id: <20210120013049.311822-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210120013049.311822-1-mike.kravetz@oracle.com> References: <20210120013049.311822-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.11 via Frontend Transport; Wed, 20 Jan 2021 01:31:20 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 19a8af24-ad53-4166-158e-08d8bce3174a X-MS-TrafficTypeDiagnostic: MWHPR1001MB2384: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LLhNn22uhhJs+6UsGmzRb3rk5UipYyG8K2oA0db7BW5LxoAu7lcCXBSuzME4Se9I4PcPm55N12qi/SZelsvr1qEUlpUOulYQoQf9aDF2eOR1I6yo4w5IR2qQvDSJQ/+SW4GbNfMNu1lsRjpkIlKqKF7lg7C96nAS0QD53uEo22V9SfrK2I4DivRoBm5rpdiHH5zNW9vWqRTbuLiVLvWUEflqFGMjBk1k/+hOmJRVbJ0IwbRiuE0eBg1JOGiyoC69CYRlu/1iTnCSgy9XBGOeSOism1zfEcHSPO56ZytZZ5qeXmY3o18dTJ7O/YAKLXT9Yk0brw7hGUU3l8IWiPM0k+rklpmHwaAPGcSqyMJ+AQc2KTxmjczsWRNBCcElIYjqlkF3Q68oETZLCK2sW/D8S2JQBAz0r+/bkCyubNYzYT2uClPCrhVTnnQ4tiVB/ld6JZDGDi4lj5ZpYSJi83gUycVrwsfi6xsLJtDezRhCUlFG1sJxJ1Amjo/zZ3imXKvvAe+lJEA/ixqjypudgo/6Jw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(136003)(396003)(1076003)(8676002)(186003)(36756003)(107886003)(8936002)(478600001)(16526019)(4326008)(956004)(316002)(86362001)(5660300002)(54906003)(6486002)(2616005)(7696005)(66946007)(66476007)(66556008)(83380400001)(44832011)(2906002)(52116002)(6666004)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: e92PoP3YqQw89+tYyHMRriUMYrErAa+aojjP1Ah0Tx3xP2WS4RsJYU5yDEFAPUFCoSj3qLFavnmLjTRCH2+GlMNWHpAZZQOQGumHrzQGKxZxFXlqowVj9pgzohTOPJU8+DUmdCp8W47BPI7RsTiqFWk4Z+/aua+H0z372J4n/JhDCc/WW3cKI5gDudSztG3CluloPt1xbE1di8K5V01P7jv8r36h9OO0Nkk8yk6x6ybIpUbKKMwGh6qpzzKQ2ZCbNmdtNaaDkmOZc3AJzV4BS+PyJpcWL1EgjvVLTVnDbkjrRuPztLMJpLh0flHPIQVzN7Ux+uaWoMTOalOYeeMezQOGbIqDT0gkPEDpR2lcAGiDcNFF3gQHh7k4nTPgWIBacF6H3cswz3oFg4W8PxLYwyO8t8FKyj7HId3uLoENlA+OWzlA1RZcWKVE4Rw7o8a2De50gYo/UxnGh+0OfdYmr8F6IMCdpfSv27MS+jN9qq2jGWxVQdhI0shApYEYFSZbFyjeMrUKF2vquBMPuqRQAd65+x50e1fIabAKk2JfnDazgb53A1/LD+TPMEKrhVcQmd2ya0yLcogDD1N5GG9kJWhiJ5CAJQOX1qFQBNxhp0YXcIM3ZznYrM4xCdJ0ZfiB1Jv8vfqh/ZauaufIihyjn8JdTHmTXCC3wruNI0ijv6aLkjQAltTgW4RHb8tBiITYyXizod3b2gKV+fxPjX64U9Odl2pbHDHwLblY9RU8ySZgjHav9BeOeujBNWLFx8k+tKLAJIgdIipBYD845NCYdGD38Exgi4Xtvi83yc3NstIS+SoAwIpdkwfdVlz4cw+lhBqFJHb6rrhh4wJYSLFJYV6UlyudZlB76Pw7g6Oc4SRR5QOAx1UAlnXF7jzV4uDhm/FrMREITaSfAxAwsIc0uY56pNx581FOcHwi1yrbLWMGrZvfU27CdiDJ9fZqP8cain5F3bwlGS4H4002lDPphjun/K1da+lEuUQvqZRy/xW9EkLJSLQpC6lSQedoN5VNucSia2dCHc9OxYjLWya64PqzQyQKSySPGSY0MDI+iwuRByqor3mdKMqWXz9KuYIB X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 19a8af24-ad53-4166-158e-08d8bce3174a X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2021 01:31:20.9403 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7hRKtlWjv4d3VfVl5ki1iKWyOWhL/9OGCE3+ciYrhIgBOehQ8of4VABIcKOjTw2XsoszuMnPok4wvRC+KvXK4A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2384 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 adultscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 priorityscore=1501 adultscore=0 impostorscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 phishscore=0 clxscore=1015 bulkscore=0 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new hugetlb specific HPageTemporary flag to replace the PageHugeTemporary() interfaces. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador --- include/linux/hugetlb.h | 6 ++++++ mm/hugetlb.c | 36 +++++++----------------------------- 2 files changed, 13 insertions(+), 29 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 1e17529c8b81..ec329b9cc0fc 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -483,10 +483,15 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HPG_migratable - Set after a newly allocated page is added to the page * cache and/or page tables. Indicates the page is a candidate for * migration. + * HPG_temporary - - Set on a page that is temporarily allocated from the buddy + * allocator. Typically used for migration target pages when no pages + * are available in the pool. The hugetlb free page path will + * immediately free pages with this flag set to the buddy allocator. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, HPG_migratable, + HPG_temporary, __NR_HPAGEFLAGS, }; @@ -534,6 +539,7 @@ static inline void ClearHPage##uname(struct page *page) \ */ HPAGEFLAG(RestoreReserve, restore_reserve) HPAGEFLAG(Migratable, migratable) +HPAGEFLAG(Temporary, temporary) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6e32751489e8..0d2bfc2b6adc 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,28 +1353,6 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -/* - * Internal hugetlb specific page flag. Do not use outside of the hugetlb - * code - */ -static inline bool PageHugeTemporary(struct page *page) -{ - if (!PageHuge(page)) - return false; - - return (unsigned long)page[2].mapping == -1U; -} - -static inline void SetPageHugeTemporary(struct page *page) -{ - page[2].mapping = (void *)-1U; -} - -static inline void ClearPageHugeTemporary(struct page *page) -{ - page[2].mapping = NULL; -} - static void __free_huge_page(struct page *page) { /* @@ -1422,9 +1400,9 @@ static void __free_huge_page(struct page *page) if (restore_reserve) h->resv_huge_pages++; - if (PageHugeTemporary(page)) { + if (HPageTemporary(page)) { list_del(&page->lru); - ClearPageHugeTemporary(page); + ClearHPageTemporary(page); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ @@ -1863,7 +1841,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, * codeflow */ if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { - SetPageHugeTemporary(page); + SetHPageTemporary(page); spin_unlock(&hugetlb_lock); put_page(page); return NULL; @@ -1894,7 +1872,7 @@ static struct page *alloc_migrate_huge_page(struct hstate *h, gfp_t gfp_mask, * We do not account these pages as surplus because they are only * temporary and will be released properly on the last reference */ - SetPageHugeTemporary(page); + SetHPageTemporary(page); return page; } @@ -5607,12 +5585,12 @@ void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) * here as well otherwise the global surplus count will not match * the per-node's. */ - if (PageHugeTemporary(newpage)) { + if (HPageTemporary(newpage)) { int old_nid = page_to_nid(oldpage); int new_nid = page_to_nid(newpage); - SetPageHugeTemporary(oldpage); - ClearPageHugeTemporary(newpage); + SetHPageTemporary(oldpage); + ClearHPageTemporary(newpage); spin_lock(&hugetlb_lock); if (h->surplus_huge_pages_node[old_nid]) { From patchwork Wed Jan 20 01:30:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12031193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9214EC433E0 for ; Wed, 20 Jan 2021 01:31:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0544120449 for ; Wed, 20 Jan 2021 01:31:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0544120449 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 79A256B0008; Tue, 19 Jan 2021 20:31:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 74A196B000A; Tue, 19 Jan 2021 20:31:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E8E66B000C; Tue, 19 Jan 2021 20:31:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id 4388E6B0008 for ; Tue, 19 Jan 2021 20:31:39 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0CE2D181AF5C3 for ; Wed, 20 Jan 2021 01:31:39 +0000 (UTC) X-FDA: 77724426318.29.milk59_2e0edd827556 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id D1D0C18086CBA for ; Wed, 20 Jan 2021 01:31:38 +0000 (UTC) X-HE-Tag: milk59_2e0edd827556 X-Filterd-Recvd-Size: 12996 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 Jan 2021 01:31:37 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1TUOr177105; Wed, 20 Jan 2021 01:31:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=JlrDjAS7DcV9GpSQiM3WLIiosxkO/pdJVoM6B4mBrCc=; b=WhMsjwKMoDDBPavyfyXNr4+gGcYMpWXV99wQi7h2tFWHIs0Vhtr8FtmwHU1lK71MbpGC zcqDipvYqNOjSh6L1ASM6riGafQhjOlIx/puRDYUTdbfZCkFevjwLab8BXD9Sxj/ngoZ zkp46pfSnKLVZz6PUvS95n/0/rUQ0Kvb57YzxcOeDtuBY6OxKxYdEnO7PRBbFp/hHmxp /M27Besnh6WZq1RjzyCIjGx5TKxQ43z/4Oz3Y0ZLZDxrUbiBAwFjlpEazq70sWL9uanS RmAiKOnpVgl5v6KrrprR1A/Rg8DEK5wMLjJhpx+Xf66buCcY0hVR/HAUk7byJqQPRJqu pw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 3668qa8amt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:26 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10K1UdWa007050; Wed, 20 Jan 2021 01:31:25 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3668qud5jd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jan 2021 01:31:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=doWsLxmXtzOywdaJXzmu0keSkS5y832YQGaqamh6D329ZaGUz9AVJ+HZ+wZoiFN84aQRywlqFL8ZxBho249Z4eoctjhjEVCEF8fq0sCaa4++3oxc3QohIqcuuyVas9vJoEPVsBOjkh0o5YV85bl9+oi6Byzd4HXCY8KKNcHCAJIC50XHmkgov1SShyFzwWnkS6mhJm9RsAogLEckaqmWsbWzQpATGbwvyClY67LoOPKOPK90h/59sNrvTPdgkzdrNB6VVk1o7SaGe2G+5M4RNG/WprevKloLkBtHGOAqLYwV5zFftefmxJMTL4FVFusl6v0PkRt6ZA7jiUbhxBmvsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JlrDjAS7DcV9GpSQiM3WLIiosxkO/pdJVoM6B4mBrCc=; b=aOl+gl+e9p1hKMfQzacPxlwRXp3y+Wx8Ip2K1lVXnbnFxLWUP6cGB0dNza8gpKpSfXC7S6Opt7HIwHaSHl+LeEC92udn2ZBCRtBS/dRS/DsUGOWJkI9fqoKDmYQER3Bkkl2/mXZD9CiaWQ5qCrB7tSuohRp19hHL4ZSjrQFjm36jmatis22SCrPpn0iIyvbD0JK56COVwmwyXSoscxaYiUGNFljI2P70ydHrNYGjgVtCF30Ntpb8Ix7tBvk6EPAw5LP+rCwhePXPKLv/YWsYvc1uqdODanlp7vVGIpmjg6EqvqZEzt0ScDuJfronmxTSLXAigym66b9MeIRp2WOXew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JlrDjAS7DcV9GpSQiM3WLIiosxkO/pdJVoM6B4mBrCc=; b=Tx6RcMc4FV3BH/b83W/QJgRKet3YWkeuFwcG+uA1pb//JDc8dpvzQyZ+tlAGd56aEj9rOC2J+mYYVp2bRmayXIru2WVB4IbXKfKmjiOQKdUI/vlNqAkM5cfR26MEH3oh1jOjYGVANmCdUW1peOnHdYDx2CzpNhhpOXiMJhxwS/8= Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=oracle.com; Received: from MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) by MWHPR1001MB2384.namprd10.prod.outlook.com (2603:10b6:301:2d::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10; Wed, 20 Jan 2021 01:31:23 +0000 Received: from MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074]) by MWHPR10MB1389.namprd10.prod.outlook.com ([fe80::897d:a360:92db:3074%5]) with mapi id 15.20.3763.014; Wed, 20 Jan 2021 01:31:23 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Michal Hocko , Naoya Horiguchi , Muchun Song , David Hildenbrand , Oscar Salvador , Matthew Wilcox , Andrew Morton , Mike Kravetz Subject: [PATCH v2 5/5] hugetlb: convert PageHugeFreed to HPageFreed flag Date: Tue, 19 Jan 2021 17:30:49 -0800 Message-Id: <20210120013049.311822-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210120013049.311822-1-mike.kravetz@oracle.com> References: <20210120013049.311822-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) To MWHPR10MB1389.namprd10.prod.outlook.com (2603:10b6:300:21::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by CO2PR04CA0178.namprd04.prod.outlook.com (2603:10b6:104:4::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.11 via Frontend Transport; Wed, 20 Jan 2021 01:31:23 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9edbbd05-759c-4eae-363f-08d8bce318db X-MS-TrafficTypeDiagnostic: MWHPR1001MB2384: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: J/p7Kxv6qb44XPIp4KJEjNCiAOcLH/goKvk0sbypLqIjAbqKKkyv7xVwTRobVRrJHFNZmwnGoS0OTeFUVD4H6/Di5Wa9rE3zIarygIIMoU5WexkpYHQKV3S9JSKZq6NcEXjILOmR7cotgbqXsZLmJI67JgejBNdVNgPlWjnJ66NWpuIO0aZARTTrjHRMOxNrKlG7cLpe3k1am0nI2/ReNvEP2HTseQwq9GJj7Ns+oIS1FteUldCEIn5GBRCsbFlUprAG3/W8gxarkNnRkxHAUGm0k43iWlOxQCqYMgIGcR4RkKLQ+vwHL+hKlXWhSPvRNr5n4QlIHZ0CsrWYc86Yi6XFnv7tea5wnBbIvt2elKyYyXP+ma8Ico0t6BXYycn5RIiNlOXMvUcRQSJ7gnQZ3xY/3KSyu27TGZcxLsdOELXnmQW3V8hYUf+L/p46GF8p8ohNhIW3JiDgaA9a0Z4vuc487rzpymQgwVOBdbdM9RR4gZ7XqKini7KAkq/g2HeE6fnfC/666dUOsDiDs4CX+A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MWHPR10MB1389.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(136003)(396003)(1076003)(8676002)(186003)(36756003)(107886003)(8936002)(478600001)(16526019)(4326008)(956004)(316002)(86362001)(5660300002)(54906003)(6486002)(2616005)(7696005)(66946007)(66476007)(66556008)(83380400001)(44832011)(2906002)(52116002)(6666004)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: L7f0n8qnG+ZpKW3rYxU2NesnNAyaw7YJPLpURtY1TRROUPEafxY8Yjd5sVbApyBEPYldO3YjIyXES+CyAHEUl0/+sUhnjw298aN0BG/Agw11D8Dryfce4zYxHLRQIG4n+tBPEpTRMr9AYi7Pt/zVQKrH+v+FmG2BKttaMr306mRCw+KVO4owoXoO7femGYo8L+lurE0OAON5FiujuGuY1Z567LUVjANAlt6zVfNe1Dsb30+mc8FV2wJdhPbWBSrjwOXQCUtMDwmegGL2i6l50v0f/0n47+OLgg14zkgAoVOVb+CTemnY+GMNipHGLHxgNRK4rHJwon8+hpmgx9bSyRXXp+n+Q3f2Q6GVq0IuJpu/nlLDU/90K/6DT63DX2vNCrcSh+uqOM851UldwVEx2HFs+JyyUf6ztaB35DtBFjkNmVTdS3eFqOwJWeEXOtFrak7KmzW8xfEoVwak59xuUjTkXmznMj2Cu4zLelK33N7ONKJE2U/QQcgudwxdLgSN1QQjwLFl5V8RUENaJSPoZVaFMltKItWPuRGvuewQaQlBt+yRXL4ITd5S9nxT2ToeElwyqEASqKhovuEkfNs9ZO2DkCpuIKYvEqJqhhNd9r6oiMJXP8PSxKylcosjHsVmBovcLWUsZ4nRshTxIk9fQVkXlriNv/+G35fEzbmNfFvHWMWMznUcpwKdHT28g3NGYhWvMiE/iVgEJQqIkNEy9xf2lmjwcpFzNscBTFaHjmopFEz2+1SqXnXPa83OuNaagShYt0jLAUKLFG9fbligxp5NewFJc51+rQE1Rk8ZSkO4uaQ7UC9HISdSTEGMMddQq85GSv1pzWvRU60Ks/SvHObrWFQjOHSunVlKjs89P4CpTF46m5fnMDeEgoP8rgI7Xi8DzNvixUB6vJQWo+O5zF2ZmdXDSSqOat1foHdKHV6K/e/nHrEFbMDkLTsRTpHZPe6uIL5yOmYG4qgalkpX3Xu/CDscoJOsU6xK+wysl+Ot47r/JKP4A/Rpvw1WNz7hHedn7E9Wb2Z6KgCX8VRb35Gj9sAQmfgKtFnHaiOLpqpVnIV7Yst25jh3ER/ktMcq X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9edbbd05-759c-4eae-363f-08d8bce318db X-MS-Exchange-CrossTenant-AuthSource: MWHPR10MB1389.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jan 2021 01:31:23.5891 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XplRZB7kAw7sz+2vrOG7W3YbWpGpu7shOClkq4GbtDMSCVeqFY1EAml7A1WltC90ppTz3wh8pR6DhMxAF8juHg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2384 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 spamscore=0 suspectscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9869 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 phishscore=0 mlxlogscore=999 lowpriorityscore=0 malwarescore=0 adultscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101200005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new hugetlb specific HPageFreed flag to replace the PageHugeFreed interfaces. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Reviewed-by: Muchun Song --- include/linux/hugetlb.h | 3 +++ mm/hugetlb.c | 23 ++++------------------- 2 files changed, 7 insertions(+), 19 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ec329b9cc0fc..8fd0970cefdb 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -487,11 +487,13 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * allocator. Typically used for migration target pages when no pages * are available in the pool. The hugetlb free page path will * immediately free pages with this flag set to the buddy allocator. + * HPG_freed - Set when page is on the free lists. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, HPG_migratable, HPG_temporary, + HPG_freed, __NR_HPAGEFLAGS, }; @@ -540,6 +542,7 @@ static inline void ClearHPage##uname(struct page *page) \ HPAGEFLAG(RestoreReserve, restore_reserve) HPAGEFLAG(Migratable, migratable) HPAGEFLAG(Temporary, temporary) +HPAGEFLAG(Freed, freed) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0d2bfc2b6adc..d5a78aedbfda 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -79,21 +79,6 @@ DEFINE_SPINLOCK(hugetlb_lock); static int num_fault_mutexes; struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; -static inline bool PageHugeFreed(struct page *head) -{ - return page_private(head + 4) == -1UL; -} - -static inline void SetPageHugeFreed(struct page *head) -{ - set_page_private(head + 4, -1UL); -} - -static inline void ClearPageHugeFreed(struct page *head) -{ - set_page_private(head + 4, 0); -} - /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); @@ -1043,7 +1028,7 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; - SetPageHugeFreed(page); + SetHPageFreed(page); } static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1060,7 +1045,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); - ClearPageHugeFreed(page); + ClearHPageFreed(page); h->free_huge_pages--; h->free_huge_pages_node[nid]--; return page; @@ -1474,7 +1459,7 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_lock(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; - ClearPageHugeFreed(page); + ClearHPageFreed(page); spin_unlock(&hugetlb_lock); } @@ -1747,7 +1732,7 @@ int dissolve_free_huge_page(struct page *page) * We should make sure that the page is already on the free list * when it is dissolved. */ - if (unlikely(!PageHugeFreed(head))) { + if (unlikely(!HPageFreed(head))) { rc = -EAGAIN; goto out; }