From patchwork Thu Mar 25 23:09:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D094CC433DB for ; Thu, 25 Mar 2021 23:30:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6D58D61A21 for ; Thu, 25 Mar 2021 23:30:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6D58D61A21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D91DB6B0072; Thu, 25 Mar 2021 19:30:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D40FD6B0075; Thu, 25 Mar 2021 19:30:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6CD46B007B; Thu, 25 Mar 2021 19:30:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0164.hostedemail.com [216.40.44.164]) by kanga.kvack.org (Postfix) with ESMTP id 9A8576B0072 for ; Thu, 25 Mar 2021 19:30:52 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 56B006D9F for ; Thu, 25 Mar 2021 23:30:52 +0000 (UTC) X-FDA: 77959993944.06.2393338 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf10.hostedemail.com (Postfix) with ESMTP id 095FD40002C5 for ; Thu, 25 Mar 2021 23:30:49 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNA64n136801; Thu, 25 Mar 2021 23:10:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=/edl4LK0LqLECQ3jS/x7z7ezoamA2mW6jrxS8EmJzc4=; b=X/JRygtpxPmElsQfgSNw5BoGZhxSHc7Wpp+M0Vc/eVOTbg209UYOCH4P8EaXF/SACUDu IcfWlu1VLDMAOv8p818oArS83vTElXvW3WmDbmhqR/hDME2QrxahLKRJNFFunsn5h9xu QRT7+OhWdG5H/GQvKtrheK6qjlld4lX1pZXDWRKbw8NGnbVPFaAubbdMCaax/2CPWAZk 4NTayKaeuL6KPeTEbdx0YagkRx5eFNdVACbTX0epbYyRoN5fPl95f1s+Zb+uUjv3vDSo qyqlO79LwLkVlFJY8z4yHUHIeZ+RuhishSHpWx/rby4RQqWbmfT+KLKjUrZkhblacBu3 DQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 37h13e8dwv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:06 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN0JKO161365; Thu, 25 Mar 2021 23:10:00 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2043.outbound.protection.outlook.com [104.47.73.43]) by userp3020.oracle.com with ESMTP id 37h14gdpp3-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:09:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PUTor0emwFTNiyMXNuA/A6jxcVmr5ufreklJ+4oJvKiNED7rg0xcetKX+dFYXQHwf94qrpBiR65a80f3vjRtIm7bcWGWRUJkBJcme0tfp9X/6/luTThIujGhVntF8QJu4EXLTaRya5GaN605pdEQ3KmntZZ9GKoyA7j3G3GKU+sWL4nbIjZJgi0l0mZeYEB2EoCxo7zup2KEox8GYK+FYxZo+RJaqYaqJqZ+3JLXwXtTFnL5IGDBaBU7d7QPUwhLjnwjWyu1XebZQVScRcdraYHAxVtJE2SMWiq0rYadPuiexp+WF7g5LF28QVeMAGLp+iCbWiLv2E0sXsXNjnPPeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/edl4LK0LqLECQ3jS/x7z7ezoamA2mW6jrxS8EmJzc4=; b=mACaqF1QvzGhySlrQ2JyeYVtbm2vJ6HnLbd27FuqQW6qrUsutSeqSYSCOX/Hey8K+m5eZU/NU0Dnypimo05fTQlpGSYLELp8YFAxj0vH158VPyikQAxcN7Ny0PDhq44gbeIFXTpJBX/rN974xDfCEMSyH5BklGGjtqu8MIV8EXDvpr9lqFiuqlyp39Hf4PrbSgl2mvxTmkeXaIGygV4IpZSe6hm9cLidgEjepzeZ//iqtqzSrM4IjF9Mjkp6C4MtNT7xobfTq2t34PEi5MbNBaaDhzm1/a8+/I3NNl9j/dJHnmy4lMt92blwPAVqsyCTkHpcv+KxLhKA1+YBEZdGzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/edl4LK0LqLECQ3jS/x7z7ezoamA2mW6jrxS8EmJzc4=; b=RExkBMiXwb5JPh9oGHArSRbrGukp+WL6tU7m62VnGZc9X5X4q7AJKeSA7OZZivQgILFzYSrl7PHObVVNk1OM02gOEPDOepv0JLJ6+Kq2CCZzT+WT/5f8SJuZR2V4gCvuDeU6+izkaqX9fXi6xuLb//KU4jfL/fn/vJJrlmJ0QOY= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB3987.namprd10.prod.outlook.com (2603:10b6:a03:1b0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Thu, 25 Mar 2021 23:09:54 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:09:54 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 01/11] memory-failure: fetch compound_head after pgmap_pfn_valid() Date: Thu, 25 Mar 2021 23:09:28 +0000 Message-Id: <20210325230938.30752-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:09:51 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 267fda0f-1be3-42e9-94f9-08d8efe319df X-MS-TrafficTypeDiagnostic: BY5PR10MB3987: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QLHA2ST4uRkJVxdULmKtdl50iS2x9tZ7gsdap5UQ89THG/nrKOEJNc5HnCSwnBMhpV92WIQ51zfglzWwmzvk0J285W03ci9g4qON5MciiLUJ+hah3ZFeiMtzr7WYF6iN5ZhGzJQppEYPGjTBboat/BgM9AWF91APsMHSvCwUUMiz3SZId37zaPJV7e1QKNNGvO2IIwCi3gQJ+PY5B1leZL4zq3YKOH2gGTrXiRGUIcxuRxwoq7VxzoCStcg7Leh/u3anSoYLcmEfmE84boBs151O6nGk/sLo6z4K+i/UjWPIMhap4Xziozs51x1v/XayCPTzLdpKKycS7LyjVw6l3FwNIwx3WsmVLViL5+b6NJiYpA54gLQUEaHQi9lwDpdX85KuRs/DPt56bLIEaSkxLomND7kWUQ4oMI8/MzLAnoFkMg75ONQ5S24Z0t6bKL2FGwpJvcoS+4TyTe5NkXB9D2qMgxkJkLjtQo/hRFWu6GySt/Ey59GwLn8jb4I7E0wLPSEd41UQErF+YjVaJ/EXzB+UpGf0vsrfI2dDnvf+j6ZPUVfM9f//endKNAUoGEA5WYeDO7qdfWWKrL5BhomGIknH85ufdXim9edeP5iwZ0NjpWSmvNPsUiAASx7ZLU0+m2kteI0B14h6I1eHd5smBMctKazPh4hjHCjs1TsiAwo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(396003)(346002)(83380400001)(4326008)(26005)(107886003)(6916009)(36756003)(7696005)(52116002)(66476007)(66556008)(478600001)(66946007)(186003)(316002)(2906002)(8936002)(54906003)(2616005)(86362001)(8676002)(4744005)(956004)(5660300002)(103116003)(1076003)(6666004)(38100700001)(16526019)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: F/ULqB6nBjBbbthr+K9wCDTC6Py7CNStvWjeNpJm8HnVlUqpfv0S1SREHyp0qKqHbjhTR4hYkvI4Np8RFvwdyFvHVypjrFpL+JONYToHhxsgv70NIA7tL52tSxHa81SzSfe6EszKLOxMiik1pVeWSzsoFkASuKMfN/tF12Db3dLp7GmuyO1Cq+9igWgZ/zIGIbTCoQlqDpvIwCfWT6dZVf+0jh8oaUtcr/hN98sfW7Yb9zygkuOpGQqlgtVcS0X04kcJu7GB/nVEnWHRwurNJ4jhw++XaEMGFLQ4hKTk+WikIH2vQZcnHV6zReihF5FI83/om7D2DbH8E8hWvFZ3t1KtQ2wOOTlQNTmWgWzhSiSJc17U2GbFpSCHADJuc4VrpJZeK9vAc6fAdM/TIgaxXpkh3N4AThwphIZxc8DLQ0Iu58+l+/ZgaH2pV+JIpB1uf4VC3RSgm+9IlMcXpcI0T/V0rNNIrDPder0ozVjyF9JDsOx6xxWUKwhOHpi7+H8fpjrOWHNMem6gxhL4KEir1Ht6N/J+7mgykaQubPrgdhi+ckawgPhxd6+2zj6yKoq/xKorHxtPLRzjq7ok8QqaTKOQ/+D7oo2zrROjmfgwnMlQPhfoj8RZzzDlQD22+SGnVZBOPrwObOcErv6cqm+ejHMsKI9PfvpGWed1ieXGr5DVLSivS/hulGb457yqMtPILraI07mCjtRXXzZPMCD37XnvFt4Yo1SAZf/OPzRvTLn9OLPwvVvcqaP0K7nzJ859WhggQl1TETdRQgbDpcyY7Y+D2mzQMzQTvdGvDRmI/ytLgsFVnrtFl3wISU96UYeqbrPUKKGrS6seRosDAdi7l3cw6GdPDtCL26Oj90uyz/Wm1FKA7qj7CsiVgG0ylexLGFkF41T1HAd1PHAGNXNi/7cy3M2SKvyw7/GAPpXwosuWVOCdc80mZPgbTkLss0aXgw4lvJBVFrkC/weXODug7Nyx1xU6zL6TEWAV5lPz3MuDmahNKvGdL3GCctjTYF+Y98/KzbO5cnpyanqBSo6yi6xdOuJxaNtkYbeHioWUFfAP6wPKDTLRZl5/8USiGduOwv/v4ZoeYGYTwZ1b5SFe0sRcvwX5rQzzJInueEVzk9+XZco0VNgnIVYUDawQtYp3Wf2MpEQMLKmB3Okdzj+rr+sWdkqND86n1FrZCjhcvaxchgBFUl00qimQf0J6RAXr1X7t9Q6XJyhZWT1UdbaD+AxxYJjO++ndNoq4WsYj9xOkDBcfs3CJHL3bppLFg+PaoVIYenaN13K8ZeRCq5fHqlBe82zVGEUtpWVj13YDwo6+zx9PAjgmp7FTxw3Q68jt X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 267fda0f-1be3-42e9-94f9-08d8efe319df X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:09:54.7180 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0oEGPP9H5EBw0VQUtHISK2JaKCykpRa64VKzJXKaZ52euYCehOT6TkScqZiRLGbSzLRjnFEtA8hwSZxU6SsGFX5pZUVWJsv4oz4Y1yM3PhI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=874 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-ORIG-GUID: XapoKmfb8yxsnSR8QRLdPQxxlG7CuzKo X-Proofpoint-GUID: XapoKmfb8yxsnSR8QRLdPQxxlG7CuzKo X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=916 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 095FD40002C5 X-Stat-Signature: rmct6i1zst7khimzxsf7b8nfh4gugxm6 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616715049-915573 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: memory_failure_dev_pagemap() at the moment assumes base pages (e.g. dax_lock_page()). For pagemap with compound pages fetch the compound_head in case we are handling a tail page memory failure. Currently this is a nop, but in the advent of compound pages in dev_pagemap it allows memory_failure_dev_pagemap() to keep working. Reported-by: Jane Chu Signed-off-by: Joao Martins --- mm/memory-failure.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 24210c9bd843..94240d772623 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1318,6 +1318,8 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags, goto out; } + page = compound_head(page); + /* * Prevent the inode from being freed while we are interrogating * the address_space, typically this would be handled by From patchwork Thu Mar 25 23:09:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B5FAC433E3 for ; Thu, 25 Mar 2021 23:10:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C823361A14 for ; Thu, 25 Mar 2021 23:10:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C823361A14 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 98ED36B0073; Thu, 25 Mar 2021 19:10:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93ACA6B0074; Thu, 25 Mar 2021 19:10:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F15D6B0075; Thu, 25 Mar 2021 19:10:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id 4C2126B0073 for ; Thu, 25 Mar 2021 19:10:39 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 151A1A8C2 for ; Thu, 25 Mar 2021 23:10:39 +0000 (UTC) X-FDA: 77959942998.23.61BF5E8 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf03.hostedemail.com (Postfix) with ESMTP id 12D59C0007CA for ; Thu, 25 Mar 2021 23:10:28 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNA629136802; Thu, 25 Mar 2021 23:10:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=2LEtq198GuV0ib2ePVeimqbTpQFbr2hIOS+83MDx9bg=; b=q4tMKNXSndodqHTzm99GJGkrEfka8XGPOxRRegJm9pYmgWvroTFm+UPPc7izS9bHYCE3 NC1CIiPiznHYXDngy35TWtcGj/IAUqa/48oTbpz7pcafVkggi9Al+NiFqN5M4BjG4X44 qEExk02SE2QbWRyguE79K5VoEheaHnRmseaOpyDanSCvGuJ9uaFCGey3RC4aTs1KUIis Y549TKpjq79jnKIW4InX3FDlwBDTBvE48O3zb/0rE9qYvttgWLdqxt9xELBuuKTxZHr5 4zev8gCGCIxTTqbC5n4Ow0Ei5Pv3DUDNnp1MjIvqVP1fh3Liye8lLp5KfEy6QD+kaxr+ kA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 37h13e8dwx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:06 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN0JKP161365; Thu, 25 Mar 2021 23:10:00 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2043.outbound.protection.outlook.com [104.47.73.43]) by userp3020.oracle.com with ESMTP id 37h14gdpp3-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AjEVAE4qa7D1N3CR1mhvhF7oTtWN0NIVr5rbauCPZpHiunAgkR8Q0ro7wAHWgQHJ8UNAdpLLJHtekC+EihAsDmRRcPt9lrgBBlmIe2POqYgIQ2TwLRrMiwCK+/rmL2/JqB+UHJQ9S3Uf8ZO9Z+a583yhA7WNJ6/RKt/oM0kqHmj+Sv+lXah5QEsZ02CycuSYoVn6B+DMPCmNMVwXbmK3Fm9CpYxeD3Alx2V3WGrlW92pSsHtKaNjmkUEQGlvYtavdCx6pM3JaWCJJxXd0bg940ZrB7js1ySvsr1XRaFYgG30LY4IX+otqYe2MvJELqtkz30SYQBFv+cbOuTkwxDe9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2LEtq198GuV0ib2ePVeimqbTpQFbr2hIOS+83MDx9bg=; b=KLGek+aUAvTHvEfpcVTkzljdzZ233MgeIZ8ot+fCO0VV4uL6xbB9nNXPHBH6OFNioGKBvYyhPwqbBG1qgOIAJqAyEujvx13j/tQoJ68SPq0ojJkk1ZNKkJR2hFfNOkYiWXqDb1zx6M9fgvILqAzaTCOFgrwz30VZplLBGAM/G+ehCVm6MkQ2iymE++B+o+Eiv+1UYbH8LaDvx7FPXQiYhR27d6chahpSM/nSHn05lm0yfB8jZg4wj8z8/19DU5V94tbbHVsvqzvAez6AShL7ijuP9RFzbFFTpqkMotRmLVk7F4yv4G+B3eaBbNPP6pg2QuoTZLmmSG78RFkYU5xpDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2LEtq198GuV0ib2ePVeimqbTpQFbr2hIOS+83MDx9bg=; b=eViCeMuTk5wN9ZGhRnhNodO7Ey1MFbMebakysp/WBoozQQD881GGAahAgKmmPCQ3edJf5wQjlcA8Gja/ien5YtVQwKbiVawuR395S8ipCmnY4NEWIzyRACDCQ9PB+HwpF/pALeCN6h5qrKuPIaDDR0dMuV6cDwNnQw9n1IQBWUI= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB3987.namprd10.prod.outlook.com (2603:10b6:a03:1b0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Thu, 25 Mar 2021 23:09:58 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:09:57 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 02/11] mm/page_alloc: split prep_compound_page into head and tail subparts Date: Thu, 25 Mar 2021 23:09:29 +0000 Message-Id: <20210325230938.30752-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:09:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1e420956-ef07-4054-fd0a-08d8efe31bc4 X-MS-TrafficTypeDiagnostic: BY5PR10MB3987: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:983; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: d0OaagaGgfZE8D5BWnwaRGh2CSRgZgcHu6KSK4maLUf3KhLV8ipVzSIG5r4OzAZ9556q3zSAmg/6jvtLBJQG6h8IArD8xvmI3b8RNk/e9fFNTqKv6t54Fp7viumHyuCQURVqxFPThThrfWcDebLje1oYK3bpKWyS4SkWej+gEcNemLGhhCzMrwfIvXFiRjrk+2J0+upb9ZlzT85UG1YyIilG8EAUILy1NsCIivwfFwujD9eSJolwDdASPqiY1cL4zaZ5QAvTBwDTK3mQr5u+EZQkZR8tbhUYWpdRBpAlXEEZ1Ob1v0CyNHb8PoYzdiEeB7XGkUiIH50aq2DPLkg/3BjmI8LqqE+9fni/CyUVL2sKiAnXyDrL4ALdCVwGMtDOgO2p/fV32YB4prvY36lpxNTdwaSP2+FeQ3L9nF2iZhCz4f67Fiq/2TzGoAh0DbWR61WTGim6pIRp7cngmy0xl3NJYPugsw7PAWH9afFdIDtwSQXF5R+A2lD7azkIaiXBP7SSM1nsMABIDbmN5IpzW4D/gOp1bL4QBDXPTevme7PqPp7mfyPM0NXNLLYEkGjbLdL6PdeNF0NteuYxhq7+CzceBOEUgIkEAwipl578RuF/1m+COPjEvG4fNCUPc/K7jHrhfta5XEPyJdHAWvVrMCJDw5Yf1iZLkDLDSz7w8ZU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(396003)(346002)(83380400001)(4326008)(26005)(107886003)(6916009)(36756003)(7696005)(52116002)(66476007)(66556008)(478600001)(66946007)(186003)(316002)(2906002)(8936002)(54906003)(2616005)(86362001)(8676002)(956004)(5660300002)(103116003)(1076003)(6666004)(38100700001)(16526019)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: c7ikc8k9fH1Oesi1Zhg0iJifQPEmTQNCzGC21WiPbC0QHNzzMr3AmzkIH9UXX4Ni/oy5NuIYsZTsAS1CIDrCMh74l29ikr8oHKwamZNMRTRTbhlHrV/59tFay5ax5w7Vii52SHbyQG6HpSF8kmRO3nC0wpobg1tZ00stxhpLMTrOzbCdntaWxNTPbdse4ht2Jdoo4wSDA96OfaS4DkXR9i0lHHy2Cxyu3V4fNER0V7t759siWN9ZyAMPctjsGpxnJqvgVDEd5TE8UaymzgDOtp+aRiWL7oyo72HjpUGBeGc2F0xoIZPFifxC8JuTY3pHVqouBHT2zQCqPiL8rH3vHOVDAjM3hBwgs44L0mK12HYJeRpBOQTmJ6yU8KwYJMIC5zx3IvtueaX6krlhq/6h0Mar9nvQhOinyQDW5fR8CVtUiuxSrb95S+VMNo5x85REYbarkgT+goauDPOhBD50Bng3uALizpeXkPI1+4Vf7G4vYoLVoFzPnRdTMnEdpQPjgk1G+EnYajomaEhizGYGa+lpVKBkmIChNzeVWS9dRNiNzIyE1kc/KgX0Pp6LOfR2HpR7aq/Zfst4UoZx3Rfpd8ymmd3pEcMn2bQw4sdk411rh1zqEeVw1IUSuhYak4/RDcw/HH43wEL8WqtG3oKEUQjAxgIULxcTi6jjg3rgRIV2a+QZNbfbSvRTxDnm7NpIlAPwHtLb19UhkIYPIyEkCk4+yzxiOkxvanECF/H8LWcN+VLO7K53V8pa3TiVvjWetuLGLPkxSHm9qlqtd8Qx0yB/p2f124nKOncshiLFMeBdtqNrf9Uo3yXuPcG1J9g71nA43tLtSITioIraJ+CqE1TrUWHwQTKne64/s9nphz/xODwrwZyVROXBpPxq2/IlMNRLalWwd3EQPhUkr+HSBHl6yaxD6MzatS09Onwi7K9Qv/oOO1Qgh6dTHD/tQOi3vC5qbpBeAiY/IWon22K+fnE0IvSYrcektgXfP0/3yKeLBgYhDbkZ2gxnhE2ZHJJp9bZ8ewVKjOpewAICABI/gPHirgOAw1yJRNeMTibvSJdEq4QTU9p9o+Ga3h9ZtpTnvVn4mKHUmrkzD23cOlF77K6m7yFcdVlsJuUQsv0E1wQyDcc9DDLgO2Jb3gmBhotfpDX6TECbXyi1sb1b7oER/DBUPE8DVJj2dqCn6PUhZqRJXsNPaAU/ls9K9kve9u2S5ymUqJj8zVRBgHetN93d9grmJe2HNIVlBZeDsKU59L3NC2PCoa9j1ERQczZ5wD1KG7DkGv/zlaO6SMfb5R+rGofyQn4lQwqtbUeplk8lRTuakCQfVg2B7sKwWBdTAGaS X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1e420956-ef07-4054-fd0a-08d8efe31bc4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:09:57.8202 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: k9xGrKpg2kynMP7u7Y/zBuy6HnyhjpEy2D1i/Wx4YjAUXX49kOjTt1Z8A9JA3Kl0EqwIp1rbmSIeYMVO/Y4zWkGFTAMDa7j7fH4DWpqUBAM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-ORIG-GUID: VjROm8v2CNCxM4qDZO-U68FCShxXr5C2 X-Proofpoint-GUID: VjROm8v2CNCxM4qDZO-U68FCShxXr5C2 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 12D59C0007CA X-Stat-Signature: eqp4e7enkp63gpocy59xzipwuz4foss4 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616713828-618756 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Split the utility function prep_compound_page() into head and tail counterparts, and use them accordingly. Signed-off-by: Joao Martins --- mm/page_alloc.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c53fe4fa10bf..43dd98446b0b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -692,24 +692,34 @@ void free_compound_page(struct page *page) __free_pages_ok(page, compound_order(page), FPI_NONE); } +static void prep_compound_head(struct page *page, unsigned int order) +{ + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + set_compound_order(page, order); + atomic_set(compound_mapcount_ptr(page), -1); + if (hpage_pincount_available(page)) + atomic_set(compound_pincount_ptr(page), 0); +} + +static void prep_compound_tail(struct page *head, int tail_idx) +{ + struct page *p = head + tail_idx; + + set_page_count(p, 0); + p->mapping = TAIL_MAPPING; + set_compound_head(p, head); +} + void prep_compound_page(struct page *page, unsigned int order) { int i; int nr_pages = 1 << order; __SetPageHead(page); - for (i = 1; i < nr_pages; i++) { - struct page *p = page + i; - set_page_count(p, 0); - p->mapping = TAIL_MAPPING; - set_compound_head(p, page); - } + for (i = 1; i < nr_pages; i++) + prep_compound_tail(page, i); - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); - set_compound_order(page, order); - atomic_set(compound_mapcount_ptr(page), -1); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + prep_compound_head(page, order); } #ifdef CONFIG_DEBUG_PAGEALLOC From patchwork Thu Mar 25 23:09:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C089C433C1 for ; Thu, 25 Mar 2021 23:10:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8EC8F61A0F for ; Thu, 25 Mar 2021 23:10:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8EC8F61A0F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D04666B0070; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C942E6B0072; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 987C16B0075; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 7A74D6B0071 for ; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 323C252DA for ; Thu, 25 Mar 2021 23:10:32 +0000 (UTC) X-FDA: 77959942704.03.D8F3052 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf09.hostedemail.com (Postfix) with ESMTP id 8DB9E600010E for ; Thu, 25 Mar 2021 23:10:30 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN0mYG136557; Thu, 25 Mar 2021 23:10:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=to3HhWm59rBOu+npVpIlkd97Z9Q5FNJcCaQyMIuXZ5Q=; b=U8iRpeiy+9SEUNI4bmy2WptwrbxEBiCkSapxnMNF3xpuNpKET4WGphkUtUl9GNQqpLdD NGuMlCs1NPZUQm+kdPaeHsQHD9Ra2pEHbgQ+dpnq4ep2JoI+s+e5NU2uXfIVFcU4aElL uruu5i94gDNeNYFPjJHDoonn3jVIU0FZ1XYMRS6Jzc/G186KyzizlAyqtWlaQtW9iFxN Nab+SPgJABk/6YmO3xvgcilfirES6YbJ0m566hnZa9wMhiCk89yMPUqDehEnEo8i+UnJ ATuDGYEK50mjwmWTFSjXwiAhSkarkZf1dPtMEtxAJTiLUlgsx/a+wMu+ifdVP01CNL/M Tw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 37h13hrdwc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:03 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN0XgM105859; Thu, 25 Mar 2021 23:10:03 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2173.outbound.protection.outlook.com [104.47.56.173]) by aserp3020.oracle.com with ESMTP id 37h14mdk2p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nW/Z/YZuMX9A2OO0pUNe23Hp3LSkdKRzKrIUWPNJ5vmrVG+HwDal3o6GUfQyLncBZ4yZvO1F/OBaH3vO94APhDgZ+u81ZUdOxJYKvOR5WnMKpozDNBTWSM/ehN6BbJ6g2oHNE56mYFzaueVjn8CvtLBPYElA5nXvO7LljEwELiYMjVzVV783Jd1dICpSmKH1cJ9WtQvvybSk8OP4eRxkWIoXh/PKojvazI6UeBrpA/K+K8D1EddfbpmQggPHDgKqD99Fi0HJccirjNxHNoMwWfxMkUj9/Rkl5gPtJwJ1kaVDx2e8iQ38JTm+wr+9YmVQpfVVtViHCc1AUnZRoKyoiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=to3HhWm59rBOu+npVpIlkd97Z9Q5FNJcCaQyMIuXZ5Q=; b=X80Bi1Tnbya7ek4BqsSXRc7WNugqy5Z1KuPvxNlFCc06d5cizaW0e4a2uP9tvOaWTW09K9nwnpix+uiQfggNbltprsMx3W6G3dvZ6I1u/xGsGLe0CkaK1T8su2jq6lobEJyN8b+jl628GfxgRcVy74Lh47FJKyrgYPEnHIEmTeTZZ7nM3IqGWkpsoe/BWfmHB1PC1bBYM+uQlxWVLSv+eJixSzbIqRpGrXnTspEAEWraLrdCkmUlJL/5zpAYq1oFRcbfCoEdtXuFuB67jiCj/Fw37vbUuaPGXEzz46orTzzzoygcLvAbqwaDYmAEGMykaFl+Juu+bFbIxCIqOLxBog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=to3HhWm59rBOu+npVpIlkd97Z9Q5FNJcCaQyMIuXZ5Q=; b=r6zYCMAx9oD7elxSWAbNLwS0KkceKcUza+o3ZUodM7+FOMJTG8sWde4M/7RtEsNDMlusqy7Cd+jq4TXNNktbMR9mqlvLDaJ8ExwyUlOi8P5bNG17Lqah41V+IrScbmnlq8QHd29NObJd/RlEEXLgpULEsOhl5Uzi3R4lsk2ZIzk= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB4371.namprd10.prod.outlook.com (2603:10b6:a03:210::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 23:10:00 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:00 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 03/11] mm/page_alloc: refactor memmap_init_zone_device() page init Date: Thu, 25 Mar 2021 23:09:30 +0000 Message-Id: <20210325230938.30752-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:09:58 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: add72cc5-7647-4b2f-328c-08d8efe31d95 X-MS-TrafficTypeDiagnostic: BY5PR10MB4371: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BOc5ifyNBQvtpWSyXJHDiW/GLgNkg7Rg5Zu01k4GviSglZzkCuCJvmVIUCj4evKlELcIG82qvQuvd49kJ0psIESuOcqrYAocajvS1dI6gstGxDorsT+G+12eL+8K8Wo668pNVjS6t7wlEiCqoG0/GI75hGWJ2rpvH5JEvqAWYD1dKGgwuUbiljuGEQh8AzkWF1jxUgbAbGZSPm44ZAT/BJJmFxIU+WFJNtla6eBzv0TlNXVQpLBLnCSiIpgKlf3pavDKsIRhQxfrhsjxSmYRKTsITClhbpR0dFTftsKW4kuhYvreJFRCf0G9M4i7z1IlqOs7rL+w6EKTCI9DEXXF+FXxKY38hYjx3+yqiP9P9onmGrtP9Ne3Opton20Q1WPqEzx0/iCi+jGiqLj0/2S/SMHCAJcReWfcAwK9blO5UP+cmSBN8NMtVlGhmVxXTlxxVU+vFceCBYMzHwHTsiXHNZWH3P8Q3Z1BaZe2ridTrYR6NvDz9jQZdAtCvUW2fTGudxku+K7VyBJCjLgvvRbo46Z7GYkAz3wTgqmqBaCdBp+nbMRlj063GLK3u/s2yqVJIS3vcInOR8tMTFGJiD04gDxo6ZjaIrEuxFDLNHsA/Py0Z7y/BEuJH63A3OTgTvL1T8/VHs3iTUuoL722nWfzCBwgDZn0HodYC82lOqgCrFo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(396003)(136003)(186003)(26005)(16526019)(7696005)(316002)(38100700001)(54906003)(103116003)(478600001)(6486002)(2906002)(52116002)(83380400001)(6916009)(66946007)(86362001)(66556008)(8936002)(4326008)(6666004)(5660300002)(36756003)(2616005)(1076003)(956004)(66476007)(107886003)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: i/s/CB0brVl5EsQlLF8CkRjco5aE29pEelCClRUum+4Eh920Gom3moj3W83gNBXgoUscx9R5u5tTMhZgMe0ZtPM5WterBhCnzuyoPAEBKgvAP1m9/8RYXEIloxFEVlrHsyD46uLd9PK1NiwXXjna/Nw75Kq660pRnGqoPOOvkdbjkYGbTl6nTR5qxvtyeJvb6affjqny2WCOr0B/I4Cx3Z+v8+sr+G6JX2Sn0ajX0AwABXQGRf9vQIGoTPfwGWxXaQrJEFoXdtuTweDSoXica+CLoOcvvkoOTeVyPlr3CPMG4d8VlohBwmi8yA8bj+UijBodfosBuqyhQBTpQIhc4F8Z+8sxeI2GTf5zaO67gcUVbeWxNx625SuyEsd/SFGZV3zYu1tjYFstiMW06FEZZeblIQxnM3qIGju6pV4SDlDy7WtFpeuc3kCMkpkbIa7UBxgJw3XCYegiiMM1M+HHGQ6l2MsRNaY5Sr0sJDulDjJ7XW8u8f0sOcXARql7yngkhRQSCirdy2BNL6gbQN3Y54xIh9mPHLraNINcEBxk42SQ2SMH0wwvUNvsdeTYR94UKX1Mm96ib230CtVnuuenyqdQkgIuQj7BoJLdJ/0cWuQoaY3uW3cQECXMKTkKyR6outCPI9YC1Mu49dL4dRB0VNwBqEu9XRkcSYjY780H4GAOy0laYp99DDh+wsopWwdKieEzeCZCEbmrGRdsbyL5AmONjDmCMkAYpD1ogpNhfBrPxCrVytJlAoZYUL1VeqPmuDqbAEuKPAtsHxBpM1shPn81HeX0fLTcF4x0FCW4bZxLZ5Un3L0+WyUQ19M//xDBCx/Xuyvk3QqchXVzy7WsNPmS9Kl8CWMKIWz1RtHva/qMmtjZTDG9ED9gY7VcRiatblAaPfK49H+c/uEMcBE1FCiRNS6cKMTHczhEKh+2PZdhW5ksOJvaYuFtaa5PNFae3/HDh+lUwI9YaTSKaoQNUYqFRCYRSt7T1IPQ8CJvQbTDvdq1doii+BihrNKRYgwhQGdxEGOaEDJiiNO/wcaFv693Rw5GwiyMU49ZwjIiKJwsjT0h50dwcK7Q3EhbRou2pdA1XuRlzWAjt6BQbUpPXKXLYiM6a6XH+anOmdU2WRB5RP7qryz77A5rTdKQAI/c+8tkKrQeJ7aHKW5JUpUFOYApCwxpcW0J05j89WsrxAVUntX7Iyq0gE8kYlxKEHJEACU59+gO0D4v09VMJbe1ildfp7ixiLAmXG7BPW2FSaSnM2vbXmCE7vV1AJsKQ/eG/DVcdnOykCo8qsv/KGLFanvubvxFZ+msCyWy1xoiwd4Er0IXRh16KmSP/IZ+4MUb X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: add72cc5-7647-4b2f-328c-08d8efe31d95 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:00.7745 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hpSb3OWgtI1XC71zeohWPPQIWNvoRX1+SOSQCvAvOMvrdGU7jyq5wlleQvW8+VZ/ia5A0M2OdcZ7XicYl6uMslH/fKdATh85VPnH4kbLWS8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4371 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 spamscore=0 adultscore=0 malwarescore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-GUID: b4DDCjW_qozs93J64QvGDVlWQcAYzpnF X-Proofpoint-ORIG-GUID: b4DDCjW_qozs93J64QvGDVlWQcAYzpnF X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 mlxscore=0 spamscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Stat-Signature: 89g34ommow3w6q38k9314y4895o8j9x5 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8DB9E600010E Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616713830-846497 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move struct page init to an helper function __init_zone_device_page(). Signed-off-by: Joao Martins --- mm/page_alloc.c | 74 +++++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 43dd98446b0b..58974067bbd4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6237,6 +6237,46 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } #ifdef CONFIG_ZONE_DEVICE +static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap) +{ + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer + * and zone_device_data. It is a bug if a ZONE_DEVICE page is + * ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->zone_device_data = NULL; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * Please note that MEMINIT_HOTPLUG path doesn't clear memmap + * because this is done early in section_activate() + */ + if (IS_ALIGNED(pfn, pageblock_nr_pages)) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6265,39 +6305,7 @@ void __ref memmap_init_zone_device(struct zone *zone, for (pfn = start_pfn; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer - * and zone_device_data. It is a bug if a ZONE_DEVICE page is - * ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->zone_device_data = NULL; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * Please note that MEMINIT_HOTPLUG path doesn't clear memmap - * because this is done early in section_activate() - */ - if (IS_ALIGNED(pfn, pageblock_nr_pages)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Thu Mar 25 23:09:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31C71C433C1 for ; Thu, 25 Mar 2021 23:10:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6D43C61A0F for ; Thu, 25 Mar 2021 23:10:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6D43C61A0F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 79F566B006C; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 73D096B006E; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5415A6B0071; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0195.hostedemail.com [216.40.44.195]) by kanga.kvack.org (Postfix) with ESMTP id 31CE76B006C for ; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E7746180AE7F3 for ; Thu, 25 Mar 2021 23:10:31 +0000 (UTC) X-FDA: 77959942662.12.BE6EE02 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf29.hostedemail.com (Postfix) with ESMTP id D5491F3 for ; Thu, 25 Mar 2021 23:10:29 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNA6mU136792; Thu, 25 Mar 2021 23:10:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=xOXVthTr7705vX0AU8WhlfFywO1Y1olXp2ZtgXcnpBg=; b=E7c4n5Leb5hqlKIJ1HK7YsB9O7XHIolA3uyevFjzAp6OwlYZH7LhqakyUk46YaUW0mnb oqUlapOD8x0g/3GUUWza3O7Aa9UqNdAalmVEmlZ4Hof8x5ri4olmWdZ1m79YlhGnqlgV jnwZfRpArYuPoc9M0VBwqIYdvpOil19LYKdXnYDnrDjQKaZ/etTNW2BNXD3Retcp2TmT 6whq4cpJhZRiJL6P1mpmLw0DeCDcKVxWuOxF237p5mYvn6gbN94/+DFQv8cHkIB03XPw PVG9Kzzgt5r1WKwwu+XkC97nOdH+jO9cHohaDv/nWmrvFQQC5htZ/sfhJ1fFLsZt5vFE Xg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 37h13e8dxe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:18 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAItK009261; Thu, 25 Mar 2021 23:10:18 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2048.outbound.protection.outlook.com [104.47.73.48]) by userp3020.oracle.com with ESMTP id 37h14gdpy3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WpKh7gUp3BMMYir80Gyb5Gm0367eooxbCKPtY4MO5nwu5uBrHy1RWF0LSdLF9tIXZyqEXcoXBcHQILaZPKL3yW51+jZJGOYGdWwJQ5kuGEywXxk8drJTgBbh5xbqKg7ssNYPRdRvvzvHFgaXG/4NHpN08sHKc3+VUoNYSw/6HRR+/HqWB+VXqk/1pMz1rRhFxQ/uI4hthhUOEzlWZhDcyrFXSP6p19x52dCkMcpDIrnum3I6CeAAIlOw026kQM0pIhoH42gaLMPvEiECE1gsSsRCH8ew0BsXU1G0loKs6WnPl2zmzUTZq+f04c3C3GEE/+YwQdBpLZVX0u/j2sfs8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xOXVthTr7705vX0AU8WhlfFywO1Y1olXp2ZtgXcnpBg=; b=JSEk8tH+wKfgTC39QQhfSPWgnjR2xBkCgFAtBJYT3h9PtZy6Q4VSL8eWBM9mcxGmDdQt5XcJWLuIGTqYX9VKlvHhdSIorrewXWFByggcx/2oX8yeB2wNlQNL6fz+oz4V4FPN5hkWtWlCej0bH+UUiQjfMXj/DZy1YpiJetDHq/TWgP6cMAbECfIZuO/QTUgabPc90tr+RurQtKdZUb2rmXHtbzkyNHordKM02G6vhKzXH1uVR5DHf574SganiKC85BYKlg2jtyntg+MYht+6BlgZFCND6zcIBX1IwWNR3HL78vcsTJJD7GeeKeOczXOan2EjpKyE+jHHWiSeIwuZBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xOXVthTr7705vX0AU8WhlfFywO1Y1olXp2ZtgXcnpBg=; b=uQ+gKmoxUzsxZrHp7vVS/8pI9JRIOVuEZLnVwr0V+Lrt1Pr9y/1clL0PqU+i/bveKZ251390YK1bWvopmGXPj9jDMtgZ9tM5PsV77JO4V3SgPlTIsDsnn1pGvCDsiFS8Kjh576xtx1DV6qsxWmKD+cHVkPeqOgHMDQB3FBw3vbQ= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB3987.namprd10.prod.outlook.com (2603:10b6:a03:1b0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Thu, 25 Mar 2021 23:10:04 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:04 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 04/11] mm/memremap: add ZONE_DEVICE support for compound pages Date: Thu, 25 Mar 2021 23:09:31 +0000 Message-Id: <20210325230938.30752-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: deec0e5f-cb80-4adf-d95c-08d8efe31f5e X-MS-TrafficTypeDiagnostic: BY5PR10MB3987: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3173; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V6fCSVfMT295TCceSsmgtBhTldM2B+q6VwiJOCCg7Y76kqsprnqdNiY8Jq4YHXQO7Xf54oFg1aJH4QsBPzfZx8FpZS8UQ7T8PAp+NFI+J6yn/nz8E0X7Z3/RlzpENXifDOxmSJQTdBmOusT3IML45eFj+X2RcHh8eBYG8hmjXd6w33YrsPUyQpUhQHDhmNiTARWfI5c71Go7o7sgd28QGlII+nNgAUs96fDgh3E/MsMv9YbKKVbL37AA3N8am8zr5o2MJGPHOtWdA9RhSuYG/OX9jzQzkiBHJD7G42UsEt3b2DesDsqfL3U9aPTeMQRxsMVhOR+rf9Qm8bB6z9+xBd1hhKB6HOB8e9LZ+PomPtRti5jo1ZI6yu+BwNLEpErqPcm2cotr7aHVQROX5BhZ/DogAgorjyiuYWa07IiDbo7Th0ldPEypagwF+4FRSlBLmKpIIAcI05hrpZFOpIRzIpAqGDUx4I8SyEAzyOdnxI7fgdlnaYjI4rRIZmqmO606hWvRqe0vpYzHo6JhsaTvqMk7CFenwpRhEFYoPbv/Lhw4M3RLN3tImJ+0t1y2i62pzHEHwwCclVD3wVauDeadHvMfrT1QJN8gQChG7rd1bCLHNwN5TLXZi29QseOXZS+S6PoajcscFkrf11QQvZ4cradZhguFvFzRM5u2sK6FgfI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(396003)(346002)(83380400001)(4326008)(26005)(107886003)(6916009)(36756003)(7696005)(52116002)(66476007)(66556008)(478600001)(66946007)(186003)(316002)(2906002)(8936002)(54906003)(2616005)(86362001)(8676002)(956004)(5660300002)(103116003)(1076003)(6666004)(38100700001)(16526019)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 1/fYXn+WUCcc63mIq/y8BhlT51f3ilIj7lqiYZt71H3jSYjCpp3rgwX3iHepWPuJxlx8MeP8AEgpaNLbslS/mDaUDPspzz7VqOSOx1zIAhgmgyYu/ULaYB6kYfwOpjXrhRzrzMehagivSTaaAzX6OLx6OldJqZQXyQguy1MfmRrCu6BHlu8O7AwAzkxlAOkAcDAe9Sl1iOydy5ZD384rhbkMOtjCOR36uHBd1zZDgjo2gHJva2x5hPp3f6Y/b13O12R2fwgCqnC28WRyh46IER8FMgTMk97U1vQyRkHDWlTd8v1saAkDHTzF5tT2fdSghSU5TOPe4qW9UtGG4k8HWm573IHnS6nKtv4D5bcwzK7veFz8Ic+/VJPwp5FN0Nxhs9nXGIl2z5DHNohq2WqNrw6v5fpaRRQ/v8IKYvCrQ8hJh3m1dLo8MDJhmA4Xql+ow5kLtOnDWmZNCKtRQ5+TmOe7u9O4X9b5L9gsVe1X5ZJwv6oMS7C2n4MDLgiDFqYcfYopjYsV8PgFg5iBFhgX3PbdPFYBwZuop1LkkktY5o0LhH0e+Koi3Irgx++cz56mvLlvtWcAtErwenDKIBcwnVKiKUe70lezD5sO0xPFRAvwSJbSzD263CtpWOamDXd9MxTBIUGzjILdkynXb/pMGJKK40Rj2SyE+i0Ye1K+7zy9Low/RnOyAIeI1tEygiZeuP+iRVO5xLKzM4vJV2/Ta2ZvGBMTTZj4da16l75NJSqJr4GH1e3Nxd84ZhIelqbUTIjDdMViLkAXjn/8BEqG0D4QFxj6ssPp84DobBf/aC00fyiZqvNt2kDAdOWuxXfXCVpbZ5mCM+xCt72UvRbLHiBdr0ZvjmckJTfqaOEG88eVLr2FlilvZtyR7+LvL6pMkTp5GkFqcoTLMTdEBiIcJ5l02ft924TcKxck/i5zlaK5ARiql2Th+26+LhieyrvEtVCrxXCceNLkkkzl7WW9Qnkq0tUf4CrAoBY28vH1eG9Zl+79ebE3OTHV9m5HKVVpcdqGSM9TgCsCG1gPLFbIO9t8ZD+SdaK06QbnD/2VJr00UQixr+8C61eKNS38NmHTTr3jQUWj2ShMzc2JDwk2+YnsL/WMe6XyxdC3mT5alN7hBuPDCWVGvw25WT5xUZ8ZEaME/Puo0TW1G1uVnjVXkcE4tjtJIL91Jwyl3UfvcKVqaTdlNQFfeS/W6hOpSvIP5w97xP1YSxiB39t+3uZzn7AfqtdJdmc6yNzJ09JdPLp/XdOIS+FI252ftjDhPtKjbFZymblRVAK0ALUgQnOM37C2oLLDNM0pQG17+4XEcHB4c3ju7zmikkBLuYM8gljH X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: deec0e5f-cb80-4adf-d95c-08d8efe31f5e X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:03.8977 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DGXIFUbY6FDUF/VvGJf3gybLLljfYqiwMIwE9AHnDd7fgOTXC6p3LO73N8ORLvB3O3DuUXL2n3N5/UJGC1dDA8HinQwRZZsm5VVYsudm9bk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Proofpoint-ORIG-GUID: GwK2E3ba2ZIN6067bHoAo3MUd9yUe7MR X-Proofpoint-GUID: GwK2E3ba2ZIN6067bHoAo3MUd9yUe7MR X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: 5q3dbjrsk1e93wmga4x9qnssat3nrb86 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D5491F3 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616713829-42490 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new align property for struct dev_pagemap which specifies that a pagemap is composed of a set of compound pages of size @align, instead of base pages. When these pages are initialised, most are initialised as tail pages instead of order-0 pages. For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages. Signed-off-by: Joao Martins --- include/linux/memremap.h | 13 +++++++++++++ mm/memremap.c | 8 ++++++-- mm/page_alloc.c | 24 +++++++++++++++++++++++- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index b46f63dcaed3..bb28d82dda5e 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -114,6 +114,7 @@ struct dev_pagemap { struct completion done; enum memory_type type; unsigned int flags; + unsigned long align; const struct dev_pagemap_ops *ops; void *owner; int nr_range; @@ -130,6 +131,18 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap) return NULL; } +static inline unsigned long pgmap_align(struct dev_pagemap *pgmap) +{ + if (!pgmap || !pgmap->align) + return PAGE_SIZE; + return pgmap->align; +} + +static inline unsigned long pgmap_pfn_align(struct dev_pagemap *pgmap) +{ + return PHYS_PFN(pgmap_align(pgmap)); +} + #ifdef CONFIG_ZONE_DEVICE bool pfn_zone_device_reserved(unsigned long pfn); void *memremap_pages(struct dev_pagemap *pgmap, int nid); diff --git a/mm/memremap.c b/mm/memremap.c index 805d761740c4..d160853670c4 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -318,8 +318,12 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], PHYS_PFN(range->start), PHYS_PFN(range_len(range)), pgmap); - percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) - - pfn_first(pgmap, range_id)); + if (pgmap_align(pgmap) > PAGE_SIZE) + percpu_ref_get_many(pgmap->ref, (pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)) / pgmap_pfn_align(pgmap)); + else + percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)); return 0; err_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 58974067bbd4..3a77f9e43f3a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6285,6 +6285,8 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, end_pfn = start_pfn + nr_pages; struct pglist_data *pgdat = zone->zone_pgdat; struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfn_align = pgmap_pfn_align(pgmap); + unsigned int order_align = order_base_2(pfn_align); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6302,10 +6304,30 @@ void __ref memmap_init_zone_device(struct zone *zone, nr_pages = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pfn_align) { struct page *page = pfn_to_page(pfn); + unsigned long i; __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfn_align == 1) + continue; + + __SetPageHead(page); + + for (i = 1; i < pfn_align; i++) { + __init_zone_device_page(page + i, pfn + i, zone_idx, + nid, pgmap); + prep_compound_tail(page, i); + + /* + * The first and second tail pages need to + * initialized first, hence the head page is + * prepared last. + */ + if (i == 2) + prep_compound_head(page, order_align); + } } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Thu Mar 25 23:09:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B591C433E0 for ; Thu, 25 Mar 2021 23:10:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D97BE619F9 for ; Thu, 25 Mar 2021 23:10:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D97BE619F9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA2C56B006E; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D5196B0074; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A15D6B0070; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0122.hostedemail.com [216.40.44.122]) by kanga.kvack.org (Postfix) with ESMTP id 44D1D6B006E for ; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 07794A75A for ; Thu, 25 Mar 2021 23:10:32 +0000 (UTC) X-FDA: 77959942704.15.D38A184 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf15.hostedemail.com (Postfix) with ESMTP id D7E8EA000241 for ; Thu, 25 Mar 2021 23:10:29 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAM4Q143171; Thu, 25 Mar 2021 23:10:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=amhTLQinMuwxgNnk2V0csPPUmxneFDqpFfpg5NELNYY=; b=XHE/u8kPXaUtlKJIeiY6xqDQOl87/RjNTfIsH39EYjYSv8/NJapuWV28rsdt8zDvUkgf cbLUu5XLlYAwGR7lIOLpMowAAsv3oeJ4HVSNeUcNxdyvEexhNLs+mYBZxXWbywFUlQh4 WHA1BkOtMfdo4kHgTEYghPml0BLSytX6XJ4kj6FJyaoC5uJ/OvituJnZ1EZQg5IrgZX3 6JXm0RkhbwbfL2NQs7Axt6Xj0fmNf4En9uZ/3/2y18mQWxt/yuSstEHwFHFOc4ZHkmaz fxWk5ywULOWSc/niKfYYT1gmOjBuV8y/YBUIxRPMXanLPCpssg4kwu/oLipFQwGdVL7S Fg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 37h13hrdwu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:22 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAItS009261; Thu, 25 Mar 2021 23:10:21 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2048.outbound.protection.outlook.com [104.47.73.48]) by userp3020.oracle.com with ESMTP id 37h14gdpy3-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OIjTzhbG2mNSUKdy87GzjqFpakuC+K0QSAhoc2Qluz0r/nNUXahFGfv6MX436mY4LuhPtWKvMwDlsij2Ur8oqvEnPJ9RcOrAV8g8dFon7788xxCNZE+jCo2cuh+sSCuZLM47phiy0EJ6/EdYTGHoqdiqTguBA0EOywS7rkDmfGnhlmytO3qKMV2HaQpVo+AS5q5bBy3HN7p1k4tMhgedI6+rYFgAE7STCPSgdeTGX/RdJdP5rGQN9vhPutVqXSn6V6MnSYgUXAe2yDkUIntj2UgOAKx238xwjUZlXENgDGOE5y2SbLlqe8zI9ukUBGv0TTKpB/sfjMxAopm6QBKI2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=amhTLQinMuwxgNnk2V0csPPUmxneFDqpFfpg5NELNYY=; b=EQ+cIFwfzxf8A6t7ZR5U8FfeKmDsYRAPi9jsJVegCrV1ETjgNrdtaql58/ysRGcrQx2hcQAD5Q0NqUQ8ONHZ3tSUy0PqQEmAE6aAp4jcz1bgRtJBWJ+B8QqnyR3ofPpQtxWS1j8L/JaFD03A+OTotYEJUKCUBOkijU4sAknvikwNzR3f1udriT/IO/kbUUFcyviHHtIO2VGyVxdyB2N+LQOEvdH3/EhTSTDV9QrGF7j7MpHomU/wC83h8JINz1UPneAJQnP3LvWCKVq+lN17qsJuIgOYB28iqo+0yZrE+9iNP7LRrygMgy90vp8OKELpxoNX0JJLt0ETtnExR43nQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=amhTLQinMuwxgNnk2V0csPPUmxneFDqpFfpg5NELNYY=; b=WTkG5W4tQbcbFthoXvdvw17d3KbFx5wfXg6dFw8bIdFwPKIR0Zoc0WWNuKvnPsFQYT6jGcejsDVQtWbtrr+nmnJkWCop6xJdWAI0Rg+eoqu2fS8Qwu6Gy51r0jPQCRppYUz+0MDCnGKUeLEhW/CuBrxupMYNAClNc/vKtjqQsKc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB3987.namprd10.prod.outlook.com (2603:10b6:a03:1b0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Thu, 25 Mar 2021 23:10:07 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:07 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 05/11] mm/sparse-vmemmap: add a pgmap argument to section activation Date: Thu, 25 Mar 2021 23:09:32 +0000 Message-Id: <20210325230938.30752-6-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:04 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 59a8e4e9-e069-4ef6-9023-08d8efe3213b X-MS-TrafficTypeDiagnostic: BY5PR10MB3987: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: naDulOLEQrr+hw7j4TmtAr4HcgV6yV2QlFAuVRpXJz4+GAmNAzNRntGPgSpLOVXOmB8MS+2ZiQfDTDR0smw747t2BqRhsKBl1P9905+dbpy9ur3BFaWvfAUg1qXKhU60VjlPoankuGncEtDM81bdhuRidnt3p00Mc950WfSSKkUolKOCUb5SU+aRGdc/DnTWjCtqH3+DFFt+xKKTOXOeFkzoJHOeGM6lOUGCsFdvUgJc7+sBqPqaB8c4Hli2kPn9uBQw6tEiLjxLMDKDZzuOdCZmH9YSFZHTjjSfPC3nHd2zVeRbpzU7jLZgW79KLe0++eStgwJdMyFkKsns7PrnC8nTS5C2S8r81G2NFa6poLNVk6SWRZBC4DsyTUSzXqm1EC7o4rZEqrvDfLPxR9vSEPVReP/itTDgFeS0rm1MLqbRmFIec128vNlyYipYJWPFdmTvwkH17edCVmxtEjgaAkI5pS2Wrx4LHLWPC9gIooaNgrSjAiz7SdaTqsC2v/Dn0dkpA4vcBuhpbNbU0basFcNNYEAXOM5dGhFUIUYLhLKvxFwN8o0ob95MKrxMDaRr+SGC2xtQkiF31BrfbmGHk8Syi0AplBcLHHywS2mOdc8V7AxDHp9faLnNPDhgt+thh4c+AbUIPdMRqnq7cKGpY76g11YNurSy36vqA+ScDHfCUYF0Pgv5mIUtlX0+CqCxPe9nStw8ciXSVtuB/GBB7D6yLYuoFljAq/dQkS8D3g2pr/m/W9A/K94UH/wM1lUs X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(396003)(346002)(83380400001)(4326008)(26005)(107886003)(6916009)(36756003)(7696005)(52116002)(66476007)(966005)(66556008)(478600001)(66946007)(186003)(316002)(2906002)(8936002)(54906003)(2616005)(86362001)(8676002)(956004)(5660300002)(103116003)(1076003)(6666004)(38100700001)(16526019)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 0thRjAN6hn5++DyXyOnfKcuYN/wcB7JaENCXa2AapuHvO+XYKrlmhfMMTmbH2zq/4N3/p0H/74RJ8TiRxAFRbmE3lSepLxqs5H0asu/ziZ4eH2xGywylfbNGn6XGzEsH+VtXwgdrH6fiSXHDWFgVUfEoOcPLKnByCYbaNuJaquW7iTAV3sYOqM77FcH5ozZLUdf8CRSffPcdAovFD0Io3WkN/LnXBDrBGqhfKXcORWRcPbC8AUd4TYmT6F2TxRqJAGBFe/1yVaCs55qvPF2LXS+0f44PbnbtH7Qr1A4DFSF4pQMRcq/NzYUhpDDJXL70jkzHVcj30cr2Vrh+f4821sDh3JB9HJG4sKKp88Fq5WrY6e4TbzHd0r3YRnrfCftEx4IVj9MYMJ021N250NXmR/aS29pHDb09OXeqG2uhVH01SO1eckxL0ojS/jtr3FvsrVlN4ydfH26gJpdbUuyxJZki7FllBFsY62gfe9ohOuuiArdhUDOSQHq1X7LSbN8KASCDts5BWHbtJMYBfYiMbyxv5qa6WtQXK2QbjDrwfiW72tE5S8gSsLy6QTpAdNOxgwt09S3pQ5X2zA3bpjLyiLxsDMJygpaNZlBLBKeHccLAHU2mRLFnFJ5gkpAr177syZWbkpWVnqP+IKxJdGpivHe26cud/MMHO4oFPkUcuhSrtkMbbWpbss0Rv2u6ehfoTIxbU6/kNWFKKnpdq/BykphkvSVZCfqivw0msH0/GALOPl6E676uc2DoOhO2FbdFwx7Ve2WjJCrmCNnXHktE1QkLiHQUBpYJixGxRxHY73QYue5OS64cCGc6yx2JX8C5XSSoGxQkSc1J1leoRR1sClJ9h7qC1oQbqtarR9AqB9kF1HCLAHnWDI9dEg0CHZK3AKXqZn17pMBNSiFLP90Vg9rynUPn0V+x+HKy+H7+CqnHFmQCRWQ7/0vevYmyE+ccT308XLLwA1tEm73tlCzZrs5XX6gwdlLp3t7jOU90YDLYlZJequuf8I4BYO2XWIZ2/qr7nUxmsn9U01AmKuk2ScmRqnopsVzHt06qxNJFMBCLLGUReHTScHdKmGqJ0adYkmgK7Qi7637E/CI4iuSdjR0yhUMZXT+M8M7IIx0R9Cf+6XC7E4OOTXxfXtevQqwYr8q2Bk4jnL+ER3PlfkMNauxpmdyJjrICXms6o0t+8zbbkOZvBmzic3ZFLS+pZUDZxwCDV1UlUSqEAw6cLiywS59JxVvhmdmhv+4WwyqPUc1/o/v/0gBj9O1iYbl8/PL8Svj4YET2OrU3gXpOIH/3pQQrqlGpvY16TOwRebqLY+nKAQz0VzdtGIrUgY0v0wj1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 59a8e4e9-e069-4ef6-9023-08d8efe3213b X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:07.1888 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3fJMTTlVRwxfl220ajjYW/tYRIcLuLbPKXAxf0TLvoDazoeV0VteTwrcgul2mZ6Q5Va3rPZAoMD7h7a8zFXKa9p0+ni/VkQ3YBU2uanY3iw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Proofpoint-GUID: ga212bBJh-fMU4fJ1xD5Peg49mTBQJp8 X-Proofpoint-ORIG-GUID: ga212bBJh-fMU4fJ1xD5Peg49mTBQJp8 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 mlxscore=0 spamscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: czy1sr8roohjy9sg58oihob6jw1rucde X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D7E8EA000241 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616713829-745733 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: @altmap is stored in a dev_pagemap, but it will be repurposed for hotplug memory for storing the memmap in the hotplugged memory[*] and reusing the altmap infrastructure to that end. This is to say that altmap can't be replaced with a @pgmap as it is going to cover more than dev_pagemap backend altmaps. So in addition to @altmap, pass @pgmap to sparse section populate functions namely: sparse_add_section section_activate populate_section_memmap __populate_section_memmap Passing @pgmap allows __populate_section_memmap() to both fetch the align in which memmap metadata is created for and also to let sparse-vmemmap fetch pgmap ranges to co-relate to a given section and pick whether to just reuse tail pages from past onlined sections. [*] https://lore.kernel.org/linux-mm/20210319092635.6214-1-osalvador@suse.de/ Signed-off-by: Joao Martins --- include/linux/memory_hotplug.h | 5 ++++- include/linux/mm.h | 3 ++- mm/memory_hotplug.c | 3 ++- mm/sparse-vmemmap.c | 3 ++- mm/sparse.c | 24 +++++++++++++++--------- 5 files changed, 25 insertions(+), 13 deletions(-) diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index a85d4b7d15c2..45532192934c 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -14,6 +14,7 @@ struct mem_section; struct memory_block; struct resource; struct vmem_altmap; +struct dev_pagemap; #ifdef CONFIG_MEMORY_HOTPLUG struct page *pfn_to_online_page(unsigned long pfn); @@ -72,6 +73,7 @@ typedef int __bitwise mhp_t; struct mhp_params { struct vmem_altmap *altmap; pgprot_t pgprot; + struct dev_pagemap *pgmap; }; bool mhp_range_allowed(u64 start, u64 size, bool need_mapping); @@ -360,7 +362,8 @@ extern void remove_pfn_range_from_zone(struct zone *zone, unsigned long nr_pages); extern bool is_memblock_offlined(struct memory_block *mem); extern int sparse_add_section(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap); + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); extern void sparse_remove_section(struct mem_section *ms, unsigned long pfn, unsigned long nr_pages, unsigned long map_offset, struct vmem_altmap *altmap); diff --git a/include/linux/mm.h b/include/linux/mm.h index cb1e191da319..61474602c2b1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3033,7 +3033,8 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap); + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index e0b499a1fee4..2df3b2a7b4b5 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -384,7 +384,8 @@ int __ref __add_pages(int nid, unsigned long pfn, unsigned long nr_pages, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - err = sparse_add_section(nid, pfn, cur_nr_pages, altmap); + err = sparse_add_section(nid, pfn, cur_nr_pages, altmap, + params->pgmap); if (err) break; cond_resched(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 16183d85a7d5..370728c206ee 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -249,7 +249,8 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, } struct page * __meminit __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); diff --git a/mm/sparse.c b/mm/sparse.c index be66a62e22c3..c2abf1281a89 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -443,7 +443,8 @@ static unsigned long __init section_map_size(void) } struct page __init *__populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long size = section_map_size(); struct page *map = sparse_buffer_alloc(size); @@ -542,7 +543,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, break; map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL); + nid, NULL, NULL); if (!map) { pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", __func__, nid); @@ -648,9 +649,10 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) #ifdef CONFIG_SPARSEMEM_VMEMMAP static struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return __populate_section_memmap(pfn, nr_pages, nid, altmap); + return __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); } static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, @@ -719,7 +721,8 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) } #else struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { return kvmalloc_node(array_size(sizeof(struct page), PAGES_PER_SECTION), GFP_KERNEL, nid); @@ -842,7 +845,8 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, } static struct page * __meminit section_activate(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { struct mem_section *ms = __pfn_to_section(pfn); struct mem_section_usage *usage = NULL; @@ -874,7 +878,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); - memmap = populate_section_memmap(pfn, nr_pages, nid, altmap); + memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap); return ERR_PTR(-ENOMEM); @@ -889,6 +893,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * @start_pfn: start pfn of the memory range * @nr_pages: number of pfns to add in the section * @altmap: device page map + * @pgmap: device page map object that owns the section * * This is only intended for hotplug. * @@ -902,7 +907,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * * -ENOMEM - Out of memory. */ int __meminit sparse_add_section(int nid, unsigned long start_pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long section_nr = pfn_to_section_nr(start_pfn); struct mem_section *ms; @@ -913,7 +919,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (ret < 0) return ret; - memmap = section_activate(nid, start_pfn, nr_pages, altmap); + memmap = section_activate(nid, start_pfn, nr_pages, altmap, pgmap); if (IS_ERR(memmap)) return PTR_ERR(memmap); From patchwork Thu Mar 25 23:09:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DEA6C433DB for ; Fri, 26 Mar 2021 02:09:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4045661A24 for ; Fri, 26 Mar 2021 02:09:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4045661A24 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C69B46B0036; Thu, 25 Mar 2021 22:09:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C40436B006E; Thu, 25 Mar 2021 22:09:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A44AC6B0070; Thu, 25 Mar 2021 22:09:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id 877A86B0036 for ; Thu, 25 Mar 2021 22:09:00 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5249DAC17 for ; Fri, 26 Mar 2021 02:09:00 +0000 (UTC) X-FDA: 77960392440.38.2C5CD36 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf24.hostedemail.com (Postfix) with ESMTP id 70C86A0009E0 for ; Fri, 26 Mar 2021 02:08:57 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNANo3136845; Thu, 25 Mar 2021 23:10:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=l+oIxM4yS48AwZPIlRoyiB81tsG53ciuXr1IICEm6OA=; b=zO4yOUzPUQygL/UJ15heZoGBpQQKMZ3H/BjE3HWfeDg93m7MJhbN2vd3uHbPFYpIwceD G+a2QLHxTyKckcrx4+pgn7e+KAUPlnVa+xhb71AVLjMSnCG09nk4Yi22SHx333dSlbR6 MMs3W13CUagS5CFFJVjCvdRYZBoJOBrC7dNd2/QzfaKNo/bPR87ZoC7GaQI3ugcr+bmR jEX1ea4Er2CqbHHWhs1G8FjV0lz0cflZaLEJri/uujCKGlr9qHdqnJxrDhVER+LA0TTD Rt0k8IAAeAFgT/K5/zCCL60VtnhCE+l+dkl7CE644pbPf/7GXA79kqeqdXmd1s3Uw3UU XA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 37h13e8dxj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:23 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAItT009261; Thu, 25 Mar 2021 23:10:23 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2048.outbound.protection.outlook.com [104.47.73.48]) by userp3020.oracle.com with ESMTP id 37h14gdpy3-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kE0ZzokvVCjJBTuzXTgQm5tC5JHCwv3VHmR/ABG54VKWW9ooPWM5MWKS6HDUuuj2uriD0HOWya2xwbHHX3IERMFCZt4RcCVEe3WSHdBkab8TdUfAsTwWooDg0hw/h/VPvHUnESzZRv+dG/lbHsGb+WS68Yyg29LfN/oMfU+E6wb8qpRWRTh882Bmqv0IhTeL6ZOEwXOfK8vF7Q2UeaxbcHYORNlLsxM+abzUcjF1xJgD+6eWxHmazm+jdM8YOniRbs7QvQFAWkQ8vdHVjPx37j+s75TJ/Yrprtf0EdboJa8BusgXeMsj/MH3IXDwQvWT13AWAYZW9Dsna/uolsS4iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l+oIxM4yS48AwZPIlRoyiB81tsG53ciuXr1IICEm6OA=; b=R4iNsOcat1TIEuSkERaRfEn14ywAAjAiqApajubP4WKt1AzdSD9KO5SKO4C9O+3DIO6wU3tZq4aZ4hnFYzjAnwo1YFp3dw2Zz+pw4zXqCXza4/yzjlE/UaMd9ixfuh9mjAFOjaVsHfh0Ba09qgdczfx33nUbM775lxBOmT74QrcwF8TZX6Rj/Li31xaOPrAdPwt4l6GdAUQCMdu0WE7bZTmItmRtTrqG6cFCzTCh9ZUMOcuzZgBqo3MEuaTQ5g0n0Uwp4fwMWpLDXc0Ze1xChCaq6yOuoR6W3GkxsmC7QQ7mLLaaiB2dDWqvMeAJQA1ejn0UfxFhmEJ069ogPlfG4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l+oIxM4yS48AwZPIlRoyiB81tsG53ciuXr1IICEm6OA=; b=FFGw5i4EeTa3bMGJh8/5nReUtUjlg9ar8raK5NSZjcHW8mFvm76GBxXujoiOcbWlNOYEXjWrWsur49HbCyyhuSOylw6amESGpYfuNetDvCOL+WuPy2d+OTa8/UGJWUFNnDG2yRROy0BL21CabRG1jSQu2rq/b80Q+PLyCI7mnDA= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB3987.namprd10.prod.outlook.com (2603:10b6:a03:1b0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Thu, 25 Mar 2021 23:10:10 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:10 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 06/11] mm/sparse-vmemmap: refactor vmemmap_populate_basepages() Date: Thu, 25 Mar 2021 23:09:33 +0000 Message-Id: <20210325230938.30752-7-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 004aeac3-4deb-4908-db61-08d8efe32347 X-MS-TrafficTypeDiagnostic: BY5PR10MB3987: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3631; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 22S/stCPQ7HqSlGdw3zLYjBa1C9pAaSHlafdKYpa5GYD1FrRcviUy3N+VyKkNP4i6xHPljmf38BpSe0aEsI42hto5FPEOlktxx07/8TCbgu+HOdFscyWli7MjiJDshQk9+POHWH2aTSiBYe83LjjwGs+SQ40C40/WVAuvo37K+WxkTttjRieSPDmlVQIINS/dbolRgSSAV1DDrOaqzG/4ChsG3itAy2UoMUvQnSYLEirhQL8Qe1ndMj82Y6LY2bX42pgwEbBJEMP8Hc4ucoUGMp3WKqhAGktVbrifDMrIyigfcaFLdGcUnOS3Ykuz4cEtRZJRUhfszXffOuWv3FNdJW0MNK9eZaSpD7VFQapBgqGudh6AHADtWdqUe333x9oJSqmTpK8l6vjgXLdeMS9PB93xKLEOLJNcxS56RzPxrA1fAR4ZmyxrYRXVLr/LXs69ikyuB4Ot9KWKrt+bDmKZ21hTKnLCnj4+i6l+xc/Hl9sqW12f2e+9KsUsMkuwr2eQjPeaHVi9ZqpHiq0WYHh+ORMI75Si7yWSyse4J3PyMLOpK8JXm6E37UfIHBq7Y3QN7thumnwYQZHpMpOIM6gjqbexqFv+UewSm16VAuqqpE+FzP2tj4faJamx8YmUzdLmeTi7zqzJSlkqVSE0d0hUvIpikvim4fMA1FiIv7isTE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(396003)(346002)(83380400001)(4326008)(26005)(107886003)(6916009)(36756003)(7696005)(52116002)(66476007)(66556008)(478600001)(66946007)(186003)(316002)(2906002)(8936002)(54906003)(2616005)(86362001)(8676002)(956004)(5660300002)(103116003)(1076003)(6666004)(38100700001)(16526019)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: nLmKkAepXl0LlUEt4E47SXSie6tdX2iGDfUiJw5/Lo6MnpHD4gf3lMdZM2LLq0Wjch11KOoLFhTf5IWJjX3lF0sHLm9eMohatHqytQghllh/WwXfI6t/ljNT/gYIzL+bTJds9fQji07rI+vChJnwOPILUWJZI4VEqhvq6b7L36AChp3wQonLPXzPLXcjpSJRNyJ/wbnajnIrAIdqmUp9FvO332d2COl7ixKm+jVMrPERTCyTjCQYYiU28pSF6c7T59sbMr2DmUDMUUaDZkm46w94Vltgr7GU0DuzdqDo2gSnnSyknSPkxjA9JBtlrP+2g2e5BvAgyaDvStAyg8rUxniovwydJhwh5IL/qJQMM2b1EbyeGpy6BMwEwQDu6t9wk6iEo9UGsk5RMlU3Tu4ZW/i4d8MxhzrQMkjTaUQs+2ydN0CLX5Fmv8zfEYc+76kaRP8tmz6rrVoQBU5fZ6Ke959UjAQlfdWLzbZ98WkOMvIYI2pf8i4gjIrFuFCis43Ft+CH8T/SNJHYuwm66txX94qOPCH5hntSdd2x2QEU9/wE1jNkH99vmnoqk838/P1gboyZV7w/oUyaRhx5jrTPr+mZ2S3zgjx3QIu+bDmTG06tj2tP/E9uB5Bt/Rfs9CV07leYU4WmP7AhXwMRjm7d44q9uxwB11sZLoR6bR74Mr/ZcQw1D7SWDq/ikCJqf67gfldHZhkMCxdrulVlJ+LvBKRMJRBV6k3kbs9B76pxqYpWH4e+Y72czl67V4Tu3XfpqxRx9G6uclxKHIK+XQeoL8aUI2zmj7BUr36gBApV6rDIgdsvRsflmB/oDn903k1QqZFhg+8sJeyK37goJHeGvmzJUW3konNoYebmCLplh2GdH75OzLpP677B76T4AyXuOqqBJgeYXZcFkYLrGntDVgvjpFU0RPWeXqWVHPmG/WXyBIGOcPZCvcnUMDbIqKniMj0QZgbyZjboaXHRSAQ4bilqkeREQ0CO8xx9v9Ty6X6FCEiEa0688fCpRq4orgf+36G/yWvl8drggoR01K+S7DaJT++tRr0HKHMttCqripZpTjX1QJXMs5MdZmdbNpDUBpjcp36F461BtZtHBmL7rgLAeus1d2Jg2HZ3A0BKwfkG6T4NVNih81fHGAT2iqIRglFLivb2zgaTMhtQGERyGWrZJsDEQL8in37HxS5/5CCUL4/9gNNnneIwYpRNPn43f7kDEVPsjGw0I3ar7DyoPfLld2yx6Ks+d+dy5Wi49xVQacGGaAMWiH9UHozWwGSz5CtMrKl3+gvGM3+qHAbSQ72SQWE+VRAOlMjMwpgLHYZls6QeffjsMwvXnaySYtVY X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 004aeac3-4deb-4908-db61-08d8efe32347 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:10.4259 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mUfBfa8QdKinRu4vL4Fwfk8yAnNuW+gMXjoZD16oja5PoEgwhs26rPend95vmPbsRGTZPAeliR3leRHN2EtG6OgxJgqrPPmMV/U5MQCWRtg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Proofpoint-ORIG-GUID: jTOgDfGZfIafMdb-8p3Md7XqE5vWZaUB X-Proofpoint-GUID: jTOgDfGZfIafMdb-8p3Md7XqE5vWZaUB X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 70C86A0009E0 X-Stat-Signature: 57gqtjff9qwc4z9hwzq69n98cntc3ai3 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf24; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616724537-970697 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the actual pte population logic into a separate function vmemmap_populate_address() and have vmemmap_populate_basepages() walk through all base pages it needs to populate. Signed-off-by: Joao Martins --- mm/sparse-vmemmap.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 370728c206ee..8814876edcfa 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -216,33 +216,41 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap) { - unsigned long addr = start; pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte; + pgd = vmemmap_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, addr, node); + if (!p4d) + return -ENOMEM; + pud = vmemmap_pud_populate(p4d, addr, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); +} + +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) +{ + unsigned long addr = start; + for (; addr < end; addr += PAGE_SIZE) { - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return -ENOMEM; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return -ENOMEM; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); - if (!pte) + if (vmemmap_populate_address(addr, node, altmap)) return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); } return 0; From patchwork Thu Mar 25 23:09:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4D0BC433DB for ; Fri, 26 Mar 2021 02:02:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 33E8961942 for ; Fri, 26 Mar 2021 02:02:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33E8961942 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BFA046B0036; Thu, 25 Mar 2021 22:02:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BAB2C6B006E; Thu, 25 Mar 2021 22:02:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AF996B0070; Thu, 25 Mar 2021 22:02:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id 7A6326B0036 for ; Thu, 25 Mar 2021 22:02:27 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3D7C318014EC0 for ; Fri, 26 Mar 2021 02:02:27 +0000 (UTC) X-FDA: 77960375934.02.E5D63F6 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf06.hostedemail.com (Postfix) with ESMTP id 4C850C0007C3 for ; Fri, 26 Mar 2021 02:02:26 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNA6x3136815; Thu, 25 Mar 2021 23:10:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=QddfEmjhRawgkFyzqoIhPYD+mvuoYVgtUhKerFl839Q=; b=cseMEbiRGLJTrL0DBT7xYPVGMVSmx1wyS6dpYXKx7syqiMeqgk52Oawf40JRXUTw60HK WqFT06Na4WidZs0E9/+s08dWhNcvRC90rlTxfjKNwO/AwXpe6bc2ls6qRl/Cgib9WoM8 2hpww2/7IDyDs91sdO3KRkNUlKpQ1k5e/zFGFT/NsG6c8wVNqvqMv+Nkjd/Rt40ABny1 5Nx6p+NFljKyybJIKaRbGL4IndeLWc92v2lZ+Pr3QaCNdioHh7NFdqNOktroks1rqhqU uC98BEIjBqC1z7erpzi5qubD1htiwRJ93lOxFHOtTO4N+YyZXoesRPDBVogJHT2U+npF 0A== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 37h13e8dxp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:25 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAJwI009531; Thu, 25 Mar 2021 23:10:25 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2040.outbound.protection.outlook.com [104.47.73.40]) by userp3020.oracle.com with ESMTP id 37h14gdqdd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XOP4jXImOpb+0P2R4nQAL8P7obdUtuV9DqcqYBdoryZlTr06k1z5MAX+6S5/4TDUHzMnx3qVtPauMkbVk5r6mloZeHLhFvbPP8YPnoYlz5blgwUVwOOq5IVOcUZF7ltvHrI62B6uVecVckGzK1QvaKI5i09gJSujqc61EtsS5lLOYWigjYKNXQutZs3OdBXt6BIun3Y5+4Hq0aH3ylFgfo73FTjyn1wDBgPh1oHJ0nfAgfXHfUldRwI9k4AgrAqouxPn0B32F4ApEsEI2GW99Qoc0Xm/jrrqmPC2mYzhXeUNeSzvmb1JM6GUWJCD97EphjVx1cF0Sdif/83U2LcboA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QddfEmjhRawgkFyzqoIhPYD+mvuoYVgtUhKerFl839Q=; b=lSpDGMA0sCKMNltrsxRB8cptI58yOapIZsKBI7dFAbkNjYz2MEso0r0TLf0BiGTlWHOfQL3Jft830xBK0phQGYen2jBgmjgWV6z4I9nW1Ell48C0K/9MuutNgW5fEiQByPNCBwLJmX4af5nQoXe5nwsORioM3ZreiV5D1MUhAZEe1bNF9E6rsV937NpcQLz213AHdJPeQXwDDA6XidreMLpq0q8G5bXGZh8VdoLUjdzDiogHN3DrASkUs2GO2xM3Lki7vT3VfhEqNf9GVNq7AE7kOnOdCtgVdTEQyqEgmuuBwGWlZi8gp7LdvplKTXK4xNlE5/iuGLX9Dmqd+PZ0cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QddfEmjhRawgkFyzqoIhPYD+mvuoYVgtUhKerFl839Q=; b=yIFQM77oG1x0iDbTWEryTa9un+LZYeYdtKPIzTvW9S+I2morDnMWtCNSXys1klq3L9N7BZvHWplW2iqSeUeE27Jt0ShYxOhp63BCSNnkGhF+vCzI9s9nfPfD2AHZeaeLVk52hCDGnRvRfd7xIwHyIwrSN7YZAOELUN/MTVaOaKo= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB3987.namprd10.prod.outlook.com (2603:10b6:a03:1b0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24; Thu, 25 Mar 2021 23:10:13 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:13 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 07/11] mm/sparse-vmemmap: populate compound pagemaps Date: Thu, 25 Mar 2021 23:09:34 +0000 Message-Id: <20210325230938.30752-8-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:10 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 308d4f06-dd3a-4cf3-cefa-08d8efe32523 X-MS-TrafficTypeDiagnostic: BY5PR10MB3987: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UP7jnQ18G31lB1kKxkF+JG5uRVEn48ShcMtdmfO+X4G7Be0lk045mlqz6SaM1dsKlJQG6Qtf/8UuhOEsJd0Pb2E9RjKibP7hw77YH/LS1EqD6AKudLtRqLe96RTIcAuSxQuqv2l2DZz4uwXALlpr9YHFWKTqisFMiBCbznfuNV7GmZXpO3mseaqmTbHYwE97dOBhszIvaA/8avEqhqTGhgx0KEqBHfetzzrMv3ZsUKbOjAQYe85wAttvHE97o92hO0y/yMeGNPUepeQp8AAlFZasvm5t761+AxHVVzZFdE6K0HP1tvG+sJ0VGBWihxIk8yv/x2pkUpp0WMY4OvJ0P3AzBLHg4GwX2OpsWsvZVYE6bOTmxpeWw+TJs7E1BGmh7qGRFpPzz3pQXGhzAu9M/LeuNVRjtIKjRz4skBVztVaq5hPZXC6ptJqJ3pEVZ7bjZ12MvW/WUrmPot5DJeI/LDqPlG4w6OCthYjVa4t5bgoD3XAGlUMoC8tDetn+J0Av4mVC0+1x5ltyetASMIBiqKHAGq/E+gaKHEqD+zzG9D0Z89NC3h7szuV8I4wBfcHo06KhL3wJ6lHdKVnv7adNyOxhZUZUi2yXXecgcyr65BPXxrJ0MsQ4+omgPa+4n5KyAOiB+P2JyGit4DERdLewLg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(396003)(346002)(83380400001)(4326008)(26005)(107886003)(6916009)(36756003)(7696005)(52116002)(66476007)(66556008)(478600001)(66946007)(186003)(316002)(2906002)(8936002)(54906003)(2616005)(86362001)(8676002)(956004)(5660300002)(103116003)(1076003)(6666004)(38100700001)(16526019)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: OuTvKmN19YGFscccWvhDRD9OgEqxOIjeUwE/3ahV/VRcpb1KkVBWky2IOs9nuj/CDNPOGafIrdZhPBu3FedFoQ00LfQt5yE2v52c8peTgHvYSTR9pOvEFRzW5QlVdOrX5+5ou/T0fWwIvefReX9CvLrsadl0M+3r0Darj8zUJyWj3AQ3+XIsvuXoodlEs3BcHhhMqpahl1lBRm8H+rrsMISRxJsmbyTM6y+Utid6azIW1x1yXxSC5I6a5HoWcASjZemH24zdpzJJL4+A5uePuI7lmd4rxN1hwFrK7xKIZi5aGHLpmvHQWs8nk2ThVC9WAIZ6PgAN35zJFl0cF9kRLb7Ol7N2+lwLZuplhY2okDZbYLUdzkUE3/MM/6+Fi6KgIdz+VWKzO3F8qtCsTTLJjjy0fN9DnWVhPpC2DBwgYoJbsVTvOPcnpDAbrNEDLM2i6RTPNlU3bsin9rXqupGNdpTdnWZmcREurCfdoQv5qPGjyPapGGznucMT8nfHfAGaxF32Lysgg8dTRa8E4oc7E9DNf/o0AFKXjds6c4n4HmLtsycezKbekwsAIMx9vyc5XPzFndGgn7TXYE5cPNze8UinEeTgxNE2+AiTXjEpeO4yAsSReFWyoYgIEHA0/zu/DCEjPmRejnoAYbRcB7Jg90U+if46gXsgg0LEa2MN4JXJxinBc/CxauCsGvQ8lm0WGPAXwFJaKjEGdLHYHuZTAHfWpLEXIq+6fYktBlXk30gH6CJqIo9cf3g1/vNHcGxHRepML+W9tZB9nzO43gxOFLJTzBOWdWF5qtDV5Oeyp/+fRup5dljc2KtqzZ1FcEKvfcOk9EVZOdPCVe3bLT1Q0lNiyCetV+CcYxXuax4NAH5r2wX9xarVs6o/GsT3XruwGJHUF8HqIb/BDRRslS7dDYJKohIhBACtsxs38aPwDnJZRYLjyX/j/mhuTGHkq1aULgIimxgo8x9MR0gKJb0oE59urJ4SRssx2e51Kw8QdmgoOumN6B7rFFyEQBufjsEHrRpoKpIagytu7TdNs/ST28OdoN3sd7c5K0n6Vh70WlkdV2hbnnb/v5x9RIX8v2Es4PdNIIjstnfqzKEEBm2dAzVLQZYkZIdmdSnKb8HBQ7L/MG952O0RgV2K4Vj6Yh7ibPq0hB4IUjQXBQ3udAMLACJAlaeunflzadSA/OHXxVi5J+gBXSVaVbK0xI8lht0ScRq4c7dlkjnolfKHe7355J4IoEZx7F6A077fhNyJxxmwMSJ7asaIK6t+d6xvzdOheqFZTNN7NJXADJGI+0Yb6gSxE76KCVphiQ3ta/9qn19cPofHzlzr05SzIXXR4B4O X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 308d4f06-dd3a-4cf3-cefa-08d8efe32523 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:13.5022 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7e5n2naJqwC3/X8ObfL2eryTm4fFOdJfU+FLtT+fyyKibQl0iz93BUvy/BP7R5Fm6y0iqMtfwqO5i/IIaaoYQ4meig/4v1u4h2BQomYnV4Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3987 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Proofpoint-ORIG-GUID: Ot2z5hX7jAYlurSVEuGHpS3W1JdAL1_k X-Proofpoint-GUID: Ot2z5hX7jAYlurSVEuGHpS3W1JdAL1_k X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: uddtgoq9sjcn1us345bk8zqeioituqnz X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4C850C0007C3 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616724146-716348 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A compound pagemap is a dev_pagemap with @align > PAGE_SIZE and it means that pages are mapped at a given huge page alignment and utilize uses compound pages as opposed to order-0 pages. To minimize struct page overhead we take advantage of the fact that most tail pages look the same (except the first two). We allocate a separate page for the vmemmap area which contains the head page and separate for the next 64 pages. The rest of the subsections then reuse this tail vmemmap page to initialize the rest of the tail pages. Sections are arch-dependent (e.g. on x86 it's 64M, 128M or 512M) and when initializing compound pagemap with big enough @align (e.g. 1G PUD) it will cross various sections. To be able to reuse tail pages across sections belonging to the same gigantic page we fetch the @range being mapped (nr_ranges + 1). If the section being mapped is not offset 0 of the @align, then lookup the PFN of the struct page address that preceeds it and use that to populate the entire section. On compound pagemaps with 2M align, this lets mechanism saves 6 pages out of the 8 necessary PFNs necessary to set the subsection's 512 struct pages being mapped. On a 1G compound pagemap it saves 4094 pages. Altmap isn't supported yet, given various restrictions in altmap pfn allocator, thus fallback to the already in use vmemmap_populate(). Signed-off-by: Joao Martins --- include/linux/mm.h | 2 +- mm/memremap.c | 1 + mm/sparse-vmemmap.c | 139 ++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 131 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 61474602c2b1..49d717ae40ae 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3040,7 +3040,7 @@ p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap); + struct vmem_altmap *altmap, void *block); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, diff --git a/mm/memremap.c b/mm/memremap.c index d160853670c4..2e6bc0b1ff00 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -345,6 +345,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) { struct mhp_params params = { .altmap = pgmap_altmap(pgmap), + .pgmap = pgmap, .pgprot = PAGE_KERNEL, }; const int nr_range = pgmap->nr_range; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 8814876edcfa..f57c5eada099 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -141,16 +141,20 @@ void __meminit vmemmap_verify(pte_t *pte, int node, } pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, void *block) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(*pte)) { pte_t entry; - void *p; - - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - if (!p) - return NULL; + void *p = block; + + if (!block) { + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) + return NULL; + } else if (!altmap) { + get_page(virt_to_page(block)); + } entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } @@ -217,7 +221,8 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) } static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + void *page, void **ptr) { pgd_t *pgd; p4d_t *p4d; @@ -237,10 +242,14 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, page); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + if (ptr) + *ptr = __va(__pfn_to_phys(pte_pfn(*pte))); + return 0; } int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, @@ -249,7 +258,110 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, unsigned long addr = start; for (; addr < end; addr += PAGE_SIZE) { - if (vmemmap_populate_address(addr, node, altmap)) + if (vmemmap_populate_address(addr, node, altmap, NULL, NULL)) + return -ENOMEM; + } + + return 0; +} + +static int __meminit vmemmap_populate_range(unsigned long start, + unsigned long end, + int node, void *page) +{ + unsigned long addr = start; + + for (; addr < end; addr += PAGE_SIZE) { + if (vmemmap_populate_address(addr, node, NULL, page, NULL)) + return -ENOMEM; + } + + return 0; +} + +static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, + void **ptr) +{ + return vmemmap_populate_address(addr, node, NULL, NULL, ptr); +} + +static pte_t * __meminit vmemmap_lookup_address(unsigned long addr) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset_k(addr); + if (pgd_none(*pgd)) + return NULL; + + p4d = p4d_offset(pgd, addr); + if (p4d_none(*p4d)) + return NULL; + + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) + return NULL; + + pte = pte_offset_kernel(pmd, addr); + if (pte_none(*pte)) + return NULL; + + return pte; +} + +static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + unsigned long offset, size, addr; + + /* + * For compound pages bigger than section size (e.g. 1G) fill the rest + * of sections as tail pages. + * + * Note that memremap_pages() resets @nr_range value and will increment + * it after each range successful onlining. Thus the value or @nr_range + * at section memmap populate corresponds to the in-progress range + * being onlined that we care about. + */ + offset = PFN_PHYS(start_pfn) - pgmap->ranges[pgmap->nr_range].start; + if (!IS_ALIGNED(offset, pgmap_align(pgmap)) && + pgmap_align(pgmap) > SUBSECTION_SIZE) { + pte_t *ptep = vmemmap_lookup_address(start - PAGE_SIZE); + + if (!ptep) + return -ENOMEM; + + return vmemmap_populate_range(start, end, node, + page_to_virt(pte_page(*ptep))); + } + + size = min(end - start, pgmap_pfn_align(pgmap) * sizeof(struct page)); + for (addr = start; addr < end; addr += size) { + unsigned long next = addr, last = addr + size; + void *block; + + /* Populate the head page vmemmap page */ + if (vmemmap_populate_page(addr, node, NULL)) + return -ENOMEM; + + /* Populate the tail pages vmemmap page */ + block = NULL; + next = addr + PAGE_SIZE; + if (vmemmap_populate_page(next, node, &block)) + return -ENOMEM; + + /* Reuse the previous page for the rest of tail pages */ + next += PAGE_SIZE; + if (vmemmap_populate_range(next, last, node, block)) return -ENOMEM; } @@ -262,12 +374,19 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); + unsigned int align = pgmap_align(pgmap); + int r; if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (vmemmap_populate(start, end, nid, altmap)) + if (align > PAGE_SIZE && !altmap) + r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); + else + r = vmemmap_populate(start, end, nid, altmap); + + if (r < 0) return NULL; return pfn_to_page(pfn); From patchwork Thu Mar 25 23:09:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16C5CC433E1 for ; Thu, 25 Mar 2021 23:10:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5E3BC61A42 for ; Thu, 25 Mar 2021 23:10:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5E3BC61A42 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F04376B0071; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8F556B0074; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD2136B0071; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id 866D56B0072 for ; Thu, 25 Mar 2021 19:10:32 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 418128249980 for ; Thu, 25 Mar 2021 23:10:32 +0000 (UTC) X-FDA: 77959942704.11.E425D6E Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf14.hostedemail.com (Postfix) with ESMTP id 94B92C0007C1 for ; Thu, 25 Mar 2021 23:10:28 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAKeT143162; Thu, 25 Mar 2021 23:10:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=210uT+K2DSFz/dK2BBxVr5ZahPOQRm22NW+bo9aavM0=; b=kdPu/YrepQSQaqj0HIQGx7iFW2Baoe21XnhNSMNf1EF1WTSiOqASQabzsN2D8dZOa+Yd hRAP8PUaI3GVCbWcPKohEEhd7un5eN2qYYbXj4lAYEYNTykJ9No1/6XASDpOjqwOY87g nqGfvAHeeM5xOARoRwF7GIIIFvYttvHKw6yO6oM7JZkDG7a6qHGshcAMIwMVJkywwwRc 7Z6QpisA5f6u2IA02tFJUSC1vW1AjqogDiljCGoUHLNXpho/otmQIB6Zc9rS++WNeXQM W5mWLILNJKyoLW6W0DlnL3E/h1qJhn79dB7ldV4/m2SrC+9vbEI3ELSwvipY0q1p4XRr fg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 37h13hrdws-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:20 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN0WTp105695; Thu, 25 Mar 2021 23:10:19 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2103.outbound.protection.outlook.com [104.47.70.103]) by aserp3020.oracle.com with ESMTP id 37h14mdkgb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S4EwmHl84jEPCq3dd1n0Jswyr9GAeesF4ZZ3KEMIx1TtZrLsiuzeCbHBJ0anGr5WZE1PCbWH/AEV9yB69FQlj07U1eTjFGGH0XcJCpotdCI6Z/Zp79KMY9Kvvlu/jPex9MNoGM7HUySZUwgvuw4VlJOsJ3b/3/osB+51SDj0molqCCs5RjnbrSRm/ke6emS3/0FGJYmXthih/zUPXutkfklJmDmLDzWJ2hm3+iXOzUUxmg5qD9RsNC/W1+2Z3iOl4pMfatXCbDhQCyyUJRkvxZNG7FIQQq6uy5nbH8P3FAIh07rnejXiCu91XlDvBE37pw6F3B3s4exFdQgf12EiDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=210uT+K2DSFz/dK2BBxVr5ZahPOQRm22NW+bo9aavM0=; b=TL0n3jnRMxix5ASyH+WWoW5l7H5rUl1yE5bJIUYH7CJndpB/ztOEIap2zlL0tmQk6fi54Uv2nKXJ/7bXdCujcIlR8uDIhRS4fHTThdRlD4inRvVAkNAUGBVABIl1ghcNWB4ZKFO24QAj9ixttdx1X/GERSweVgcZKGGxptSIpfcuOY/vt/3MROz6ZcANAOCyfABqcMsgHBmhG6hj6npHogbKgpUZjiW0Jne1RjZfnqLAM6ZG+7gDd56e0BGFUgUgQSo2CRkCdSgHl/z9NPpyADTPDatup66yRRp9m/0/anqprRWxG68yvlw3WLqhlv7Slvjlc3JoYUnsT0LgQi0qvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=210uT+K2DSFz/dK2BBxVr5ZahPOQRm22NW+bo9aavM0=; b=uXpeRyXQ6DO3S2sBVSxBnc4/Yurn8PP/1Tgm7CCjz+tkihXQla3WScVk/QhtY9vccAbkxne8urZy7m7Zw0YJtAjtFNESb0fbKHDpqPIfwbn610cQIPMlG9rEbxhng124+etUUtSZTbsDZi+lGQ48EUpS+k98TCd2an1fYErz0qY= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BY5PR10MB4371.namprd10.prod.outlook.com (2603:10b6:a03:210::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 23:10:17 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:17 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 08/11] mm/sparse-vmemmap: use hugepages for PUD compound pagemaps Date: Thu, 25 Mar 2021 23:09:35 +0000 Message-Id: <20210325230938.30752-9-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b4f9bed1-1e3c-45aa-eae0-08d8efe326ed X-MS-TrafficTypeDiagnostic: BY5PR10MB4371: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4941; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: djcUSWEml1yOso3W5Th50VtJhm+hJyt1lbzaL6HiZKMhFn71uf6yiou6ArV/pWL37718YVhOR0PrmA9mAhFXhzuqBT+9xDnjitbEl4zXcSaNAxx9onkwBGvdwjO7skC8Znj/2ysCNlhuuqOjTXaovVeojh/gfyTlG7KK/WQygewmed8cylWISkryxISKMo9B57ogTwHtgPHiheM/+gEejRLqx1eHr1Uw0WrJWHdxVxKVzn7LKI67uA2K9NQ2/xRGJntc0/qBS4TW1fBOaLs5Crz6MovDqeIgk6w4btxb6v+ncNGrAP9ntpqxqFWUk5HrHtL/sn9xpVQe4oln4HJZPHUo3TQ8ptANEDUsggvEzpOkeF6+ac8D9vpt+Bw/PjjfhUc3vkYgVuS6xH6Ho0B/vsZqIaPcfEKddTCvB2XV/n5B9i2SG1o7r4MFkn0NJi+8JakFAn/b9RRzDRxpzRA29tgRf7FIOQcCddj9qUvNeBhRHnAnjDmBBjzb5y8aFakumyqGhPJcmPULaf5yIk14fKfJaAgb+xyBlQmvNGEaCzx4SzoR0ftDwiJDOtbKMW0rUk1MWnywWzk+nN3z+pbV0lu03k+xa50bRSlRN9Jrml56R2xq3lDeZY7KjIri05SJMFQDRHRZrwHtUftprtQySw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(376002)(346002)(366004)(396003)(136003)(186003)(26005)(16526019)(7696005)(316002)(38100700001)(54906003)(103116003)(478600001)(6486002)(2906002)(52116002)(83380400001)(6916009)(66946007)(86362001)(66556008)(8936002)(4326008)(6666004)(5660300002)(36756003)(2616005)(1076003)(956004)(66476007)(107886003)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: FQPG9J9y7QJJWqXCZxiRR5Ll5DO9Tlwqktt32JHEjVMSCQH3AqP/8U4WB64IarQAwOvKSA5qWIj5FA4ZVmftmfYAPz+UhAKLrqcWymVcj940f1ZYL2aPfUGkbhALXqy1OOwFj2Wg7h4ZnxpVKiWC1VVSZV31AMeq7O8EWD8U3Pt4Ju9houHKSnltLPG1Ig5lYh+y+Pma54Jt8K39sHi6MSkMRA/4ZgX+V0D92RpDShNUSvzVT8ooSbj55nrUncmBFoesZGVTo7z/cXRqN72ww0IyEfeww3Iw41XRYj+NBTNoc3MIuFWoMtI2kwpHD6ROGXgDlz669lZOWwUjgLrGriPYHCenYEdi6BKFpGejfeSvK9uoXYDvX3eMrIfZZjsRMl9G1cRYlECMkUtWcQiuGxePkI6DnzCF5VHJQ4gfwBnSyxvoVRAJIAbKZQcBWYakSKDs4w4jowJa97vCOiZNU/wKTqGAtLdZFVe5u+CpSj7iScgzT+oR0pSGNk1CcON2i6WL2B8/0lfl4vqJIbCaW0uC9FCwMDYpYbub1+8KiOPblgpUIwzzmYQV5v6M28D6dhu0gtWd/cMD+Hxdukz0hShM1tT9sM1bkXKr9sUKTlgRMnj63EJXWgGsfgRfRX3nWuKQ+PIS8FGye8PiyQSdLDdo7PS6rmR5iWtdyZe9lrx1FsOBgW1d3kAEB6cFuhoilca/X92boMQ/WxyAq/+NMeAnscnTRZePgYz9gOl21SNsinN4VYR/fYuF1Tr8Lz4zCHrsNqb/unL5JpGu8ySp1sApc0G0Cdb4P9ONGAmx1gOD1Q2t154LR4gfile/d3HdAaLcOi8e0FPjC3T1xGk04CNtk53Rk2tU1tZR9XGIBu70yykJfLcnOmkLAPIDkkC7cO7EJqbI4URuF0Vd3R6YqItrr1EVFbpM9jfCbI0T61XyhJMxcIeXvYH5bUITXC5MI+PqhBz8FuAx/KB3acvAiAuAssXDhIK0db9JyvL1zvofZcr0SC++LFWXsSMtA5K5iC5R0g/9GCS9xbDomiX41aE+MxQIJRTmJMrKiAwD5sT+9w8GosVZWYv2uALPojzIUt7+70ZAX0wci3pmdMLFe2RxaOLyonQmgq1WzXAzDr/rw3X9bhYx48sX9jmujSFOY+JXgDXhagY94JYzPOz9u6Fexh0d2zcCgGLDwVgRQHy8b0hkgmIZ1K5PDKgBSqHL41ySd3rbeJBhpK8mqcoE2VHZuBPfmh4wB2gG1dgnxWnOs3pD2Z+enIsAg2LJu5ahrJcTnGbmtfKBvRkkxJH/4OkvSdEtKdUPCUV/cESYPA9HnPKrpta8APu3+/A8sQqc X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b4f9bed1-1e3c-45aa-eae0-08d8efe326ed X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:17.1081 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UiulKvIbBb+B7+f+HtLl6XtCm8azBpeb3Uo3lxOk8uzKfF83sULDvvsmP0pPXtPzACGJ0wTrMq7LJ/WmWklizvfX4Is9oAVeOsuIRUhHUbg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4371 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 spamscore=0 adultscore=0 malwarescore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-GUID: HRpf4phBP-1AKrMwMP5g_gznjuJfHVdy X-Proofpoint-ORIG-GUID: HRpf4phBP-1AKrMwMP5g_gznjuJfHVdy X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 mlxscore=0 spamscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: faidf7ocxmzsoytmxfachkabhar7z6hb X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 94B92C0007C1 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616713828-958304 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Right now basepages are used to populate the PUD tail pages, and it picks the address of the previous page of the subsection that preceeds the memmap we are initializing. This is done when a given memmap address isn't aligned to the pgmap @align (which is safe to do because @ranges are guaranteed to be aligned to @align). For pagemaps with an align which spans various sections, this means that PMD pages are unnecessarily allocated for reusing the same tail pages. Effectively, on x86 a PUD can span 8 sections (depending on config), and a page is being allocated a page for the PMD to reuse the tail vmemmap across the rest of the PTEs. In short effecitvely the PMD cover the tail vmemmap areas all contain the same PFN. So instead of doing this way, populate a new PMD on the second section of the compound page (tail vmemmap PMD), and then the following sections utilize the preceding PMD we previously populated which only contain tail pages). After this scheme for an 1GB pagemap aligned area, the first PMD (section) would contain head page and 32767 tail pages, where the second PMD contains the full 32768 tail pages. The latter page gets its PMD reused across future section mapping of the same pagemap. Besides fewer pagetable entries allocated, keeping parity with hugepages in the directmap (as done by vmemmap_populate_hugepages()), this further increases savings per compound page. For each PUD-aligned pagemap we go from 40960 bytes down to 16384 bytes. Rather than requiring 8 PMD page allocations we only need 2 (plus two base pages allocated for head and tail areas for the first PMD). 2M pages still require using base pages, though. Signed-off-by: Joao Martins --- include/linux/mm.h | 3 +- mm/sparse-vmemmap.c | 79 ++++++++++++++++++++++++++++++++++----------- 2 files changed, 63 insertions(+), 19 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 49d717ae40ae..9c1a676d6b95 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3038,7 +3038,8 @@ struct page * __populate_section_memmap(unsigned long pfn, pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); -pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); +pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + void *block); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, void *block); void *vmemmap_alloc_block(unsigned long size, int node); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index f57c5eada099..291a8a32480a 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -172,13 +172,20 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) return p; } -pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node) +pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + void *block) { pmd_t *pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; + void *p = block; + + if (!block) { + p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + } else { + get_page(virt_to_page(block)); + } pmd_populate_kernel(&init_mm, pmd, p); } return pmd; @@ -220,15 +227,14 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - void *page, void **ptr) +static int __meminit vmemmap_populate_pmd_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + void *page, pmd_t **ptr) { pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; - pte_t *pte; pgd = vmemmap_pgd_populate(addr, node); if (!pgd) @@ -239,9 +245,24 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pud = vmemmap_pud_populate(p4d, addr, node); if (!pud) return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); + pmd = vmemmap_pmd_populate(pud, addr, node, page); if (!pmd) return -ENOMEM; + if (ptr) + *ptr = pmd; + return 0; +} + +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + void *page, void **ptr) +{ + pmd_t *pmd; + pte_t *pte; + + if (vmemmap_populate_pmd_address(addr, node, altmap, NULL, &pmd)) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap, page); if (!pte) return -ENOMEM; @@ -285,13 +306,26 @@ static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, return vmemmap_populate_address(addr, node, NULL, NULL, ptr); } -static pte_t * __meminit vmemmap_lookup_address(unsigned long addr) +static int __meminit vmemmap_populate_pmd_range(unsigned long start, + unsigned long end, + int node, void *page) +{ + unsigned long addr = start; + + for (; addr < end; addr += PMD_SIZE) { + if (vmemmap_populate_pmd_address(addr, node, NULL, page, NULL)) + return -ENOMEM; + } + + return 0; +} + +static pmd_t * __meminit vmemmap_lookup_address(unsigned long addr) { pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; - pte_t *pte; pgd = pgd_offset_k(addr); if (pgd_none(*pgd)) @@ -309,11 +343,7 @@ static pte_t * __meminit vmemmap_lookup_address(unsigned long addr) if (pmd_none(*pmd)) return NULL; - pte = pte_offset_kernel(pmd, addr); - if (pte_none(*pte)) - return NULL; - - return pte; + return pmd; } static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, @@ -335,9 +365,22 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, offset = PFN_PHYS(start_pfn) - pgmap->ranges[pgmap->nr_range].start; if (!IS_ALIGNED(offset, pgmap_align(pgmap)) && pgmap_align(pgmap) > SUBSECTION_SIZE) { - pte_t *ptep = vmemmap_lookup_address(start - PAGE_SIZE); + pmd_t *pmdp; + pte_t *ptep; + + addr = start - PAGE_SIZE; + pmdp = vmemmap_lookup_address(addr); + if (!pmdp) + return -ENOMEM; + + /* Reuse the tail pages vmemmap pmd page */ + if (offset % pgmap->align > PFN_PHYS(PAGES_PER_SECTION)) + return vmemmap_populate_pmd_range(start, end, node, + page_to_virt(pmd_page(*pmdp))); - if (!ptep) + /* Populate the tail pages vmemmap pmd page */ + ptep = pte_offset_kernel(pmdp, addr); + if (pte_none(*ptep)) return -ENOMEM; return vmemmap_populate_range(start, end, node, From patchwork Thu Mar 25 23:09:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB1F7C433C1 for ; Fri, 26 Mar 2021 02:32:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 679C461A25 for ; Fri, 26 Mar 2021 02:32:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 679C461A25 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EE71A6B0036; Thu, 25 Mar 2021 22:32:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBDB76B006E; Thu, 25 Mar 2021 22:32:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC19E6B0070; Thu, 25 Mar 2021 22:32:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id AE2306B0036 for ; Thu, 25 Mar 2021 22:32:28 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 728C28249980 for ; Fri, 26 Mar 2021 02:32:28 +0000 (UTC) X-FDA: 77960451576.34.193E0CD Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf18.hostedemail.com (Postfix) with ESMTP id 609092000243 for ; Fri, 26 Mar 2021 02:32:27 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNA6go136814; Thu, 25 Mar 2021 23:10:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=lPo7Ti03CLdH4oj2DM8QS4tfFwfDCyyGc1h07TMfb2U=; b=SCJu1NnZxzGgLQ7S7QXfreRj7QqgRg2lsJnIyOXFE/AUTpKi5EdiyGtOGs7ZFLgsbigw 0PC/bH2LCQPzPkRc4epDpPN+QDTPj/i3cTjIb/dltxRmwdte0s6KSVmToBdFRRjt1CRX HA4dDXNwWxK5A9+OClMW52QQyx0X+lBGG0IXwmWcq9NEzmtTXjaqgoPfndFRCB/IZhtz QQOg6IFMnHjZCJTjlLnWR0eEZHOcGsjYRB54qFYZnxhvqtSGXKBuUyEaCLCepK0O3Mcz v3iIgaOebqxsMOy/qlYsg2f0QJV338vIlTfUyGP5URY2ymr/UED42eUsiCf3RsCZ55J9 nw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 37h13e8dxm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:24 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN1Ko6002948; Thu, 25 Mar 2021 23:10:24 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by aserp3030.oracle.com with ESMTP id 37h140nhh8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FlNOMrkhrGa8uNUt9EJC6kYU9JVSQi8xV9ga4v7MxEjy4qTAmsuUg+tBHx5wMizNXLd43tP9l2OjGl7R4qsI8lsObt4hnPzH03KYpprvKD1tbpmwPXMtFk++685K6xio/RCqTMqCewhSSc4KQuTmnp07WI9Vc9AFhHgt+w9IHR6noj4H2ioG1X7/LIePA/npZMY3X8MwdQEVHZsKVmgvhP3Ig9BYMRpCzGeQXnBPrmupwvBczZ2dbijqXgCIdFWKNHvxolmn8P1c+X1t5XgXqO4KkWA7mbExmrggBCnNGzSOOzs02hIvgk8iS+OxdSaEe720dx/Ak3llTAAzlz7KGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lPo7Ti03CLdH4oj2DM8QS4tfFwfDCyyGc1h07TMfb2U=; b=oOB69t4RKHex0+j9nt9T+TZlayMIb7cN/AJXiHA/YuVTNo2j+Dh4LNQELGO3cnkcB2Smw4lNYMTzZ7VASEQ2t1uWJmbd47j5KMvbmvgS6xNLWfIPFlhUMmdYueS+NSFaHlP6MuF0zY/aBspyKclWM1ALm1Yz98bbxxpntzd29TClIW0VGAQzPL5M+rJZ2C0CBHgcP1I3hsBajWb8QqgE4p8OEVLhY5BPO7LMh8owyKtTOlMGNUklVvhKQopc+ELzT0cIWONUdZ7SKesEW6rRtQTGvB+171rTmL80ad90QeUMe4Evuy544hRJmRGcP8MopfkaTDIO1uyzcYtRfEbHRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lPo7Ti03CLdH4oj2DM8QS4tfFwfDCyyGc1h07TMfb2U=; b=bYxTxW3KR0QJo2ptTtesjdAVIsZQb3SuLXpWlx9aK3F2yk+8wvvjkToU1tz0pgc+Wi1YmabefNTF9i7d0VIrSw2boZ8e4cAUIGPiSOk0kcJmy5hi4mU/prlHGZfQ0NzgD3v3svXQtHVoXs/nJGhtdzsFBUpzn1fHE438kALAgHc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB2853.namprd10.prod.outlook.com (2603:10b6:a03:92::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 23:10:20 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:20 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 09/11] mm/page_alloc: reuse tail struct pages for compound pagemaps Date: Thu, 25 Mar 2021 23:09:36 +0000 Message-Id: <20210325230938.30752-10-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:17 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ad2cc3f9-8144-4bc0-5b37-08d8efe32952 X-MS-TrafficTypeDiagnostic: BYAPR10MB2853: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: C+le9RwC/ssugtmFYnZxMmv9+VJj3BKbPX2Oxefp2BU8ZZbjgd9m0lxPaaNzw3rkUKyzp0mnYYPI6uKUXBTK2RkZgs9QDdZUsKTZUkSr5A7MNJmI7sTf61ZT8h2Osq9CWyW3SbE0pHiwMq7Nxr0yM3lsWKYS2lt+d6kt01VSeJqgojENHdELCyZBlmCwp6/gWjkrNesKJI8U71ofc0N+HArGp44Po2huE/GQjVK94MTJ4IkMuKoD4bys9SDCAd1c/WxOYdy01tK/ZYvtnbYK+5koiH8daIKuTW6pUm4MyhyltAnwIs+TAl7Uejm/eebO8S3itcHbSHOb7gr/4tUfPpSOQfNka7PlX4HpEu5dPY8kiPVJDaKtwPT3D7YJAdV+vXLb+Pe89lVuamN+0sv/iwudFo6iBG7yljvMxWz31n45iSLy0oVRhUXVVl19K04JcpqKhKEKzYeRN/vHJI7fydCbU4PK5j5g2FlO3Jcbgw50G57fRQfkfjjVvaCY+FyWZo2o0lDiI3doQot2VaovTaNkRM9fAIWRcWjuN241tZwP1aPLKct+WkAHyj95BaBEcaQSmepuPlhthAVWHTWebj+uC6P6Kf+tP2oxbq6jh3NTWHwegfQetRPLg8BSo3o7UxWtB9WYcgi2pZMmwQZZIG+Yj99sPE5jnVC3LNzO9R0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(38100700001)(4326008)(8676002)(1076003)(2616005)(8936002)(956004)(6666004)(6916009)(107886003)(66476007)(478600001)(103116003)(86362001)(186003)(83380400001)(7696005)(5660300002)(52116002)(316002)(66556008)(6486002)(36756003)(16526019)(2906002)(26005)(54906003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 15hNaVsw1F0i21kGlUWylfbw3kiFhsYcxUlbqo6x1MxYxMidB1BkXXVbHWZ3mmNQTV1QMKuxmA1GpxZcqTF46kJ3MUuaeQhI6SWER/DxTLSD+MJbJcJtlaTy2Rb8YlwtIAGgVb3zi5Ev4uvP9ptD1+U1eXiouQkuu+IYv+QIqNntH4on3orO/uf+Q0bTXHdjnDy0tM8IaX7iLeFe7EsfFDyHceaeugUBw/SJIIcedppD7bkwhNMVNTHhybsvVM6pW9sr2be6syngH7Rv9ZunNDbWrLGOjPmzKJIl7TONRfGXag6qeYibkEEiGSrrq3KFyWv5AAE0hJ5TRnr9R4VFEVC5Q/U2YXkfjjOyu1uPGpGVZCDvjrOsWfNxUNY+XA+28fdzW0a+lCfDYVInxkmllwoJaIamAhF6w5obHs2NyLVEH76gPRO/J6IAcFDvYubMvZqVd3YuIo62/kGYs0lQppyV2OgVyuKu95b3zDSJ5qMzRtlpUflYqCJP6bkYfA7zSZLLNCOtasNH/qLD0Lp8jpIW4qyOiNJMjdxiW11WIkatkhUwV9ooQHrNjH9WgAVEaT8dgvWOikuIxp8XjPg7mXBL1Cm2UQV6hNrFB2Msc9f8GwRFUKwTKS0meJ19gP+5NuKa6ExbssZoA5N+zb+3DC90eKwzWzIz0F7sFDPIyd0OshZ80h4LUDhrK7lWENnSva6LeLZik+WdNu0FvKTWdsxEokrU6K7MqrJd4F1fJF+ZjYpAFov9fIIwhleF3ETJvmUpOmDWzSbSrtwGi6tJFaxuw3384O52XvpbehAV7rzsHo3JPoI9OruPuqS23E7il22dmn133ft2/DcKR2JDGOfvao4TQ65//nIIknkj0MHBFb72G7YHT+iDwXlH81nVN3j/RDXUKzC5HV83rsZKF8cywLcB1FaiS+hZATERvOGaQd7b0aNJXLPLZlRfyrZljkyiAKuNLpVQr/IDgp8xdYuRABF+DFxaKHNJICmyJNfUioTFC98blfphdYxpJNt5L5lYFzvGbYEDvmXySvMDjnHhNmOdfWHX2+ybwlybXMkcAPtFU26lg/f9RlQxGcVv1O9gXypYpL0B07iE+bbHsb3DkmloFtei5BAqFb/btqC2mYfWYG639Gd4humfSIPtGOw04DUbhfpTp3YghZITCX2+q9FzyX101HceOHpk/mNf9PGFPsLWNWSNCeyXWsXEnpHLEGnhPtVo6hayeoW2DLbDcee3HQdI3NFLueGB/txpX8nGKqXd/ZFhfB5WaGVWlpBkv+5RacnFYDUjdXIDQeda8vmv4xwaCIAVeXYu+Elabcxc8kJdRB8zf6Cy74fC X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad2cc3f9-8144-4bc0-5b37-08d8efe32952 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:20.4681 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5bZPo4M1iS9RArrgZTQ4scNML5a0J6knRCS+/uOZ9gNge7AH/s+p8QYPJjr5HY5qCSByHfy43kQzC7adwUxIVpJugzjYql483LLDw1vo7WA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2853 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-ORIG-GUID: 9SFP2AaPkjf1RiUFbRtNNwvkEne0BrnQ X-Proofpoint-GUID: 9SFP2AaPkjf1RiUFbRtNNwvkEne0BrnQ X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: e8gei9ju7j97jnriktn7xyumgjq475g9 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 609092000243 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616725947-114629 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a pgmap @align is set, all pages are mapped at a given huge page alignment and thus uses compound pages to describe them as opposed to a struct per 4K. With @align > PAGE_SIZE and when struct pages are stored in ram (!altmap) most tail pages are reused. Consequently, the amount of unique struct pages is a lot smaller that the total amount of struct pages being mapped. When struct pages are initialize in memmap_init_zone_device, make sure that only unique struct pages are initialized i.e. the first 2 4K pages per @align which means 128 struct pages, instead of 32768 for 2M @align or 262144 for a 1G @align. Keep old behaviour with altmap given that it doesn't support reusal of tail vmemmap areas. Signed-off-by: Joao Martins --- mm/page_alloc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3a77f9e43f3a..948dfad6754b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6277,6 +6277,8 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, } } +#define MEMMAP_NR_PAGES (2 * (PAGE_SIZE/sizeof(struct page))) + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6287,6 +6289,7 @@ void __ref memmap_init_zone_device(struct zone *zone, struct vmem_altmap *altmap = pgmap_altmap(pgmap); unsigned int pfn_align = pgmap_pfn_align(pgmap); unsigned int order_align = order_base_2(pfn_align); + unsigned long ntails = min_t(unsigned long, pfn_align, MEMMAP_NR_PAGES); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6302,6 +6305,7 @@ void __ref memmap_init_zone_device(struct zone *zone, if (altmap) { start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); nr_pages = end_pfn - start_pfn; + ntails = pfn_align; } for (pfn = start_pfn; pfn < end_pfn; pfn += pfn_align) { @@ -6315,7 +6319,7 @@ void __ref memmap_init_zone_device(struct zone *zone, __SetPageHead(page); - for (i = 1; i < pfn_align; i++) { + for (i = 1; i < ntails; i++) { __init_zone_device_page(page + i, pfn + i, zone_idx, nid, pgmap); prep_compound_tail(page, i); From patchwork Thu Mar 25 23:09:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C561FC433DB for ; Fri, 26 Mar 2021 02:04:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5CA6661A3E for ; Fri, 26 Mar 2021 02:04:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5CA6661A3E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DCD4F6B0036; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D7C836B006E; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5C056B0070; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id 9522C6B0036 for ; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 521CFA8CA for ; Fri, 26 Mar 2021 02:04:58 +0000 (UTC) X-FDA: 77960382276.23.1DF1542 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf05.hostedemail.com (Postfix) with ESMTP id 142D1E0011C5 for ; Fri, 26 Mar 2021 02:04:56 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAHj5136830; Thu, 25 Mar 2021 23:10:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=IYSeFT3PKOSLK03Sr/oxNuTZOq3HJR8yrGxzOY6hTwI=; b=GEN96YFXtN/4IlQoktWOMyZoFCCRu1E2UVbMfcajqTgoDDKKTfPFo9nEpb/qMUDaXyrI 4F+USNKKaLsRESYdOyl8zFAFNKIwl7e7zyzTW/rK07kjiCsUqbgWXW/jvNbLnK50N18O EIccLjRRwHdF3h7PyNSvru7zcQpxNxmEWr8IRq9/eVtbzbgJCG+vmKdJ8NhO3NmEPMle Lt0ZH8VgOsj609pHjOF5C0+vz4w1UIN1TjGxvyiYSLkBIZRRf6l1WFzoNhr7HykbFFVD FsffFDBjvxVBxbOjxEIhE+3En88bxWJHh2qr27akljYcyIUkxkI2KlPCzR4KcrBxkxiA wg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 37h13e8dxn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:25 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN1Ko7002948; Thu, 25 Mar 2021 23:10:24 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by aserp3030.oracle.com with ESMTP id 37h140nhh8-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=etm+xDll18SMSsVBEfeinS5EOTQ50wTmCvmaxnIe2sVKUHgGjP6NhioZYetCnq0TTUbkRyxSRRMOo3MeLwnHcObKwkpZW7hH5EVgPgJ9Ubiqgfq6MGRkQD/TVO+Nfb/4HYAu7dGxRTdfJ8UIb1uEiOYYi0+J0xMjr2haUe3RXKPYNUeROfLbWdIOuSUvw06A6GFL5JYPK1pbblD1elhZZjp4bkB/fr3rr7rQxeJ2JU7Adis/cDmiltDhraweziZ2ayH5+XlfMOJ2r/iYo8SuTKOQDovfVmsL4mMuWuTVHrjNfFh1Hi0o9MGbhEE8J9Bh+u2pR1ukRrypCSlVHG055Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IYSeFT3PKOSLK03Sr/oxNuTZOq3HJR8yrGxzOY6hTwI=; b=T85ZBa0Gz3qbf752pOT4FOu0vCkMUOvtqe/EcKYfCuj96JceLxvAhDcg+wUAeTZoTqI0AYe3GrUrT0FNvDxjD0y69bzHtC2IO+2QAsIozfOWaV75ZnRZKR2N6/trdGgSzp6fsANFx3c/ynwWwfNKgTX2AEedcxR6O5eJ1hHkfsfeZds0qm/QFcSVly9CJtTXbGQm4K/lchkd6dJ2DjocVbSUpHwZWWA+lwsVuEWnPmJj8vAcNvwMoTF0yL4vkVqP3nRBF3p/0oLC6AkrZNh6423HtHkSKaB8BGrLFMuA8Rg7sKB9JYIi0bxaSuWuSmYe9DCqvZ0zBHekOxWnfEuv6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IYSeFT3PKOSLK03Sr/oxNuTZOq3HJR8yrGxzOY6hTwI=; b=YcwpZyuV2XyzZD+zkOO0EkfWtcSfTodB6hlDcCWRfAykzGswLlWpBLByihKP+nIqk6Iae2eufYjmdgBbGifqJjL2rIBI958XsWo35iWorU49D1KUXShkSCT6Yj271pyGvT6nLsym9E8owW2pwsmToLhOI2G7N7mFJakYQ0xHIxc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB2853.namprd10.prod.outlook.com (2603:10b6:a03:92::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 23:10:23 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:23 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 10/11] device-dax: compound pagemap support Date: Thu, 25 Mar 2021 23:09:37 +0000 Message-Id: <20210325230938.30752-11-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:20 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b525738f-d968-4734-015d-08d8efe32b25 X-MS-TrafficTypeDiagnostic: BYAPR10MB2853: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2733; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ybcPQ6fIy+mt9RcJtx0XBBAcpAvkrl4AbMoC/yLkVIKUg7+E4bMlBmf/Yc3+aKTgvMPYEuS9xtZnlY8OIiwmXY1EJtLuiAi2Kr3ld/Qq8R2K0gwPfYVz5Rjj1lphB5uqmTowrqFNrikzqiQLdh+e3N4vBaWSldy7SSQ3skdv4xK5NFMvXQBCdkvFWWqx3Tg62E7sipbErbsY+a/pu9U44BvGufQdEW/DjN5U9VcZYPAZY9BFD3Xqo7sTYskBN+MISzozLl42C8titX2I9FGvkJbgX/t6kMoYf+oPjPH38qBkMWnbhOb0eFutkFzcUwMmLUhSdjffU14zAKpsmwogbjxXdYBKu7d8RFfRHEI9CD31uytD8I/+2sRXbnnjNjRQusGIj7egWxv3scIKcWVK3GnZLioXMjhskTvAA4OPaGg/NPNATzWffnqQdQsfdfGNOEoF/A4V9wcawijRJP5OlJNRbUfANk+5M/FWU6C3AS3u72SkjQ7TYi1zHE9vVXOXx67ALHYts/PkT1b5DbEUepVKpm5LgRMuaOHowhDilcA84jv18YPXoIY7X/idVdtL6nFsQTUKAs0jUvgFHNgxHrExq9V+0F98dCl71BSctn4JNTLfuma0w+QgEOKzwbvPDWk99ssSYexkzTb8JbWX4Pv5DvxGeu8X2daIhm7A15w= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(38100700001)(4326008)(8676002)(1076003)(2616005)(8936002)(956004)(6666004)(6916009)(107886003)(66476007)(478600001)(103116003)(86362001)(186003)(83380400001)(7696005)(5660300002)(52116002)(316002)(66556008)(6486002)(36756003)(16526019)(2906002)(26005)(54906003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: OtgS/otz953nXBf8nYoWlr2cJWomyvpNjI2tsPyduyDTawTaducu1Yar0d1ROLCSJUC0SRkymJY6KHcZ2KvIbQVSebAB6TzcSdAqEwvliz0PIU/Cdm6m7dT8nnnms2GE5lEHn5OpaTCclx7e8kMnkrcM6kNcPMjdnyTee96WBQ6zreMIzo5WWPpPn+tSuoC9yx7TB1Utu+7iGfMfvmrifHbCVOcPcMEP3EGCLpA/el3JRlNXgJfdmQxjGX3Xm39KrXW2t/NBMcs9adCLMESDhuSFKz1NgPz01nMJF8GWI00L4PExa2jPVJ0/De8sGdQ2Kc6MtvtZz8cYeE4LNO+DCHXXvN6M837pAWrn5pfu6C52yL0WvBqArzBypCmwt3Ba1pXJaqznLA8C4Fbc6sTkoTsqYZi+kIRqoHtCqAfNzKyd+BZiFTFwRsGJhaayM4Gy3FPJfK1FLFM/aNppgf1YS6yH/YO/LvBfnfQfgLr38M6tFAB/3pwkTvEk2ks9MkceXh0S8AQCVj/wBDbn0Z6quAMUPq24tzI6PpLWIb/fZ9a3XMPttEO39LFsvIc6xH1HWBHar/p65wKmOeHtexnXprkh1GI/A/bhfbOQoEYkUaAsqzUX5uBR5kIVW2BToJJFNzyxghHm8u+vfEa4E5hc9EckMlSTzW+ogMstfb6x2Dc+OCI4bqgSdQz8u/FehkN5Iqy/2FC1p55WFEFky9i6qqSYJuDhm+JYH/QocvFzkNpLJvoXq6kOozSS1a7+cxle192ET3xRh5dlyWReTmXA/O90gN5jG6fJ2clYfxRlDIVPdghwXd8nOdgSL7Ql3moKdLJ/J7GPqizcbasxAIFwWJd5DLUd6TgPhMhDYVU1tr7G1X/Lh8ex37wmU/v1Yf+MIyWakMsDaefQ3Qq4+IG0nei+AE6pmSxlBODtgLZyJllCS327Cef6jYsoniPWVyyO3GbuAFjifaLbjk6cnVcvfEAR4Tv0Xj+R72Dlf6MhodzdgC5zy636e2aoP3B2Cmss+68R/CH7QLmFfwsJQXbzHmPv5VAqMMD5n3bK0L9vZ0XkA2PGaDFODXjJHHqhq+F0EniI1lkLarW+sneeUy6Qo74V1rNi5SDPlC/7TnkpRY4FCELOwfKibXMvgpVUfJk7fnOIopdXZ+wKudOBTgsU5EUtp6EJDrJe5UcOotIn1aUII5lk1nnyxxhB5HLikjFhI5kPO5YMrIUInexatpNbsGqHypQuQP2PxvnLlckflGn9gdD0A8SgCvrllbe36HTiVLpfJ8CrwZhRRl9pHdWpBZvYHBKpKa8roHxfjEqgZPsc7M34eUmNmPNrNlNCyogM X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b525738f-d968-4734-015d-08d8efe32b25 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:23.5753 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /t2V/bTRWG92xMai7Tdg4ewVrfHPUrAL9vBp2nLXV6BWP9YiruPBxPuWfQLdCcSYXpLPHWqIzrnQhTcKX1NpJjHNowQSuPq52QrNEh873ZY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2853 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-ORIG-GUID: tuYdsBbc0DzjLg-8akuONTWhRb81AwUa X-Proofpoint-GUID: tuYdsBbc0DzjLg-8akuONTWhRb81AwUa X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: z8jbyjrsuqhx5muzwzughjuour9foxw4 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 142D1E0011C5 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616724296-145757 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dax devices are created with a fixed @align (huge page size) which is enforced through as well at mmap() of the device. Faults, consequently happen too at the specified @align specified at the creation, and those don't change through out dax device lifetime. MCEs poisons a whole dax huge page, as well as splits occurring at at the configured page size. Use the newly added compound pagemap facility which maps the assigned dax ranges as compound pages at a page size of @align. Currently, this means, that region/namespace bootstrap would take considerably less, given that you would initialize considerably less pages. On setups with 128G NVDIMMs the initialization with DRAM stored struct pages improves from ~268-358 ms to ~78-100 ms with 2M pages, and to less than a 1msec with 1G pages. Signed-off-by: Joao Martins --- drivers/dax/device.c | 58 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 45 insertions(+), 13 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index db92573c94e8..e3dcc4ad1727 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -192,6 +192,43 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax, } #endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void set_page_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + unsigned long i; + pgoff_t pgoff; + + pgoff = linear_page_index(vmf->vma, vmf->address + & ~(fault_size - 1)); + + for (i = 0; i < fault_size / PAGE_SIZE; i++) { + struct page *page; + + page = pfn_to_page(pfn_t_to_pfn(pfn) + i); + if (page->mapping) + continue; + page->mapping = f_mapping; + page->index = pgoff + i; + } +} + +static void set_compound_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + struct page *head; + + head = pfn_to_page(pfn_t_to_pfn(pfn)); + head = compound_head(head); + if (head->mapping) + return; + + head->mapping = f_mapping; + head->index = linear_page_index(vmf->vma, vmf->address + & ~(fault_size - 1)); +} + static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, enum page_entry_size pe_size) { @@ -225,8 +262,7 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, } if (rc == VM_FAULT_NOPAGE) { - unsigned long i; - pgoff_t pgoff; + struct dev_pagemap *pgmap = pfn_t_to_page(pfn)->pgmap; /* * In the device-dax case the only possibility for a @@ -234,17 +270,10 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, vmf->address - & ~(fault_size - 1)); - for (i = 0; i < fault_size / PAGE_SIZE; i++) { - struct page *page; - - page = pfn_to_page(pfn_t_to_pfn(pfn) + i); - if (page->mapping) - continue; - page->mapping = filp->f_mapping; - page->index = pgoff + i; - } + if (pgmap->align > PAGE_SIZE) + set_compound_mapping(vmf, pfn, fault_size, filp->f_mapping); + else + set_page_mapping(vmf, pfn, fault_size, filp->f_mapping); } dax_read_unlock(id); @@ -426,6 +455,9 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + if (dev_dax->align > PAGE_SIZE) + pgmap->align = dev_dax->align; + addr = devm_memremap_pages(dev, pgmap); if (IS_ERR(addr)) return PTR_ERR(addr); From patchwork Thu Mar 25 23:09:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06E91C433DB for ; Thu, 25 Mar 2021 23:10:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8969261A42 for ; Thu, 25 Mar 2021 23:10:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8969261A42 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 459E36B0075; Thu, 25 Mar 2021 19:10:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 431C06B0078; Thu, 25 Mar 2021 19:10:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2398B6B007B; Thu, 25 Mar 2021 19:10:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 023826B0075 for ; Thu, 25 Mar 2021 19:10:41 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B81ED824999B for ; Thu, 25 Mar 2021 23:10:41 +0000 (UTC) X-FDA: 77959943082.09.0CD6979 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf17.hostedemail.com (Postfix) with ESMTP id B5B7540002C4 for ; Thu, 25 Mar 2021 23:10:39 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAaFO136887; Thu, 25 Mar 2021 23:10:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=qk7F7RrEv2ndA9z1VMQVsJEgAKfhVBs0GSQxMoEVwTQ=; b=fgWr2A6mEqCMD0HxhwGorbc3riuHs+oShrccwGj1BM43iDqiWW2W91zG/8NeVTiCckN7 GEvWV5M900qtip6FRVwPOoa46H6kdr9IvhfF8GDp93jtP9DUIGfMQouo2kDL8lWKhipH FtJkfG8whAPVyxKpCr3WxRdYyX/jKb1ihnKy1dTiD5fv17uR37HT7dybDeyYIyfxG+/Q 2ruOxw2T7UFB7GxlB4s8n5x0lAE8+rFuDXHzCqT4akOSM6QoJlTB6zO2cdhEspK6mBf2 I4ngGDWHiLG0IYikz8aq1KLS8BDU0eJ/jPFGif3SyeKhya/jjW/UdoaeEblW9WhxpByq TQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2120.oracle.com with ESMTP id 37h13e8dxt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:35 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAWNB148442; Thu, 25 Mar 2021 23:10:35 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2174.outbound.protection.outlook.com [104.47.56.174]) by aserp3020.oracle.com with ESMTP id 37h14mdknt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:34 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kmlTxv3jZzPh+QJugh9/tekA9DHu3m6c8VwenJRUfBnrpPMp3E+cA0frSEBkWLNTVg94qyXs7YKWPtb7i3Bb5VWAS0+pZj7++FyQXsaB3M10rm3CEzwxx+YM7xBrN49MpeUvnzHwV/o7+dqgkXTNH2ZGBWfeqoB2H3m+oJjTp/ba2tRG3IbarivxRyMZDA2FQjbIXMtdu84YlcVY3g6p3U8v80ULV4v5Ovy3LCSiAvw4zljkGy1nuh9fLijVH6W/BEDNej+Z+P+DxrVZFJYPSeTwjSOLpNxzL99c9Q17iF7v77Yf1++wxjIKPhKIJXdB6ombq876pFT2ndYSmlPdKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qk7F7RrEv2ndA9z1VMQVsJEgAKfhVBs0GSQxMoEVwTQ=; b=bnEj++WX93Q5A17vyNlqkOBhtoGiPlwd//3r6J7CjFbRXMf2uaLtjRFwYL6wxLarjo7gSyQMXKpK2+dCGM43C+hb4+YEtk1+ZlRefoLp3s+7WlmxvIHeh0XMia8cKworwcQEsmTJQiMZMPyQ9O98VpZEVyboujNgA54ruLSeGU9otckVTMuW1YRgB6fRpVCWUlPWL8TjwSnXlTg1wTKmikqOir4aBTmpNLv03hFLRqlcBZthlNgxcTpCiFzSSogyZxXrb+AGdmCHnRbpgc0MQi6I0ihijTQMNTtBuICySdZJWXb1/mGT+4E3aPsaHebBb+TvRSV3u3BgTbrSFjMk0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qk7F7RrEv2ndA9z1VMQVsJEgAKfhVBs0GSQxMoEVwTQ=; b=fBzX7mbksXE+EHZWYy8ASp0plUSey9LMhBrtoXuzIYdnCoVCnhX/cCwG+NzhozAZEME4T2SwEe0QaYJgqgA+g+TnyZmPAlp6RqUwTAP+dVVQod7wkhSYNcl7J172DMsdH0DrV5cTX5w311OibD4Szo8be3+1voLYFxHvahiTFr8= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB2853.namprd10.prod.outlook.com (2603:10b6:a03:92::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 23:10:26 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:26 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 11/11] mm/gup: grab head page refcount once for group of subpages Date: Thu, 25 Mar 2021 23:09:38 +0000 Message-Id: <20210325230938.30752-12-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:23 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 221a6aea-71cf-4e5f-91c1-08d8efe32cf8 X-MS-TrafficTypeDiagnostic: BYAPR10MB2853: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1923; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EYuU23tNWZ80qzM2+DfwO42DD4Yhj9miD655jmaIj0jpCcOl9CQLom98xWiqIuje6CaYj8LvRwBbw/C1YguvrLCV48p7QvcXjaQYKEPHZAk/jtEIYu1HZqH3Hp1/0eTZJME2VHyfaJzrjT6w1hWcNUCjNb8SKCnOirSreOjuKuG7hY0Ip1gfUHwMJD60ldMeZAKtBjFgq0tkpBWBU/CGhxLkyG4+mWz0LmcIUwAi5i0vypXH1zDf+ZdWOpYHxYq+MMFD5SzXKg0PtIlLQOJU35fO2zUXvpgOj9043X+mhzeEOVOj9vFJCHf+prg5awawKCItvTxjiSiwP/2roy308ao3+XTj2KttntfTGRkipmRR2i4ba0eoXd0qX5hRqxtJ8tO2cZu0xFlX+ezGA5vps3ePWvg1L2lVTNSMPvcfCfL3tNVlzb9AI6LViJBYcYUnhBOi0OSNpy4SZniCABFd5tquR1xYDJ/mttXIkNOOqYwOVo5GoWSDaiVrCYdtl8jnAtmht3AlVQLp6ckbPdBybf9DQYTYCIHjLJBEhgQGurxl97Qe4NPXX1HS2lYQnLqfCszJvUe+4MOgp28n5ZMGAKhVfwjsX6pVBgWtXjotgpQG++7snJgKndGaYCtHRzZeZYJ+/arj0j+ta3h12BhhlL4307HLGEv8Kf33aKz5ybY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(38100700001)(4326008)(8676002)(1076003)(2616005)(8936002)(956004)(6666004)(6916009)(107886003)(66476007)(478600001)(103116003)(86362001)(186003)(83380400001)(7696005)(5660300002)(52116002)(316002)(66556008)(6486002)(36756003)(16526019)(2906002)(26005)(54906003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 2isDJkI4qsg/jnyQOjoGN9esyXmkSUQD29d9JfKiv4zkl6y1Ng/8OQ+bU2Lcqf9hw9HXXHxXgO2s7gbwwLvK0aKuZtRMkbMCLncStoWZynApwTBKhYW1z6nRFUQQjg99j0m2IJtH7DhX+uUYSbjckvubRH63n7Zcy56yKe4ccAsqI/c2d2vRnTJf8Xx/0VsNhajZ4UTRdyLw3qH49nYLG5YSharN5ytdetfruWO6qiCyQyYk+Ou+CLxtChdv9UpkBdlzeMfD5vJpyZjpBHy9TEzyWr3wooyIeEYF0GyMcuByz84/q6YuTox8ISNVLkobH+xBb19x0BYFTCkYq+120Nq//iphHlV1oGIzC5wTXi61Q38WqiQfDSFuOq4UUJX/AtsMtcQXa30a3/OzFdtAd+jWP7TCaCMF+8gu7jsM4cGKNJFc1sezVd4xsuBJFGOAd0UtypOkQRFF6+/RGWYRLIBSEVtYXYAhrdm1YuF8DDoiF2c1Y2LXsMGwsNtYlnQG+CeTvPwdY3HOjbcYuJBNyqX5Y/e7BARtJA1UJ9I9ByVq/3yp3pFj4h5oFrKY+UAadvP2W4rwA3MwKpMs2tM7QZWQmRjYckBuW0XiLiN/jL+9C/o2nHsl+1PxsRjBO4qrshdiTjfn9E3KT2VinGXES9Q/yjxLf7HF/0pnrYe3zWoQPr8U1Tgu84W3pD7UMBBJBoRdVu9o+LSCm00c4XAqC3ZpDNiXbJnaVDUY/CEf885ItdEwly2yyFWPeg60RfZqhiP58ny8Y0nAsHyS6xsYfwV1ZZI2QfMaS/jbmMuE9RcEFn7JvkS8AhJ40En0fYD02E29bOarqZVD4fIW43AC2UI2e8n9MTlvLKYN2oN5yyjFTiA/6dSDScsOh8Qub64TF5dbUsGM5WTHArZopeyQtiUtvQMRs6SxvmvQnO8PitupJys0PUQRlkgkl3UPDubydUVF4ClS3B68gJVTj1N3Nh/73FxoxfPQT5eUKB6eitOO1sMP5ivMEype6D6OSsnZixtU+xRUcahrAr/g1EjwJr6ADcjAz4CIvrGWXmjpgPDbB3ggeI5LBkOJjQH88uRdCKB1b7j5d1pd0JcFvqdDC6GLFJRKxZSdh1iEDtCntwOn3xgAPzqhEF0Z7YQ2PsEQK60hO88j1+iCg3oF+3PHIvMVA0lehHM3QcP6zuxeAmYl0Bdm97XodV6spwG6VM9SwjFjXmLxNHbKq0mak1uspO7SZS1BCSMMatSRk4AcQH0N36pu4Qs3YhZnM4uIJxvQwhQ0fVOq+oSogdOCyU7paNh1jNJdZC9tDJki65fCO5iEVsyMOBQi4qdKGyDD0GrA X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 221a6aea-71cf-4e5f-91c1-08d8efe32cf8 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:26.6436 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Gi6J4BBKjiKl8UzwyISKg8gZk+MUJhdniLCTo/IRxtvXcP6McP6s5tqQOa21ZuJiNqgFdJcycWlv10ii4ejH9I4g3t+ykZSWbXEpxEhLJdU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2853 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 spamscore=0 adultscore=0 malwarescore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Proofpoint-ORIG-GUID: dnxdYPMyScT_rAeFICldTrJex07kmz5i X-Proofpoint-GUID: dnxdYPMyScT_rAeFICldTrJex07kmz5i X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: prw74pposrbyzbxq8ijwxg4ywedmaubd X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B5B7540002C4 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616713839-188292 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Much like hugetlbfs or THPs, treat device pagemaps with compound pages like the rest of GUP handling of compound pages. Rather than incrementing the refcount every 4K, we record all sub pages and increment by @refs amount *once*. Performance measured by gup_benchmark improves considerably get_user_pages_fast() and pin_user_pages_fast() with NVDIMMs: $ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~59 ms -> ~6.1 ms (pin_user_pages_fast 2M pages) ~87 ms -> ~6.2 ms [altmap] (get_user_pages_fast 2M pages) ~494 ms -> ~9 ms (pin_user_pages_fast 2M pages) ~494 ms -> ~10 ms $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~492 ms -> ~49 ms (pin_user_pages_fast 2M pages) ~493 ms -> ~50 ms [altmap with -m 127004] (get_user_pages_fast 2M pages) ~3.91 sec -> ~70 ms (pin_user_pages_fast 2M pages) ~3.97 sec -> ~74 ms Signed-off-by: Joao Martins --- mm/gup.c | 52 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 32 insertions(+), 20 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index b3e647c8b7ee..514f12157a0f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2159,31 +2159,54 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, } #endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ + +static int record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages) +{ + int nr; + + for (nr = 0; addr != end; addr += PAGE_SIZE) + pages[nr++] = page++; + + return nr; +} + #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE) static int __gup_device_huge(unsigned long pfn, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { - int nr_start = *nr; + int refs, nr_start = *nr; struct dev_pagemap *pgmap = NULL; do { - struct page *page = pfn_to_page(pfn); + struct page *head, *page = pfn_to_page(pfn); + unsigned long next; pgmap = get_dev_pagemap(pfn, pgmap); if (unlikely(!pgmap)) { undo_dev_pagemap(nr, nr_start, flags, pages); return 0; } - SetPageReferenced(page); - pages[*nr] = page; - if (unlikely(!try_grab_page(page, flags))) { - undo_dev_pagemap(nr, nr_start, flags, pages); + + head = compound_head(page); + next = PageCompound(head) ? end : addr + PAGE_SIZE; + refs = record_subpages(page, addr, next, pages + *nr); + + SetPageReferenced(head); + head = try_grab_compound_head(head, refs, flags); + if (!head) { + if (PageCompound(head)) { + ClearPageReferenced(head); + put_dev_pagemap(pgmap); + } else { + undo_dev_pagemap(nr, nr_start, flags, pages); + } return 0; } - (*nr)++; - pfn++; - } while (addr += PAGE_SIZE, addr != end); + *nr += refs; + pfn += refs; + } while (addr += (refs << PAGE_SHIFT), addr != end); if (pgmap) put_dev_pagemap(pgmap); @@ -2243,17 +2266,6 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) -{ - int nr; - - for (nr = 0; addr != end; addr += PAGE_SIZE) - pages[nr++] = page++; - - return nr; -} - #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, unsigned long sz)