From patchwork Mon Jul 10 16:08:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD4BAEB64DA for ; Mon, 10 Jul 2023 16:09:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EDC66B0072; Mon, 10 Jul 2023 12:09:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19D0B6B0074; Mon, 10 Jul 2023 12:09:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 065A08D0001; Mon, 10 Jul 2023 12:09:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EE80D6B0072 for ; Mon, 10 Jul 2023 12:09:20 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B850712012C for ; Mon, 10 Jul 2023 16:09:20 +0000 (UTC) X-FDA: 80996186880.10.E13B888 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf21.hostedemail.com (Postfix) with ESMTP id 42F551C002A for ; Mon, 10 Jul 2023 16:09:17 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=d4fgA1Rm; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf21.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SO8r5VbGmdwJ3q2RNf1QQ4rNQ+vbk/APFGvtlYo2uGM=; b=PEh6SFDrzUIFocT0WVGDLUijP5rm9qXUhxzuR3jEdLHYiTuMOLShn5aTwxRWf53+arD/YT Ncc40x87jUNh3JO9SpLQEXWhWhsjFOTBYUaOONPAixeNl5M04I+uhDz5FYo49iAspC4C6U 3JTS2k6ofE0lFpfuGzMas8kFA1MeqrM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=d4fgA1Rm; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf21.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005358; a=rsa-sha256; cv=none; b=ffAy5KCoXmzoyhmwE+OQguiOC+UrtqnFe20eSyWc8uepGj+9uy2uzbBgkEr3P2ICoNM95A ZOKz0KClEa1Uk320RIyPviV6az6ZWyQoqjNuN35DzofLgPi6SknHrqsMkIBUrSWgYlOvl7 6NkTQRZ7MYZwAdKdLZAHuAHjw9XR3+A= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFkiBM019039; Mon, 10 Jul 2023 16:09:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=SO8r5VbGmdwJ3q2RNf1QQ4rNQ+vbk/APFGvtlYo2uGM=; b=d4fgA1RmClJ6cJ4Rb0uGPCIn9Db2ye25xE+vqH2MAdQWs04W3vqm6mCBiBKfW32upwe8 keK9b0vdksVrtGy9JXe5Hb8SC2xfB2BhpIBGMli3I/dEfcgZ26uUHBuplCnLjKxw3Bdi HZrfMxpmQJW0swVWghM+Gp21Dfu5kFeQZbUMD0YvfiKy5oa7lvRfTQgNQNsRAFA0xaqJ TCIWOqir+mbTAl766Jjwc9oPOUJvfNdid2yTtIW6ZTzveeshZ65Ic/+pjZyj+/2W3KIS GpP2pfQImI568JZ/smyEfMAahkQviU8d66LqfZZIxPgCyacK3GRJ49/9zytKgeboRypE yg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx39069-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:01 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AFknv6019651; Mon, 10 Jul 2023 16:09:00 GMT Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx3905c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:00 +0000 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AEJ87o005249; Mon, 10 Jul 2023 16:08:59 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([9.208.129.117]) by ppma02dal.us.ibm.com (PPS) with ESMTPS id 3rpye60kkr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:08:59 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG8vDm5243450 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:08:58 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CB17958056; Mon, 10 Jul 2023 16:08:57 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 48C485803F; Mon, 10 Jul 2023 16:08:52 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:08:51 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 01/13] mm/hugepage pud: Allow arch-specific helper function to check huge page pud support Date: Mon, 10 Jul 2023 21:38:30 +0530 Message-ID: <20230710160842.56300-2-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 1tTQ6MZX0f5Iyn8TK6o_wksTBUJECtQ7 X-Proofpoint-ORIG-GUID: 1ry6ZdJIbNBtR9f_VBxmEdWWbRe7-BlB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 bulkscore=0 impostorscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 spamscore=0 priorityscore=1501 clxscore=1015 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 42F551C002A X-Stat-Signature: a5g9s6isw1nmqfhcuppewww53d3zkr5p X-Rspam-User: X-HE-Tag: 1689005357-24713 X-HE-Meta: U2FsdGVkX19D/lAeZNX2v0gHFNwrnL01/k42qq2bcEyay4QSX+PNm2Gkr0/ElGMlmqwaOB+watmE2INQE8UYbPsDkidQAo8VXJPxiT58KvZ70E4QOf1k4NN4naH49mn2Ps8ENtFPgktyfE4bRuWNeL/dvGz/kg++qDTs4ZRP4giFUMjPMkzns3JvGPturH2UNWcvHfbv9XEK2ZMSWhp3vyYp/Qx4rURHvmw/pYlW+KQTqiZs8kTCk9FlRpdghT0j2+zUE8A7sWEpnkqNFGAmOBS+DFDM9qfcn9jCu5ORTTK+Q1k86zX5FWtGr0NedFM6XEXFT3tnADcz0Wg6D58L9UK4dKMDOhgbdQk/B/Sws8tDnaXVzTytd7PqafRSBOxUgj5kqj0XJwTQKAGNXCAqNwwjgXJYBtvXs2DzeXOWcQg+UOrHGNDx8div45ICfXBk95uMdTh8+k7TXDSZA+UbcG40L2F0Lvn/dxbJ45qwnpIuLXRd8edpiczYvdyvrT5vtLjMBqnNMXy0Xru9zyj/3L0r6krH0iVf2diMQYFtIoFF8rgn26qhzt/+y4J4KaI3/y4LVsUEdWIfyAWSlO4lS3Zu820L95lzFGd5cbbJdeDYmNP9Rq8hWhaZwg2JUpTh9hAOC7C/LBtu2keY8ZTWiYSb1o5oIlkD9kTSUnroDYUyWLxHH1ZA07ywp89LQaMmdosaBQKS5jJ6F0ayoEcWHMCz4TlV4W0w/VZXumxC6x9jAG2jDCXDbQvehqD0tmNyFbJhNW6ztCH2ohuTIjP3VrMait+OMKp/+EQumxRaxMAEyoGEU2zEFtOwTsrlC3sU+v2k73m9Srbk1IOEfFZFr6J/3g9731t4UrDu9Ay74vm9aKNReGfuLByOLKTC4zATSHraNrJOIOcdsY/cBt/YmlT6G7XvQOVRioFic5eiuChKgqBtQ0OpCY4nGTBErN9zPa2Zn97WZO+v8ORZb/g mBpFrNmF TqIXSCR7cuaTsoHGLFqdKsKgN4TjsGL8JOJHpNy2h0iZA8RvBPy4vABSF5ZoSlETTfLji+cT4WiAfe6qo7yzJOQm/WEbkSa8MHwby55FGbZMkQE5Vmk38X1wFIJy8nlp5B2wkv/bDpSnOXxs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Architectures like powerpc would like to enable transparent huge page pud support only with radix translation. To support that add has_transparent_pud_hugepage() helper that architectures can override. Signed-off-by: Aneesh Kumar K.V Reviewed-by: Christophe Leroy --- drivers/nvdimm/pfn_devs.c | 2 +- include/linux/pgtable.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c index af7d9301520c..18ad315581ca 100644 --- a/drivers/nvdimm/pfn_devs.c +++ b/drivers/nvdimm/pfn_devs.c @@ -100,7 +100,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) if (has_transparent_hugepage()) { alignments[1] = HPAGE_PMD_SIZE; - if (IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)) + if (has_transparent_pud_hugepage()) alignments[2] = HPAGE_PUD_SIZE; } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5063b482e34f..cf13f8d938a8 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1499,6 +1499,9 @@ typedef unsigned int pgtbl_mod_mask; #define has_transparent_hugepage() IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) #endif +#ifndef has_transparent_pud_hugepage +#define has_transparent_pud_hugepage() IS_BUILTIN(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +#endif /* * On some architectures it depends on the mm if the p4d/pud or pmd * layer of the page table hierarchy is folded or not. From patchwork Mon Jul 10 16:08:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A173FEB64D9 for ; Mon, 10 Jul 2023 16:09:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 073456B0074; Mon, 10 Jul 2023 12:09:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE41D6B0075; Mon, 10 Jul 2023 12:09:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D85268D0001; Mon, 10 Jul 2023 12:09:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CC2A76B0074 for ; Mon, 10 Jul 2023 12:09:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8FFA612013A for ; Mon, 10 Jul 2023 16:09:29 +0000 (UTC) X-FDA: 80996187258.10.B5F9744 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf26.hostedemail.com (Postfix) with ESMTP id 882CA14004E for ; Mon, 10 Jul 2023 16:09:21 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=KwskIEIk; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf26.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R97SYAHPLPjbUcnqCpDHlj9OaI8fme9l4271zttXIHk=; b=phr59AoK/MS10QetqnutH5I1N2zY6uJa7x8KrRrDpf2LiNIHM0ntzHwZCxntOIj52+xOxv /zmmWZUydEprTVdS0FTUBhQ9Dji21Wa8yHWOhjyNvFYZcN9vrgvmRqdfxV+uVg+WQMut2y iKbS/HANUki63oKNgetdjNItfXM1JvM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=KwskIEIk; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf26.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005361; a=rsa-sha256; cv=none; b=MNFhgXYoSw/4bixi5GkxATb3ynRD2vrK1x7sNEOHB2ZKXMzuVfAzuU4Dl+S07vWiVgvWJf LNyYzQbQP8tkYPrTL/wWrfptj0o6Pa7VBkrkoUJgoaSfKfoJHrNmQNqkaWO8X9gbtVd1iu bBZSr4lya6PIF14SdzUJ71kts8r8QcU= Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFlL93023765; Mon, 10 Jul 2023 16:09:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=R97SYAHPLPjbUcnqCpDHlj9OaI8fme9l4271zttXIHk=; b=KwskIEIkkIV+aqQwFzX382fAIa7SLYXvfa0kyvxCAHqpxhfqQNG+0qtui9kdLz0V4JBT CWUL+fkptfiHqCcj1QYsZx6XGRBFd2/1GpFzIIPRLZyrU3IHNnpxrRf+6uoVDOVfHC7B Dny9fNdoN7OpiXBlpVu5clWTD1gKhYEnjdpzWNsnqP4Pr2sPq4J9f1woLu0LMmmhKtHy QKdKjfzZ4ScD9LyKGwTBg7hGYsp+xVA62F8kBXoYBoAELRA2InXIcFS8su0i99BneBOm WRkxM8TQcaf5UkOd2pu5aE0VYezv3eHTp7E3rxKeIW64EsJQaPBOcuwSBRvHh8HuL7iE 7Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxfgnbk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:07 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AFlNFX023882; Mon, 10 Jul 2023 16:09:07 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxfgnb0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:06 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36A8vxnA026453; Mon, 10 Jul 2023 16:09:06 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([9.208.130.97]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3rpye5kb75-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:06 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG94gE19530414 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:04 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C17305805A; Mon, 10 Jul 2023 16:09:04 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9DA555803F; Mon, 10 Jul 2023 16:08:58 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:08:58 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 02/13] mm: Change pudp_huge_get_and_clear_full take vm_area_struct as arg Date: Mon, 10 Jul 2023 21:38:31 +0530 Message-ID: <20230710160842.56300-3-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: iLZ8Yq_G01MfDLpzxucUkVl7nzK6S6G7 X-Proofpoint-ORIG-GUID: Y7_DABXpJy3T1IElMeTpP7bBWxwNNyzf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 phishscore=0 suspectscore=0 bulkscore=0 priorityscore=1501 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspam-User: X-Stat-Signature: er1y7xwgagx7awcyqai71ojii7w3rdzo X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 882CA14004E X-HE-Tag: 1689005361-470679 X-HE-Meta: U2FsdGVkX18887lyITULBEm2PnZk5cWJ7YJhV/pM8mkTqTJ0maskkVTLrKPBDGSfV0WFA//HFZXaTWoiJAhomBJc2A+pkvwI+Q7wxRKBDa+56ajjp9BuPtCodx5Rjn4QN8CE4EJlzo4v0t+RPHeq1VkH4eAU4550c6S96f9fVgBSWnsewivUq15CPOBWR8AZctaHfi0ItVCs1mXckObo4t71RCOEczDw23r5KzHfa7jLHSl0g7ZVFO34S2W6Kj06d0EhizZNVUbIKZBE7Vnizb0boZ0GD5f26k5sbFBDipTYhsutIp8bl51IpW9AGMMqEWtxnoIYIi5h1LXvGUlv/HEh3N4D4q1wcpB0JCdBWNnQSzTAFfhhy+1cwM861QRxMz/TNVxniAcSZFVf87vAoT/B9fySovv8G62TtM7J+AwC46GmZnMisix/NMlCUtjqg3UbEbeQ4GdMDWQ7IGAdcXXXoqJjErvSctUeJlKse+D562fY34+oIFGOPqkBm/2pErmafbgTXL+vGqEvjrFnr7n6RqzNMlriWikZ+7oIFduB0lV2pPhl/BY4U7+7tzI5RJdiIwLP3MkN8kcSha5EuHejA2AoMtM/cBadpD+LdmdzF34AWCQ11pGRzLUX7jnsAEz7GmAoIzABgb545AlaWRdop41Pl+/Cqd+3AvJUVnny9SJIDP0HMSxlKfa6ELRL/Ev/uq/0EqaSuVK7XAOkKsVrJURwf2oBMpKNr8LyMPMu8COk41sflDmWGKVYaOxOu8qd4hKX8+mVyOzaR9NZHCRXz3iHkD9x5ktuovlen/fhpo2kjGCnAcMuAGmrgWrGbIgQVT/6QHF0i9vT1DMQxartzT9e9lyQQdm1gEB28IO/2glnzJ13vf9vaa92KaJ+Xx9zguyrRSvEh1MfP7BgxuFuXh44vEefr6mcAqKluqo8m+447IcuIpPF3aPzpx+TevI2CKGcs1LE0wLclWQ m+HIg4VO VReYSMSRawnTZDI4SN+1yTdxgl/gQaMS6HDzB9HsjX7CyjsjOEBk1LBNnjSX5sCtTJk2eMPiK78e/vlHXX+YdgcUmYH/rNuCCGqCLblMvTZh7npSaYNEcqgWUjeOC/kGGVxIVMgmFrVJQ7bpVm7XWUQTk5G7x42lqmYF5fIKaX+JVApaaIzRXbbsB+VuIfFm1AuUA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We will use this in a later patch to do tlb flush when clearing pud entries on powerpc. This is similar to commit 93a98695f2f9 ("mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg") Signed-off-by: Aneesh Kumar K.V Reviewed-by: Christophe Leroy --- include/linux/pgtable.h | 4 ++-- mm/debug_vm_pgtable.c | 2 +- mm/huge_memory.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index cf13f8d938a8..6fd9b2831338 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -450,11 +450,11 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL -static inline pud_t pudp_huge_get_and_clear_full(struct mm_struct *mm, +static inline pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long address, pud_t *pudp, int full) { - return pudp_huge_get_and_clear(mm, address, pudp); + return pudp_huge_get_and_clear(vma->vm_mm, address, pudp); } #endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index ee119e33fef1..ee2c4c1dcfc8 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -385,7 +385,7 @@ static void __init pud_advanced_tests(struct pgtable_debug_args *args) WARN_ON(!(pud_write(pud) && pud_dirty(pud))); #ifndef __PAGETABLE_PMD_FOLDED - pudp_huge_get_and_clear_full(args->mm, vaddr, args->pudp, 1); + pudp_huge_get_and_clear_full(args->vma, vaddr, args->pudp, 1); pud = READ_ONCE(*args->pudp); WARN_ON(!pud_none(pud)); #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index eb3678360b97..ba20cef681a4 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1981,7 +1981,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, if (!ptl) return 0; - pudp_huge_get_and_clear_full(tlb->mm, addr, pud, tlb->fullmm); + pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_special_huge(vma)) { spin_unlock(ptl); From patchwork Mon Jul 10 16:08:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 571BDEB64DC for ; Mon, 10 Jul 2023 16:09:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 438066B0075; Mon, 10 Jul 2023 12:09:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E7C16B0078; Mon, 10 Jul 2023 12:09:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12AC96B007B; Mon, 10 Jul 2023 12:09:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F14A08D0001 for ; Mon, 10 Jul 2023 12:09:29 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B5E4A80194 for ; Mon, 10 Jul 2023 16:09:29 +0000 (UTC) X-FDA: 80996187258.04.979ED93 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf10.hostedemail.com (Postfix) with ESMTP id 2180CC0002 for ; Mon, 10 Jul 2023 16:09:26 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=O638G2cP; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005367; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wRpOvZEenjvDphBufXXJXjaJM9fWCo+EyUTVYUwsbTE=; b=fMPQLG8vrGGzvz7wMAKnKO41xS/KMbzU2Bs1gw7xXnS6D0WKK5Z8yBbbtUUspSTOy8zGZr wmJMUkh2u8uXS+MgMIj6Zi6HO/z5oRI16ATuBs/Whv2R2DYlRPUNxnFwo2zJuPdEK7/VPX a+u+ajpeXLWRCN+4FCHCtFSWD1UCgfI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=O638G2cP; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005367; a=rsa-sha256; cv=none; b=Vg1V/0rn3wb+E4XfEdpvOa181JMXFjfbiLRXVCiumdSEojIaEA0JaQd/CEfQXKYByMebfk eMoOSf6zXreqOrxfKSL/x84GT7aLTmjNzlSK7k5EW7czUO0VDZOmwmgjaKk3gH7GrxT4J6 58egl6r9jdAQ56MkcWWgV8MSWGCL+gY= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFkhEI019026; Mon, 10 Jul 2023 16:09:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=wRpOvZEenjvDphBufXXJXjaJM9fWCo+EyUTVYUwsbTE=; b=O638G2cPIoGZnzfBQJPxTu6OkA5gVpYNqI24oa4F7EWZ9QgKnpBzrJw6MLlUlvVdIMPx Lli92okcckFlG67SeCLoYFmxD5z/wGb5yWy1bbhMRAIhuMh4tzQQZaaF6BNLC7RfqVDw +XLdhztBb5FMvX4/2JcBREmhGLFXcXDFjGwnletCw2ZVQ96iVE/kNmCiwSHovNRvmZ9T REybt/GrItqdjpoZqFnbiGwNw0A7niXKJq72IGzmFeQ38JOujsaZsQs3B9KbYP4ZXd// cfb9JJhdI1b8LwISmfZuI9dJTcGnhboE+EtngsAs2MeWsLh+iDw9frNauAc5v+QCi3fc 6A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx390d8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:14 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AFkog4019835; Mon, 10 Jul 2023 16:09:14 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx390ch-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:14 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AE3Z6S016659; Mon, 10 Jul 2023 16:09:13 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([9.208.129.118]) by ppma04dal.us.ibm.com (PPS) with ESMTPS id 3rpye60kpp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:13 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9B8065995100 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:11 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7DD3958060; Mon, 10 Jul 2023 16:09:11 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 948DA5805A; Mon, 10 Jul 2023 16:09:05 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:05 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 03/13] mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to override Date: Mon, 10 Jul 2023 21:38:32 +0530 Message-ID: <20230710160842.56300-4-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ra3Bm4wSV_iLhHplkAxn8CpPXdTq2wyR X-Proofpoint-ORIG-GUID: aT04GZjYabtywjSq7KdAhTBcwRfAWDTY X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 bulkscore=0 impostorscore=0 mlxlogscore=918 suspectscore=0 malwarescore=0 spamscore=0 priorityscore=1501 clxscore=1015 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2180CC0002 X-Stat-Signature: mpkrm69epms81tr7kscu64bgoxtb3m4o X-HE-Tag: 1689005366-727313 X-HE-Meta: U2FsdGVkX1+pj7vMH0d27uyR2Ysr8wvbF2KVsqyFUxB4xL2K87Nm6Hew3zkoz+2rHsBCVu7zUngRgKcv2jIhRiT5NpPA4h5duwsrm7FllIGV3wuqnYlRSjtCF0VkwWCeQHBRfPBXjJtBjQ266+Wqir1DN/VpoAR4ZlM4jawGplIqPDpyX80TjvKOAcnPAqVbpNlVkeWUV2LFjZBACM5CkGdkFU/zJxtTA7ETCWCUOsIwB7OZVu2Z3RnO9gpVC8mtWzoV9dvhHMGjAWajBmSxYflhgHlaRWP05vxLnkFC0XeHi2JcvT/CUKUWEXHSe0KaV2wQymgK/aWazy4xvq6XidaqduSYX3i6SdE7QChJnmivQT3XwiGBs3XnLXbj/Bt4gTc6HXxoLEPZ3YCsfqsQ4uR/Tp9ywiz9XHw6IbHxpDUL0FoquNTMugmD65uGX9WbVKLFCAj/Gwkun/FpgmlBlL/Tqo7TMo6647OMJ8FOjIy4qeGynQRVB2dEzdw9AdrK/v2zqFyiNq4ITa6qPh/tfBYdbtWGzAwygUfU3jd67j1vzMemjl/OsSu2wCKOb9fI1oIqfyDj0h9ISduBSxlQLSGvmaiLOuDxwIRSmHOnXOztdlexfG/EHBNWTJM/bMMVtbusGpgi/Y4FfxIgAh8Dla1eNZgb38aXLxTj+ISQ0bSjpK49Hhm1vcMLlSGISfCwPJkr8O5F/JQSIXuATr1l9r6Nw5dQwfNIrU2bMQ/gBfMfPY39XajiLWp3cLWN+n3RPIwLWe+eKrTHs22r987+ngJXhmX1zIEAEnkMBWAhVQyZUQ7KmqjW2khtvw8PVRoV4o1I2nqZltE3R6wfmcUUlyNthTG794fXwAkmajkzYKLJiSwJV7Sr4lfkJgpdCwMzEtzxccMCagq30oUbQmmItB4glxX33CnVBRK4jwZGCaSrxP2ziy/hnPFJjUURnWyTpvoY+uLnT2vHa29Jni8 SOXd3MqP sWJY5Ou71rTik1Ca8A7WkffP2VUfBK5S5WK3NDT4iXHhWd0lWFiP0ichUM93pGHLCRnJ4631wirGnSVl2/XCzH0m4gB/JoAKS/0tAkGfgTWFRZsHLZQHDQ4kitF6GR291Ha48KewECtJT/btT7ZRWu43uYdp4dzG/UijrQ+dcpavh3w3wNu9u5Lzdn09SW8CJ/Q4h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dax vmemmap optimization requires a minimum of 2 PAGE_SIZE area within vmemmap such that tail page mapping can point to the second PAGE_SIZE area. Enforce that in vmemmap_can_optimize() function. Architectures like powerpc also want to enable vmemmap optimization conditionally (only with radix MMU translation). Hence allow architecture override. Signed-off-by: Aneesh Kumar K.V Reviewed-by: Christophe Leroy --- include/linux/mm.h | 27 +++++++++++++++++++++++---- mm/mm_init.c | 2 +- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2dd73e4f3d8e..1a2234ee14d2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3639,13 +3639,32 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap); #endif +#define VMEMMAP_RESERVE_NR 2 #ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP -static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return is_power_of_2(sizeof(struct page)) && - pgmap && (pgmap_vmemmap_nr(pgmap) > 1) && !altmap; + unsigned long nr_pages; + unsigned long nr_vmemmap_pages; + + if (!pgmap || !is_power_of_2(sizeof(struct page))) + return false; + + nr_pages = pgmap_vmemmap_nr(pgmap); + nr_vmemmap_pages = ((nr_pages * sizeof(struct page)) >> PAGE_SHIFT); + /* + * For vmemmap optimization with DAX we need minimum 2 vmemmap + * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst + */ + return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); } +/* + * If we don't have an architecture override, use the generic rule + */ +#ifndef vmemmap_can_optimize +#define vmemmap_can_optimize __vmemmap_can_optimize +#endif + #else static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/mm_init.c b/mm/mm_init.c index a1963c3322af..245ac69b66a5 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1020,7 +1020,7 @@ static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, if (!vmemmap_can_optimize(altmap, pgmap)) return pgmap_vmemmap_nr(pgmap); - return 2 * (PAGE_SIZE / sizeof(struct page)); + return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); } static void __ref memmap_init_compound(struct page *head, From patchwork Mon Jul 10 16:08:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB2B0EB64DA for ; Mon, 10 Jul 2023 16:09:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45F5B6B0078; Mon, 10 Jul 2023 12:09:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 435EA8D0001; Mon, 10 Jul 2023 12:09:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D6836B007D; Mon, 10 Jul 2023 12:09:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1AFCA6B0078 for ; Mon, 10 Jul 2023 12:09:35 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E052A8017E for ; Mon, 10 Jul 2023 16:09:34 +0000 (UTC) X-FDA: 80996187468.26.CEE8309 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf20.hostedemail.com (Postfix) with ESMTP id 6A80F1C003B for ; Mon, 10 Jul 2023 16:09:31 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=pETylCo3; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf20.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005372; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=au27mQkFBCX0j7yag6NPyyKfAD/tDcJxx1jGYEmeYuM=; b=mKf4FT2DXUBt72qTT0Jk1QMU1F+jyFrddh9RTUirPtOBYBZ86MKE2JfQgzeoPDG7HxCa6y 7bRGccv7CJkqKI54OX76rIe5389LqNjAnfURQm1AVVxI2gmyHdOvYZQT9UVvbc0yVvnwa8 LDegmosId9qEpb/rDSMkBo82x/AGZcI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=pETylCo3; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf20.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005372; a=rsa-sha256; cv=none; b=RUXKjhF7JPx3HMRS7lfis2SVk9/nvIXnUQPQmUXJiqL7waJMh/Z0yAlDfj0UtbuX2cBuav W4WGEWiTGK6+RBsPa9Q/IWz6zEaP0pC7fTuOaJJurcUciaBaQdwu89l2h1+KYJdM+m8ZPH HISKJB89mUOu7Gx9tDx+zFAZQbX+9to= Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFx5Qb023924; Mon, 10 Jul 2023 16:09:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=au27mQkFBCX0j7yag6NPyyKfAD/tDcJxx1jGYEmeYuM=; b=pETylCo346mzxsSiBMhZIiNYOUhgwtzEdJK61Gdj59HhQaR8AxNPr3xbDDB+qOf3ihwz BRaXNWcsxAIL8HhGX7doCzsIkg/RM6PCtC+4UobeyNV8I4KGawey8M4SQKn0+mMMgPX8 5Jgw6qzKJmG02BSNZrzzVJ/d7qsyfFEeieaav7TJahOora7Aag0Ooq8gqQaewGmT9pWq BC0JsiQiC4+K55gGWIMRcPNZ2Riy0FWYcS1byN2VfWt+0QVLG0m7IQ46jKL5YnLdGVCQ NLXFVzm7Uppz0oelpnFmamK+hkxVbn5g/+ljIQTLL/UaIHOL24XSq7ogNkee67l+FN8i nQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmyhrp90-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:21 +0000 Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AFooOQ028326; Mon, 10 Jul 2023 16:09:20 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmyhrp86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:20 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG0Pts014032; Mon, 10 Jul 2023 16:09:18 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([9.208.130.97]) by ppma05wdc.us.ibm.com (PPS) with ESMTPS id 3rpye5jnec-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:18 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9HEM32047546 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:18 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B930658060; Mon, 10 Jul 2023 16:09:17 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 27CEE58056; Mon, 10 Jul 2023 16:09:12 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:11 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 04/13] mm/vmemmap: Allow architectures to override how vmemmap optimization works Date: Mon, 10 Jul 2023 21:38:33 +0530 Message-ID: <20230710160842.56300-5-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: dM5DL3hhhBtduq8RvnuKV3eVsFqycPq5 X-Proofpoint-ORIG-GUID: y_vdRo6apOzkb5rydDSrPDBitq_Nf2Ek X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 mlxscore=0 adultscore=0 clxscore=1015 malwarescore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 impostorscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Queue-Id: 6A80F1C003B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: y4rfzpbx4u83ziy3m5159s9ki1ko6tyf X-HE-Tag: 1689005371-595800 X-HE-Meta: U2FsdGVkX1+rgkcaCMYXUxsWQeGRAFkRjKItm5tk0Z+yI64OE6kyouoFiwxb+OC58q2l6myNq+HIjOkVK2+qoAFSj37Lai1KIdXufJ2YpyPybPC1kYl/w1GZRUUoIn7GAAop/u0A7/Ihn+jaxbMOW8Lwmaw1+OrADQLvDgvVWUBtzGvlerY0I0DJx048uCfDO4QVHYzy8oVbXWKs1gQFJczvMhzcY6xRb49hZe3AGn5Jj1zWYUHJegydZ+kY+z57JQApYWFVCFTOKG8AuVG8VzHwK/WdQlqC9k1jVme6U4g8mimMFizGG9LozDQeM245AzYjRKsNenj67M1UIeXqovVq5SsrYcjT7n5p97lmI0npPn3AOXyY8eTbkfiyB/6hydlGcuD7NYA+cXop5qGQEMjWxqSpF1B2p1F4/q8Tms2eFd7NNgZHjcotTbQ8sCE0efKeCQIbC5PmYeKBBtGCnQIV30B5FqYYDBmTi5zGxO+vtMhXab70jtljqUFEprK+oEn1sMnzNAttBqkopD9+3NQ0JBbmlASb9STS1e18/htqKU12zbqRU1dVRtdTbPzqnV9cXGi6I6tIy03KlNblDED3Cp/xZjSSbto9b9VK7u1MWtxXsWsmQjhE/ev06IGYe3gnLHoKDexGeWTHAq8CmMKJxtaNiG0JcklZsRbzSub+MjcAHa30VM1TE/eVOz8ziJFBXEKVw7vpXsjoSSb2ARnXnbBBZlP2ybplU6XlAH4USpN/PJ+zwlZFZ01qvU8ypapR+DPMViVhSncfhCi7e97B285t+HUqgNgWtm3dJrHD4epEGhDrDLy8qmBCzhu9MYM59cw5ivCqRKwgHPsO3q2H+llILLgBbgUXeOyqpZQ/Mnlpm5q6sTH6SkEzcUkFAP9ZZMIT7rB2+Nzzj3IWVXUicITXirdweXPhQW77NRojAVNGZPLwZHJroi+sna7M+G82vv10zcvSpqBJ95t L16sSs9C nCuHzcds/BkmYst5e9+daKpwoLg4Rebo0f+VSMQ6CsQ22Gl2SvQaq93mOBtliPepd1f2nlaxkAy6A08ArRpqjSE4bRC6r95v4QTY+ZBa5Na9rf+w9lj4vbcA2qRlTx0v/v7HVCrHCIDsn+mw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Architectures like powerpc will like to use different page table allocators and mapping mechanisms to implement vmemmap optimization. Similar to vmemmap_populate allow architectures to implement vmemap_populate_compound_pages Signed-off-by: Aneesh Kumar K.V --- mm/sparse-vmemmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a044a130405b..541b3f69a481 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -141,6 +141,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } +#ifndef vmemmap_populate_compound_pages pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, struct page *reuse) @@ -446,6 +447,8 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return 0; } +#endif + struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) From patchwork Mon Jul 10 16:08:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92D26EB64D9 for ; Mon, 10 Jul 2023 16:09:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D8448E0001; Mon, 10 Jul 2023 12:09:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 287FE8D0001; Mon, 10 Jul 2023 12:09:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 176E48E0001; Mon, 10 Jul 2023 12:09:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 067958D0001 for ; Mon, 10 Jul 2023 12:09:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B938712018F for ; Mon, 10 Jul 2023 16:09:41 +0000 (UTC) X-FDA: 80996187762.30.A8760B4 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf22.hostedemail.com (Postfix) with ESMTP id 1E4DCC000A for ; Mon, 10 Jul 2023 16:09:38 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=md2T9uPG; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005379; a=rsa-sha256; cv=none; b=zCwu/AkGHRT3ntIytlg4yGUPrt2mL+BWvX7Ilq6zN7GV0RdGboCCX//7RBqx8X/BIApRBv jNOmq+3FBh+JiiHRf1QXVaec2/hwVnZqs5Pz+5x/uEj95OmDJ1qX7PTLzHFXYGt9qg5wJA 0b6cvSlFiM8AP5s6Ox1qgA5TM1ymp6c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=md2T9uPG; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005379; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v4rgjlG+j4BFMpAKCsUxYL28soNQi0PCLcozMdYF/S8=; b=XMXc4+tTBwwJo4KEJPtE8rgwcljESx9lGUG4UHko8ZVDIznygmPm43mQ8EPc7EmisfGkD4 dDm5FNcgUiwFW+qnXSYdj1a/nwcrFpHEDo2kN+GttrOjhAQaOXyLCfIdyCx3Iw/sZf1Lo2 igTPP/m40M8szOi6IvvakMIAXYiiTEA= Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG6soQ022182; Mon, 10 Jul 2023 16:09:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=v4rgjlG+j4BFMpAKCsUxYL28soNQi0PCLcozMdYF/S8=; b=md2T9uPGftRbVK9CfhrVJapQ0o1bm7j5jOIE+CzN1DkCcsYCGEz0I9/m/jKbVuwMkKaL 5nrpFOWooTkjYU0fp5bOC4GJGg5YVta4Gk/Nvcf1CYcif3FSdiSx65zvAuJqnIRnWfuZ PzXDikMyUm3pSzFxioQJJUO7MghkKTbPBxt/UKX+KGylGOIsScBiasEH8Xt+XDxaTkbK aTYU9cyI5D7QLI1B322yYtEcFCwSdTik6Lkm8mcKIOR2JxBBK5u7occ/Xkd7Qii/s6Oz MFiycysGLQk7JYX7x7HECqrFp3TlxtiyDyjeHYWCWcFOFiV0iDQJP6B3DqZFFtTRFYsA Yg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8g9g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:27 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG70Mb022874; Mon, 10 Jul 2023 16:09:26 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8g93-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:26 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AENBHQ016652; Mon, 10 Jul 2023 16:09:25 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([9.208.129.118]) by ppma04dal.us.ibm.com (PPS) with ESMTPS id 3rpye60kqx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:25 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9OW12228832 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:24 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3CF9158061; Mon, 10 Jul 2023 16:09:24 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 960805803F; Mon, 10 Jul 2023 16:09:18 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:18 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 05/13] mm: Add __HAVE_ARCH_PUD_SAME similar to __HAVE_ARCH_P4D_SAME Date: Mon, 10 Jul 2023 21:38:34 +0530 Message-ID: <20230710160842.56300-6-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Bh06hjFuM7DxUmQhCuruqsfL3LAtml0G X-Proofpoint-ORIG-GUID: 9aG38rov3Qn_HHyvRr6antLwCO8gZHZG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 phishscore=0 malwarescore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 1E4DCC000A X-Stat-Signature: scwaje3zid6354bxtnj9qi5hkti65gw6 X-Rspam-User: X-HE-Tag: 1689005378-815916 X-HE-Meta: U2FsdGVkX18g8m91a4HR7+XYtJS+ckzhWuzGEzW1JvESNCSw4OF/TOEIdRQcafRYzV8LGzKKVJLdKEEn2I74GXhEwJ5TDFPWpqyJdUfTjoP5rNCid35RKpxIbBtxvJx69dVM3M0fUCZS8YMb0pdL7PwQLvPgmrZY/BpNc9reIpCxA2plKUwEy/6qnP1aclXKx74kbwKDZa4ZoRG5M0sK2Hz9deJrndhx2wK9C+AyDe5bkORmCCiLxOr9AHRuHpM40PjtBTd3DItusi91fjeSrGxIyjd/6ood5tXkGBh9wX5ZQUggWi3+8cogjywQOMXX5oSRP5QdqvKKq+1iIk9HiBeWW5qtQtjV5sS587IJk6xdBzLBY1gvpxQMHTWCAiQYBhA/4wltxg9kP4iVwDJNDZLAHd0qiHPsv/JcYXr6k6gN/8cbULPUxEJoO6XIaCxldPi3NkX9Xtv9tFV63dKEyyvoB3VfjT2g/senTWDGREcWF7H17FDxCb5Ton7RqVpKTNvMg1VFE0PK70LhVRu+LmX2ibEIITgI5yb8Y17zAs0eJ9YHe8RittXmebtpvM0kVH5vYPVoZSw1bCPuH2opWCgljAInTLxykf3hqAj6+4OsBdt+nulp4lk7TNwZZ1tNFWFA53pa6ki5wO+LLDgzL61zjgEA1cVBN2Iz+3NDxOguN2wO7TFoc2mOp+c5M/YqF+WPpoKj+ma6hcaS0G/EKq9/1MO0o7cjw5TaNJXXFxqNXvqoTelGTccXrO2Q71spaEjRsC6+SPb3Yw4bScISxxQxYDgE3Xb1WKHJt5FWG1T8h8BYauP1OiM6fY6h9mt/zzJgLN6TCCDbanxgj+kVQ589xr144DLu9xGWYgcOcXrCgyddg0sACdNPEHpv7utDheBc3o3dgeFm9TxhOmF4nDPvr1WjFcaZHymJiY9Um30bwsLvmHye3kxIK2aYtm6SQJ3XXd3VOdyOcrwStYg IHYnUAFw PPVWr79qCoYSS2/afq0aA7Jp5BLJi2pjB15LBoPWxV1ky2uhDHeYS2MhYJqqEloPyMHU5dN5HkAB82uYx7EboMTgPohbzOxUyqYoWd8Pd9Fh3vd8W5EXsMauHrk6yvyfhFMND7iux+qku8+L3K8T/tkswMdI1UX7enrt8W/OpJur3G0smHmiLU4m8KsYMbUSOFr0g X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helps architectures to override pmd_same and pud_same independently. Signed-off-by: Aneesh Kumar K.V Reviewed-by: Christophe Leroy --- include/linux/pgtable.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 6fd9b2831338..91def34f7784 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -693,7 +693,9 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) { return pmd_val(pmd_a) == pmd_val(pmd_b); } +#endif +#ifndef __HAVE_ARCH_PUD_SAME static inline int pud_same(pud_t pud_a, pud_t pud_b) { return pud_val(pud_a) == pud_val(pud_b); From patchwork Mon Jul 10 16:08:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307368 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84D66EB64D9 for ; Mon, 10 Jul 2023 16:09:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ED8D8E0003; Mon, 10 Jul 2023 12:09:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19E628E0002; Mon, 10 Jul 2023 12:09:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03E608E0003; Mon, 10 Jul 2023 12:09:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E5CC98E0002 for ; Mon, 10 Jul 2023 12:09:51 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B5A7A801B7 for ; Mon, 10 Jul 2023 16:09:51 +0000 (UTC) X-FDA: 80996188182.17.6AFC17E Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf11.hostedemail.com (Postfix) with ESMTP id 40CD74001A for ; Mon, 10 Jul 2023 16:09:49 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=AI5YZKyh; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf11.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005389; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CCRKKwBNmbNEXrHZXrT40NjFgPyXV2Q+00nl9gdum18=; b=qNWd+bhy+8DMj5r+pdr8SVo/i0dlt5e6UrS0TjJ89Dv3iUDnSLWBpV0w0XOmB7vHXFdwU9 nEAFYizadyYfikQQbu+qg6qr6NW9bjArUNk4isZPaPaRbyN28DEDVVqTBW02D1JxTJ0iZF kDPDnrc86j8B9VyXvoXGgQO/e9Bh4lA= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=AI5YZKyh; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf11.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005389; a=rsa-sha256; cv=none; b=uVAx5/cgEWwbVSFjNc7+KDfpVF4IVpPOzWAoEPgT5AjFqrkvgu4Vb2A2IYxtORGfm1EKZY OSaR3DNFJhf3g3cYwITApgPq2fZf0Ika84y3WEpqHpq9H8/u96qq46Kk3S5MAQn1eY5Axe NvElQCsWehnMk9Tp/8iU0L33Le6/veg= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFkjC4019114; Mon, 10 Jul 2023 16:09:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=CCRKKwBNmbNEXrHZXrT40NjFgPyXV2Q+00nl9gdum18=; b=AI5YZKyhZn6tnvPcxFvWHX/sp7ZrL65oAObj0hYnUUKE9aHgfbBdoPnvBL3xYSxaLi9J XkYMQqp6Cw/sjpCZaHdlXMnTSrFqNfCjgHKNjm5DD9PU+cp4F1uSPpLao//z5vEoWNng TVoj8fZGeqV1XChW/gBB9GaFkh3UBQ39qPCM2Khvje6YUJlJVUrLOrM7rHMFvIAZeMtB 0YSsd9h9IV+/D/gpK0ZGmuXxs1P2Qk3JFU8g37b2LH3PfDItALnrxeqznJJs4ga4ktJB ZPOdxAprsWQHztVQVPg/FDfvuNK7BDowa7Tu8eG0RaQnywTp/3OuM46O4jP7UH3bDWQH Eg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx390pr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:33 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AFkj0Y019118; Mon, 10 Jul 2023 16:09:32 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx390nw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:32 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36A8QX4s026617; Mon, 10 Jul 2023 16:09:31 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([9.208.130.97]) by ppma03dal.us.ibm.com (PPS) with ESMTPS id 3rpye5kb7n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:31 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9USk35389712 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:30 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 60AE05806A; Mon, 10 Jul 2023 16:09:30 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D467258056; Mon, 10 Jul 2023 16:09:24 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:24 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 06/13] mm/huge pud: Use transparent huge pud helpers only with CONFIG_TRANSPARENT_HUGEPAGE Date: Mon, 10 Jul 2023 21:38:35 +0530 Message-ID: <20230710160842.56300-7-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: dbhA-LWUYgSgWwbUI2uwwftIoNmPxiD3 X-Proofpoint-ORIG-GUID: 3-U2rhMjLxVOMQog8i67w3UGZPyIEpc- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 bulkscore=0 impostorscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 spamscore=0 priorityscore=1501 clxscore=1015 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 40CD74001A X-Stat-Signature: ftepahcaf6iittnwns8577juq4j5r8br X-HE-Tag: 1689005388-792617 X-HE-Meta: U2FsdGVkX1+m2hETqLXmLZFyN2r9e4C5+a1wXGmR1dsIxdOZcGA3oo9jCnFWqP/OVe+dUcIMo7E9smmusTSbcXlXyTfEPYzWmSH/C6Xi99zeZFGFQ9rcy7VSEMdariZgnK3W6ETd61vHjWZhJaOYAi+v1MJdSitJ7OpTChN0P/EMyJySMZPVVc6pA0XmqN8yQebOQmHc/dpgUTYPs/5z96y1ma5xn+T60DgHK4cby1/HXGTuMoO0aRwsFhE93Sx++b1x0GLGhZiFoo3JYA7vjve5itG0CKQ/lZQGybzFd+B2XNcvxL8hLiGvhKocJVYyyspSUTdcLkmRv30ff/R29+J98JleGZgaNq5gsr2IQN1KSP07LMlVKgFN4KKvWW2nlpw4vpved1A+vatieWZgp89tvVVdsqaWb2c5Cd0/U7CL5KvZNEDnDMttV2AJ1sh7//zhGkngVUlvonJfI57/ZOQ9SJRLBOH4And4/51y15dg57ce34lzn06A0reHjYvjo8zMvJAfCc4nIXWodL1KERgMO6qyM7TvUBrnL1YEMk10vhfPunZkQiTWGWLIq72NPSCPNgPIQ4kiJSo4b8Ir5A4xTJ/p471ypy8tnGqplZtUBEtqs5Mm9aGBh5/B1DSIioRul8Ld2HfcjGeWIjBy23ZDFCUqUMoxm0QnH4KRIMIcB7QOlYzBvksxTucGqKUk6KkZv6r2FyjRrB2SJE6zOkF9b8E5aNHdl6JeoXrHELYuwjsIH44ryn7uLbgmOQs7HiX5Kpt0wEAMslWjfILnL+JpQLjTWtCLqVUIuYq36kHkY+PrxeaN0RI5XuzCGbPF2JkE8MJrWlynHMIBAoqvYD+HWjrhwdvj/MoE8b7SPfJSS50vhEV+aifAYmoSztdZPWDCWnS1qWPoSzshyqpDKkftbo/vv+R6zGia+LlBdndHUc9ai+egffnXP1W1MKMd4C7vTpABQFc17/kWjSQ wBxuEMP/ QuW5YOq2nfv9qS58YtHrwaBU+xHM9kZnj2BrdI3YBCN/l7Chdv8j/LJA+iIUIgYqhBiHm2a2BHMet+l00z93Y7cse7hjbXGsJoeh0Qf0R5ab9gEXJv/UFr6b78YCOIbZFScrTvLTVQ36FuQpBLoxhg+KRCr3U4+H0t+z1fpfBLaALJUUzzxPBjNYzJPomfVEXrLEW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pudp_set_wrprotect and move_huge_pud helpers are only used when CONFIG_TRANSPARENT_HUGEPAGE is enabled. Similar to pmdp_set_wrprotect and move_huge_pmd_helpers use architecture override only if CONFIG_TRANSPARENT_HUGEPAGE is set Signed-off-by: Aneesh Kumar K.V Reviewed-by: Christophe Leroy --- include/linux/pgtable.h | 2 ++ mm/mremap.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 91def34f7784..b5af3e014606 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -558,6 +558,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, #endif #ifndef __HAVE_ARCH_PUDP_SET_WRPROTECT #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +#ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long address, pud_t *pudp) { @@ -571,6 +572,7 @@ static inline void pudp_set_wrprotect(struct mm_struct *mm, { BUILD_BUG(); } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ #endif diff --git a/mm/mremap.c b/mm/mremap.c index 11e06e4ab33b..056478c106ee 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -349,7 +349,7 @@ static inline bool move_normal_pud(struct vm_area_struct *vma, } #endif -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr, pud_t *old_pud, pud_t *new_pud) { From patchwork Mon Jul 10 16:08:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BBF6EB64D9 for ; Mon, 10 Jul 2023 16:09:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAD2A8E0005; Mon, 10 Jul 2023 12:09:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5E7F8E0002; Mon, 10 Jul 2023 12:09:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFEC78E0005; Mon, 10 Jul 2023 12:09:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B03F68E0002 for ; Mon, 10 Jul 2023 12:09:55 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7454A80186 for ; Mon, 10 Jul 2023 16:09:55 +0000 (UTC) X-FDA: 80996188350.03.0B1A97D Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf12.hostedemail.com (Postfix) with ESMTP id 03B0D40016 for ; Mon, 10 Jul 2023 16:09:52 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=kCArlejN; spf=pass (imf12.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005393; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P+AZIqnopT1dy9laW4Ph6z8M3DAEFIp4FvwjX5DVsDo=; b=2J/U5TLBhAdBw8ooi8RLVOOpc83sCwLHGifotVrpMEdf94eFiHOvDwBKc25mBTBX7Hur7z DYz1tp53WpLMyZqb2SL+/3LpXsFLVrkBFxuIqR2W/TGG5x9voVFTJAtN6pKOTyBRwhIByf 1PRFd6GgoIIYXotLJq47yuN4pxs5qjw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005393; a=rsa-sha256; cv=none; b=lR/8SLlLuo4JmWy59W6OjJin3B1uYXnbeR5L4Tu2V8Eou+mRaRF3INBGXOHrYgkfiEYDk1 Nib7DDuHvkQ52GbTeL8lleklAm/sZJx7vF9Rpb+OlET3Dk1kJ9GE4lbFVlZB6Ztgh/HoJ/ tYppPLlsbKATJYp7ms+FBoO9e6oXCrU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=kCArlejN; spf=pass (imf12.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG6u0P022271; Mon, 10 Jul 2023 16:09:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=P+AZIqnopT1dy9laW4Ph6z8M3DAEFIp4FvwjX5DVsDo=; b=kCArlejNyJEpZ4iK2UfmX2NNpjTVANeJgWBikfNLsoExMDAGqt5nCx+/I7r3BwS4UeHL 5cM+40jOJiF8KFrCyyfN4AMx89/Ffp/fAq0pAD4o1DXSEyklWdd756qdr2NhlWWsfn0t YO1sDCIcaVWl39NaWLzv09qeyEyt1v3v6aOO5OtZDCb9HIUOogvKfmcGTAqD/31SiAAB 41DdlIqva/p1JudgPy5Kir0dZ2W3fDKabNjgLJv8s8a8smjPqnFxE5bYqofcKy/ZdwQy GA0gSofCX+i0dQAP9oIlMT/uKkjViaSRCFf9KwQ5CLOIFjUSbzKu3ULo4Eln1UlLbm0/ iQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8gh8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:40 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG70di022982; Mon, 10 Jul 2023 16:09:40 GMT Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8ggp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:40 +0000 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AEJ887005249; Mon, 10 Jul 2023 16:09:39 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([9.208.129.118]) by ppma02dal.us.ibm.com (PPS) with ESMTPS id 3rpye60kq9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:39 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9b3o3474092 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:37 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 56D5858056; Mon, 10 Jul 2023 16:09:37 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1C2C05803F; Mon, 10 Jul 2023 16:09:31 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:30 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 07/13] mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization Date: Mon, 10 Jul 2023 21:38:36 +0530 Message-ID: <20230710160842.56300-8-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: isLGc3SvCmJ6YoWaUNk62wHBU3Q0HnO0 X-Proofpoint-ORIG-GUID: 3_EhPL9KedALYPy_LZ7tceuWpHNNmP9K X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 phishscore=0 malwarescore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Stat-Signature: ij3d5usp4eutfsu4wkjtpkkrd4xf89zr X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 03B0D40016 X-Rspam-User: X-HE-Tag: 1689005392-60149 X-HE-Meta: U2FsdGVkX19megStikK3C5dYQqqqVLesHZFNSRByfItG5o6dFm94+e2YGyQ/z6Nv3woIB0I6XOHjI3Ar/OGqMUHmSWVy6sQIMWjR3eI3rkbpNClANafWIu71zOEcJ237sqx9/WavOeu9OHqrTuK9s7jAzP7XK8UjcBWhQxJ3icuowIz8iADMV5pdrtqWMGSBzyW1YSd9ZkN4TUvCPhbaTLbR7mhOz+GCKziMr8rS4bi2KzepdZTrKleMIFckuU6KWkYNkkp/easRqD5aK6RouOr1o9qA8e+TAE4MwA0mrQNcIAVc1FTVTJZ5HWVn7y1k5u8ShIQl44SiCSwkwf1LdU3gOQhonfL71IJSWRwMgjJVX8I53aLbIKRVpDFPcOIS7game+eh7RphK9M9mC5uTo6k547sZMwN/LAgSIEluSB+NB0z3MABvJ2LEAHxeUE5w3pseFh4J8QfDHCAc/nOdjHuWirmQxcSS4mXfu7KOQdV12GEszvZB2iB/2So7Fj6vIj1EB6xELaxbwl5u+7zUCm+JSXJ4UofXqE5JbabqNeWKlPvsch07BXtQOPvlSKgmNuBC18BC1m0qdEXpm7PaxhteCnPmxP5RJHR07gxyG7ijUSecUTaZ4OLZy31z3oXCu4dsywBTcLB8G0Y3aEFqVrUX/MmenobZAnedGK/cOaIpwyxh1aksxZI+9kvJYkiQxwFskv0XQFU9bRVuf1r0qsmH6fYJY1gA0b/Udboe267YN9bS8RpTlvJ3LrQ9qVkDrUNffcgyXI7GvauiNCO39xi7JZl/TbpthGXYAWi0ER1QBYtz4ySGb0vXrN+gTUq/mVm6AoxwlWZ75ZaIwnGoB0jGAIhlQdIDzP2LfFbwm+UizqSsC2F8kyJ3wvVSnmZXjDmPo/UGkWr8NJbO3LIQlhQ35nc6z/t4ICnGI20SDitW9/mWsitDLWG8z9tb+xVmHVMcyy7MioAI1jYSUx 1sJYH3sp kANqQ1I2RrWVESDkSmhNJzy8i9QbjJCz7priwP/36SYSD1Si8jd0s/jE6HNmlQwXvUx3y1Kin6/BSRrCEo/tRNaUBzZm0Zsh2mUmEhzKGwyXmt4kiwtyENCf9dsJe/8lXPD+p/gcqSvrB7VrA5BaRnG54UffYrxqcIEcISyYTHudFMmU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Arm disabled hugetlb vmemmap optimization [1] because hugetlb vmemmap optimization includes an update of both the permissions (writeable to read-only) and the output address (pfn) of the vmemmap ptes. That is not supported without unmapping of pte(marking it invalid) by some architectures. With DAX vmemmap optimization we don't require such pte updates and architectures can enable DAX vmemmap optimization while having hugetlb vmemmap optimization disabled. Hence split DAX optimization support into a different config. s390, loongarch and riscv don't have devdax support. So the DAX config is not enabled for them. With this change, arm64 should be able to select DAX optimization [1] commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") Signed-off-by: Aneesh Kumar K.V --- arch/loongarch/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/x86/Kconfig | 3 ++- fs/Kconfig | 2 +- include/linux/mm.h | 2 +- mm/Kconfig | 5 ++++- 7 files changed, 11 insertions(+), 7 deletions(-) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index e55511af4c77..537ca2a4005a 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -59,7 +59,7 @@ config LOONGARCH select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select BUILDTIME_TABLE_SORT select COMMON_CLK diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4c07b9189c86..6943d34c1ec1 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -53,7 +53,7 @@ config RISCV select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 5b39918b7042..975fd06e4f4d 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -127,7 +127,7 @@ config S390 select ARCH_WANTS_NO_INSTR select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCI diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7422db409770..78224aa76409 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -128,7 +128,8 @@ config X86 select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT diff --git a/fs/Kconfig b/fs/Kconfig index 18d034ec7953..9c104c130a6e 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -252,7 +252,7 @@ config HUGETLB_PAGE config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE - depends on ARCH_WANT_OPTIMIZE_VMEMMAP + depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON diff --git a/include/linux/mm.h b/include/linux/mm.h index 1a2234ee14d2..83f51ec0897d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3640,7 +3640,7 @@ void vmemmap_free(unsigned long start, unsigned long end, #endif #define VMEMMAP_RESERVE_NR 2 -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { diff --git a/mm/Kconfig b/mm/Kconfig index 09130434e30d..923bd35f81f2 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -487,7 +487,10 @@ config SPARSEMEM_VMEMMAP # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. # -config ARCH_WANT_OPTIMIZE_VMEMMAP +config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP + bool + +config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool config HAVE_MEMBLOCK_PHYS_MAP From patchwork Mon Jul 10 16:08:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307370 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56826EB64DC for ; Mon, 10 Jul 2023 16:10:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE6368E0006; Mon, 10 Jul 2023 12:10:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E95CC8E0002; Mon, 10 Jul 2023 12:10:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D36D68E0006; Mon, 10 Jul 2023 12:10:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C55C68E0002 for ; Mon, 10 Jul 2023 12:10:02 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 80FED1A01A5 for ; Mon, 10 Jul 2023 16:10:02 +0000 (UTC) X-FDA: 80996188644.12.DA47A61 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf22.hostedemail.com (Postfix) with ESMTP id 143A7C000A for ; Mon, 10 Jul 2023 16:09:59 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=N3eehmBS; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005400; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HJ59ItgsLbPBAoIVj3eoS5U1bMjw9YWn4IbBZYPC7LI=; b=yTZHbarurVKsU090wWn6B0Y41orlS4hOwQ1R9UkuL7VKx7O82CR3QioMR1YjSRRE0nzySg atEHYvMo8/LVrg8yMtJ8jA8cCImtxVZHt6uHy1AJ/pmwHUXKMWZxE7y8S4u+ulaMcZ18Kd SyAf1WQ86DgW1bKOZtXNIpsMLGhhPzQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=N3eehmBS; spf=pass (imf22.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005400; a=rsa-sha256; cv=none; b=KgK9aIp3CpLQLe+ISb0uSWIiOL5r2NjOPOeKxozH++b+KDiaYYZnNSc2fq5VMbbdqjcHWj zw56GrRjLoSWD0omP2saSMHaM4AxIZLCfiVnWcg827EOKoqy0tu7P+AsMc2Ir/RZuzVs6n fRoDxBALj8cJKoihUgxers7oCnrtRsw= Received: from pps.filterd (m0353722.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFpttZ011328; Mon, 10 Jul 2023 16:09:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=HJ59ItgsLbPBAoIVj3eoS5U1bMjw9YWn4IbBZYPC7LI=; b=N3eehmBSP6qbmJhBAUpjitGZsh/vmDgwhEU0MKMpOFfmzdgPy9bgpG5W2zeoAvgbbiRl qbRqa7qa2qUNX+1Jd+I3yet4BLBkLxIje5ElivHYiVtTGlXUzMjtDnG4MULfOEBC2kr7 7bDpKU0VArjSELKCgmtEVRZ/v7dpPNhxlC7br3LaKz2kXMiIP0pysKxH6UcBueDb0YZ5 6xtWp3DeaYhtxiDHBa8fINMdTnIAQ66ea0LdmYeuDxyPwZzZMthjCsuM5Vw8i2njo7ra +gd46ZyoVALujPPLgkotxb2zla3j+fqknn+DqvDbnndGgFNlhVbmo+wjIAWnYWDBUhAF eA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrn0g8g91-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:46 +0000 Received: from m0353722.ppops.net (m0353722.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG6JnO025767; Mon, 10 Jul 2023 16:09:46 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrn0g8g8e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:45 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFruN4009334; Mon, 10 Jul 2023 16:09:45 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([9.208.130.98]) by ppma04wdc.us.ibm.com (PPS) with ESMTPS id 3rpye5jq1q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:45 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9isW65601860 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:44 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4E44358056; Mon, 10 Jul 2023 16:09:44 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1F98A5803F; Mon, 10 Jul 2023 16:09:38 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:37 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 08/13] powerpc/mm/trace: Convert trace event to trace event class Date: Mon, 10 Jul 2023 21:38:37 +0530 Message-ID: <20230710160842.56300-9-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: -Egj92rGfboxJPAtNBSO5w5B52mL9b10 X-Proofpoint-ORIG-GUID: pt0JEvLwCyZlrON52wZa7F7HapELbV3j X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 lowpriorityscore=0 adultscore=0 spamscore=0 mlxscore=0 suspectscore=0 bulkscore=0 priorityscore=1501 mlxlogscore=978 malwarescore=0 impostorscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Queue-Id: 143A7C000A X-Rspam-User: X-Stat-Signature: f1ui195ugsjs366riggurarhrc85194a X-Rspamd-Server: rspam01 X-HE-Tag: 1689005399-432746 X-HE-Meta: U2FsdGVkX197AIeez0As/IacAbxD8Ez7sREUh08qWWF9NV+p4F2UdRhkep+EpM5lf6oQXFT1pPwGveL98pr9xTC4bJ91d4UJpeJRQICDPLeR3u0XUMbOjElGM+eTGfzneBqCP/+SD06u/6ZYB0eMPwhLN2hxIVDvYFnakG7ebjWfXRKGeuBOT+ArYIvFd+mA8MX7YRdjOF/1pqplM1k3PHgfJQfcbX4M8kqDQKoXue5xwfKox/DLZSbDdYWe4I68n7AFeMbsD6ycTY6XjIHwQY48CNVDk90618GfE1+Wr0HOX/JEU8iRyHCPW67b2Yyyw/nCtn5KxQBQ9jqKchq+NjOqIVFggptCGorYJ98qULC/J3SxYVI2xtYyF76GSbcgX2GJZmyc0875ygG538GNXHciS1N6mXBoIKzYceJpfoDFiSNmCnZftpgO5yqUhii+NJ1irdXNWuz9MJCdQeOpD97R9t3fWlAxOTPRc2KqZ/FgxUnhtnIu+C/CSWVlA3GUj5hhsi6psemM3yQbRDj17mKVK0kGFQCsQ/t0nZ0fb/oISXF7XYQGepjzEMGhhLrZQFt+PtBmr2GXzqLhq6v1nup8p9NUlf5b8gVpOU3WoqZGS5Ab8BHCUBapZy6xHjNFGGuKrAGa4cVSbxbYnLIGeRX10+rn87nlxh23Kl0ru2Dsmo6cWWmiBZ5tbDiP25bnDjzJT0wQNBZ4YCJV0j1HgX0D/eGqXYoZz+sJ2wmdVJhDiK7z/KodC22z/LXM53Ilb5nyRgeHtynTyn9nDxNPYwwrIzdB7bm0e+y5JNmv94NDu1DJtX4xEyJ+QkmZ9gGbexR4ylbp8T289O6FwseIe5Swv4OYFWGn+QlJgPgp6mqr1oCp8FUP9QhxljewKlQMP/HABQylyICp4frxUtaj5TunA1j3HrrwPAx7Q3c7b6Yg3LoEKr1aCYhaRi3ghE3PhrZ3b8DwWmwKJR3Vwsx WkEK081N LAUl/zMpvQXvzyiwhowrAt1BEfFNqiX/NEzjlQVt4PJeUzzB++thwaMvEYTHYYXbEtAo4H3MmWlMr+u2Ndn8XBCVuRT4pE6JzCZsCdi+pI4IZNGbK3WCMk6+5fVzPf74f5SqMr86fR3oYJcY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A follow-up patch will add a pud variant for this same event. Using event class makes that addition simpler. No functional change in this patch. Signed-off-by: Aneesh Kumar K.V Reviewed-by: Christophe Leroy --- arch/powerpc/mm/book3s64/hash_pgtable.c | 2 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- include/trace/events/thp.h | 23 ++++++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c index 51f48984abca..988948d69bc1 100644 --- a/arch/powerpc/mm/book3s64/hash_pgtable.c +++ b/arch/powerpc/mm/book3s64/hash_pgtable.c @@ -214,7 +214,7 @@ unsigned long hash__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr old = be64_to_cpu(old_be); - trace_hugepage_update(addr, old, clr, set); + trace_hugepage_update_pmd(addr, old, clr, set); if (old & H_PAGE_HASHPTE) hpte_do_hugepage_flush(mm, addr, pmdp, old); return old; diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index e7ea492ac510..02e185d2e4d6 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -962,7 +962,7 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add #endif old = radix__pte_update(mm, addr, pmdp_ptep(pmdp), clr, set, 1); - trace_hugepage_update(addr, old, clr, set); + trace_hugepage_update_pmd(addr, old, clr, set); return old; } diff --git a/include/trace/events/thp.h b/include/trace/events/thp.h index 202b3e3e67ff..a95c78b10561 100644 --- a/include/trace/events/thp.h +++ b/include/trace/events/thp.h @@ -8,25 +8,29 @@ #include #include -TRACE_EVENT(hugepage_set_pmd, +DECLARE_EVENT_CLASS(hugepage_set, - TP_PROTO(unsigned long addr, unsigned long pmd), - TP_ARGS(addr, pmd), + TP_PROTO(unsigned long addr, unsigned long pte), + TP_ARGS(addr, pte), TP_STRUCT__entry( __field(unsigned long, addr) - __field(unsigned long, pmd) + __field(unsigned long, pte) ), TP_fast_assign( __entry->addr = addr; - __entry->pmd = pmd; + __entry->pte = pte; ), - TP_printk("Set pmd with 0x%lx with 0x%lx", __entry->addr, __entry->pmd) + TP_printk("Set page table entry with 0x%lx with 0x%lx", __entry->addr, __entry->pte) ); +DEFINE_EVENT(hugepage_set, hugepage_set_pmd, + TP_PROTO(unsigned long addr, unsigned long pmd), + TP_ARGS(addr, pmd) +); -TRACE_EVENT(hugepage_update, +DECLARE_EVENT_CLASS(hugepage_update, TP_PROTO(unsigned long addr, unsigned long pte, unsigned long clr, unsigned long set), TP_ARGS(addr, pte, clr, set), @@ -48,6 +52,11 @@ TRACE_EVENT(hugepage_update, TP_printk("hugepage update at addr 0x%lx and pte = 0x%lx clr = 0x%lx, set = 0x%lx", __entry->addr, __entry->pte, __entry->clr, __entry->set) ); +DEFINE_EVENT(hugepage_update, hugepage_update_pmd, + TP_PROTO(unsigned long addr, unsigned long pmd, unsigned long clr, unsigned long set), + TP_ARGS(addr, pmd, clr, set) +); + DECLARE_EVENT_CLASS(migration_pmd, TP_PROTO(unsigned long addr, unsigned long pmd), From patchwork Mon Jul 10 16:08:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FB38EB64D9 for ; Mon, 10 Jul 2023 16:10:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82F0F8E0007; Mon, 10 Jul 2023 12:10:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DEAB8E0002; Mon, 10 Jul 2023 12:10:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6590A8E0007; Mon, 10 Jul 2023 12:10:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5663D8E0002 for ; Mon, 10 Jul 2023 12:10:10 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 23C1E1401B6 for ; Mon, 10 Jul 2023 16:10:10 +0000 (UTC) X-FDA: 80996188980.22.642475D Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf02.hostedemail.com (Postfix) with ESMTP id 8945580023 for ; Mon, 10 Jul 2023 16:10:06 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Sb5AOUb6; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf02.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005406; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a5RHy5LCyBWAffq3u5fxbbAOnH+hyHn5PiZXR0j4vyU=; b=pKNMUyHV/Mlp0h7av8XB/KPukJ0NWbPB8ZhyRXUw/GGs9d7ruj6KiwsLYC7HRdR98BOS8x 84QHn50R2bEYOiMyI7GI4y4pMtprOVSUbvpM581baJrOMxy9EuppCCUjTB1zl+qcY9R/Oi 5vlZl0wfIpN4Jm2MrF/JUTypGAm9/Lc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Sb5AOUb6; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf02.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005406; a=rsa-sha256; cv=none; b=OHH8gRn72GILeP4GgmnQXLFWrHYMCo4i8RANX5Q6zxyt1VHp6JGonEUgMP5piIBIOUFUIw D5iz+0KNyFw4peAKYwVoJibh8QvSUvTnChhFjSgMvruQKQUfDylkcNdVhMuKBLukWsbtD0 evsV+IGrBHSLMd90AXpY/NV0vwFGLfg= Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFkkTg019202; Mon, 10 Jul 2023 16:09:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=a5RHy5LCyBWAffq3u5fxbbAOnH+hyHn5PiZXR0j4vyU=; b=Sb5AOUb6HP6fPoJDej+Y8FsoLuGJJZkgEUghu3T4jWq/edEUwlDjLq3rJg2/SS/Vyxaj RRKCw03kBf2z5wRco5VTj0gFv6oAk3OHZS8Dd5y5XnEnFjVcGjm0ZNfFbESiQSQ0Dh/x d8DDAJ9pdaUDFEdLClkEIrEZt7MzwaoSKVvGOAE/7kM9qnTg+HGeqZAdImEipr9W3q+Y BY7Js1dh9qUd/qzjvVqjDKKXbgQ+f1eVly4T5SwjiIqn+SewxS/IBx9rRCJ3sYaf3LSI Or2coNKjl3Rn2otzfmQ5zJqdfp6gjA9KR+jPRSdJbn+DV+3I1DhjbT+ju8ACo294k/EV DA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx3911e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:54 +0000 Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG0GDk025058; Mon, 10 Jul 2023 16:09:54 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmx39111-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:54 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AEDLtP016682; Mon, 10 Jul 2023 16:09:53 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([9.208.129.116]) by ppma04dal.us.ibm.com (PPS) with ESMTPS id 3rpye60ktf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:52 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9p0s43385156 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:51 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1A94D5805A; Mon, 10 Jul 2023 16:09:51 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E2CD858056; Mon, 10 Jul 2023 16:09:44 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:44 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 09/13] powerpc/book3s64/mm: Enable transparent pud hugepage Date: Mon, 10 Jul 2023 21:38:38 +0530 Message-ID: <20230710160842.56300-10-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: zJpQrL8OE9nUpnjmq_QgJIFIHuqNbEG5 X-Proofpoint-ORIG-GUID: gPvhSyLdpi5Xf6-O7AKHXh8AX67CfH9o X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxscore=0 bulkscore=0 impostorscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 spamscore=0 priorityscore=1501 clxscore=1015 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Queue-Id: 8945580023 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: zyxqisk5no4q4ea5n7p4dttwn4ugxw68 X-HE-Tag: 1689005406-457140 X-HE-Meta: U2FsdGVkX19XAZu8PnapXWp5BXJMSI/Dy//U/k3cHciHrZhpssQQLfsuPObGNFWhOw58SWHIt5BZcIMgyU8qfWdW4FXEw+WJi6tPtbv8JMoZir8vJ0YVXtGPm0vNNm62snSUKQAgOk0EagS6SkaraTr5Ac1e85kXAtW8t9SXwGh5INDML7R/ZRQP3FncpVMCNmX9++hyMUd+NLCcBiSOpB6fm7fH/kFBPF24g0o2WG/kTAShAS5AlYPMHJMERWUvfVo518vNn0PY3pQEU1rEr5pQ4CwPgqvP2k8EZYC0Fbyn9OAZJaaGWL76ccHbHflMeL+gm1U2IXrPnqddX+RpBry2TOCmtit55kwWnlPAZfArnK1JzwWurv+S52E8ydS3rfOG8hjC6HQqrYviLXpc7QrvMohd96lvPFtHZf6CelQHY06pyU12ZxgAp16vtRCWfrdS1NmSmofSgzwMLnqyqUjEEJtxtVrZFnxNnao+hblcL/eq50J8emlSGvlrbZN/mMnQ9ys23NE6hYEEFAqO7+XM+a9KGUX5MW04LaOYe/pwWhasXPO5wg7MeLDq0JxKWQbeWZ2vy3+9L2/PCZqJqB/0JlRDFKXDZW5HmqV7/OdwOEZ8n/4bkEIoHsDoL/Xjzj62Ggo4WHQvtvI411ZLuOB8iiwvuTBcq5M+mjCPCUOgNoeU/eqW3MqZVrXiq/Js4rw2wfAICe4KUQ5QrErnR80tfDWfbQVbX7d68SV4cdO8uzmrX2jtaeKhr47+IdgU+HuCRiJc3AsjwIFLqeR/OnH/vCAOJso5quBkVU5HplhJOIm2fFT1Sm/Ddmuzy/gH71tXLxbJhkSxwC8WMQqktw4aSphv/V9y+pskmElpeAlkgbfgiuUnRRjC+GzRfZULZMyyQsOhxTrWwJ+lrWPRXO/JyVIEvI7G9jGxAtVPBdBI4YaqPc+E8mNI2jv2VR1bqQqjineoTaWydSHvfV4 GRBOxj+g UOC1wb/zLpUqQ1KU4AxZSv9DmVvFOXP6WDJ1KpwsVZ+70w9vciJ5fZqgCl6pinAWrdOtipQnMBOExVoua3qc8LtO9j/FlIDohmDyByscd+vKub5oa+X545aCr8Mx+scSpySHGvfqPoB3fBmY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is enabled only with radix translation and 1G hugepage size. This will be used with devdax device memory with a namespace alignment of 1G. Anon transparent hugepage is not supported even though we do have helpers checking pud_trans_huge(). We should never find that return true. The only expected pte bit combination is _PAGE_PTE | _PAGE_DEVMAP. Some of the helpers are never expected to get called on hash translation and hence is marked to call BUG() in such a case. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/hash.h | 9 + arch/powerpc/include/asm/book3s/64/pgtable.h | 155 ++++++++++++++++-- arch/powerpc/include/asm/book3s/64/radix.h | 36 ++++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++++++++ arch/powerpc/mm/book3s64/radix_pgtable.c | 28 ++++ arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/platforms/Kconfig.cputype | 1 + include/trace/events/thp.h | 10 ++ 10 files changed, 323 insertions(+), 11 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/hash.h b/arch/powerpc/include/asm/book3s/64/hash.h index d4a19e6547ac..6e70ae511631 100644 --- a/arch/powerpc/include/asm/book3s/64/hash.h +++ b/arch/powerpc/include/asm/book3s/64/hash.h @@ -138,7 +138,16 @@ static inline int hash__pmd_same(pmd_t pmd_a, pmd_t pmd_b) } #define hash__pmd_bad(pmd) (pmd_val(pmd) & H_PMD_BAD_BITS) + +/* + * pud comparison that will work with both pte and page table pointer. + */ +static inline int hash__pud_same(pud_t pud_a, pud_t pud_b) +{ + return (((pud_raw(pud_a) ^ pud_raw(pud_b)) & ~cpu_to_be64(_PAGE_HPTEFLAGS)) == 0); +} #define hash__pud_bad(pud) (pud_val(pud) & H_PUD_BAD_BITS) + static inline int hash__p4d_bad(p4d_t p4d) { return (p4d_val(p4d) == 0); diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 4acc9690f599..38ac50279199 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -921,8 +921,29 @@ static inline pud_t pte_pud(pte_t pte) { return __pud_raw(pte_raw(pte)); } + +static inline pte_t *pudp_ptep(pud_t *pud) +{ + return (pte_t *)pud; +} + +#define pud_pfn(pud) pte_pfn(pud_pte(pud)) +#define pud_dirty(pud) pte_dirty(pud_pte(pud)) +#define pud_young(pud) pte_young(pud_pte(pud)) +#define pud_mkold(pud) pte_pud(pte_mkold(pud_pte(pud))) +#define pud_wrprotect(pud) pte_pud(pte_wrprotect(pud_pte(pud))) +#define pud_mkdirty(pud) pte_pud(pte_mkdirty(pud_pte(pud))) +#define pud_mkclean(pud) pte_pud(pte_mkclean(pud_pte(pud))) +#define pud_mkyoung(pud) pte_pud(pte_mkyoung(pud_pte(pud))) +#define pud_mkwrite(pud) pte_pud(pte_mkwrite(pud_pte(pud))) #define pud_write(pud) pte_write(pud_pte(pud)) +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY +#define pud_soft_dirty(pmd) pte_soft_dirty(pud_pte(pud)) +#define pud_mksoft_dirty(pmd) pte_pud(pte_mksoft_dirty(pud_pte(pud))) +#define pud_clear_soft_dirty(pmd) pte_pud(pte_clear_soft_dirty(pud_pte(pud))) +#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ + static inline int pud_bad(pud_t pud) { if (radix_enabled()) @@ -1115,15 +1136,24 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write) #ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); +extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); +extern void set_pud_at(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud); + static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { } +static inline void update_mmu_cache_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pud) +{ +} + extern int hash__has_transparent_hugepage(void); static inline int has_transparent_hugepage(void) { @@ -1133,6 +1163,14 @@ static inline int has_transparent_hugepage(void) } #define has_transparent_hugepage has_transparent_hugepage +static inline int has_transparent_pud_hugepage(void) +{ + if (radix_enabled()) + return radix__has_transparent_pud_hugepage(); + return 0; +} +#define has_transparent_pud_hugepage has_transparent_pud_hugepage + static inline unsigned long pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, unsigned long clr, unsigned long set) @@ -1142,6 +1180,16 @@ pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, return hash__pmd_hugepage_update(mm, addr, pmdp, clr, set); } +static inline unsigned long +pud_hugepage_update(struct mm_struct *mm, unsigned long addr, pud_t *pudp, + unsigned long clr, unsigned long set) +{ + if (radix_enabled()) + return radix__pud_hugepage_update(mm, addr, pudp, clr, set); + BUG(); + return pud_val(*pudp); +} + /* * returns true for pmd migration entries, THP, devmap, hugetlb * But compile time dependent on THP config @@ -1151,6 +1199,11 @@ static inline int pmd_large(pmd_t pmd) return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_PTE)); } +static inline int pud_large(pud_t pud) +{ + return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); +} + /* * For radix we should always find H_PAGE_HASHPTE zero. Hence * the below will work for radix too @@ -1166,6 +1219,17 @@ static inline int __pmdp_test_and_clear_young(struct mm_struct *mm, return ((old & _PAGE_ACCESSED) != 0); } +static inline int __pudp_test_and_clear_young(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + unsigned long old; + + if ((pud_raw(*pudp) & cpu_to_be64(_PAGE_ACCESSED | H_PAGE_HASHPTE)) == 0) + return 0; + old = pud_hugepage_update(mm, addr, pudp, _PAGE_ACCESSED, 0); + return ((old & _PAGE_ACCESSED) != 0); +} + #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) @@ -1174,6 +1238,14 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_hugepage_update(mm, addr, pmdp, _PAGE_WRITE, 0); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long addr, + pud_t *pudp) +{ + if (pud_write(*pudp)) + pud_hugepage_update(mm, addr, pudp, _PAGE_WRITE, 0); +} + /* * Only returns true for a THP. False for pmd migration entry. * We also need to return true when we come across a pte that @@ -1195,6 +1267,17 @@ static inline int pmd_trans_huge(pmd_t pmd) return hash__pmd_trans_huge(pmd); } +static inline int pud_trans_huge(pud_t pud) +{ + if (!pud_present(pud)) + return false; + + if (radix_enabled()) + return radix__pud_trans_huge(pud); + return 0; +} + + #define __HAVE_ARCH_PMD_SAME static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) { @@ -1203,6 +1286,15 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) return hash__pmd_same(pmd_a, pmd_b); } +#define __HAVE_ARCH_PUD_SAME +static inline int pud_same(pud_t pud_a, pud_t pud_b) +{ + if (radix_enabled()) + return radix__pud_same(pud_a, pud_b); + return hash__pud_same(pud_a, pud_b); +} + + static inline pmd_t __pmd_mkhuge(pmd_t pmd) { if (radix_enabled()) @@ -1210,6 +1302,14 @@ static inline pmd_t __pmd_mkhuge(pmd_t pmd) return hash__pmd_mkhuge(pmd); } +static inline pud_t __pud_mkhuge(pud_t pud) +{ + if (radix_enabled()) + return radix__pud_mkhuge(pud); + BUG(); + return pud; +} + /* * pfn_pmd return a pmd_t that can be used as pmd pte entry. */ @@ -1225,14 +1325,34 @@ static inline pmd_t pmd_mkhuge(pmd_t pmd) return pmd; } +static inline pud_t pud_mkhuge(pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM + if (radix_enabled()) + WARN_ON((pud_raw(pud) & cpu_to_be64(_PAGE_PTE)) == 0); + else + WARN_ON(1); +#endif + return pud; +} + + #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS extern int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty); +#define __HAVE_ARCH_PUDP_SET_ACCESS_FLAGS +extern int pudp_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, + pud_t entry, int dirty); #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG extern int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_TEST_AND_CLEAR_YOUNG +extern int pudp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp); + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1243,6 +1363,16 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, return hash__pmdp_huge_get_and_clear(mm, addr, pmdp); } +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR +static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + if (radix_enabled()) + return radix__pudp_huge_get_and_clear(mm, addr, pudp); + BUG(); + return *pudp; +} + static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { @@ -1257,6 +1387,11 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full); +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL +pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, + unsigned long addr, + pud_t *pudp, int full); + #define __HAVE_ARCH_PGTABLE_DEPOSIT static inline void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, pgtable_t pgtable) @@ -1305,6 +1440,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return hash__pmd_mkdevmap(pmd); } +static inline pud_t pud_mkdevmap(pud_t pud) +{ + if (radix_enabled()) + return radix__pud_mkdevmap(pud); + BUG(); + return pud; +} + static inline int pmd_devmap(pmd_t pmd) { return pte_devmap(pmd_pte(pmd)); @@ -1312,7 +1455,7 @@ static inline int pmd_devmap(pmd_t pmd) static inline int pud_devmap(pud_t pud) { - return 0; + return pte_devmap(pud_pte(pud)); } static inline int pgd_devmap(pgd_t pgd) @@ -1321,16 +1464,6 @@ static inline int pgd_devmap(pgd_t pgd) } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline int pud_pfn(pud_t pud) -{ - /* - * Currently all calls to pud_pfn() are gated around a pud_devmap() - * check so this should never be used. If it grows another user we - * want to know about it. - */ - BUILD_BUG(); - return 0; -} #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION pte_t ptep_modify_prot_start(struct vm_area_struct *, unsigned long, pte_t *); void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long, diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 686001eda936..2ef92f36340f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -250,6 +250,10 @@ static inline int radix__pud_bad(pud_t pud) return !!(pud_val(pud) & RADIX_PUD_BAD_BITS); } +static inline int radix__pud_same(pud_t pud_a, pud_t pud_b) +{ + return ((pud_raw(pud_a) ^ pud_raw(pud_b)) == 0); +} static inline int radix__p4d_bad(p4d_t p4d) { @@ -268,9 +272,22 @@ static inline pmd_t radix__pmd_mkhuge(pmd_t pmd) return __pmd(pmd_val(pmd) | _PAGE_PTE); } +static inline int radix__pud_trans_huge(pud_t pud) +{ + return (pud_val(pud) & (_PAGE_PTE | _PAGE_DEVMAP)) == _PAGE_PTE; +} + +static inline pud_t radix__pud_mkhuge(pud_t pud) +{ + return __pud(pud_val(pud) | _PAGE_PTE); +} + extern unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, unsigned long clr, unsigned long set); +extern unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, unsigned long clr, + unsigned long set); extern pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, @@ -278,6 +295,9 @@ extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t radix__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); extern pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); +pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp); + static inline int radix__has_transparent_hugepage(void) { /* For radix 2M at PMD level means thp */ @@ -285,6 +305,14 @@ static inline int radix__has_transparent_hugepage(void) return 1; return 0; } + +static inline int radix__has_transparent_pud_hugepage(void) +{ + /* For radix 1G at PUD level means pud hugepage support */ + if (mmu_psize_defs[MMU_PAGE_1G].shift == PUD_SHIFT) + return 1; + return 0; +} #endif static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) @@ -292,9 +320,17 @@ static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) return __pmd(pmd_val(pmd) | (_PAGE_PTE | _PAGE_DEVMAP)); } +static inline pud_t radix__pud_mkdevmap(pud_t pud) +{ + return __pud(pud_val(pud) | (_PAGE_PTE | _PAGE_DEVMAP)); +} + +struct vmem_altmap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); +int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap); extern void radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 77797a2a82eb..a38542259fab 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -68,6 +68,8 @@ void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); extern void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +extern void radix__flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end); extern void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); extern void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0d0c1447ecf0..a01c20a8fbf7 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -50,6 +50,14 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma, radix__flush_pmd_tlb_range(vma, start, end); } +#define __HAVE_ARCH_FLUSH_PUD_TLB_RANGE +static inline void flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + if (radix_enabled()) + radix__flush_pud_tlb_range(vma, start, end); +} + #define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start, diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 85c84e89e3ea..75b938268b04 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -64,11 +64,39 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, return changed; } +int pudp_set_access_flags(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp, pud_t entry, int dirty) +{ + int changed; +#ifdef CONFIG_DEBUG_VM + WARN_ON(!pud_devmap(*pudp)); + assert_spin_locked(pud_lockptr(vma->vm_mm, pudp)); +#endif + changed = !pud_same(*(pudp), entry); + if (changed) { + /* + * We can use MMU_PAGE_1G here, because only radix + * path look at the psize. + */ + __ptep_set_access_flags(vma, pudp_ptep(pudp), + pud_pte(entry), address, MMU_PAGE_1G); + } + return changed; +} + + int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { return __pmdp_test_and_clear_young(vma->vm_mm, address, pmdp); } + +int pudp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp) +{ + return __pudp_test_and_clear_young(vma->vm_mm, address, pudp); +} + /* * set a new huge pmd. We should not be called for updating * an existing pmd entry. That should go via pmd_hugepage_update. @@ -90,6 +118,23 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd)); } +void set_pud_at(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM + /* + * Make sure hardware valid bit is not set. We don't do + * tlb flush for this update. + */ + + WARN_ON(pte_hw_valid(pud_pte(*pudp))); + assert_spin_locked(pud_lockptr(mm, pudp)); + WARN_ON(!(pud_large(pud))); +#endif + trace_hugepage_set_pud(addr, pud_val(pud)); + return set_pte_at(mm, addr, pudp_ptep(pudp), pud_pte(pud)); +} + static void do_serialize(void *arg) { /* We've taken the IPI, so try to trim the mask while here */ @@ -147,11 +192,35 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, return pmd; } +pud_t pudp_huge_get_and_clear_full(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, int full) +{ + pud_t pud; + + VM_BUG_ON(addr & ~HPAGE_PMD_MASK); + VM_BUG_ON((pud_present(*pudp) && !pud_devmap(*pudp)) || + !pud_present(*pudp)); + pud = pudp_huge_get_and_clear(vma->vm_mm, addr, pudp); + /* + * if it not a fullmm flush, then we can possibly end up converting + * this PMD pte entry to a regular level 0 PTE by a parallel page fault. + * Make sure we flush the tlb in this case. + */ + if (!full) + flush_pud_tlb_range(vma, addr, addr + HPAGE_PUD_SIZE); + return pud; +} + static pmd_t pmd_set_protbits(pmd_t pmd, pgprot_t pgprot) { return __pmd(pmd_val(pmd) | pgprot_val(pgprot)); } +static pud_t pud_set_protbits(pud_t pud, pgprot_t pgprot) +{ + return __pud(pud_val(pud) | pgprot_val(pgprot)); +} + /* * At some point we should be able to get rid of * pmd_mkhuge() and mk_huge_pmd() when we update all the @@ -166,6 +235,15 @@ pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot) return __pmd_mkhuge(pmd_set_protbits(__pmd(pmdv), pgprot)); } +pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot) +{ + unsigned long pudv; + + pudv = (pfn << PAGE_SHIFT) & PTE_RPN_MASK; + + return __pud_mkhuge(pud_set_protbits(__pud(pudv), pgprot)); +} + pmd_t mk_pmd(struct page *page, pgprot_t pgprot) { return pfn_pmd(page_to_pfn(page), pgprot); diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 02e185d2e4d6..227fea53c217 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -967,6 +967,23 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add return old; } +unsigned long radix__pud_hugepage_update(struct mm_struct *mm, unsigned long addr, + pud_t *pudp, unsigned long clr, + unsigned long set) +{ + unsigned long old; + +#ifdef CONFIG_DEBUG_VM + WARN_ON(!pud_devmap(*pudp)); + assert_spin_locked(pud_lockptr(mm, pudp)); +#endif + + old = radix__pte_update(mm, addr, pudp_ptep(pudp), clr, set, 1); + trace_hugepage_update_pud(addr, old, clr, set); + + return old; +} + pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -1043,6 +1060,17 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, return old_pmd; } +pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, + unsigned long addr, pud_t *pudp) +{ + pud_t old_pud; + unsigned long old; + + old = radix__pud_hugepage_update(mm, addr, pudp, ~0UL, 0); + old_pud = __pud(old); + return old_pud; +} + #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep, diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 0bd4866d9824..79049a0194f1 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -1461,6 +1461,13 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, } EXPORT_SYMBOL(radix__flush_pmd_tlb_range); +void radix__flush_pud_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + radix__flush_tlb_range_psize(vma->vm_mm, start, end, MMU_PAGE_1G); +} +EXPORT_SYMBOL(radix__flush_pud_tlb_range); + void radix__flush_tlb_all(void) { unsigned long rb,prs,r,rs; diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index 45fd975ef521..340b86ef7284 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -94,6 +94,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE diff --git a/include/trace/events/thp.h b/include/trace/events/thp.h index a95c78b10561..f50048af5fcc 100644 --- a/include/trace/events/thp.h +++ b/include/trace/events/thp.h @@ -30,6 +30,11 @@ DEFINE_EVENT(hugepage_set, hugepage_set_pmd, TP_ARGS(addr, pmd) ); +DEFINE_EVENT(hugepage_set, hugepage_set_pud, + TP_PROTO(unsigned long addr, unsigned long pud), + TP_ARGS(addr, pud) +); + DECLARE_EVENT_CLASS(hugepage_update, TP_PROTO(unsigned long addr, unsigned long pte, unsigned long clr, unsigned long set), @@ -57,6 +62,11 @@ DEFINE_EVENT(hugepage_update, hugepage_update_pmd, TP_ARGS(addr, pmd, clr, set) ); +DEFINE_EVENT(hugepage_update, hugepage_update_pud, + TP_PROTO(unsigned long addr, unsigned long pud, unsigned long clr, unsigned long set), + TP_ARGS(addr, pud, clr, set) +); + DECLARE_EVENT_CLASS(migration_pmd, TP_PROTO(unsigned long addr, unsigned long pmd), From patchwork Mon Jul 10 16:08:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307372 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36D46EB64D9 for ; Mon, 10 Jul 2023 16:10:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D108A6B0072; Mon, 10 Jul 2023 12:10:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC0AE8E0001; Mon, 10 Jul 2023 12:10:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B88786B0078; Mon, 10 Jul 2023 12:10:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A814A6B0072 for ; Mon, 10 Jul 2023 12:10:19 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6A8AC801B7 for ; Mon, 10 Jul 2023 16:10:18 +0000 (UTC) X-FDA: 80996189316.10.64DF47E Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf12.hostedemail.com (Postfix) with ESMTP id 920D740018 for ; Mon, 10 Jul 2023 16:10:15 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=EGMkQ7R4; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf12.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005415; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mVE6MWZ6CEbZcrEA9t5OvKdceeovJvFe0GbC+65p1Oo=; b=PS6vCPgJxtYFod9D+wBAoQMn0kImjU0wYT4nO4aGbz0U5oaFhhhty1kQcT/yD+f8jqll0Z c9aWBlmffL/BiMMYQele3KRaYLSzaCHUnowURooabIkstKEhYRN0fv0H9qf+QfwbawWwXG R8dB7h0pmTnR0WRn/faArezBK9/YY3w= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=EGMkQ7R4; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf12.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005415; a=rsa-sha256; cv=none; b=fwXtVv7M520vq8S1Dn9i+FgfXrGWkPkzt4IB07zNCKy1xNXbE6FOY3cc5iv4I+x/Ipuk4W liZPcQ/DX9EjLuPAyc99arNVAndzTXEr+V4nnJmRAKd7+zvhNQ7SYSQqkwpgdCAIDVOmE/ ltwDY0evYliGL4kgw3O/VSoQBgp5iQ0= Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG6uuN022348; Mon, 10 Jul 2023 16:10:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=mVE6MWZ6CEbZcrEA9t5OvKdceeovJvFe0GbC+65p1Oo=; b=EGMkQ7R41mnMyCC4Cl0k0M2Uubwq4CIhAt31YcCLerASZCdQMDOlYsAIeG2Coj/iMSzY DupR3sytqcMPdl2IAHoC2sftiKZsrlrI2NYRY80o5/d6/ZnHf3eSGTTZszol6JXPPIur YDEJ7pfo2qNuPYrNqCB0U0ED/UQ1+YheDQoHZQPfQnPe8izpZewONtzo9obb2KndSybp obP4NVyEtiUFbkXH56iOoWrrnEWToLM/OeGlIYBHmt0DIY/7KJpahC/XSVQBrGEMdCzX 4BIT23HCE030hvNKoCvJFwDq1OlAeAd2dVt+n7nfJqnwyWDky5vXXUngjLooW6meqrLp lA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8gvm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:01 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG8rB0032504; Mon, 10 Jul 2023 16:10:00 GMT Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8guq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:00 +0000 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFsIYB005859; Mon, 10 Jul 2023 16:09:58 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([9.208.130.98]) by ppma03wdc.us.ibm.com (PPS) with ESMTPS id 3rpye5tnsx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:09:58 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AG9v0i54460682 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:09:58 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CC03E58060; Mon, 10 Jul 2023 16:09:57 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C177358056; Mon, 10 Jul 2023 16:09:51 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:51 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 10/13] powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap handling function Date: Mon, 10 Jul 2023 21:38:39 +0530 Message-ID: <20230710160842.56300-11-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: -XRuNFkB9lztOxlt2nXOrDsHA4Bq8trU X-Proofpoint-ORIG-GUID: 0VtIL17ao4O78rC6kw4o4tv2Pk3UHQGv X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 phishscore=0 malwarescore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Queue-Id: 920D740018 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 8tsorq5sqynrgm9co11ypbacbo4d8ztu X-HE-Tag: 1689005415-893890 X-HE-Meta: U2FsdGVkX1+nRx7nWgZ3q+V6BzFrOhM6D6bsThBEvLEhXtlfHQvCO6+4aHdeJLxFYS6XXCehIBtfGHZ7UkXR/6wNpMCMztzojeBxP//DuHkcCN6D+Yv3Rz6hvXl1PlVtI17rIZt322cGp1c4CvlRbhFDma3wCj4o6uP/4cZrq5FHxq/62wz7EPLs21aVILBSlXzVRGktCsB3ZRDs2Faio8tUZs3dPVJVFqStXhokvOxxI5V5KwaixhwjtLzjIXt2kj2ztI4o4NlnQ7sZYG6Cxk5xOPyCE9h0AJTrKvkQxrMqNzGQusJS/hyYUgIpUFFp1zo22YVGQlYz8dDQPU+ttK9//+rjll/OU0mCKzzG7VOALLjhG0+T+TTMAMSK4L+ruD3QPQ5cea16UxGGYTPu8x2w8UrnsHnob3Ii8PvIbZMxSdq3D6SpNhZ158Gt9IBzCjU+QK2fuhc8OV7FJpld8wNK8hdjIkX2VXjVwbSXYgS5FQ/hfNWiNKfGfCGFz/fogWI/09eqOIrbNW6knQud3B6tHKj1CauLJECfSE6Qpd6Es08mbaciwHTy2A7uJjgOjNrJrqGNMHaukfuK/qA5nkO2HCN9Qq3YETiDk0griMCFbPbX6EN3lPTnR/Z2PjPNP7agc9wXAEDyUcoq3Bxdz2b3hnLDhDoC0vVvaXbka2MQFilQnBx4xDAZupUpRSs2dkGilM8enooDO7Ug/uPsfh32nr15cDQ6xTcvDDvGtriSJKqxPxaK4y49/tWlouurQDAv09HdDIHFtlJK8A8pwpvphQXYcexOrPWJsVMOtpTsb+b86nUH+/BZefc4rhC5JkgpP6nxe9jehGGtOLSkAZ6PTh0iPkEbSmbBS1viHxmnIW8LDrx3p04NUZt7MEsPh3Eq9kCNPp78+DzjATjUhp0AN8EZ4s4+S4+IhVJHSR3off0UaDZAmWNB0g4N7mDwpuqzS4Pn4tVu7611Z5A k9lz4Sbz jk9gW8Gf3cLUnfD7x64YoeM/ZBZDpMcRqSCUPLFO1ZUffoOkgJ/5kl1/21XUjtUjMbWHnW9EYZcpIqnhG3WS74uJmMS9lyXwNdEwsUSwrqHSnwVObKV6f/+pk6Iq1OV4fseRh4fYNRVZvUSw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is in preparation to update radix to implement vmemmap optimization for devdax. Below are the rules w.r.t radix vmemmap mapping 1. First try to map things using PMD (2M) 2. With altmap if altmap cross-boundary check returns true, fall back to PAGE_SIZE 3. If we can't allocate PMD_SIZE backing memory for vmemmap, fallback to PAGE_SIZE On removing vmemmap mapping, check if every subsection that is using the vmemmap area is invalid. If found to be invalid, that implies we can safely free the vmemmap area. We don't use the PAGE_UNUSED pattern used by x86 because with 64K page size, we need to do the above check even at the PAGE_SIZE granularity. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/radix.h | 2 + arch/powerpc/include/asm/pgtable.h | 4 + arch/powerpc/mm/book3s64/radix_pgtable.c | 326 +++++++++++++++++++-- arch/powerpc/mm/init_64.c | 26 +- 4 files changed, 327 insertions(+), 31 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 2ef92f36340f..f1461289643a 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -331,6 +331,8 @@ extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long phys); int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); +void __ref radix__vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap); extern void radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size); diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 6a88bfdaa69b..68817ea7f994 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -165,6 +165,10 @@ static inline bool is_ioremap_addr(const void *x) return addr >= IOREMAP_BASE && addr < IOREMAP_END; } + +int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size); +bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, + unsigned long page_size); #endif /* CONFIG_PPC64 */ #endif /* __ASSEMBLY__ */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 227fea53c217..9a7f3707b6fb 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -744,8 +744,59 @@ static void free_pud_table(pud_t *pud_start, p4d_t *p4d) p4d_clear(p4d); } +#ifdef CONFIG_SPARSEMEM_VMEMMAP +static bool __meminit vmemmap_pmd_is_unused(unsigned long addr, unsigned long end) +{ + unsigned long start = ALIGN_DOWN(addr, PMD_SIZE); + + return !vmemmap_populated(start, PMD_SIZE); +} + +static bool __meminit vmemmap_page_is_unused(unsigned long addr, unsigned long end) +{ + unsigned long start = ALIGN_DOWN(addr, PAGE_SIZE); + + return !vmemmap_populated(start, PAGE_SIZE); + +} +#endif + +static void __meminit free_vmemmap_pages(struct page *page, + struct vmem_altmap *altmap, + int order) +{ + unsigned int nr_pages = 1 << order; + + if (altmap) { + unsigned long alt_start, alt_end; + unsigned long base_pfn = page_to_pfn(page); + + /* + * with 2M vmemmap mmaping we can have things setup + * such that even though atlmap is specified we never + * used altmap. + */ + alt_start = altmap->base_pfn; + alt_end = altmap->base_pfn + altmap->reserve + + altmap->free + altmap->alloc + altmap->align; + + if (base_pfn >= alt_start && base_pfn < alt_end) { + vmem_altmap_free(altmap, nr_pages); + return; + } + } + + if (PageReserved(page)) { + /* allocated from memblock */ + while (nr_pages--) + free_reserved_page(page++); + } else + free_pages((unsigned long)page_address(page), order); +} + static void remove_pte_table(pte_t *pte_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pte_t *pte; @@ -759,24 +810,26 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr, if (!pte_present(*pte)) continue; - if (!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(next)) { - /* - * The vmemmap_free() and remove_section_mapping() - * codepaths call us with aligned addresses. - */ - WARN_ONCE(1, "%s: unaligned range\n", __func__); - continue; + if (PAGE_ALIGNED(addr) && PAGE_ALIGNED(next)) { + if (!direct) + free_vmemmap_pages(pte_page(*pte), altmap, 0); + pte_clear(&init_mm, addr, pte); + pages++; } - - pte_clear(&init_mm, addr, pte); - pages++; +#ifdef CONFIG_SPARSEMEM_VMEMMAP + else if (!direct && vmemmap_page_is_unused(addr, next)) { + free_vmemmap_pages(pte_page(*pte), altmap, 0); + pte_clear(&init_mm, addr, pte); + } +#endif } if (direct) update_page_count(mmu_virtual_psize, -pages); } static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pte_t *pte_base; @@ -790,18 +843,24 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, continue; if (pmd_is_leaf(*pmd)) { - if (!IS_ALIGNED(addr, PMD_SIZE) || - !IS_ALIGNED(next, PMD_SIZE)) { - WARN_ONCE(1, "%s: unaligned range\n", __func__); - continue; + if (IS_ALIGNED(addr, PMD_SIZE) && + IS_ALIGNED(next, PMD_SIZE)) { + if (!direct) + free_vmemmap_pages(pmd_page(*pmd), altmap, get_order(PMD_SIZE)); + pte_clear(&init_mm, addr, (pte_t *)pmd); + pages++; } - pte_clear(&init_mm, addr, (pte_t *)pmd); - pages++; +#ifdef CONFIG_SPARSEMEM_VMEMMAP + else if (!direct && vmemmap_pmd_is_unused(addr, next)) { + free_vmemmap_pages(pmd_page(*pmd), altmap, get_order(PMD_SIZE)); + pte_clear(&init_mm, addr, (pte_t *)pmd); + } +#endif continue; } pte_base = (pte_t *)pmd_page_vaddr(*pmd); - remove_pte_table(pte_base, addr, next, direct); + remove_pte_table(pte_base, addr, next, direct, altmap); free_pte_table(pte_base, pmd); } if (direct) @@ -809,7 +868,8 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, } static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, - unsigned long end, bool direct) + unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long next, pages = 0; pmd_t *pmd_base; @@ -834,15 +894,16 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, } pmd_base = pud_pgtable(*pud); - remove_pmd_table(pmd_base, addr, next, direct); + remove_pmd_table(pmd_base, addr, next, direct, altmap); free_pmd_table(pmd_base, pud); } if (direct) update_page_count(MMU_PAGE_1G, -pages); } -static void __meminit remove_pagetable(unsigned long start, unsigned long end, - bool direct) +static void __meminit +remove_pagetable(unsigned long start, unsigned long end, bool direct, + struct vmem_altmap *altmap) { unsigned long addr, next; pud_t *pud_base; @@ -871,7 +932,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end, } pud_base = p4d_pgtable(*p4d); - remove_pud_table(pud_base, addr, next, direct); + remove_pud_table(pud_base, addr, next, direct, altmap); free_pud_table(pud_base, p4d); } @@ -894,7 +955,7 @@ int __meminit radix__create_section_mapping(unsigned long start, int __meminit radix__remove_section_mapping(unsigned long start, unsigned long end) { - remove_pagetable(start, end, true); + remove_pagetable(start, end, true, NULL); return 0; } #endif /* CONFIG_MEMORY_HOTPLUG */ @@ -926,10 +987,223 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } +int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, + unsigned long addr, unsigned long next) +{ + int large = pmd_large(*pmdp); + + if (large) + vmemmap_verify(pmdp_ptep(pmdp), node, addr, next); + + return large; +} + +void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node, + unsigned long addr, unsigned long next) +{ + pte_t entry; + pte_t *ptep = pmdp_ptep(pmdp); + + VM_BUG_ON(!IS_ALIGNED(addr, PMD_SIZE)); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + set_pte_at(&init_mm, addr, ptep, entry); + asm volatile("ptesync": : :"memory"); + + vmemmap_verify(ptep, node, addr, next); +} + +static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long addr, + int node, + struct vmem_altmap *altmap, + struct page *reuse) +{ + pte_t *pte = pte_offset_kernel(pmdp, addr); + + if (pte_none(*pte)) { + pte_t entry; + void *p; + + if (!reuse) { + /* + * make sure we don't create altmap mappings + * covering things outside the device. + */ + if (altmap && altmap_cross_boundary(altmap, addr, PAGE_SIZE)) + altmap = NULL; + + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p && altmap) + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL); + if (!p) + return NULL; + } else { + /* + * When a PTE/PMD entry is freed from the init_mm + * there's a free_pages() call to this page allocated + * above. Thus this get_page() is paired with the + * put_page_testzero() on the freeing path. + * This can only called by certain ZONE_DEVICE path, + * and through vmemmap_populate_compound_pages() when + * slab is available. + */ + get_page(reuse); + p = page_to_virt(reuse); + } + + VM_BUG_ON(!PAGE_ALIGNED(addr)); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + set_pte_at(&init_mm, addr, pte, entry); + asm volatile("ptesync": : :"memory"); + } + return pte; +} + +static inline pud_t *vmemmap_pud_alloc(p4d_t *p4dp, int node, + unsigned long address) +{ + pud_t *pud; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(p4d_none(*p4dp))) { + if (unlikely(!slab_is_available())) { + pud = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + p4d_populate(&init_mm, p4dp, pud); + /* go to the pud_offset */ + } else + return pud_alloc(&init_mm, p4dp, address); + } + return pud_offset(p4dp, address); +} + +static inline pmd_t *vmemmap_pmd_alloc(pud_t *pudp, int node, + unsigned long address) +{ + pmd_t *pmd; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(pud_none(*pudp))) { + if (unlikely(!slab_is_available())) { + pmd = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + pud_populate(&init_mm, pudp, pmd); + } else + return pmd_alloc(&init_mm, pudp, address); + } + return pmd_offset(pudp, address); +} + +static inline pte_t *vmemmap_pte_alloc(pmd_t *pmdp, int node, + unsigned long address) +{ + pte_t *pte; + + /* All early vmemmap mapping to keep simple do it at PAGE_SIZE */ + if (unlikely(pmd_none(*pmdp))) { + if (unlikely(!slab_is_available())) { + pte = early_alloc_pgtable(PAGE_SIZE, node, 0, 0); + pmd_populate(&init_mm, pmdp, pte); + } else + return pte_alloc_kernel(pmdp, address); + } + return pte_offset_kernel(pmdp, address); +} + + + +int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) +{ + unsigned long addr; + unsigned long next; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + for (addr = start; addr < end; addr = next) { + next = pmd_addr_end(addr, end); + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return -ENOMEM; + + if (pmd_none(READ_ONCE(*pmd))) { + void *p; + + /* + * keep it simple by checking addr PMD_SIZE alignment + * and verifying the device boundary condition. + * For us to use a pmd mapping, both addr and pfn should + * be aligned. We skip if addr is not aligned and for + * pfn we hope we have extra area in the altmap that + * can help to find an aligned block. This can result + * in altmap block allocation failures, in which case + * we fallback to RAM for vmemmap allocation. + */ + if (altmap && (!IS_ALIGNED(addr, PMD_SIZE) || + altmap_cross_boundary(altmap, addr, PMD_SIZE))) { + /* + * make sure we don't create altmap mappings + * covering things outside the device. + */ + goto base_mapping; + } + + p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); + if (p) { + vmemmap_set_pmd(pmd, p, node, addr, next); + continue; + } else if (altmap) { + /* + * A vmemmap block allocation can fail due to + * alignment requirements and we trying to align + * things aggressively there by running out of + * space. Try base mapping on failure. + */ + goto base_mapping; + } + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + /* + * If a huge mapping exist due to early call to + * vmemmap_populate, let's try to use that. + */ + continue; + } +base_mapping: + /* + * Not able allocate higher order memory to back memmap + * or we found a pointer to pte page. Allocate base page + * size vmemmap + */ + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return -ENOMEM; + + pte = radix__vmemmap_pte_populate(pmd, addr, node, altmap, NULL); + if (!pte) + return -ENOMEM; + + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + next = addr + PAGE_SIZE; + } + return 0; +} + #ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size) { - remove_pagetable(start, start + page_size, false); + remove_pagetable(start, start + page_size, true, NULL); +} + +void __ref radix__vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) +{ + remove_pagetable(start, end, false, altmap); } #endif #endif diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index fe1b83020e0d..5701faca39ef 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -92,7 +92,7 @@ static struct page * __meminit vmemmap_subsection_start(unsigned long vmemmap_ad * a page table lookup here because with the hash translation we don't keep * vmemmap details in linux page table. */ -static int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size) +int __meminit vmemmap_populated(unsigned long vmemmap_addr, int vmemmap_map_size) { struct page *start; unsigned long vmemmap_end = vmemmap_addr + vmemmap_map_size; @@ -183,8 +183,8 @@ static __meminit int vmemmap_list_populate(unsigned long phys, return 0; } -static bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, - unsigned long page_size) +bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, + unsigned long page_size) { unsigned long nr_pfn = page_size / sizeof(struct page); unsigned long start_pfn = page_to_pfn((struct page *)start); @@ -204,6 +204,11 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, bool altmap_alloc; unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_populate(start, end, node, altmap); +#endif + /* Align to the page size of the linear mapping. */ start = ALIGN_DOWN(start, page_size); @@ -303,8 +308,8 @@ static unsigned long vmemmap_list_free(unsigned long start) return vmem_back->phys; } -void __ref vmemmap_free(unsigned long start, unsigned long end, - struct vmem_altmap *altmap) +void __ref __vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) { unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; unsigned long page_order = get_order(page_size); @@ -362,6 +367,17 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, vmemmap_remove_mapping(start, page_size); } } + +void __ref vmemmap_free(unsigned long start, unsigned long end, + struct vmem_altmap *altmap) +{ +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_free(start, end, altmap); +#endif + return __vmemmap_free(start, end, altmap); +} + #endif void register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page, unsigned long size) From patchwork Mon Jul 10 16:08:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C551EB64DC for ; Mon, 10 Jul 2023 16:10:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 373D08E0002; Mon, 10 Jul 2023 12:10:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 323E58E0001; Mon, 10 Jul 2023 12:10:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19E888E0002; Mon, 10 Jul 2023 12:10:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 08A198E0001 for ; Mon, 10 Jul 2023 12:10:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DAE1DC01B7 for ; Mon, 10 Jul 2023 16:10:22 +0000 (UTC) X-FDA: 80996189484.19.83CBCCB Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf10.hostedemail.com (Postfix) with ESMTP id 40FABC0010 for ; Mon, 10 Jul 2023 16:10:19 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=J9OjxajT; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005420; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8eWjFBDf+yRxnTJPxKOQhUOIQyyFB6yshvJb80Wv8oI=; b=0hgjHnIjJ69kWTwaYdmOI0CpZ0Tyqvy+Cq38uK5F3AAAvTFJBSVWQo2iEU8D2InZuhkevL O4i6UZbYhacOB53go6hYzKCbaAWuEON2cmk4d/uB/7sUsFcX6NjYASjrst5jbaPHxSZyN6 YGjN571l6VxEtwnvFZ/ttcOKu/BIGpQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=J9OjxajT; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005420; a=rsa-sha256; cv=none; b=nzgFCkg00ia7B8cUYIW9pLxf4Z+CvyHZkb+rBqsVRv10bTrOeKFwH+z7VLTN7B5B4lbeXJ o7w9nF/UEpuvMTaHVQrF4q503UigxWR7eraE+sWVRDGmNXi9PkPTHiOcgPgMTmyRQkqoae g8DZgeQDIHkRFJ60K2y1M7Q0IdJcjqY= Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG2W2l000361; Mon, 10 Jul 2023 16:10:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=8eWjFBDf+yRxnTJPxKOQhUOIQyyFB6yshvJb80Wv8oI=; b=J9OjxajTo61iglypLYbDg6nwWp8fJtVuc2KdB8EtsVmF5YzGCkaDo29cOpb5YEEV+4UT et97iVDsIyH7qjPKOHG9mEl8kCLB5VHsyUgvUoBdnUCGXvP2SQYJbv34vWHMW6jkUyr6 kagNgAK2gCkG/3VirbrXa2oo39jPTy+Fe68/fJ5MCoyZWsuDGiVrQpf/Djv1KEzzzdi+ jfQnDaL7xyk5AXU2LNuIdSGg75Kg74WEFLgQ8+YUFtJfT7/b7+ke8x5KRfqvc9mDLUy+ N5hPUKjQwysvTV4dUzvhw9WrpueFUrNDuKKjnPdkGI2alU11aJfEXANG3kIUttPuif1L 1Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrn5k08bp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:08 +0000 Received: from m0360072.ppops.net (m0360072.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG3CmA002316; Mon, 10 Jul 2023 16:10:07 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrn5k08bb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:07 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AFeVWr009382; Mon, 10 Jul 2023 16:10:06 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([9.208.129.116]) by ppma04wdc.us.ibm.com (PPS) with ESMTPS id 3rpye5jq2y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:06 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AGA4d350790782 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:10:05 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B6EF258060; Mon, 10 Jul 2023 16:10:04 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9F4A65803F; Mon, 10 Jul 2023 16:09:58 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:09:58 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 11/13] powerpc/book3s64/radix: Add support for vmemmap optimization for radix Date: Mon, 10 Jul 2023 21:38:40 +0530 Message-ID: <20230710160842.56300-12-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 1ezUKs4p5mrWP-OhtWV-y-IFI9hLWlnl X-Proofpoint-GUID: 2WTYTIbQhjMx1pcM7ekG2EPTpSFKXuMh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 spamscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspam-User: X-Stat-Signature: 9zadkuzjg66epfrimgdj4fpj9ugr18es X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 40FABC0010 X-HE-Tag: 1689005419-769926 X-HE-Meta: U2FsdGVkX1/ZQ1iFIq70ozMO25D45qLkiubQjC57MsGvROA+zRK4jQv7SQeMjdfRhbFILKzBPDHwz1GpvjVcE9dezTO7kbIV7pOVGHwLMF9rw88UW5vi+SDK8UqYVQ1Utbn0pMOalq0eAvdsS6i1Q5B4HmXFMx9nCC5EpfBYBD3HgZQUE5ceoVQKDTtQa/AQGXhF3fqSsO6Yi6yOtx8uABybWCm/fdnP4tu3HOOxh/TX08ZHhwwjxXUd264Lx2HZ37Q4qa7qbtpr09jY3M27enuXcjUyk7pH3Qek55F9C0uOlVpjRoCTjtnHFX+3tNY3w7Q0Q1uB6urdAmlL/deEtK46jkkBhFh5YnAki85GRMOL/BeZIQ58dW8+nn10mS1BTNcyLcvXe7gDxldPV9RLk/JibJJGgB3mavM/SGE31pEHxU119TRQNjTqmq60prRLVGQVFmGOr9CYDG5K9G5L8Pmn2l0jfsD6c7Abq1EaCiE/YM/eazGGRurH8pFA0Fl0g9M/gDS621lzpFAh2ycbABk5Ht53pU9sORocMMznobeeTWcVRRYplP/TIU/YZr+0bWtI514fUFD7QTeZfwwUrhdQWjwdtOp3hhthvOHycQYpzeOBDidx03dgn3Lob306zK57J6QWHkJSREZEQxii/JT7+KrdarZZoASUJFiIhDah1ExEWO9Vg01NqTTmgY6qHjxiNpX3KXjGzd29n4E+bO+9db0vUUZJf0fxFJzBwqsoE/G5TG9RDK6qa0xQMU+zOhY9fE1xYkgp27EHrEEJE+UKzzf/BU17CK+0X/r3NkbyV6zoIt07+AJACPqrMwyvXnHRG7QnG5IwsKm41zjCbUVS2zoIb8ywk5/TBbSOVu1XLc2A7A/A4Dp0lTk4aqe4095Uniw+JiceWMHtVAy8C5iL40SCPTyJqKafyHXJ/H7D3coKQrh3vPX7kFCCFVYXsedetEqldLIf+oyQgpk SaSKO978 PpKYT8IYpZdh/LHpfl1FBVHi6lNNxPjiQ1Fp7VOi9O93iex0idQLD1sisjeAau1ryoIr8E0XHUujfX2c2toIYixcxfftOOEJdlDL00Jb4c8LUtzsDWQoxiUFC50+Xxcdb0DATxXR2qFY9q/A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With 2M PMD-level mapping, we require 32 struct pages and a single vmemmap page can contain 1024 struct pages (PAGE_SIZE/sizeof(struct page)). Hence with 64K page size, we don't use vmemmap deduplication for PMD-level mapping. Signed-off-by: Aneesh Kumar K.V --- Documentation/mm/vmemmap_dedup.rst | 1 + Documentation/powerpc/index.rst | 1 + Documentation/powerpc/vmemmap_dedup.rst | 101 ++++++++++ arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/radix.h | 9 + arch/powerpc/mm/book3s64/radix_pgtable.c | 203 +++++++++++++++++++++ 6 files changed, 316 insertions(+) create mode 100644 Documentation/powerpc/vmemmap_dedup.rst diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_dedup.rst index a4b12ff906c4..c573e08b5043 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -210,6 +210,7 @@ the device (altmap). The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). +For powerpc equivalent details see Documentation/powerpc/vmemmap_dedup.rst The differences with HugeTLB are relatively minor. diff --git a/Documentation/powerpc/index.rst b/Documentation/powerpc/index.rst index d33b554ca7ba..a50834798454 100644 --- a/Documentation/powerpc/index.rst +++ b/Documentation/powerpc/index.rst @@ -36,6 +36,7 @@ powerpc ultravisor vas-api vcpudispatch_stats + vmemmap_dedup features diff --git a/Documentation/powerpc/vmemmap_dedup.rst b/Documentation/powerpc/vmemmap_dedup.rst new file mode 100644 index 000000000000..dc4db59fdf87 --- /dev/null +++ b/Documentation/powerpc/vmemmap_dedup.rst @@ -0,0 +1,101 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========== +Device DAX +========== + +The device-dax interface uses the tail deduplication technique explained in +Documentation/mm/vmemmap_dedup.rst + +On powerpc, vmemmap deduplication is only used with radix MMU translation. Also +with a 64K page size, only the devdax namespace with 1G alignment uses vmemmap +deduplication. + +With 2M PMD level mapping, we require 32 struct pages and a single 64K vmemmap +page can contain 1024 struct pages (64K/sizeof(struct page)). Hence there is no +vmemmap deduplication possible. + +With 1G PUD level mapping, we require 16384 struct pages and a single 64K +vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Hence we +require 16 64K pages in vmemmap to map the struct page for 1G PUD level mapping. + +Here's how things look like on device-dax after the sections are populated:: + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PUD | +-----------+ | | | + | level | | . | ----------------------+ | | + | mapping | +-----------+ | | + | | | . | ------------------------+ | + | | +-----------+ | + | | | 15 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + + +With 4K page size, 2M PMD level mapping requires 512 struct pages and a single +4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we +require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapping. + +Here's how things look like on device-dax after the sections are populated:: + + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PMD | +-----------+ | | | + | level | | 5 | ----------------------+ | | + | mapping | +-----------+ | | + | | | 6 | ------------------------+ | + | | +-----------+ | + | | | 7 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + +With 1G PUD level mapping, we require 262144 struct pages and a single 4K +vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we +require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level +mapping. + +Here's how things look like on device-dax after the sections are populated:: + + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PUD | +-----------+ | | | + | level | | . | ----------------------+ | | + | mapping | +-----------+ | | + | | | . | ------------------------+ | + | | +-----------+ | + | | | 4095 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 0b1172cbeccb..116d6add0bb0 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -174,6 +174,7 @@ config PPC select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if PPC_RADIX_MMU select ARCH_WANTS_MODULES_DATA_IN_VMALLOC if PPC_BOOK3S_32 || PPC_8xx select ARCH_WEAK_RELEASE_ACQUIRE select BINFMT_ELF diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index f1461289643a..3195f268ed7f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -326,6 +326,7 @@ static inline pud_t radix__pud_mkdevmap(pud_t pud) } struct vmem_altmap; +struct dev_pagemap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); @@ -363,5 +364,13 @@ int radix__remove_section_mapping(unsigned long start, unsigned long end); void radix__kernel_map_pages(struct page *page, int numpages, int enable); +#define vmemmap_can_optimize vmemmap_can_optimize +bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap); + +#define vmemmap_populate_compound_pages vmemmap_populate_compound_pages +int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap); #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 9a7f3707b6fb..b492b67c0b7d 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -987,6 +987,15 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } + +bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + if (radix_enabled()) + return __vmemmap_can_optimize(altmap, pgmap); + + return false; +} + int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, unsigned long addr, unsigned long next) { @@ -1194,6 +1203,200 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in return 0; } +static pte_t * __meminit radix__vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return NULL; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return NULL; + if (pmd_leaf(*pmd)) + /* + * The second page is mapped as a hugepage due to a nearby request. + * Force our mapping to page size without deduplication + */ + return NULL; + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return NULL; + radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + return pte; +} + +static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, + unsigned long pfn_offset, int node) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + unsigned long map_addr; + + /* the second vmemmap page which we use for duplication */ + map_addr = addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; + pgd = pgd_offset_k(map_addr); + p4d = p4d_offset(pgd, map_addr); + pud = vmemmap_pud_alloc(p4d, node, map_addr); + if (!pud) + return NULL; + pmd = vmemmap_pmd_alloc(pud, node, map_addr); + if (!pmd) + return NULL; + if (pmd_leaf(*pmd)) + /* + * The second page is mapped as a hugepage due to a nearby request. + * Force our mapping to page size without deduplication + */ + return NULL; + pte = vmemmap_pte_alloc(pmd, node, map_addr); + if (!pte) + return NULL; + /* + * Check if there exist a mapping to the left + */ + if (pte_none(*pte)) { + /* + * Populate the head page vmemmap page. + * It can fall in different pmd, hence + * vmemmap_populate_address() + */ + pte = radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL, NULL); + if (!pte) + return NULL; + /* + * Populate the tail pages vmemmap page + */ + pte = radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); + if (!pte) + return NULL; + vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); + return pte; + } + return pte; +} + +int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + /* + * we want to map things as base page size mapping so that + * we can save space in vmemmap. We could have huge mapping + * covering out both edges. + */ + unsigned long addr; + unsigned long addr_pfn = start_pfn; + unsigned long next; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + for (addr = start; addr < end; addr = next) { + + pgd = pgd_offset_k(addr); + p4d = p4d_offset(pgd, addr); + pud = vmemmap_pud_alloc(p4d, node, addr); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_alloc(pud, node, addr); + if (!pmd) + return -ENOMEM; + + if (pmd_leaf(READ_ONCE(*pmd))) { + /* existing huge mapping. Skip the range */ + addr_pfn += (PMD_SIZE >> PAGE_SHIFT); + next = pmd_addr_end(addr, end); + continue; + } + pte = vmemmap_pte_alloc(pmd, node, addr); + if (!pte) + return -ENOMEM; + if (!pte_none(*pte)) { + /* + * This could be because we already have a compound + * page whose VMEMMAP_RESERVE_NR pages were mapped and + * this request fall in those pages. + */ + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } else { + unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); + unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); + pte_t *tail_page_pte; + + /* + * if the address is aligned to huge page size it is the + * head mapping. + */ + if (pfn_offset == 0) { + /* Populate the head page vmemmap page */ + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + /* + * Populate the tail pages vmemmap page + * It can fall in different pmd, hence + * vmemmap_populate_address() + */ + pte = radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, NULL); + if (!pte) + return -ENOMEM; + + addr_pfn += 2; + next = addr + 2 * PAGE_SIZE; + continue; + } + /* + * get the 2nd mapping details + * Also create it if that doesn't exist + */ + tail_page_pte = vmemmap_compound_tail_page(addr, pfn_offset, node); + if (!tail_page_pte) { + + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } + + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*tail_page_pte)); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + addr_pfn += 1; + next = addr + PAGE_SIZE; + continue; + } + } + return 0; +} + + #ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping(unsigned long start, unsigned long page_size) { From patchwork Mon Jul 10 16:08:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEC24EB64D9 for ; Mon, 10 Jul 2023 16:10:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F6428E0008; Mon, 10 Jul 2023 12:10:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A5F98E0001; Mon, 10 Jul 2023 12:10:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56ED48E0008; Mon, 10 Jul 2023 12:10:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 44B708E0001 for ; Mon, 10 Jul 2023 12:10:30 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ECB2D1401B6 for ; Mon, 10 Jul 2023 16:10:29 +0000 (UTC) X-FDA: 80996189778.02.DABECE6 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf29.hostedemail.com (Postfix) with ESMTP id 6DAF7120018 for ; Mon, 10 Jul 2023 16:10:27 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=CgvRAlcw; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf29.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689005427; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Uid4soVR4vf6s+0fcrZuIzbN65q5Imh7atR87IkTTko=; b=hS3YZKjBDy+AISEcXEvPBeoyflQtgJk8cjQdQoz3YZkRRMv/ZfJLoGrulXMymHqB6Jy8az ErbtaNSEIWTfXm+j+5DlSKRDZt2cNKmCI9oJE8Jq4iwVoCFdfa+kMt2fo3AbkPHB+5Tfeq P5pkPN4/UknypJ03MGsjlD1myMuzBms= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=CgvRAlcw; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf29.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689005427; a=rsa-sha256; cv=none; b=TM3A3WzmZxcp2isFJNU+fUBDW+ZRjEUvruSskEOHJJt6RAhEZKIs6GblsfqYHa+RgXOg26 zUWfRh6N8mOAqaiAzlu2kRV4/4BSIIzXjBm79so8REAGKExvXH9GtQ0E1EfNuw1/Jz5DXW k80sX1yKOnaYwtFuO7fjLNvjyAoEnNM= Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG6u0x022271; Mon, 10 Jul 2023 16:10:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Uid4soVR4vf6s+0fcrZuIzbN65q5Imh7atR87IkTTko=; b=CgvRAlcwyba1I197R2EOhpJQIIQVKF+9IGqLiULxMDRskJR9D/inatzkhGOQZ2GPG9J+ rdm+mnfXhmoixwMMQvMbpnD58/0K9DEOviooBXEq72y+M9jM95GjX+T+71dvg+YCjk27 GacmcR0bESDn7l3XaHVIFP+Ax4ysvCTz6lAkX3TWpupX4o1PPl6bQe1uMfJ0k1z48Trj oriBxHPl4hcFuUYxMOWXLP1olwqYRzOg6fx2MJI0OaN8uNwCO8hWb0oOdTXuefA3vRm+ fQcrOWNukJ3Y8FhQDHiE+O4eTj9DYQTjwV3vio+TWt8hQgDy4D0HNVdrUesTShWFsJ2n NA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8h4r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:15 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AG6xaN022700; Mon, 10 Jul 2023 16:10:14 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrmxh8h3t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:14 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AG0ZUt014443; Mon, 10 Jul 2023 16:10:12 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([9.208.130.101]) by ppma05wdc.us.ibm.com (PPS) with ESMTPS id 3rpye5jnjq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:12 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AGACGF1376908 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:10:12 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EBB7458056; Mon, 10 Jul 2023 16:10:11 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 656F85803F; Mon, 10 Jul 2023 16:10:05 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:10:05 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 12/13] powerpc/book3s64/radix: Remove mmu_vmemmap_psize Date: Mon, 10 Jul 2023 21:38:41 +0530 Message-ID: <20230710160842.56300-13-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: VUzHjfLKFHgqXDBi2xMZTPUKtkSnUPyD X-Proofpoint-ORIG-GUID: vs491LrjLAIg5rN828IjzCyS8EodEKDd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 phishscore=0 malwarescore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100145 X-Rspamd-Queue-Id: 6DAF7120018 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: cwhtkh1fh5oj91r3k38kdw3ssgieqk6n X-HE-Tag: 1689005427-766338 X-HE-Meta: U2FsdGVkX19EyGM+sbUv7dIaUNOx1ExbXzG+1moZxXSJLd31is32duyCAdEtgWMt7oAeyAC6yTFExITum36MsdFW5jJ6+v02iLksFEGes20Nxlp6Zjbi3rOT0YbujdD8jApwBO/+Ua+8MAQxYd0+GbxmVGG4sLnsHGFpZKvY0xgJw6KR8Zvr3Cvw33X4VsNnSs/fVq2ut2/Pbq+oHKJbrXQfKF3fsrN6IBWCeJPY1Psk8U2cayEpykULTNmw2JTdIe2s24ao7d5l8oGI9HjbVtUQCus3ycOZUOk5cSfW3F6nI0cIDGYnYj/eeeh+Uc/jcgakJjr2ykuBm6RKW6TJjlV/JXWANnZd7swHXloFsV5MQ9WojkwvBxFHcy5Kg8PCcY4rwUvhtqeyG8NJIRCPVEz7wIdvjtA6iunpM1Cy/VzpxF9fqy8YOYM2gi7MeZkJZ7HAfyctHAFrOuYJwv8HLh69gwuGKQyNAIGEMvqyFxjoUbFIvMEex3lfwmy3fBbgh87sMbCRinfdsYA/K5XOjw11ZOvBwAixM92fxo/tf0YRuZtGeNYus1h2iFAu8F8UUUJRVCwQqp3scM01AjcMrNw6hPebiEgmePyzTjTxyx6tYhg3kx+9rxXkLn1Lb5CbdedzGcFirZMn4bl0mgTs0pRW0xNdh3qHSeiWXKwkdQNQOvW3/x3HBfj0Zk1Rzq95mMZTuqlrtGsZFm6rrB0KHSwGfBMx/9uqmde+QC6saBdDBsyaP6l7WJGblp8l3+CItBffAFtQdHTx7ers5hRLqqLNgexGzyWKs+zva0Sca394ng1zfeDM46IV7AMtQBPBeZGaDVFSZi9CmaRW6Cv+vuxlzJAefkbc9mTCuar3DmowoO86pxYB2Vs/DbfoDmASlWZpT2DhvA+HrN6fXhvuNDlodvE6uHRngoyag6FIHsHF5Ea0s9102WHS1it7r/CBm2KQWL1ZLRa/AgfdFLf y2rQoOZX OD6tKGlnWFSwRcFC0NpTFpy79yp306vaqoVwTZ5kZrE2dDSbwwhDOU4da3dZ82LTZ0Bm4fdo4XyxrxpFHxQCl1iwaKyf9UWvlAgqbjxk62ZD1SHh/RMBsdWTV2mL7Mz4Rki2Z/7zdyUotuBJLILhjsM8UfgnI4OKe6bb9IjRA4uwMNreB2+J2i8vmZqRgHOj47pkq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is not used by radix anymore. Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/radix_pgtable.c | 11 ----------- arch/powerpc/mm/init_64.c | 21 ++++++++++++++------- 2 files changed, 14 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index b492b67c0b7d..73d0987369ff 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -601,17 +601,6 @@ void __init radix__early_init_mmu(void) #else mmu_virtual_psize = MMU_PAGE_4K; #endif - -#ifdef CONFIG_SPARSEMEM_VMEMMAP - /* vmemmap mapping */ - if (mmu_psize_defs[MMU_PAGE_2M].shift) { - /* - * map vmemmap using 2M if available - */ - mmu_vmemmap_psize = MMU_PAGE_2M; - } else - mmu_vmemmap_psize = mmu_virtual_psize; -#endif #endif /* * initialize page table size diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index 5701faca39ef..6db7a063ba63 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -198,17 +198,12 @@ bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start, return false; } -int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, - struct vmem_altmap *altmap) +int __meminit __vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) { bool altmap_alloc; unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift; -#ifdef CONFIG_PPC_BOOK3S_64 - if (radix_enabled()) - return radix__vmemmap_populate(start, end, node, altmap); -#endif - /* Align to the page size of the linear mapping. */ start = ALIGN_DOWN(start, page_size); @@ -277,6 +272,18 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, return 0; } +int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, + struct vmem_altmap *altmap) +{ + +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix__vmemmap_populate(start, end, node, altmap); +#endif + + return __vmemmap_populate(start, end, node, altmap); +} + #ifdef CONFIG_MEMORY_HOTPLUG static unsigned long vmemmap_list_free(unsigned long start) { From patchwork Mon Jul 10 16:08:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 13307386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6750DEB64D9 for ; Mon, 10 Jul 2023 16:45:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA6306B0072; Mon, 10 Jul 2023 12:45:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B57236B0074; Mon, 10 Jul 2023 12:45:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1DBC8D0001; Mon, 10 Jul 2023 12:45:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 90E866B0072 for ; Mon, 10 Jul 2023 12:45:04 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0AB64401B1 for ; Mon, 10 Jul 2023 16:45:04 +0000 (UTC) X-FDA: 80996276928.26.C214026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf10.hostedemail.com (Postfix) with ESMTP id 5F3DBC0016 for ; Mon, 10 Jul 2023 16:45:00 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=DUaNIuXj; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689007501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XCQvO4AMwAl3UVUAgFYIr0Tyqbjq93TFj9IeImIxS/c=; b=1CcGC4h4+6xiKiyY8x84TAjbART7nRbMlLOcSQ0AA0SZec/BInwnLAc/skmwHsCT3yOJd7 vEMkLk0m3FxeX4Ya1KTOscRrLOIlm7ZfdsLe7uh49usDGXkItRXOMcR2IYSWavUuBu6RaF 2KcU+h0Gu2AkxSGOumnXuSvxD3GdArQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689007501; a=rsa-sha256; cv=none; b=ZZGEGlkf4IEeYeU+RIr/EMreNmkd5EL7DLke9AI61QrQVO3zGakfLoUEOvt7FVkUsvLuL4 ADmLVH7xLZ9ZTmVy3588sw0mUROZ7o4rjtnA3UeLILvn1QHSYH4wa6t1DGanLb1xjkIJot jPQ+m+zKFMGhqg4+QgKLzqmgDdIN1m0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=DUaNIuXj; spf=pass (imf10.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353723.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 36AGHurW010516; Mon, 10 Jul 2023 16:44:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=XCQvO4AMwAl3UVUAgFYIr0Tyqbjq93TFj9IeImIxS/c=; b=DUaNIuXjPrGAiW08/9O5WqZ2YaewIiMP/qiGskSPK7RG0ORefyt6HgdTiE3dSMgFhPBG xlYhxs+iW9uF5yKJ97lyH0Ev1WTLZ4qjPM2SdRSIPejtS5zLInQM5fwCrGJomDTC2XqN MPelDvTB8c+uRLWwuYs1C+WiRMO112fllJWVDrWN/YpEv9FXyPD9T6AtMl8iV6jc2RnG nQ79QDbpYRxvhtS+pG9jWQMNhzpk/E3uGNkbCpPjVGlfajJnmRLNU5djEnhrmI8V3RGB 6Q+SI5vod/8mFiyltET+jWxOlR6FlbkW+QLW/urL9n+WTdw6TGg0vpidvsywBJAZc5w8 tQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrnckrk8t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:44:48 +0000 Received: from m0353723.ppops.net (m0353723.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 36AGbN0v001960; Mon, 10 Jul 2023 16:44:48 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3rrnckrk89-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:44:48 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 36AEDLte016682; Mon, 10 Jul 2023 16:10:19 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([9.208.130.100]) by ppma04dal.us.ibm.com (PPS) with ESMTPS id 3rpye60kx9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Jul 2023 16:10:19 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 36AGAImU64094606 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Jul 2023 16:10:18 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B0F5D5803F; Mon, 10 Jul 2023 16:10:18 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BF7E058056; Mon, 10 Jul 2023 16:10:12 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.43.9.86]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 10 Jul 2023 16:10:12 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org, mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, christophe.leroy@csgroup.eu Cc: Oscar Salvador , Mike Kravetz , Dan Williams , Joao Martins , Catalin Marinas , Muchun Song , Will Deacon , "Aneesh Kumar K.V" Subject: [PATCH v4 13/13] powerpc/book3s64/radix: Add debug message to give more details of vmemmap allocation Date: Mon, 10 Jul 2023 21:38:42 +0530 Message-ID: <20230710160842.56300-14-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> References: <20230710160842.56300-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: UKYBQNzylw66poHeRX3bseGikd3W8RyS X-Proofpoint-ORIG-GUID: GCRqle8WNpa5R1TL9XzHcq0-XjHrQdOb X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-07-10_12,2023-07-06_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 lowpriorityscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 clxscore=1015 malwarescore=0 mlxscore=0 adultscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2307100149 X-Rspamd-Queue-Id: 5F3DBC0016 X-Rspam-User: X-Stat-Signature: b17ubqy3y71xcs48gok9kzdrfpzu67do X-Rspamd-Server: rspam03 X-HE-Tag: 1689007500-820664 X-HE-Meta: U2FsdGVkX1+B+HTjNbBcqeic6MaH+7qOCi7iJMDgcXw/+vIHN4NlQhsK9m1PiNCMrgc3gLFc78j/0kgU0JrMAqGNCEHbFnUbXeD6oeeNsVhWWEjQrPOQS0cCZISfBw5ayg83RDjKE3CF0+9ntwPVcUpfKPvrgtaDgLICGEltIaL0QHA54FZO8VKlKnrnyWgjuLe96ZawrOWZQueama4jnUSmoDt33c6/SGmtf/cIPP43dA3O+NYgjSoXWtUW9jBd/xM5oF7GPO4jNPNsDyOgucgEGF3dA/RStLrOXfdsGDS1Z/onJrHqWUqsDHScOxPtSWH/8DxFGjX7K2RZDnNwZROYIA76Y3WuAWFnaFLSQoCq6jIFjefpO/tkDXNAmIJ/lrRliRtEArFBgitSk0HUdsfoALdPqX1LZYOxAdpdzzvSCR6ePVD9zUxNdMMIICJdeoc9sY1TWcPDjcggCWBmee8X6LvgdLP/xa59FeqQ3JKiCRF+dkQSa9X71E1rCQyPbY6n91bZCuKbzSHa6WjryNSZg/Fu9tWXltkEnVHLeIzp7nNDuJMgelYopWZ2IpkscZ67Q2Vg6NMhOCkh3hGhjjVVltbILf7JoUnM8AKEWEnZd/XOGarr3RDUt54wH3hisLJ6wXfeUOFj6swaVHijkOZ0t0pEjYjyE42r2D9Ft1CeArAU2emVnbujZEWeXNOwLDN6ZI7pErc/QaPG0mgak4FSHGKhdqTpMv9MZ6ewgNIOeiQ8qqmOBqt5MXxGHH81BEMDxFQBENyc5eZkPfblhrzMM4rtmc1PfhkuRMPq7TBmBJtADzwZEAxkrPxSVcP8/0eNPwYXiLCnxBdiXjql/qf+5hjd1ZcJAGJ2E3oiNx6I9PZZdA1vjkKdyU1YVR+H6okeiySZC9y2U759tNait4InJSbni9ZbIBDVijkBOdg2htKK/poA/F1Pe/VcrCm0YkUojDqKvALOEhmyNuZ /QYcOGmd vkxdRQwhYMMZbzd6HIx5IvpCMmicKB4GYQj2xUZZSSWwoZBgdvAhB297TklyWi9RQ9VyAjteOENKXwdPT4kn79UAPdJMydR9j0W2wB7tVNUmvkaBDrGrPzwelHLm6pZxXHMgTwiYIGoKTnOE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add some extra vmemmap pr_debug message that will indicate the type of vmemmap allocations. For ex: with DAX vmemmap optimization we can find the below details: [ 187.166580] radix-mmu: PAGE_SIZE vmemmap mapping [ 187.166587] radix-mmu: PAGE_SIZE vmemmap mapping [ 187.166591] radix-mmu: Tail page reuse vmemmap mapping [ 187.166594] radix-mmu: Tail page reuse vmemmap mapping [ 187.166598] radix-mmu: Tail page reuse vmemmap mapping [ 187.166601] radix-mmu: Tail page reuse vmemmap mapping [ 187.166604] radix-mmu: Tail page reuse vmemmap mapping [ 187.166608] radix-mmu: Tail page reuse vmemmap mapping [ 187.166611] radix-mmu: Tail page reuse vmemmap mapping [ 187.166614] radix-mmu: Tail page reuse vmemmap mapping [ 187.166617] radix-mmu: Tail page reuse vmemmap mapping [ 187.166620] radix-mmu: Tail page reuse vmemmap mapping [ 187.166623] radix-mmu: Tail page reuse vmemmap mapping [ 187.166626] radix-mmu: Tail page reuse vmemmap mapping [ 187.166629] radix-mmu: Tail page reuse vmemmap mapping [ 187.166632] radix-mmu: Tail page reuse vmemmap mapping And without vmemmap optimization [ 293.549931] radix-mmu: PMD_SIZE vmemmap mapping [ 293.549984] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550032] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550076] radix-mmu: PMD_SIZE vmemmap mapping [ 293.550117] radix-mmu: PMD_SIZE vmemmap mapping Signed-off-by: Aneesh Kumar K.V --- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 73d0987369ff..2828e7e0802c 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1034,6 +1034,7 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL); if (!p) return NULL; + pr_debug("PAGE_SIZE vmemmap mapping\n"); } else { /* * When a PTE/PMD entry is freed from the init_mm @@ -1046,6 +1047,7 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long */ get_page(reuse); p = page_to_virt(reuse); + pr_debug("Tail page reuse vmemmap mapping\n"); } VM_BUG_ON(!PAGE_ALIGNED(addr)); @@ -1155,6 +1157,7 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); if (p) { vmemmap_set_pmd(pmd, p, node, addr, next); + pr_debug("PMD_SIZE vmemmap mapping\n"); continue; } else if (altmap) { /*