From patchwork Fri Sep 6 17:59:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13794493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 836A5E6FE38 for ; Fri, 6 Sep 2024 18:00:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17E326B0085; Fri, 6 Sep 2024 14:00:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 105B86B0088; Fri, 6 Sep 2024 14:00:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9A706B0089; Fri, 6 Sep 2024 14:00:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C1F256B0085 for ; Fri, 6 Sep 2024 14:00:25 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7328C1A0E0F for ; Fri, 6 Sep 2024 18:00:25 +0000 (UTC) X-FDA: 82535078010.15.02F12C7 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11022078.outbound.protection.outlook.com [40.93.195.78]) by imf05.hostedemail.com (Postfix) with ESMTP id 9970E100026 for ; Fri, 6 Sep 2024 18:00:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=os.amperecomputing.com header.s=selector2 header.b="u6nxFC/0"; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf05.hostedemail.com: domain of yang@os.amperecomputing.com designates 40.93.195.78 as permitted sender) smtp.mailfrom=yang@os.amperecomputing.com; dmarc=pass (policy=quarantine) header.from=amperecomputing.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725645525; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=1hfMzbTuMCk2BcCguQ6lifNwEvSxz/uuz5HbU3VKr5A=; b=lDlgdxnob1phk77wkhVnqDqoJDmJSPWCK0LbKsZOR7mkquYMii+7422mOiQ8yhaFtzyeub r+gAAX7Zt9klrLCtD7vPcxeZR2AuxxOXj6aJ8Tbwqj2y6jZl5jEUzR0zLTTxtne+ijj23r mEej3TRXyS7ntjfSvOZaW7rx91DeFyg= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1725645525; a=rsa-sha256; cv=pass; b=7itc2mlGBj34AXngWs48ahehsiuFTeYOhSx8J4yv7+E5SpVtgZ1/Kvvt1iyrQMaM7clTZk BuZl0xgzh2ASCAi8UjmuTlfXp7FZ/Ouyy6uHP8/BggawlT0mbq5WkOcKZ1HLV7pluNmFDH NIGncaTmrEoenb9IloLVv26b/R+oA70= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=os.amperecomputing.com header.s=selector2 header.b="u6nxFC/0"; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf05.hostedemail.com: domain of yang@os.amperecomputing.com designates 40.93.195.78 as permitted sender) smtp.mailfrom=yang@os.amperecomputing.com; dmarc=pass (policy=quarantine) header.from=amperecomputing.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=C+YKa7u1GzjmNM2703Ob7xNtHm6XOroRTtAxzrT6d7W3Jf0GwGPi9LMyXJ9eZs+R6iKUElw2YZ81zkvaKwyQOXnxAp7SDgsgfuPkc4ZCuYWFa5soQ2BGFtbO51jNlRMma7GsPuy8Yr87u7pkLEs0aFESqQWohx2EcMtfrJGIjT1KN4laIAk3trmRrI4eGYgq+edioZMa5t5FKv2Fa6Ajnn9RcCUliPuksDsLZg7+iB/HW1dJ4dkemnVy3A/ap6TCne6qspNMBQkk4vcL29ZbTPZY7Fffg6lKYn872jNIOBn+hJmenvLfaqK8U3mcvL2wTMxEIaqE85vgo6ZGsRS11Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1hfMzbTuMCk2BcCguQ6lifNwEvSxz/uuz5HbU3VKr5A=; b=ubx1hkyrXzW9MzvfEaQlqSncZpcjedR2tKTekupEOBbgOHBAhkom/AbDPLOzKjARtg3i5KmXdA+949f4evo+F7GcvreCTlfkac4le6PIue4qm3Wbbv3NgBRjVUYjCpG3/RjYCAQOEAW3WKT8oJbeUpXnKGoS+F8C+hx+DnwDt7UAVEuKYbZNlk1eJoHKnPDToMPWdZ7sM5ICseXcTwrq1egS0kHIAZpW1WkjufTSvlr4Da2s8u5SRdt43lNgSkzTlK7lT0njyPQjdhvCgeQphBRLDfd1fqkRRkzXYkfXgHZOSk76If6DyZIiA8QueRVZukOYfUKuOZctIptqpzgktg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1hfMzbTuMCk2BcCguQ6lifNwEvSxz/uuz5HbU3VKr5A=; b=u6nxFC/0w4hZNOXtqkiBlNSdG+pM5Zx0Gi9gFWaacIM6pU3s323kcOamow7Z2bvDkfmRFChbVV5sOmymTxwg1ZE/S7B/5ZmasEAWN9rb15wOm97k/T3D78DEfMSUCtMIFO4vqORQVX/keq2qjH/q/Qr9Pm07zz+qFYuog25xTAw= Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by BY3PR01MB6561.prod.exchangelabs.com (2603:10b6:a03:362::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.27; Fri, 6 Sep 2024 18:00:04 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%4]) with mapi id 15.20.7918.024; Fri, 6 Sep 2024 18:00:03 +0000 From: Yang Shi To: catalin.marinas@arm.com, will@kernel.org, muchun.song@linux.dev, david@redhat.com, akpm@linux-foundation.org Cc: yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 1/2] hugetlb: arm64: add mte support Date: Fri, 6 Sep 2024 10:59:44 -0700 Message-ID: <20240906175945.3636193-1-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.41.0 X-ClientProxiedBy: CH0PR03CA0241.namprd03.prod.outlook.com (2603:10b6:610:e5::6) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|BY3PR01MB6561:EE_ X-MS-Office365-Filtering-Correlation-Id: 444664e6-a4ed-42f5-dbed-08dcce9dbbd3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|1800799024|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: Cl9uEdUoNIElHfS93p2NNRcWaq6AbWgLtFtu623Ph7/Fu97nT5dYEIPdCiDy5rfCqz5Nupq0yYlPePOH/hololzzGFD+gm1piqF14CNfAQ5NEyQzRlfw0bVRcLGQtWrNX+xxkKfAqJ0lWJA709Rb4Ai1EXIquuXtlMAU+aquXiZBW93Yur6mNsfKtogjM/M7HLF2U1XegMjwzEPoeyD4QOmxhdr9GPaP9lyRI+CO8v3cCzrJodzhgm4EUtD1UHB1BkgWj8iRHK50mLMwU1EMREIr/lnmuvZoeTR/Oz7F1O+FIbhzOkIZ28utwonz/dzHpU0xuYsedAX/qCTcQ1t8HZCG6WOENBQJxAyUy/TPQ6zUNX7MU7UMFUE9Tax7H9HmPBHRIhsZlQbSqM8//10QQVButJwEOS/e96D0Ygknao3jNTqg8vL1eCkknJmFeJMRiEzeVhovXyzJUTYXZ982d/+lbVhTYL57TtkF8ipjTIv7urt2YDPhqxlBhvbFoxLzgIq/nW+ogEwRIF2YWkpyliDtUSrrWrhJCoiJbEhyFimxLEZTNgT6ewkDoE0TsuhMPgS4g/sP1ddHqtna6XNrsQX1RGnGqyI2Db0zmO+6JyYgYSiWHMSmhd1fqmrTJaL3BvCJ6hU0RKlYOiNi7CeyLac/jvB348d65ib/YEXmNKtI8aktb2h38lWBHAkqtZBFZkJOB405WneEUC1DT+JLcMtZl+M6uYLWH5P68ykPPufa/5+1qPXMYo2A6tJQEf9nTeSMxSsyl0qEmDCHxKrg6zg1biypuAOcylFFjUpqLYOOIlGQlzQ5lpJkM8qUXuaHbkNOzWOu21fa60obJph0Drkv36RUG0Z1rM7F6zPuAdQndXNHLZH+oW0ZuJDAhlc03az133Rjm8sFst2efzHf30bp+b8LlT9kbDC4YgORBDOjmWs2iRYRvQq6hOg/sh+xQy0fSIZ1T5uzmY8Relvrr5lCTnH7jy2wAPEmWAweTfq4AgXs2wJy0Tr+8+h5IbYaN18AZPwDCC8sUcqUl9/mZj05jqO5i3lwu67+Bcf21+4YvShXcaJKw8jc1xTfH/bnwqzCGZDQNn2phccKSuzorYHmHF+ECGfuJHjT8jSMX9w+lO+AS77C50OKQirWUFmlMWEiBeW39R+JuzP0BEUSAM2jsin3LrJ9UtlZzsPuhNESm6v1JpjAwc5OfbHZpTQIKIuEZ4ZkuzFe7Eg4VWiocbMtcKlGNXqwMnyyXII3bB0szE6t88XvZc5CF9xXLQTSuM9GYRlXDxfPYCI78ZKVchgPA3W+Hat+vLG6MnxAsWrUZOT0g45gOdQ7jVkWgjyZAjswTsKvz9QR8+lkRP1Hki7zF6M554SCb4gLML8OiwYsVFSPItdMOeAcFDqlXDixB++wPtqmBCaLZLGCEm1i2A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(1800799024)(376014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: J6glGUrqq82Qzk60n5X30RfMGWihMaWfwTcHfaNUeJuc3ua/Qs2IiKFQztG9Z76eGGXSIuUPL3b0hjDuaL3DHBF3h+U1WVzTzPcs+ZxpG6uM+eilmFsbN5R1gHqjqoOizpHIu9v0k7RpceElReWjjWpdfM5Y0DYIGZJdSacfG/HoRDy/0Hxd6tG1I3vIFDf46nm9DMiDggmm48bisBpbek+b+QedwB2xJSK3XWCoJZ6q66OBYiDQYxcjpQYgTjAjKDd7O3JZH5G1un05oJNVVl7PXJw0c1dPbqnJa5ctvYgkj9WEn39UTmyMXnCBFBAiXkECHps49CwJD0r5gaoLnFvuvfnOcX/hfj+sIykpg3Ryu6jF+3Jkfrw/f65uAnCCAjFzVxKyFkYtKseHkmOuHK4C+jIbE9rd0KOatUb6DOHhkk2NITGKnFJCXYvLT3uJLBASAgARlOWMBrtQi2+9LhEIevSyk9UH+T4deUxtlf4y8ihRSLkNENiZs4Z0j1oq0Yz2M/sLoRw0/sSN2/6tOxlv8vi934Vk4wHmrkc2sJVwksDlowkOXlCN57Z1kF1qMaS1dLptN6lcW6S/eLjDr02u1LwtZdrxrOseb5lfBsngo/8rNVN/ZE8Qa837kpH2tBkI2XtmnPLCdEr+erVIX5ERLmPPWcTg59cWTXpxwsjJhjiGJA0P5Ef8ne31+1GZ+9q+UJLeP2LkmYfPxRblN5Nt5CO9ObfezYrX7TmzNM3ii3LuEOIRQsJSJkluvrTfiX5znRuLrghSWlXo4XGwtKZL4C7sBAPKQOihuTbNakuWFfrvc5njx00ukH57VI6eI33LMei8cXZSz2UOlWQEIG4iVc945ERoO8X7uOw/0hBZFUk/aCQe/F2f23mjT9751koFJsGB5fIsBc1F1B9bvDOmKe0IHyIilPy2xW/WD474/JaQh4RopU1LeO3bwgSbl+21YNVKJCgAYhPEBDHtfYo4V235jNcf+pHvmhDjsnSDqW69aYc107hh6/qpe57Zdoh4vw8N8fQHP0NQeF3hRf0HH7fS8cGqY9Yk8f6jcyUa1GPIWpE/Om2jVDQyAsF/YxBUZvCAFwGYajoj247IIxKz1d1OZHa8pGvyec4DUY1jYJH6YxzcjV0rSTySS5wygjDZZCdGAo4qMZ+MR340dFF/+f6Ebe0hrBO6gaVHgX3RHo7GEUFI9JQz3+XiBcvDCVln/TQ1fctQ1Hb7D5DWhR7LXktX061zm0Y8f06HTtLodBSwdbmF9IIbBBdElkXPhJBUGx7+iClkxPIKVzvpaTB71eyyS4aaTEoANKcNMcyN4IpujxUwYKY4Z78oethcmintWxRNkEJ7FEX4yzh/8A7uHU++k3FpxNUNysnQfUQtVuPUInH9nM3PmpLtbhBc6NjS5va1XrJ+idp+mpCx7wS0XAUEM1Ml4CEVd082TYuoq5xqu0mV1Bui/+as0LkMsxFvKbzgLHebHKEWUU6jAIBqlNbOQpocbwRkR4hnfh3NSzo/AVUDCG4jg2LNxkVYjimJLhh7VaKCV7jc87z641ldBr6MnZf3aL11h3FeqzPnkN0kVVqIKyxvazdqQ7HYtzjNhBKLIdCDOwfX3yMhY+frD+noFUszxYwGqrCd0I0= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 444664e6-a4ed-42f5-dbed-08dcce9dbbd3 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Sep 2024 18:00:03.8296 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AxUzZnjsb1T08PUhBRNIY4mwWohY9ygtlpmUI5CeOtNRKw/A+gYiWMRXgQIY3Dq+MhTchae+2+SwYSx93yBFZeOaGaATBzRJSAF6pizlRik= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR01MB6561 X-Rspamd-Queue-Id: 9970E100026 X-Stat-Signature: q3pd78rfxqm57xpbjqnyganbt96wmazk X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1725645622-851405 X-HE-Meta: U2FsdGVkX18mzDwUEdPTNHN7WN0FNDQ4mXwpfb7DjELYqU2NguR/s7/UhEPpFeVHEmHp8aVORTSGsuh7bwOsK0sTiNSwycXQug3Vugkpfnbz0LptmFdBoX1Ms+qza9hH/MvNnx6UW/XEd10OZfMNSzAaY5kkLiVqHVBfj6OV99TAbRhXXHgvBlnB3yy0t3PjAsKSFDTMJKj1KAcbKDG2JSeb3ie6a2f5sRzZTHrnNBRq9DSXZwrn0jJc7Pp4QYn/OXXuaKfWKJ9upMqefs/Hmgyfkpc4KOXfVfBKN3t+ue5o6ojssU4H26Py2ab8VFu4ur8VlUcB9c0UBxRoLdoBMrigClERtueXNYpkHxqSpiVyyv8de0dBnxVeebfggBgq/THs44K6Vd5fUlMk8BbD/8veko9tW0Uc71HOtmyGjKDD9sHYn24ErFz5s1GrnfLpxYRVzA7mJwzxvc2SUlW9nxgh8N0rt6i98qGh+mdsVimhPl/4izXgIdNDhCGI1xkwVdBlF6/EFXMJwGqKoSgeRtcWFI/P36jUacrOr7k3y8jo8UJeJjHPetVanWX9i4MKopeHhVjzuhIWpxbShUBU4IPhXPJiN13vC/IbN9BdcFejk8hcpMykCKsJswufgpGGhbclqcJrePGTwvcByK+Zh4EiGEjuOGgG96unL7zI0/6qmzeyy+jwrfGKPEH1oAt1ZvNqTkAAhL9E4CJfSltnMxz5NGaHl55h1Cz+l5W3mX+272aWV15gDFdSTQAptjC3bdjhRmVvGUCPnUERqW+2H9t50auaSQf3sJdRHHSGkwewgqcgbjQ06gWJq5MC3w0s5da5xJ8kdiKwbQfNHtj0i8gVFcGCDHSmuM8SMYoDMedzi6PPIWP1RxPwnQn0B9nAUtbUvARAqZVkE7WLyUKoEkTQ0q5vWzOZnbGagvhwfn5B2w5fUwipAaJR9kZPc88NLcgy/Ux/V+rgH9x4G2x aT6jDG9M MrzkBqGuIOinixJGkWcxuaz37is5MRZSNYrC0WNHLHpiuG0MVWyBWJxhBY3mcAclYqWxxYejft/FzdftMU82GJEWtn/JIx8UGYedGM4hQ2iM4Dp9e+egoCOxpuUjatKMcg9uxpX9tcIsC1mXlYBA6wnfiIAZx/PH1lSpw/wH0Cr8kupuMjIEknhYe1F5dIigWmydFYeNqxH2kwClas2RlC8IDopoAWJWSFYuvhvxc+rE5VNzINnCP+pHGv5muttvwXcuCoBldPqUuwsmN1pYOi185MsCFAJtdRzCjUpFuZB2vl7bTWM4aPh3DaYZXTcejM725M4ayC1WsNcMKp6XADxqQqjfg7p83PW5He9l2aJwCkNXmfwylGIvjdO5EvlLtgEIr3H+vPYLREh6BmtzjTY3Yt9j30xHtEeuA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Enable MTE support for hugetlb. The MTE page flags will be set on the folio only. When copying hugetlb folio (for example, CoW), the tags for all subpages will be copied when copying the first subpage. When freeing hugetlb folio, the MTE flags will be cleared. Signed-off-by: Yang Shi --- arch/arm64/include/asm/hugetlb.h | 15 ++++++- arch/arm64/include/asm/mman.h | 3 +- arch/arm64/include/asm/mte.h | 67 ++++++++++++++++++++++++++++++++ arch/arm64/kernel/hibernate.c | 7 ++++ arch/arm64/kernel/mte.c | 25 +++++++++++- arch/arm64/kvm/guest.c | 16 ++++++-- arch/arm64/kvm/mmu.c | 11 ++++++ arch/arm64/mm/copypage.c | 33 +++++++++++++--- fs/hugetlbfs/inode.c | 2 +- 9 files changed, 166 insertions(+), 13 deletions(-) v3: * Fixed the build error when !CONFIG_ARM64_MTE. * Incorporated the comment from David to have hugetlb folio specific APIs for manipulating the page flags. * Don't assume the first page is the head page since huge page copy can start from any subpage. v2: * Reimplemented the patch to fix the comments from Catalin. * Added test cases (patch #2) per Catalin. diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 293f880865e8..06f621c5cece 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -11,6 +11,7 @@ #define __ASM_HUGETLB_H #include +#include #include #ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION @@ -18,9 +19,21 @@ extern bool arch_hugetlb_migration_supported(struct hstate *h); #endif +#ifdef CONFIG_ARM64_MTE +#define CLEAR_FLAGS (BIT(PG_dcache_clean) | BIT(PG_mte_tagged) | \ + BIT(PG_mte_lock)) +#else +#define CLEAR_FLAGS BIT(PG_dcache_clean) +#endif + static inline void arch_clear_hugetlb_flags(struct folio *folio) { - clear_bit(PG_dcache_clean, &folio->flags); + if (!system_supports_mte()) { + clear_bit(PG_dcache_clean, &folio->flags); + return; + } + + folio->flags &= ~CLEAR_FLAGS; } #define arch_clear_hugetlb_flags arch_clear_hugetlb_flags diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index 5966ee4a6154..304dfc499e68 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -28,7 +28,8 @@ static inline unsigned long arch_calc_vm_flag_bits(unsigned long flags) * backed by tags-capable memory. The vm_flags may be overridden by a * filesystem supporting MTE (RAM-based). */ - if (system_supports_mte() && (flags & MAP_ANONYMOUS)) + if (system_supports_mte() && + (flags & (MAP_ANONYMOUS | MAP_HUGETLB))) return VM_MTE_ALLOWED; return 0; diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 0f84518632b4..cec9fb6fec3b 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -41,6 +41,8 @@ void mte_free_tag_storage(char *storage); static inline void set_page_mte_tagged(struct page *page) { + VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page))); + /* * Ensure that the tags written prior to this function are visible * before the page flags update. @@ -51,6 +53,8 @@ static inline void set_page_mte_tagged(struct page *page) static inline bool page_mte_tagged(struct page *page) { + VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page))); + bool ret = test_bit(PG_mte_tagged, &page->flags); /* @@ -76,6 +80,8 @@ static inline bool page_mte_tagged(struct page *page) */ static inline bool try_page_mte_tagging(struct page *page) { + VM_WARN_ON_ONCE(folio_test_hugetlb(page_folio(page))); + if (!test_and_set_bit(PG_mte_lock, &page->flags)) return true; @@ -157,6 +163,67 @@ static inline int mte_ptrace_copy_tags(struct task_struct *child, #endif /* CONFIG_ARM64_MTE */ +#if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_ARM64_MTE) +static inline void set_folio_hugetlb_mte_tagged(struct folio *folio) +{ + VM_WARN_ON_ONCE(!folio_test_hugetlb(folio)); + + /* + * Ensure that the tags written prior to this function are visible + * before the folio flags update. + */ + smp_wmb(); + set_bit(PG_mte_tagged, &folio->flags); + +} + +static inline bool folio_hugetlb_mte_tagged(struct folio *folio) +{ + VM_WARN_ON_ONCE(!folio_test_hugetlb(folio)); + + bool ret = test_bit(PG_mte_tagged, &folio->flags); + + /* + * If the folio is tagged, ensure ordering with a likely subsequent + * read of the tags. + */ + if (ret) + smp_rmb(); + return ret; +} + +static inline bool try_folio_hugetlb_mte_tagging(struct folio *folio) +{ + VM_WARN_ON_ONCE(!folio_test_hugetlb(folio)); + + if (!test_and_set_bit(PG_mte_lock, &folio->flags)) + return true; + + /* + * The tags are either being initialised or may have been initialised + * already. Check if the PG_mte_tagged flag has been set or wait + * otherwise. + */ + smp_cond_load_acquire(&folio->flags, VAL & (1UL << PG_mte_tagged)); + + return false; +} +#else +static inline void set_folio_hugetlb_mte_tagged(struct folio *folio) +{ +} + +static inline bool folio_hugetlb_mte_tagged(struct folio *folio) +{ + return false; +} + +static inline bool try_folio_hugetlb_mte_tagging(struct folio *folio) +{ + return false; +} +#endif + static inline void mte_disable_tco_entry(struct task_struct *task) { if (!system_supports_mte()) diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 02870beb271e..ebf81fffa79d 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -266,10 +266,17 @@ static int swsusp_mte_save_tags(void) max_zone_pfn = zone_end_pfn(zone); for (pfn = zone->zone_start_pfn; pfn < max_zone_pfn; pfn++) { struct page *page = pfn_to_online_page(pfn); + struct folio *folio; if (!page) continue; + folio = page_folio(page); + + if (folio_test_hugetlb(folio) && + !folio_hugetlb_mte_tagged(folio)) + continue; + if (!page_mte_tagged(page)) continue; diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 6174671be7c1..c8b13bf36fc6 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -38,7 +38,22 @@ EXPORT_SYMBOL_GPL(mte_async_or_asymm_mode); void mte_sync_tags(pte_t pte, unsigned int nr_pages) { struct page *page = pte_page(pte); - unsigned int i; + struct folio *folio = page_folio(page); + unsigned long i; + + if (folio_test_hugetlb(folio)) { + unsigned long nr = folio_nr_pages(folio); + /* Hugetlb MTE flags are set for head page only */ + if (try_folio_hugetlb_mte_tagging(folio)) { + for (i = 0; i < nr; i++, page++) + mte_clear_page_tags(page_address(page)); + set_folio_hugetlb_mte_tagged(folio); + } + + smp_wmb(); + + return; + } /* if PG_mte_tagged is set, tags have already been initialised */ for (i = 0; i < nr_pages; i++, page++) { @@ -410,6 +425,7 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, void *maddr; struct page *page = get_user_page_vma_remote(mm, addr, gup_flags, &vma); + struct folio *folio; if (IS_ERR(page)) { err = PTR_ERR(page); @@ -428,7 +444,12 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, put_page(page); break; } - WARN_ON_ONCE(!page_mte_tagged(page)); + + folio = page_folio(page); + if (folio_test_hugetlb(folio)) + WARN_ON_ONCE(!folio_hugetlb_mte_tagged(folio)); + else + WARN_ON_ONCE(!page_mte_tagged(page)); /* limit access to the end of the page */ offset = offset_in_page(addr); diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 11098eb7eb44..c0f4c2acc503 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -1050,6 +1050,7 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, void *maddr; unsigned long num_tags; struct page *page; + struct folio *folio; if (is_error_noslot_pfn(pfn)) { ret = -EFAULT; @@ -1062,10 +1063,13 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, ret = -EFAULT; goto out; } + folio = page_folio(page); maddr = page_address(page); if (!write) { - if (page_mte_tagged(page)) + if ((folio_test_hugetlb(folio) && + folio_hugetlb_mte_tagged(folio)) || + page_mte_tagged(page)) num_tags = mte_copy_tags_to_user(tags, maddr, MTE_GRANULES_PER_PAGE); else @@ -1079,14 +1083,20 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, * __set_ptes() in the VMM but still overriding the * tags, hence ignoring the return value. */ - try_page_mte_tagging(page); + if (folio_test_hugetlb(folio)) + try_folio_hugetlb_mte_tagging(folio); + else + try_page_mte_tagging(page); num_tags = mte_copy_tags_from_user(maddr, tags, MTE_GRANULES_PER_PAGE); /* uaccess failed, don't leave stale tags */ if (num_tags != MTE_GRANULES_PER_PAGE) mte_clear_page_tags(maddr); - set_page_mte_tagged(page); + if (folio_test_hugetlb(folio)) + set_folio_hugetlb_mte_tagged(folio); + else + set_page_mte_tagged(page); kvm_release_pfn_dirty(pfn); } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a509b63bd4dd..fc96becdc395 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1401,10 +1401,21 @@ static void sanitise_mte_tags(struct kvm *kvm, kvm_pfn_t pfn, { unsigned long i, nr_pages = size >> PAGE_SHIFT; struct page *page = pfn_to_page(pfn); + struct folio *folio = page_folio(page); if (!kvm_has_mte(kvm)) return; + if (folio_test_hugetlb(folio)) { + /* Hugetlb has MTE flags set on head page only */ + if (try_folio_hugetlb_mte_tagging(folio)) { + for (i = 0; i < nr_pages; i++, page++) + mte_clear_page_tags(page_address(page)); + set_folio_hugetlb_mte_tagged(folio); + } + return; + } + for (i = 0; i < nr_pages; i++, page++) { if (try_page_mte_tagging(page)) { mte_clear_page_tags(page_address(page)); diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c index a7bb20055ce0..5c8ef1a4a772 100644 --- a/arch/arm64/mm/copypage.c +++ b/arch/arm64/mm/copypage.c @@ -18,17 +18,40 @@ void copy_highpage(struct page *to, struct page *from) { void *kto = page_address(to); void *kfrom = page_address(from); + struct folio *src = page_folio(from); + struct folio *dst = page_folio(to); + unsigned int i, nr_pages; copy_page(kto, kfrom); if (kasan_hw_tags_enabled()) page_kasan_tag_reset(to); - if (system_supports_mte() && page_mte_tagged(from)) { - /* It's a new page, shouldn't have been tagged yet */ - WARN_ON_ONCE(!try_page_mte_tagging(to)); - mte_copy_page_tags(kto, kfrom); - set_page_mte_tagged(to); + if (system_supports_mte()) { + if (folio_test_hugetlb(src) && folio_hugetlb_mte_tagged(src)) { + if (!try_folio_hugetlb_mte_tagging(dst)) + return; + + /* + * Populate tags for all subpages. + * + * Don't assume the first page is head page since + * huge page copy may start from any subpage. + */ + nr_pages = folio_nr_pages(src); + for (i = 0; i < nr_pages; i++) { + kfrom = page_address(folio_page(src, i)); + kto = page_address(folio_page(dst, i)); + mte_copy_page_tags(kto, kfrom); + } + set_folio_hugetlb_mte_tagged(dst); + } else if (page_mte_tagged(from)) { + /* It's a new page, shouldn't have been tagged yet */ + WARN_ON_ONCE(!try_page_mte_tagging(to)); + + mte_copy_page_tags(kto, kfrom); + set_page_mte_tagged(to); + } } } EXPORT_SYMBOL(copy_highpage); diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 9f6cff356796..f944e8e7126b 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -110,7 +110,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) * way when do_mmap unwinds (may be important on powerpc * and ia64). */ - vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND); + vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND | VM_MTE_ALLOWED); vma->vm_ops = &hugetlb_vm_ops; ret = seal_check_write(info->seals, vma);