From patchwork Mon Apr 14 03:46:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 14049623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C754CC369B2 for ; Mon, 14 Apr 2025 03:46:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6D9628003D; Sun, 13 Apr 2025 23:46:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACCF2280037; Sun, 13 Apr 2025 23:46:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D1A428003D; Sun, 13 Apr 2025 23:46:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6699E280037 for ; Sun, 13 Apr 2025 23:46:39 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EFDD01615BE for ; Mon, 14 Apr 2025 03:46:39 +0000 (UTC) X-FDA: 83331262518.10.E6F3972 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf06.hostedemail.com (Postfix) with ESMTP id C2ECC180004 for ; Mon, 14 Apr 2025 03:46:36 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=dbtWDngw; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LO+ai67A; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf06.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744602396; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b=vSOzs+eoEnjfebOiiQBjEMkzyx8Ah1syaPXN+6tQZEpmYd7H9ejS9WfooOVdthHYaW/dlI TQi/+IYZdX5Yg4ZH4eCULhND8JOE46j1JWh4XDXUMunJDvkKMnOVmz+/wYWtURjXDg5tTM 7HiFN+5E/Qc0brzJUsPXLIrZNC1uvrM= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=dbtWDngw; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LO+ai67A; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf06.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1744602396; a=rsa-sha256; cv=pass; b=Y/Km2UTaXXfm5Ii5VrsUDsMWo+BSXZLiKYEnn7nurFqRiBtPQNqwJHQ3C4Oi6cQaPXvW6K 4fJ/Oh7o7Q8fIQCUicxGUQ+PGPwmcGsKrwHsxBEQzBhiaWMAGLhmUwUzZk6Jvsf9u1d94e 01EEsKfnAZ0oz5NaBQZ6XHBL+xjtDl8= Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3MKj4000707; Mon, 14 Apr 2025 03:46:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b= dbtWDngwTs+RsuXUaN9vILsT55WI9i8RdCISt8IqVmkpUsO6pXrYGiOF3QyUY/ti 45nzEfcaIGt96fu+aahnOsU6S0SRybrcG10ZaUcoNJeVsLAWOWyagbPw9FOR3jmN RFRBEojRlS3zYckyfk1rpQKDm6iQiJ7pKod7hgi5BHwxl886FCC3HyUNnaYoXr42 tn3SOjAUV8f8xZzNIi0HGqne7q7F+KavtXa2cYIzPwyEACZ52OiUuGKO5BO2PlNe dnmzkz5+NTiP5IxUuTR4YAJp23cG8mOASKHdUZ9HQ7XhEEz6MA55XMcrtpeWqxcB cqtgQ1fvPGEnTJ2kkTJa3g== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460td100ka-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:11 +0000 (GMT) Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3UUVV024638; Mon, 14 Apr 2025 03:46:11 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010003.outbound.protection.outlook.com [40.93.1.3]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 460d4x6e8h-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MWaa8ujdoblk6M+e5EKgIuqr9daqTVPeyH9xJFgV6btYWMsy8XG9tCyQmAmIA9jfNaPO/XMQyJU52wurOzjeSYT7O9Y7VQkRpbAW/j9ipbVsekP4rR9kxk/PFNA2J5ZiRa98Xa+BRF/a6RZKgnDVq2rj0Cw4Z0HiO46aRc2NLHKx446MTSXRZlohf0TgfoCAlzUFIyJ/QLF6nmPYatLn3Te09+0gdsBzaZM5WGDpxQe4LIW4X28Fb5AmZlpD44pnCWtxii4CNVLySh/jMkApEI9Ax+I3iJ+VDg6Zx1j4NNoupN4D2g6dDsb3TD7y3GWSi1p6WqMI7SSIuMOZDit/MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b=HGnaD2eXzsqZH/5Cm3lB4P9DqplnnmkUbtWRlTpLxttkSiLSRbM2Fqf65VAPALSKn8QPTUAupcEnyjpl1xBawqdFUJrrtSNZ44Y42iw4wwmVU6XeDLxWNbHditRW/LrWVZqyT6nXhVgqRjsHqCKumb1LBMznFyvQiT58MFISR991WejXLEyVTuusynHrHO2DiWrgu7WRB2bbPEbCCFuQQ5xRRX7kJTUxP6r/iRcpiBKXEg6nvJxqvMsXPxZviagaHt0oGF2PQ8HSzC1sE4GpgxKpFETQMMgSioWDCF7X0CjhZ5vi/dw1yw9/XfiaNffIOysb05Hls9OVfcp00p8OJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IvLmEHZA0zPv9FTSfiq/rKt9GX75OUBIareZZvifBFE=; b=LO+ai67Au/iZi62Ae9P0yreDOdT2NQPy7g1sLX+kEPDgUIua9B/SvDm4CIWFhsEDfZmjmb9/Z4hQnlc3ss5B3i1h8WAcWcoeUtq8xKvkLVC90SUKHSEcX9B+yA7mP3V7Co1xT4ebx4Zg5jXNQ40xUBMMMymHdtme+M8Gz2LCtBk= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:09 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:09 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 1/4] x86/clear_page: extend clear_page*() for multi-page clearing Date: Sun, 13 Apr 2025 20:46:04 -0700 Message-Id: <20250414034607.762653-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0340.namprd04.prod.outlook.com (2603:10b6:303:8a::15) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: 28f644c2-a573-4d97-876f-08dd7b06e4bf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: dYGOXcbeHkDqStIQjEDZ0jf7mSYRvBSWtSqYUFE0GmY/A6lt5f6EKMlniizOtc4lmuy4Tj+ai72Sl7AwQZepiw3fuG2nl2A0DQH+uqx4xRLacPt49ThU+O8R746MyiZYcfGTGEBTEemqBl5MykU9yxgTOxLIBAEPe+yxb2vgO5Whd6dfTvZ5Voc0k0ousF/KynkHbAJ5vS7uwJtc/4bZwSeuIjdIkY6614m0nXyaI+DvCx3OUYUUhFC96e58ieQP46bEkvn29sFFhZCZEUMRtIEez/a/TXZv8H0O5pV+fsw+5W2gjUiP04Yv1ikUH025RApx5yXJLOkmPXJPQXeA82v55UgK2JAiA01lgnZTga9k/VvOLiLPv2OhUgQD4wU1j9o6PoU3aeT9GRcY0KI1MXNK0KjQmtAlaYMlLmhseSJwiIdlKECBFOkXKw+KVbVAljSZoAY5JDvVc8VGjw4ApbVUFys9V0NTBk/7BMgIj5Z+DUMiaDcTK4/xjfvsXvAGlLQ5Tgk8WDrtrmj+2Kura3owqAtuPJnFT7i5Zx3ncZuyxP7fSyJ+OlYGYi7eEybsP+ENZl9zovaqP4Q+HZVH+kC9h4Pa7YKRRcunyO3uzz471K6Ma2NE7Ip4pf6BYYha7T5/HIiIKXtUchTp6ZmsB75VDq6NjoZlKGn9WbZapvMHMV62B+V0whWft4kkcmkC+kfLqr7MoK9Al0ehyiRWNZrqrxdePi5/67x7uSN7KU60cqCAy2UDwhueIZSLkkL397Gf/GcG7UirWt/y7ijcyCTSF7hVTHcLGKS2etDSsZDTMIqRVzmq4D1avKQJzDoVFDKbKnYWBWE2119mxNR2sUrin0lMJy9HmP6RVTC9EHB5RdF4p1F7uPObEtDx1Y6JVtxQA4CENARfJtBVlQgOTZv6Mh0i+P3R+lmCnioTktFfV9zfpisyHbqryLRGXIzI54aOefjbjob91yLNhcyq8KvWK1eXXfTIZ8KFhcOgwwzN0aCtc70K6VLLjbaprsJMJI8tYw1yWQBjEqMKgY4b40FtvgMf2G2nktgYsKjaRDFvw84TKdlF4kFb2nWOzQ+/WqdWbwv6eCAn/aauzaXyBNc70sTPh2TGdbC/EQiSCK30V7llpjRE0ZJnLyAm64zq91n6udPSHhlpco4A9uKekTKQfHoam6CV2Jz58Nyj1tn2BP2/udSGLS4eqIwBsqq2hK/4lZvi+hdiTygqa34ePFry8WgCQBopuy4011HDnyvHoAQ4nHsU0Rw39mPQxuYyCTRVzo6LwH6+5TZGcCsXpATkZcnORiJrZIFwGzpwPV4Gu0cnnlo1wlSterVKS5J7au08Pp2Gc9PoaB3vzO54ghQeuOw2x4c6Bf08qWYtOG37a6lZrq7ho3RtIRlLGt8WtI2RSKHw50Eu7X1SCLvr/zTXhH5417WIdp7LtqPZc8w= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: LDYjfWQJTjlfGAHYdVbJk1/gHQsC//UHNTyjERnG3d6aa7HWLHEXS7oEDbjLML/tyjLYmvyzg64htSG8LpaxCyD+gAmSxc5yOr9bg7q/zTY+/mLxnp7Eb7YE4qrAhA30b7dy/BTc5nU+KXK4/Vj6QuumIo4ZShe5Np+33c8R+kuY2OSYoJo9xE0/ivxQyGgTzLvKzOP6G5c9JKgn66zYyoOwQF3Vvh/Rq5xK949RFCPY2jmEafdWMWODZIYGZmmhJZu15q2kGrTMxkao2t1eTnLl6n3uBccE8zjTrgowxOP2Hi8S2H8kOJa1utu0dtJDypaRk2qlQD6YWxqBwbvy6m7bMVr1AAUbFTS2rh9m4oMPKYuWffCa9B0soXj+WhbA5MeeuqWR3pFYcx2ttUqgZvW9jgv0TwGrAksICz6Fshkbpj+Cdi+jiSAoT7m8VwzgTXcDtZnh9R1Fn0NJ0egOV68yjMu046xVqgZkTa2TPbR1vipFvnrfGeYCojn4N2lKkbpOT6+awH5SRssNy6Uvgrqli/Vx+mIGs4/H+nCU21s9fMq7EmXW3pW+Qe1bhqTMW46D5/3R3uN93GzMUtRMRIyOmTbop8zlcW+ead9SgTzuFduHNrPxQIgDv1v4pgW7GKWnwYcYxJughJj5SttiuLQd/8cJen5TgVzX2QQCrqqEbJ5DD6NHvFVyLVIL4bWVvaoe+tCLh4pnljJ8C9e3W8XD6ygeXOlrMPgICWh55GVdxu31I0X3+noc/bGIXg1twglc6cZRrXkewegXE7QmCs1Py6FmGShSPRT1oqL5ocLooTVLz/pVbuBMAF/kQc7bXw0462P4CyD9GLxx3vR9oNOC2vgr5unophu5KUi9JjOwjyT52S5NdSoDOsuvryT+6NmpAy0sw10RNuWbfi7owUNcQrlNsXc6NG8jcZkduTazQsTnnz7PJslxfc0MQ4CY5N8N9uC2EGxNRgJXDWqdtam9EQi/GXDeIZcBMxxU+bwmHeA8NKsvFZLzIBQv++sOssmtRzd8W8JuTrKDCNUKUlH3PRC0abTqGQ5Xq55vIuf0oCPT4MSxq/hLegLuSd0AVzl6eob8xsJUamwyYaE7Mlio99cXuXUrzNPe6TWkTpbQvrSV1YXoVLkvQeBMSrYgZIREuWPPdRlpGLfDpnfOD9U9mg5ijWReO324oSjSy/ELQ2d2Hw9IpiiEXnnqJHcBDJfpMBm4tx/CSdvWthiV+ThnDVhqf8kC2YupQzDOIVzaXazEcJa7rhubnDWas1niaSOCN5jh6Zp21ToglPuAnDxa4fCxbubgVVj1vwk7NFOsHOjv34zR5e8kflf/qSD/recRGWa1vXgW7uS5YZkVyvLwGhIh+h1KQZxc5ybidCbPMjVPMOmOy1tdx9G8qAZP/bR5hNtVLVRhOHScbq+Lf3CZ3g3/m8cDM9q34SpRL+1o4kBJsrBNMOOKxSgdGakm0DWpRYUMxjt7HXx7BfFQ8eNGC6veUbDQz0y/dz5VrHKsAsRRN6cLzwZtEEYv1WcEkQMCqRMK5m4Y8RUBDF2zfDjsOjAZALFayi4kAFTdr7dp5EE4kYToOM+FklY3hlqD8wZkeuvHaPqMT9PBDa87bg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Afan5+0HLc2jaVUM/eBz+uxTKESeV84DMkXLqmXTp6Qk7tImQC7fojNnUiehIS0YZXput7Xu9PGJEh/6XluBCwYTAE4+qUkvHpn2+BvKb+Li5g/pBLPDEqHhvwwNKbXDmk2H1d+/QwWFZ+zwHHlRnK5Pxx07TgG5vPZ/TVx6V6VWpBnikrxNnsTOxzejx+wVYl2wZA8IvWZFdqi8p1HDiQLo+MJ+nzcVLRlkXOcoiK/N4TD9B2FjVTMjEEp0pFcK3MvjIP6JzwIQnSOUEPYItvYyRK6GbnVRtkj3Z3R2SkOUKyY/7PeYmvhHBFto7wTXyQpW/f4o978BWuUVDXy6OYqwxRa2J0BCk+WXkQoD/bKSJV3m/7aIhKZC2bwukAdree+UmfSHe6i1wTgazI4EEaEtmuMI6jp1AtBBXscGMlQ+n61a98g+64iz02XBF11rDxSSrq0h8+1gJqzs/bXE0XfodLHnx6qApCod/ysmge4C6SuI9jmKIgZMqCB4vfrZt2uHuRBwXbaEamiUK/33KSgQ541vZAoK9j2cZtwXX9D0OPSx20L9EPlcMSUjSQAFF6UcAS8HvH7hOMAnIkczG9bWWLky7ZmTZdr6lNUsE0g= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 28f644c2-a573-4d97-876f-08dd7b06e4bf X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:09.5459 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3NRgNwNkOqwC0V1/AJVVknSxFhHFjy3O2O6Yf5Bh1GEtT9J+izpX5dLsK+6AJRYrZIPds/6T5kUgVR29oIxQsk4l8ehUqboI87073ZIdaX0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 suspectscore=0 mlxscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-ORIG-GUID: hqmA70WF9qnVQQHmjhfAeWjqmhgfLc2D X-Proofpoint-GUID: hqmA70WF9qnVQQHmjhfAeWjqmhgfLc2D X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: C2ECC180004 X-Stat-Signature: eyd743t3sc9tuuu6dn8je44hkexq9qof X-Rspam-User: X-HE-Tag: 1744602396-202914 X-HE-Meta: U2FsdGVkX1+WYVBCCk/3P59a7T253n7uiUvr/TDPkknI780o0W96+Oa8Gi5Z4Q1RxwaCH1A5sXJWa+Y2z8xoLHrljnAfhaHAOcQHS6gtcUCF875PeD6gWtfMe0nUYSD0C89OIDV3MqYg/DlKrCIl75iiryGbOMNxmeQWUH1/TwHkqzSe7V4zn6bLA8buN26aOpvSqI3XmMYds+Kl25QwHVVm+8dp70BbckbXck+Fn2sCIDhuGsh3xqw8jyy93TT3XuE4H7kNrmd0XK8rwzGQO7SXEMvRc+7HcFaU38Wqv+C1g5HI/VbpJjBFrdJriqeiTDONWVJZG1kF+a3JQi1UHTOIGrhP3h7G+7koQXE3vaIM0YnQ7vin0VM3ouHEu9n8LU0a4oIzBwUHLsCWTDSjzKTbqonTWXApXKNwN0//E6iHvy/h75coGrMvblC2oBw8X2VZL53Ea/ZQl42DA+R1AmWJ3b3PhCEi/ccGNV7n4F0dXhg7Vjjl5p/IbJw5mDe6O3LYXN4RCRcz1B4KP7sUFf3/7fWOjuRpC7fJkBf5nfzVLy5jE2JxR5Qj5eQHAr/TakICstrx/JAkY8Bb9G6ITLhi1h8L+FCDjSGkASb4i320rOpm+HtNFjeDb9wjjIkwBVAElA68yFl5jhV8acciGUv72q/bGq3bis1100IxmJ2R1rX9uvRHp2GHJ1pWofx0Q0Tu5oV8FBdpcTwObD9w13eZePiBuG01o5PSezEF4kX3d1d0w5Hep59oINwYohACF26cfj+AaAVkvyw98bt69DnBqSURMbPpfXshsL/DvbLc/tW0uIqcvaEIZmXMSZRZAWleIoWXVxhz30gzWrdY1A3qXD41jQYrJZi9ho7F8mkgmO4gn/DqVezJni+LhtdLHHH66ZHGWCwpl6zAJXHt1lcfx+VOuZbjpnf2nmn4jCSOoae+n4iSrPjOo0+Ynkhe7Rgcl5/PnPdcqy1gSqC h1gp8aCJ X5/PmVbmLtvdiG1gK6aa5g5t/u6rYsQJeX2qOmB3Olu0A8fJmQV7Z0UGfCEEsCNFM5LRHCso90fkkMrjq0mLnmHnX8wAK2liBt80pinwERheo25T8sexW1x/0Kgvezl7RyM28gXYjoyxhbsokhK6zFowJW2fBelLDEuWMPNATPjk5xQYlAzp8GwyzyjgThlMHZ1oAiP1ZMk6OUh2Tx/6qZ+JMUL7hPjEWGN0efl/LIPKp8xb+nUHZQuGTfC6jyHYhh7ZThyhbVqbZIvu39c0zsoBI3ck6eKGox/FBQPJ58dSJtEQgbkUAlcn57YlY5AHLMA/6dAgrJGDBXLhUf0ojimkTtPR23S1gMv766cpX9Y4MK9FDVjGA2uShw9VU/F6y7HqYb4MvSlib2hzxA2DuE36o9l3qKZYZkYer0kxIStpMIg5z57pv/h9hyP5eJobyhKKJbj7ZG3V7GYyfEzOKMf3NHi2bfBpsqbMfWkmOdVWdgJeVNNr8S3Z8CIRRl+AZ5LGv0tuJhYMS9miJ7OCsqbDP37bgxjQvJ4xswF90HZ65Uc1ymbvL+fWnX0IsCvAPuw8ZKnsY6qzfvQyYiZkSbFHRWJ0XPY3DAxi+73Lh6y8xjZjEsnUcuvsoex5XqWWn5qungxrIiyBgDxtwCLngIUVZieYFJm4ykuMr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: clear_page*() variants now take a page-aligned length parameter and clears the whole region. Rename to clear_pages*(). Signed-off-by: Ankur Arora --- arch/x86/include/asm/page_64.h | 20 +++++++------ arch/x86/lib/clear_page_64.S | 52 +++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 23 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index d3aab6f4e59a..45db74db9021 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -40,22 +40,24 @@ extern unsigned long __phys_addr_symbol(unsigned long); #define __phys_reloc_hide(x) (x) -void clear_page_orig(void *page); -void clear_page_rep(void *page); -void clear_page_erms(void *page); +void clear_pages_orig(void *page, unsigned int length); +void clear_pages_rep(void *page, unsigned int length); +void clear_pages_erms(void *page, unsigned int length); static inline void clear_page(void *page) { + unsigned int length = PAGE_SIZE; /* - * Clean up KMSAN metadata for the page being cleared. The assembly call + * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. */ - kmsan_unpoison_memory(page, PAGE_SIZE); - alternative_call_2(clear_page_orig, - clear_page_rep, X86_FEATURE_REP_GOOD, - clear_page_erms, X86_FEATURE_ERMS, + kmsan_unpoison_memory(page, length); + + alternative_call_2(clear_pages_orig, + clear_pages_rep, X86_FEATURE_REP_GOOD, + clear_pages_erms, X86_FEATURE_ERMS, "=D" (page), - "D" (page), + ASM_INPUT("D" (page), "S" (length)), "cc", "memory", "rax", "rcx"); } diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index a508e4a8c66a..bce516263b69 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -13,20 +13,35 @@ */ /* - * Zero a page. - * %rdi - page + * Zero kernel page aligned region. + * + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx */ -SYM_TYPED_FUNC_START(clear_page_rep) - movl $4096/8,%ecx +SYM_TYPED_FUNC_START(clear_pages_rep) + movl %esi, %ecx xorl %eax,%eax + shrl $3,%ecx rep stosq RET -SYM_FUNC_END(clear_page_rep) -EXPORT_SYMBOL_GPL(clear_page_rep) +SYM_FUNC_END(clear_pages_rep) +EXPORT_SYMBOL_GPL(clear_pages_rep) -SYM_TYPED_FUNC_START(clear_page_orig) +/* + * Original page zeroing loop. + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx, %rflags + */ +SYM_TYPED_FUNC_START(clear_pages_orig) + movl %esi, %ecx xorl %eax,%eax - movl $4096/64,%ecx + shrl $6,%ecx .p2align 4 .Lloop: decl %ecx @@ -43,16 +58,25 @@ SYM_TYPED_FUNC_START(clear_page_orig) jnz .Lloop nop RET -SYM_FUNC_END(clear_page_orig) -EXPORT_SYMBOL_GPL(clear_page_orig) +SYM_FUNC_END(clear_pages_orig) +EXPORT_SYMBOL_GPL(clear_pages_orig) -SYM_TYPED_FUNC_START(clear_page_erms) - movl $4096,%ecx +/* + * Zero kernel page aligned region. + * + * Input: + * %rdi - destination + * %esi - length + * + * Clobbers: %rax, %rcx + */ +SYM_TYPED_FUNC_START(clear_pages_erms) + movl %esi, %ecx xorl %eax,%eax rep stosb RET -SYM_FUNC_END(clear_page_erms) -EXPORT_SYMBOL_GPL(clear_page_erms) +SYM_FUNC_END(clear_pages_erms) +EXPORT_SYMBOL_GPL(clear_pages_erms) /* * Default clear user-space. From patchwork Mon Apr 14 03:46:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 14049626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8020CC369A2 for ; Mon, 14 Apr 2025 03:46:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1C2A280040; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9CC7E28003E; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73D2B280041; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5335528003E for ; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F279ABBCEC for ; Mon, 14 Apr 2025 03:46:46 +0000 (UTC) X-FDA: 83331262812.16.6A75E5A Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf11.hostedemail.com (Postfix) with ESMTP id 9577F40004 for ; Mon, 14 Apr 2025 03:46:43 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=Y87NErTq; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Dv4KFfMf; spf=pass (imf11.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744602403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b=6dk2Lr93Sc5YXf7nW/7/OZjR7PBBy5pggMzdhcJ/WA1seQWkvBxVyTPLJEDqd4yG36FsFU FSfmyeK1Tn6Dxi9EWXAXBIUf8E37bID5wcx/N8NuElaeMe7pIxeqGQY+0U6csp9C4OpaJA 0RWiwEI9uBfj6/VEffCyx35TaVwBTWk= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=Y87NErTq; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Dv4KFfMf; spf=pass (imf11.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1744602403; a=rsa-sha256; cv=pass; b=f84H1wUA5AlN4GWfjmkRgzgCPGjdFOqC2EfoanKcSMamE8cB7rK1PiAcNIRAdnSNhQ/ThJ lDQ+AnoqA4CfY2WHpQiKtIeLeSeX709kdxwSn0O16VbAiVwwlQbML4ZRWOJzIMXNqU084E x5Ol9afovAZlrPal7SxohG4o7KFOLjs= Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3buF0007908; Mon, 14 Apr 2025 03:46:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b= Y87NErTq8wZ5b2DQKAxhMI0zqI6c6JyGPX3FoGZXfhtFmmJv80ycOD57LOaQOdjs 4tp1UYHDd3ydhaodNJcrNgfbkCdRDRKtU0tPBROwqD4EbM81ZHZ2ukHCS+EyVAFG dvilYI2MSg4SP/lsmY6eJBLNXVo+NrruSOpYKsTTuYKcQKWS7/I/mQUJin3arhVK b5weJTYlhIqVGcwvaxvuF7NTZgJkxmhZcdPsyvxte4rN4xFNBr36ZRSgQl/kAblG +CikJooDvnsCGQyGUL2AoV/6IDL7BBXhwifpK+QhB+5DhDRH1GnpgDAMsXfXD69f /h6drV8O/LhNjaooxpQ2Jw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460tm28076-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:13 +0000 (GMT) Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3SnlE009213; Mon, 14 Apr 2025 03:46:12 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010001.outbound.protection.outlook.com [40.93.1.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 460d3fxhkp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gdeWhyktJQH66khcJm+Q0NEGsFc00RX/AZ09rutvmSzXnQ7g82IjWCQAc2A2YfknoJtMJ9hBUiSy5NHjZTlq0d8tLsXsYgO42sfapZL3KDaaXkU/WSm2+5qXcnwd0gGjHQ5tLPy5+9jstT+Oq5a2p3wfxShGldxrxIwhZA/FRg/+5F/oIrYgJWYFFyxJouj7LyjsVVt34aWGOMygr2d6lhYBC/FBbb57XKpC1HnxRRxo147XdgJfbO19fQcVs35jXxiUVgNz8nUsmxfNgwaYoD9mcajxKp4JUue6fW4GO3caxaPAjWuTfNRIxgrt/TOs8zn9Uc7RX54Srxg5zzYvjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b=mRbNjlkhanSiKnqVEyJn0eOl5Emw8OdhGipngJVLGfDdbNUwiDjf3taJVu8gX/JzThM+fDPJmZ0/Gx7dNXfL/fZH1dbGsrL46MGJIzhS0iVLn1QhzCOxG8j5KBnDDeEFB52CZy8t/TxAcyAPOcAKylxm5lgaDFrKT9ChXEGmaVZV9SNx33RXPJtU6C/nNdLiaiANvThERDAs8tSBRM3nzzkA0WJOSCtg+YGlEzu9z/9lZQmj1X370ZP2QBw5IkIKmf6hb1+h0x0uhqJ2Jdbsmo+lCah8CgrCANJwTqWDDS+qvJDJ09800KZCJ6zPKVRSvDgTnh/uQ8Buq/m16elIEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jvpnIooywk121/A9ba5yAbNWiYQIegMHpNkfFCXsNmw=; b=Dv4KFfMfCDduwfkY/iFjxj3gH4605U+hwLdpxvNbywcsW4m8mon8A+i3nWxDf57dN+YkUC9J3Nktl6LHXiJY4Hu3wamO132gSWZ/Y+iGrWdq+ZO2ZxCkxCXyl9oBCv5rAwaO20wdHad0xAbS3UCCRPRGyNmSyrWErohic4/M6jk= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:10 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:10 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 2/4] x86/clear_page: add clear_pages() Date: Sun, 13 Apr 2025 20:46:05 -0700 Message-Id: <20250414034607.762653-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0343.namprd04.prod.outlook.com (2603:10b6:303:8a::18) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: 2fc85a70-6231-47ca-13eb-08dd7b06e57d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: TJ3d+Ipdo8geV6wal543AUJagHZoDD0TFyxPmOTSKlS6biNqeOKhoyd7HE6yYvWzJRKiDGKH4cV591P0qIgnTU6eT+nr1NUJu35JOD7MvHzi9OABh84bmgYzV2HNqsYADEADDWkdnRiR90nMzWn5vN3gX9/uEht4wr2FBDliIrGhq8gsB5QVK3ZxGs1VU+c1ul/JcJwAhIousUbwvHkg79gqqE2N+3ZIXfR9+YOGnvSJPqOwbid+CBbsZ6fxCSkhXGDiHzaGTqhz3AqzZd43tBtwGPSXvjcHZFUfEeWnCZTbb5pq69jLdXZO6DTNnKfu8+w8KB0ZLRfctBhac/Zy6lOG4XSU6jOdFH4GC6d0BLyNjHeWbkhGQZ7xAJ48Joa5cEAkp/OX91dI1J39pQ1KP4qu2+A5aC3Uwz8rpkxHaWsR7sRJow3u8WdfeM7LdMtGzlFxqZXyCZ19u3FDIz/9ENrFerM0UVm/Gk3+7FG/+YYQfDE7O9UV5So9fOkpsnhTpDV/HxfSSrLMUXgzwt65/58jynOH9ryyQQHlQEkkJVHYjgkB5kkVPldHP6pzEJRY9QMuyBJ2x747R+2W08NBlBmVE1q7UiYAkZdVoelUQxK7JXX2RP/ciy8NHC6b6I3HnSgK+QDF+PMZNvIYch+BQN8lfJmkAyfM1bMEwj8tJ0DFYucXa2JNeQyqL5c90+AQQKJjIM64WiGzADO51VN6UtBcHKwAic5CCSbO0wuJC+idTZj/mPXsj4Za832GkJtDfTDjVxmtzHQkNMfUlqrE+rrZevvYmwyANvlQ8iPMQANk9LY5HOT8SmPOujYsZ+aNr7+swtL99+7vN6CFbiXRlbnsMGFkD/x1yaK6sbRqTLKCwO4s4K15eicAly2FZ+YCTH9e0K3A4gCb7Bc42AZ1XdqAJALQz1kAPFpbS4DvCLvIpP1WIb1wlqcz0FYxUN0o6sxynNg371c1ohR9UWffVCUCqoj9E/9Cmtkd/E2vlzmm22YG4hMxkFtmEIemAf3wFgSlyBpfUQj0E/ifZ6/C4Pa4cFyAdwf2Avj6sn12SCvnjXHtAwu3ESGcsYspG2nsGTFVzj+MqEEa5fHQJToee28yEebzGytHk5VmslnPj7lWqAZ8dKX8C9kEh3Zuc0eX91TFEbiIfAdxvy8GT78CWJK/FH4oEqRNKpK70MZYTEF2jVqAO/ZhE/rk/nkkAxkPJ5E6PiXL2aT1XIDy1S7+kebuY4g/lz3JbGwREmPn0J8oEs8TXcQdu4Z0OnY9Re5Shp/IJMnX/8wxvmfA3Ic1ck7KIKO826lap9IjIyXDtZEaoE7H3l4jxxrWjZLDGMRSgyoTTskc0iD9ww78Asv9E9noxlbs0RknKVQNsaKrWZzh7OYgglNwsIRiWXwHGdiIO9/Q0XIqFuazL9qet345Zh46gDkm+1NYa/E6PqHFwCA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CT0ghD5wFdJm5m4aFMoqJcd9xBYjvFTuoW4mHltkecX+p48icsihDG1fsY/m/eYX5n4QJy955onoPFSolmZPiSMaTPBfOfCxnUmV4v2q+F2IbhMOSaknaA32PqD6N5jUGcn8vOyTdZLTS/X5I0OFtf7TmaJTDPe9bVTH6vellwvtRAvD6h/TXLq6M1F3C95Sg9URieB7h5So2laevb3aWQvlKE+u7O0oyLPTJs6s471HrBA0tfWrdFXvdo1c6e4LHt1iaLUQFp9uM6ombsvICdee7wl+1qdv+CpvXjA0Dpx8nnrLlcI118Hn/bm3Q1AgATxf/xldUDEgH3hJegEF/xMetpdopXnwdLqa/aLRLsiA9L9nGnmJif7oJnjjdhBuEPGRTK9u57hgpkfDlChYOYIWpkpzb3AeG/2crJ8/khZGc/8yz+iBBgWuNnZNluMvWfUGkXJjOH3rG5PN8N3Y5pzaQJtzSgKjoOFhicrKHIKpa6Jofa+/e5O8UmIEJRgGO+US4xdWSrJsICf/TDTM9I+lpaAq15h59Hnj0RmuLjiLMzLhQLcn+5BB9JHP8g8kp6t04uPyEvs6rgvhYtT7SU/aPE+wuDhuoUS1dU3v+FcdDPYeAXrlkPLPYUGVcjOl2s8Z6NZbG9oBlhyQN7Zv06dM26t3H+R+zLePZ7kQUBgXin9SHxPmckj3C2TBX3KS8dz0zVp8AyiDGIHcd5BsYSfj8LefZpGhRhoYVqj+aG4nohGMgvZRTVdWBNAXhJooz+pw3vWBsgmiEQG48M00L7VvK2yvm8lCc6eUzF/sgXuQJdUnekPxrjxh0p73C3oCpTg6ojg0DYR8FLuTwDIzIXBd+TGoj17ZAm6F2nsoPxs64AUIGyea0ebI3N5bdKh0AVaOs2xxq9JriJMDyFoIbDR9JrlpLPX+KtqJs9F/ga8u7HiW5oIM8h5KemJ3a7Ivg3b4EZY8S7EWIYwMliQ1QZvEhn1nRegadZY21qXgOekyMJ4ZBAPTbiSUSyoXHcOKHbG5WzgqUHNHcDWQbsIoGsA15wVoja+j8hxIcFi7JbXIjXI4IlXo48m/2+DNssIZmTk1r/T4oFT1IfrvLD+nizk8Skn7FHaw4uk6sI+yU56nBKDwgpV/1Tx58l62+fZGi5/tSxiQZeyCggd115S4dzfP3wJyy9pOumz3L+xUe+mO8H8bn5g85GHVPFHe4aYCii+r2QUQO5QI7gjIpJ5cfKDhHLHE4BAIHVnb7wMR+OKbYTBVz7zdAR8Ci5rrS05EIoRIqzyP/eiLNf33JuOCSLoojQnHpoXVzxMizPBS2O65o3+i7oAUJ3pmw5nlI1U7KqRwkA+1+nIYRK4jYk87kLpaOBkDXAfEWJFBbcrSOTTGuurRs+GzSlkMNba0mhvYpgLwZ9BZMio6ZB5YJgJSnYS1qhgpOnUMFOqvGTJvaK1uFZdhqj7nlf+w4yUJdlGodiHOphYpqGeGwbeJ8OMAPCzqSVCOXH4UC8X134n2vOg/GJYPbUrqWGSfgymYl06IDukZd9mq5GZ3MXvycB8SOS4aQASts8FDGahY4wu2SpOt3EMraNy4mlfHfw0TzgQ/mZSWtTciFe/9KUa5s+22Zw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: YBaYVvS1ZPeMu4e0dx0wudRV3k2DH03oUMFCz5Z+sZRzOS8RvXmz3vUfs6yapvTYzoZouTliQvdaeqQ9m4J3EoNZWUn4xN1ofkBGkbbb8NT2ANuNTYgFkwjsZZb1O02aGe9hHImzdOk6XbN7V0GS5N8MY8AdbYamTgxUmhNBrA86s6bAw1NSyA28cyNv8iKxg0hReU9NIrbjdwK3WSwHUFOwDvXbWR46UAY8j9Z6YjNJkP5AUA/oQWgKxWQ05F6S9ouw0Dp/wOMh8ofuHxcqD0DxggLczipnXYcK9nvZbclsQiinnE/G73DMtJPkMva66tsMVTmF9T9NUj2s9yu3LwkA6Yuc4YB/6lE+bZSMOTgtkmBHnHaxYTER/nl3CnBQP28cxQkm33e/gGmc967/Q0V6+xXu2GYzro7ACRfFqSvf3i9gyMxhv1CWfHD9s/YkZ1WqvCz50EcpytCT73CoCDC2FB5Za2U2Wosam8d9cqbBlcGYRkdc69C6e/cu06cwgg9O9bNLwXOUt9feiBL3iFlGoOu1HHeUBjRJWwEe9TDvEv3uCG59mD1ZFgYVp5deVH3St2Xfw7wZgw/D8JsUWAB2r4KOsbGAM9toR/oLPqE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2fc85a70-6231-47ca-13eb-08dd7b06e57d X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:10.7653 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FukM796r1seXONLhmUE1D7qqNom6wgQjnd+yr2U/fsKc3f0QX+2BGeWDxtx9VlzdslMxuIeL4dEg7HXIt8gNegVE5rVb8KZ+QRAWs8pU1UE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 suspectscore=0 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-ORIG-GUID: Q1WLOQTmTucUHw8OqWOwXPddsF9slvdC X-Proofpoint-GUID: Q1WLOQTmTucUHw8OqWOwXPddsF9slvdC X-Rspamd-Queue-Id: 9577F40004 X-Rspamd-Server: rspam05 X-Rspam-User: X-Stat-Signature: zq6b9kc8pb5bi8x4qr7xfpebogyi5uc7 X-HE-Tag: 1744602403-634890 X-HE-Meta: U2FsdGVkX18ld582f7U7NeUZE2WITEDFmMObJLp+nmQIGVSzeHcXliFnDwaQjoeypVaDrQ4EYgq1rEElKThE4Iwy71xW0xTKzfGIKKwIzknK06Yj06J4j3itHbkJBK7GdO+IiosO0h2kC7c36r/gnzsqhJRM7DZTFBdXMzhhwguHAkCiOu0Jd0bKCjWQf3KcEFfUZSWf+1Y4Ji7eDDqD6t3EMEbGaea2EQaQkQBanUYVMcpzhgHgGXY2EYZTiryAMhdQhcZ4cnFiiOJ1Bqs7T6iucbPNK6m9xs/XQf6h6LIXpTaeK58a6x8n082pvGymws5GitWBAATiK5z+ZeiqUuJh0qjrr5R0huhZbZE8QmRKCJ+/jsXFgjHacZslxlKUFotcIMZ6CQqvZ+mXs5xKDphSJce13tjwR/+aaKVK9i4CXZCG8GBA5s8q1XCHURbV6qX0BeVEVVrB2jbKsTTK5hbQOrNDHTDC+iAsxwYR4FjivHw06X4NWgjgK1tBdhU7Q1xubMs2e8wltlKOLZaQVuHorhagoqjwmATGTvbJPvhuUtVzL+L/xczRMphJ2hXV16vEvNObUEYsCvFji7PTAzuRNL0CLL1pv+v0wXXLz15a5Plf67Q+AlF50COPSjSmtSY5WTZGYPcH3tKl9wWXEOBu85Oi97RUzZqdHdqd1W3HSCSURgWiv1s1c0b4M8YVytI3+f7EX/ZlDkpGL7fa7xRd45Ei3wKMp4lQDRyDUluaK5cLbD05qs3bTSOr9+tgJa0SMhNHzPvFXuhL1LiFmVvdLHRiXHL4a8XM9nS9W31tuvPq0sd/xFxCQTQ+em9U296C4jvYjnf9u8hhcdtKkN+IARzjecHZ06+024TncNwi+vs7oNcmGsOVZ8GbuyUclST/TyiEhzy/wfaU1Xqrl4uaTBaaXjnUNZwNmlXEmuE85FdK6iuYO4phnQiUuR2eGkVLzakkeneow6Soanm d9Dc2tcf 5rESFogABNSeD9OC4XKpmVnXyZmUvT3NKs1tcsEp9SQhzZn2LZMsq1w0UKU9JME6NC/UZBbbDMlrylI7EvAzJpkwzcxdgCokmDq+X2H9PZrYmOLEdrVOWGD7oYGHjgwISGu81uenO/9VRT1ZBfCBbSaP63F6hMyYTMTbiXPpHSVn3f0IsU+yWDDLmaMeMiBgS9Riqi4rMuSMFABImWr9BhCaKWlEwZlOC4mZguVFaGJN/B50UEZDl1IQLwigYB5CiMjCw27dK40qPg6tg8vYAsnI0PA7nTPlAZVZNcX5qJLdFfH0UqLk2YGuvnpW/pyIm//DG5TrpyxRDwbm5nCIbzEdpvY+m1LcWN/Y4c26tvLvmFuoXvsEb8lgCgxr7mADKCed6JyJEUn7sJFXsSqeqL05sCRl65luUKBKTFR8eoc6W7MbuA8cqLKp5Af732JZlmKpc0j6ZSoCafyq3CCS/Z4H6H2VWBBO9glq6h/KSyf26EvnesJDwiL/rcPoIskiZAgYpwAjeEx0TLOAXeeT9/y5NJKGqbEKv/biENOhTfo4Aw090gDVENTamgr70VSd1iHKtoQUTH/HzSfABUb+rSTAl5iahAhZYZ/CgERB53WF+IC/QWceQ7HmhWEQwhGV/X5aXARQ9i/YmQDR91XT5ezBM6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Expose multi-page clearing via clear_pages(). Signed-off-by: Ankur Arora --- arch/x86/include/asm/page_32.h | 6 ++++++ arch/x86/include/asm/page_64.h | 9 +++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/page_32.h b/arch/x86/include/asm/page_32.h index 0c623706cb7e..73c94c9e4eee 100644 --- a/arch/x86/include/asm/page_32.h +++ b/arch/x86/include/asm/page_32.h @@ -22,6 +22,12 @@ static inline void clear_page(void *page) memset(page, 0, PAGE_SIZE); } +static inline void clear_pages(void *page, unsigned int nsubpages) +{ + for (int i = 0; i < nsubpages; i++) + clear_page(page + i * PAGE_SIZE); +} + static inline void copy_page(void *to, void *from) { memcpy(to, from, PAGE_SIZE); diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 45db74db9021..6781cdee6541 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -44,9 +44,9 @@ void clear_pages_orig(void *page, unsigned int length); void clear_pages_rep(void *page, unsigned int length); void clear_pages_erms(void *page, unsigned int length); -static inline void clear_page(void *page) +static inline void clear_pages(void *page, unsigned int nsubpages) { - unsigned int length = PAGE_SIZE; + unsigned int length = nsubpages * PAGE_SIZE; /* * Clean up KMSAN metadata for the pages being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. @@ -61,6 +61,11 @@ static inline void clear_page(void *page) "cc", "memory", "rax", "rcx"); } +static inline void clear_page(void *page) +{ + clear_pages(page, 1); +} + void copy_page(void *to, void *from); KCFI_REFERENCE(copy_page); From patchwork Mon Apr 14 03:46:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 14049625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98B2DC369B2 for ; Mon, 14 Apr 2025 03:46:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49569280037; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 38C50280040; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13F74280037; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D0C1A28003E for ; Sun, 13 Apr 2025 23:46:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5D120B19B3 for ; Mon, 14 Apr 2025 03:46:46 +0000 (UTC) X-FDA: 83331262812.12.2590B94 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf15.hostedemail.com (Postfix) with ESMTP id C3AC4A0005 for ; Mon, 14 Apr 2025 03:46:42 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=cJ+48OTb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=vZ9cUUFx; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf15.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744602403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b=KBnvN3QyzF49hAQwAY/uet9lUlWIE5e8KNrPI4f/wHaZbIpUaWEwqzcNt9BTbvMtQt+Q9l 7ngzZgShvcu21o/fSYutG3J8PAVWaJSmL+cOhIdf028rw/4GUTgqkmedl60OMxbtxGYX9b MgJqi1nebwhvPagVpj/OiGtj9ssTRYs= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1744602403; a=rsa-sha256; cv=pass; b=A0Hm9346xxNxNpNI9SloggI1coHkQaWNasRMPWMOiTpOX5StAjzhzMgKOINNHViHJso5zM G38Kd65LKl4DuBK0o0owsvwaeppE1HEZ9PxNCooMVyZamwy1za0slpZgIpwK7w0llKG1eC bTDdT/kAhdGSdqbxbneign7DiALrc+k= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=cJ+48OTb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=vZ9cUUFx; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf15.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com; dmarc=pass (policy=reject) header.from=oracle.com Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3bnSB007627; Mon, 14 Apr 2025 03:46:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b= cJ+48OTbBWbtocjQ+mWKg2lWXX2UzDzZzSQi6a6Hqe9n3/doQPCEaBY5EHUBYwOR ++cT4bEj3UbXXdSWlBAmxOm2gp1P+HhqsifwSjMdnBsB2vnTq59zHwCxsbo+b7jE vaMu05dm1jyb56CXH1e67J8EjEAtt70yY1ndHdGm2++DymiXQWZiMU5OKbnjgXyI AlwPSSsv+VZCsQUttzx1m0cwWFkPFplGvX/ar85fmcZ1KgCHayA3kjBHKoB3iHiE Wv96N1EA+np6FR8+xf9TrsR2BzRzFBhSNqPLvqgmnPiuiihXwDCcf4ypucwqLUWL llDFu4CXNC23ZiJZJp+BMA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460tm28079-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:18 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3bADv038784; Mon, 14 Apr 2025 03:46:17 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010002.outbound.protection.outlook.com [40.93.1.2]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 460d4p6fw5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E/hV5lD5nb9KfzWOkE1J4SuyOkwyYyG8zPvYgMeLA4JGFuw3nRcADsTL/wV7vCEwXYLFwN95LPcWqn+bhlJQlAKZInu6McmtI/VYAYsRtYUVujvv7BOQgNQ1xd0DWd9MNRYLefH++TriKQOSWe7IRFGzolGXZHwxZEOSv8GzG+eHb7UzSKKfXt42tVp1qop7buoowybXCzQ7AK3wAx3pYpCXIp0J/qKP/RMUUSm4qccgKPEpeAYOBWX/GsB7HeFIWA27wbnF7C6vcerPPlp7H84Gixqo4D3h9Kl3GceBR3oT73UboNqAbV1hZt4gfrp+tAmLpIThkBbwZFutZRZBaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b=tu9XXATBZyWtjb5uPKRjvIJweqiwXm+CrlEcoI8dcZggES9QG5AK8S+nTZ8lCEfxqoa+jiuiSofhsgrTsfN36wVivwtitlLd4lyEKbNF21jNK9R4L6XL7PxODMg/NayxmvZBXtMbtg0U1O/6xvdkMR6y+GViT+Wsvn9CGJ5cGt73swNYIAxBYdGnYBI7tSO4jC69QzV/kn0s1CE04sh8+e1mxAWKFbW++3/JRwAOTHJqCeSFrnt1N7RZvrqlC0PU6acNwZuWYVqrTzkJnuSig9aIlI8AZCapB8RN2Ld1utx/WW5X2wIEFQPABPiX+ORhrsqnCkv+zUl26U+qdk23GQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u9P3VpLIvdp+NSb70dIKhiVvV1dVwGDeX3t3RWLaGjs=; b=vZ9cUUFxtoPR2vj4NvUhr/rtkc/UelCi1SlyormO3sKtGx2gR6z50hRI/vyw5ul1Shv0/jttlIUULWfv29OYPS5A1M6nGjRBE09VSS/8H2bIWHv6UyRmPvgfqD8Il8PwEKdo7nB+1bSSIz83X1//f+EJ7t8U8TBZ6E4JsXpB85g= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:14 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:14 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 3/4] huge_page: allow arch override for folio_zero_user() Date: Sun, 13 Apr 2025 20:46:06 -0700 Message-Id: <20250414034607.762653-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0360.namprd04.prod.outlook.com (2603:10b6:303:8a::35) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: bf3fd167-6a40-4b0c-1e11-08dd7b06e7f0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: s5au/PX662NJWAMp2e9APYavoAzfzZB4foB5+U6xDCe1YCa3sngr6UgRNnhMdqI0KzyegPNjvh5NbSU+i8nJZskF54O3tXmewJels5A4dFwHC2gy/JFbpqOAzEOrgvBLL2i6GLwXBO7mj7XPGcCxsAsmPEV2BQtGrNEft7rMYIhAkhJPLxZqAEXbBpL0I6Yg4ePvIBpTShVL9kXCEgWLclY88qaja9ER8GHa2MgZSjjAt+9Y1WpheKsC3AKH3qz8FG+Jcb2A+6vwucQ9uczWg6G5uekgQPqYngs9WyNkDViL2Q4d9ZNjnrvt/sI7ZtuGSqZz75xk+LXuL9NSvYrhN5DDPWvS7OGpUGBdvbfaBBH4RX2IwV2fE1Etk+TLr5ac1wntMmEhAwsYyKAmkkFNwGpkDozqq8rrPinkEaITeUNETvEYd56kNIXe/ZuEImHH5Ndq9FSSsXJDjUJCyLAcU7rNc6J/JqbE9hUbcRFyTaey11bnBAa0kJL2Ot3BPyRboAMneLZLptPY3EBxeTC3eeJCxxZomYMAdNyUO1+WaLDptNvjxa4elLv/wlIRNft0cYJSUzFpu1qYaeJ2AFaBmHccSK4YlKZklTATJpCw6kDEsqn2s+yj9ghUibD3AOZuLRp+vNC17VGBaZUuZ1XMdT2deAHgBK2n8EJdwEwIIsHD+qSQEmgBpGWXjSJyRaGR2mf/O5qew3+Pp/3LEMh3CEAmkmZDs+cGGnLVZBl3hBY92PS8upvlg9WHoEQxgPv7ZqQbJAa5J6KR8SH9GLhAKY1Je4jLx+ehyMkDtk9BudQBCGAz78Lt0T4WdeM8E20WW2JGL6xWdVZ+tYLKPKDJ5nA7xJClRyfy6bo5rrmWla6LxoT2WC8coxoqIGSkqy6yPGdAPhiyeeMEX+CWjd4a6DN0RdqFx9sk5gFjGht4dBazuqb1LHI3XGUKOlANzLdb28yRPxhqbkUDdkGoZKsbJijTMmfo68Yy4Ht0r5puHG5KrFutH8JihKDXrlKMsepNh6zjVjWoK40njn268dxzPMQ6113FjQo/PB5lt72XzwV6Sb9NQuMUNkoOlw92VoVT6J2feWSlBfG7CX8ZOSfrmEcPUrFHk3/RdRWsC/2GE93xQiXUVwoSSrtHWKVG6buRfAUzhp0RAHCzfS2pndRn1gu0Ilmpd/j/E8z7b2V+mSiQcDpkA17GCTx8J8wh7anxRqruowC/r4Zi3yew7CjqU+Z74BpxhQK4FHNI9GmtrYuHBgNOSS7NVESDe6y5ji1r5AxoR28Wz2i6WPhY+81gMY7fdGswiuX44B9VmBJ5wpzPY99iwR3YlS6qNC4tzaBJ8oeZqQYnnCgdD4+wNN39u6TLQLv67eCvIYR5FDiBqzTjzV1raKNI3GxoOs+pST9cRCKoyRf7Ye/LX7ubWTSWDl+94KYWDPXyzXKR67VOgDU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QMUDcZdUwgCup14Ej7pnU36bgHMFUNZzdHpfOZpQlRiWD8Jvb0TRJoB3Ke0zR/5tapaNHmZIrDYRqWuf94DmnX4T7m8wi4MShtMeoP07bWr0eEh/PjT07oWyZbctuS1ZSkmkNIRuAhOZRcpxI1nLaWwKZrq6ocQLEI9QeE9YlOyMqyd/8ffQNKaF9gjWd4kz8RtPK2PvjHxSw4YODoHE+H0hd3J69397dO3Vm5ymJK91H4l/X6vvMZrIvBTIFG8jLRXF+cC4fmgice6YctCQf3QZHe61txGx462ukxQHQzSHOm8p4eF+c+oVJhD8xA9eYSz6yjhlhkuItosX+EZwc6QF8WhUqhjrBwPlHMy40ngPA0bsDIwYJbGgmrXyK9p4JXTIOT1MQ7dyQM2UDOsFadQiF4hp5t4a9gRIyMPZud5zWAmCc0NqpYD6i5GmxTqJdGrOiGvxIcPY5BMvmVdboUxO/mnQZkvquXr74d8C5txBDyUhIX/gbOjXr6SZtG/Dtl4mtWV6NGMH+Fai7n6wSq8fFjYirMGf0/7t5Am9pRq/jEn3BZlk+2wQqLwShtcWHsV95U7gXXBQnKJckvHFHfa5BC3sZO3bBYVKePYmW/5fllxDMaAsbP7KALMULHwKlFPBbeciD72RmAGft1xamGyxqFB/CTPAXWywU+qmqaDgC7cz5Or5NSj+vVQgk/AQVfpu5nz9fFFiJO9FYsWSckwDk0lZiB/zmTqDKWMZZ+G5sS4qWwFJYXrZPqYUW7fqSgTReFR/Zq7Jgg0h+9c1JVybD8aUbJ4gi+Sfe5BKowf1iI83TCuaQ1HRSststeKgLlTdco9bawGK/uaKbVLkD3t2NLhZ4E86WDgPEBW7mi5zi09S/PzLqRa5e+GnGSbwzuUBxOCkhe2GPbnk5CgrvRfWFQFp6z09TI9NJprYu6v+ZneitF3ixX/LaIl7h2Q2xoZRPeGkGYMjyn0ovuDMkEeHyiBRTZZ/7pO7xd1jy0T8HqqHPy5bhDMifvyMBB4IR+VXdcTpvu7lveW8AB68xIt8FmN97c1uPPgNg711p1HgqinUNfOOo6UyBABaJHgjCZHT60r1n5WmNmFvoWaXoG5ex97o2PWErlymsgEZB5JpZFdy8o346QICB+qwESu/Y+MAcvnZDIrvVoTN+FFwTNZY2v9Y1/45FEag4cbsXUhjAi9AvYzlr1dngN8wsovVzZ/6Kq0vajolUqlVJEILdPTe8YSgCnjUQWo7kxJlj0lWJLMc7J206f7rulx9T3kFpxdyYvyklH10qweS5GNdpsFMSYfF7J2+t8h9OiiyYKJKiQdNXMolGzHjpmjcI7xc4BAC08TcDGGmOBsDoMg2HmzuqnYclBzvqdcwMLVOOzl9/EtdkeyOP/rdtKC8+CGIvt60WpQjDGJOOdNom+RmMuBZSu+L6MMKYVAQIQxjZpj1jktuzfOBI/2hxVxXQO3NOr7WRydu0DNqAueSaDJs+BXzKSeM2srU8c3Wws8LhoQo563bPPbQu/3YJzl3lneq2SvvZtWgFfdn3QjAORibDYgAXdgdLdEqvwgsLX1Cc2WWrBLUCXQyEy/SPxnCUvhKbpVelCztilfyiK4BRNvFew== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: t+dRBiyBM8Q870hwtlLyjXueWRR7o8NK9Vuj3Hr6BQ2vyUojWLfxVAnUvXnugNIJUGNrEkga8prodaFdNtu9M/qKpweDqeY8M9VZ3bwdJ72WKMbf+hML04CXRY3BdoqItAxbTQqIuILLybFmcJrhWATkM7MZiOHvJkUAQoXyJHGJA/iNLRsign7ABzsMlXQtRoYWYe2iRsbDS3HXoXcm0W5lXGs5keWLOrkUkPF1QSYSvXzuuLCSM0jbMeFDCGifZ2Il/xa/+kbHg4tdctdZyj70xvxlgfsgELDSAxw5K6gE+4gnW6qzv0493gMUoxH7XXsPGEZpucHG1OINIiHFGJXEJ87yESDwpWkRHApf2WP7aD6D4xVNuWunCM3wbU3oMGsCEwuvYc7JCIdEla3lDkH0VHD8MdcP7+SH4CGc52/kOwssqWNexUFrrShqEMla8DD3OlJ+6SFjM00+4VsQt6PqROvbznND7z/OQe5/LEPizpU9ZGjxJ0/DRwYOhkCWrAy7qZURdCG5rjvFHvoVQHEF9bDpLSE64G3nNyB3DK5Y9QqUuNsG1pQCr8vRjKQPub1nmxJp8VCVJFvufHgjCCXnqOq/goJWZO04zirTcrQ= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf3fd167-6a40-4b0c-1e11-08dd7b06e7f0 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:14.8770 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bU4gXNVzG/GowSFa7H1OFHyXbgeuIP9Eghq7rrdFPHYJS3fLdcakeuneCMGESPRQTDEpZjezCeSJyIRvGJNs9+yY2SR0w06Zz5Kp5zcg5d4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-ORIG-GUID: RudJO8jlJ9QWLlvv5h5ga7DrfUtBWKCO X-Proofpoint-GUID: RudJO8jlJ9QWLlvv5h5ga7DrfUtBWKCO X-Stat-Signature: sd6uypobhhrpm9hha6yfozrqouuy7j9p X-Rspam-User: X-Rspamd-Queue-Id: C3AC4A0005 X-Rspamd-Server: rspam08 X-HE-Tag: 1744602402-642667 X-HE-Meta: U2FsdGVkX18Cv5mx5aWXBu9hlFLxqjc64k6KqBkV09TnbN8TDTiZzgEdbTK7GAhq4ZTguCoV0bB2goLDQys5QqR4cjYLJ8rk03cua6Se53xx36NCRLDGmirOKOa5U4EJYCfAk+i77X5TCZ0BboQDX60wjXnLhoDnjScyTZUoOCzhhrOufutBE8u1Fal+NIXw0R4hi4eAV40EHafv235a9QeRRMCWHo2FP8DCWjq1DHiekMrGZdzmX81AxmEfRpH3hcsFtAB9T1hleDnhjgNd5JF2LGRYwy7MgKhzv25taexfNT4KviiBBbp6rc+XlL8eyohQtjJ0NdSWrgLzFA17AoCa/VhGTjW6ocfEZ+eUByc7hogV5r3YYksR61TNge9kG2NtRVW/JvPFMORMyE5gd6NrcMv0zoqRfqa97UbNiXLk63+PJU0+wmfkZgQaeEKaLFCSk4EpqhkErSg6SuGIg/jmRb2zxvxt27uTBM9ebbKM2bOX1YOEaBsaW9davNUHXCD0ZhzQSQHUtYqFaF9hb2MmSCz5uiuJ5YQ258gX+mE1oMtnClhRJ1b0X1HP+VDNaZgX2RM3ORwAeBwRcr6vwK0mLUtF/qIYG9BpuiaWwHh8j9Z+7t/U2/eylDGc94L1EG9NrcQK+IYHAhWkttI/l1DffnFmKuZl2gGb66nXo3HjyuZz4ZfoigwwCJ6GrXYvTyIXaxlaZxgun2Qsi1v6SREk03mX5upl1NRjr+4PtoV1X2WZaXwwmUpFU550WVrRND23GAcxpUJ5QV0qS6EhZ3WI1W1nUE02mMl13Ip2356isGpST3mQKx75T7VlGwyKVKMagb95dMkJmhy1BFwF/vTgv4Y9sV8MBCVN4Fp0J+r65cFmL38jhqXeAKFuQfY0cdKXKzap5FuFRt/RJE4nXptsvcSQKwh+5bojwvKmtonf/YNxo8/3h0Hw/+fBFSsor05GuB1k6KlHRmRhvcB Ijqo/KHQ PKsBHReMv5mgulLbVWMe31AWvCTSqgdOn2aW2ib7GQpcaHam61sAM/rd6zlCOKhgmmg4YgXNkdyYCoOjWPI6DEnQ9u5OCWHQj4LZrrMIt8GqLBy0QDGJYBnk+ujlhOzbN06Nx8c8wSg7XXaIkEqCZbBaade7b5TCLXZhrMMsB/VEFQD5NmC5pGLSPNjegsNKMbI84yp3wRMFizf4qj18QtfTnGaw98kXjSJ9NKF6z+yFOCDHhDLScyvSlEm/WhRJn9s/yDhaqGeHhsbDJu3VLjqgdxJmDBOqSF2VM7R5AQclhqx3Ht5qYPFORN/L0VexX9JgOOHzWWf0A1aomBAtwx+2qGR1np1WsIf9sG+mG+iRtZYNdVwm9wABzQK01mk8cPMnhwwTYVPX6PR0hIZyVeMB9QzWBCEPGCQdNqyXt5cSAfeu4tlXiP9Vi0TWColcZWaDClmQafQZV40NcpZ9YvOpP7wVUoZ3k2rtNMF/Ee1S5ppix3dBVPSEGj1BlzUopbnCKWDuXLMJOqLnttq9U7XCyyXJhtb43uYBr/sJGF9Ft0RKYOSwHGET8kHvlr9ZHYNn1j5YaCSDYN9Sa6IkKHqNKC4wR3AmGei2VjVqq+vfTao5pzqsmxqlTtliqIt4eUZGoU/kLw85GKeEUhDTjuugfa2LYrA5nQkB1Za4XgQSMvsv+sSnBkqUvB11WqTivjPIHpB4jvENa1u81/Pi5Skc2BKQTX7R4qhl3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: folio_zero_user() is constrained to operate in a page-at-a-time fashion because it needs to handle the CONFIG_HIGHMEM case. Additionally, cooperative preemption models (none, voluntary) need regular invocations of cond_resched() which limits the chunk size when zeroing. Move the page-at-a-time handling to __folio_zero_user(). And allow an architecture specific override. Note that when running under CONFIG_PREEMPT_DYNAMIC, we could switch between cooperative and preemptible models at runtime, falling back to __folio_zero_user() in the first case. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora --- mm/memory.c | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2d8c265fc7d6..ac6a19d7bdf4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7235,6 +7235,32 @@ static int clear_subpage(unsigned long addr, int idx, void *arg) return 0; } +/* + * __folio_zero_user - page-at-a-time zeroing. + * + * Handle cases where we have nothing better available. This could be + * for a few reasons: + * + * - the architecture does not support multi-page zeroing (no override + * for folio_zero_user_preemptible()): because there might be no + * optimized zeroing primitive, or because CONFIG_HIGHMEM is supported. + * + * - !preempt_model_preemptible(): need to call cond_resched() + * periodically to provide reasonable latency. + */ +static void __folio_zero_user(struct folio *folio, unsigned long addr_hint) +{ + unsigned int nr_pages = folio_nr_pages(folio); + + if (unlikely(nr_pages > MAX_ORDER_NR_PAGES)) + clear_gigantic_page(folio, addr_hint, nr_pages); + else + process_huge_page(addr_hint, nr_pages, clear_subpage, folio); +} + +void __weak folio_zero_user_preemptible(struct folio *, unsigned long) + __alias(__folio_zero_user); + /** * folio_zero_user - Zero a folio which will be mapped to userspace. * @folio: The folio to zero. @@ -7242,12 +7268,14 @@ static int clear_subpage(unsigned long addr, int idx, void *arg) */ void folio_zero_user(struct folio *folio, unsigned long addr_hint) { - unsigned int nr_pages = folio_nr_pages(folio); - - if (unlikely(nr_pages > MAX_ORDER_NR_PAGES)) - clear_gigantic_page(folio, addr_hint, nr_pages); + /* + * Use the arch optimized version if we are preemptible and can + * do zeroing of extended extents without worrying about latency. + */ + if (preempt_model_preemptible()) + folio_zero_user_preemptible(folio, addr_hint); else - process_huge_page(addr_hint, nr_pages, clear_subpage, folio); + __folio_zero_user(folio, addr_hint); } static int copy_user_gigantic_page(struct folio *dst, struct folio *src, From patchwork Mon Apr 14 03:46:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 14049624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F49C3601E for ; Mon, 14 Apr 2025 03:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 206C028003F; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 18EF228003E; Sun, 13 Apr 2025 23:46:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E39DC28003F; Sun, 13 Apr 2025 23:46:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B6A5B280037 for ; Sun, 13 Apr 2025 23:46:45 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1C2C68151E for ; Mon, 14 Apr 2025 03:46:46 +0000 (UTC) X-FDA: 83331262812.24.5ED1FEC Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf04.hostedemail.com (Postfix) with ESMTP id E53AD40009 for ; Mon, 14 Apr 2025 03:46:42 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=HuwvCD9X; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oPEXvWHB; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf04.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744602403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b=yLdJOcPiYuOI9Fv9DmAKJuoOTReD7ZebpkhU7nBwOttfammLtd1TKug4iPA74CCpq6xqGh StEMUJAEoVOKyksYr5QRTo3l6ZIAmGgHslEl7XPLHOLAYJAJMCaAS395xNUkxL0AAnBUTR PceG2/t/Ww1vV48o9bMDuXZTsEI+RGw= ARC-Authentication-Results: i=2; imf04.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-11-20 header.b=HuwvCD9X; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oPEXvWHB; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=oracle.com; spf=pass (imf04.hostedemail.com: domain of ankur.a.arora@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=ankur.a.arora@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1744602403; a=rsa-sha256; cv=pass; b=FLW5T9dnNO7gSL8EC/3XW2lIkPkQl3EqxTKzy52ClRBeMBUjvWlt+S7xrA61pu+IhseoM+ PXWbjeRbvuFl0hVqGlAXSGG9nihesCiDwXf61JoMGnOcyTIVTalzg4hMj1i02VGYw3oxSe 3F2rZmpwZIDRA0ApJY3EPRWs17TEKxg= Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3NRmr023153; Mon, 14 Apr 2025 03:46:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b= HuwvCD9XHblyct3eaf5GnHGIMgBvHJaDKfPk0TySlDakwYUlvkzwygOWjAh7tukC 8fXULszVaMjq4znh7QdePFkuW1m7Bd3spk7+/1pqbI6kV2RJt/MHKeYy/0T+D37N mQMX5eMQsMuqaaod96LMtCkHtrKK/ejVYk5gEw+jsLgrnyfnUWfRvg62nnv2oKmQ x8cF3VQq1OLNP2UhCVjBicwJCcfw5NWhIwF9sfue3QIK01AN4kCftUYd1ye3n09a mFwV1NBq5YGbwMv/QJGi0nu+GQne7Y4EudIQXJWQGmAderkQ/IZp/Jk0o3e2FerW eJ0spCeDaYrNpT1G0Qo+7g== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 460td3r0hw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:19 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53E3bADw038784; Mon, 14 Apr 2025 03:46:18 GMT Received: from byapr05cu005.outbound.protection.outlook.com (mail-westusazlp17010002.outbound.protection.outlook.com [40.93.1.2]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 460d4p6fw5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Apr 2025 03:46:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hj8Qw0PP9dUeGa7INdfx19Xwzwiq4ArwfxtSHXSyTT/yNJZjXH4hmxk74ERO/o5tkOiWZSVsB3ci95/kMWRW4oJxDkOmGpBwPB9Yu2YLSSEpOI3aqVJnmpsFDyLGMzRnaRo479fIUCHyPePT7grSo+xiOeOOIia9oyHLxc6ke0afx+ZGNx/jaZrLoR0bAE6Hg3T3L/WPdbsw3Plu1GEds7braFdBSClEFsTB18wkJte9Mli0CnvrVAofC1CWXgxmVbJjc3FSQhSFFzJlF6H6CGUs6BuGNGXGTn/49Q9cvpz9w0+gRk3oEa2F+KE24j8AHkNFXifwpzA9wK5eAao01g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b=yCvBjA+Tf2SsZj4vSI1DVb6VfLUi3iKi6hBCeB+2xb5e/5+EhoEzfFx844PK5mlHumx97dFZqdUzjUpLWK3KNopae854nXFGTbgpvG1VDlRnhfPgidlnPseA2lF8UV8smu8xZCwIA9x/8+zjC0P/vpWPEhYxGtEQfLU9tcOMGyFdiUiTMitmE+/21/OttMjOQ/LudKE+cNskjFYZWmtz71BLQ10UwbKoHlbxz9DMRy5DM6V7wxReygxOmAWZqwH+21//WJfxErk1sYt9grAQJ3uk1bkaDptJqRlZ7eYQENPVwRVgZ0j2LkIcXZ1kGWN1iXrlt5k0H0/0jiRS9Wn6yw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QS9jq8IqfRXva5KezIzWJiDSS2AnC/XV5bo2nTV5Tvo=; b=oPEXvWHBf7o4U1zznlQSyTnpb1r+Lfqci/Hr18sOaWouCIzLU6UxLHivcSBmtl9kuGhQuV7r689xJWYyK31on3Kt8sFle1FSya+fiJXoZlfFoqpAslZ4HQy825FQKIxg/HvoFnUjdamIeSrSf4uJuRJG9LKg+7ZL+2R7E8RRSwM= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by DS4PPFDB583D803.namprd10.prod.outlook.com (2603:10b6:f:fc00::d50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.33; Mon, 14 Apr 2025 03:46:15 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::25a9:32c2:a7b0:de9e%5]) with mapi id 15.20.8632.030; Mon, 14 Apr 2025 03:46:15 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, luto@kernel.org, peterz@infradead.org, paulmck@kernel.org, rostedt@goodmis.org, tglx@linutronix.de, willy@infradead.org, jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH v3 4/4] x86/folio_zero_user: multi-page clearing Date: Sun, 13 Apr 2025 20:46:07 -0700 Message-Id: <20250414034607.762653-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20250414034607.762653-1-ankur.a.arora@oracle.com> References: <20250414034607.762653-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0340.namprd04.prod.outlook.com (2603:10b6:303:8a::15) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|DS4PPFDB583D803:EE_ X-MS-Office365-Filtering-Correlation-Id: f7f12fac-2d80-43c8-e642-08dd7b06e878 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: YP2SOdURxkuTdWp1K0h/3iuOT1h+7RjbPvBMeHVnh8hVO+OEU0tbdSUtDXnnqmrsFq1qFdwsCH11mabru5+ui21Cr+CmIUGJ79mPacX9Rw5EfMrOQ5NDsKCUnvp0jpueAxdUHKEFUaWlw6DdimqkPUqln/8J1OriJeCFZb9IbPhBaYt5RkiNUKG5gzmhcNwKVsV12KF4HRTbglMpSGOW0xNj+Yr4z1s6MqZBPrsph5om0wC8Sy4lYIqhrQPV1MhlQg4d3JViy/Zd0T6aHybB26l9CP1HH5VA+uWuS8EOGwrLMFjJNDtMEaHMYGUyjseQu3lvcDjSquaeRBjU9THPE/m3APIU2kdIJvo1x7HWFpGXiUn4aIoou7iXs3cJrTltgTYuwtcjjA9U5Y3qXGncc5qzoDl3Ys0A6YKr7nteko3KQVgQyXbJ2tYjWMnubbwAFSTQOwXAAS1/+e2lNb0kIOvNjk051klFznnSTUjgOc2yjqYvxHTsv+RvwaQqcZLmGmyJZoiKHxJNjnNjTUT5xRf0S4CNtec17SpiP9tqDxPAacgTLuFSjldKxlQpK/jcVspoYnExzc1Armyr9xmcHZ8mPOIPaaEw4jTcq6PVGlrYvOtO8+1zme9vqhITNgk4xjhaGMSTeg8ZS3AXq3aoKdqunEzwobNVi7k3229uvCQbo55tlII/f+HaE5hNEouI06+0MwB9msnR5PIAIPA22tvV4oZx8lIDQVi8XczYXRi5Xnz0IynahBVusiyZCkS1U4OLbA4c/b3kxlz8UhbVif0lLWmm9PfAxa08qQAN72NQJjWz0EGvdQdman4e7sq9v9QkDWjw5zEViwvw6Cggs7j1icO2Ll3SfeWLufa/ExBx9Xirhj3WDflBnOQ0/WD2jsCJHQXSvISlDV1MHRn/0p5yLYbEquS9x4yWPZrUORmitYwkDyauQlJHT0cyzrfiGni9A8nBTW7Hz8P4hrs99Ve7i4aKydhR11hBa6FvYoUxF4X2aknecbIdE7To6jbj5ufQ6yahCcrMhbf09K9QJ6Jyb56rUywIH5lLuNYT/ydu2NZHIexGWDsw1kISc4f7lRMb0EMTMeCz9Du+hXxSI4iblUo2YMfQR9Gh8vgYGjJDvRunA1okKJaJ46Dy7euaCs9oMzh9iw4j0brcjayQXp/PG+/ZKSIewPV96oE9xVwFwu27eq81PGkfjfWirnx+3PNvFUHVhCs6kEuE+yEZJUx56502Rdef5U/8T0hRy+g97U3LU49Zg9ZnEEZP/HhVc8BmlpQbtOBEoh7eQGFs8bBZKaQ6RoweQF4IZcdKTFU7jWH1HjtKztqd3xma2Grguk9NRPO4CbPi0neDYzz1U+iXtyRdQCblFua/tvu/hqdBzaRda/4gI7JRQcanZxNKPhKyzDR6s9NoPaeh4r+zu3x+SJjAtQVa9HqVIJwcelM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3/OkO2bHFRYajEHPHwI+KReLd85ayYYWDZwrGME4TGNwZVP0diVs+ZngEzbSh+ee//yU+3BftitsHDkm5scT+TjXXSmo+bwzt3WypDrudLhqumx8HuhlYoemdUchB/YEsDpxxrX2CIb/krOnI7LeqQRY16Rm4yccjfDapmY5aPhKLG9yToWb4B+Qb2Yy2PkxbblzVav+3Sc9XqTzDwoH28etfZ8/wpuXQ2oofkYtwvzYnjnJ9TLlf4OqieoN+Od7aoAZoVw+g0ogk7S3wC0iad1UJqMzDGsTf/I5cbBHmpR0TA5zbrdMqBBfkqxpojqPaE/pKooLbSnZia06aiSZgwu8MUmTG5pR6fzwINfKUzdjy7ElFQ3tOxu5wG9uVTAouR3Iqqs23XBKxYnBLwHa4w4Yrv4QwiayMQ4ZsynB7RH9x8bvoD4vJY/ADj84aEeMGm1ajW2ywqt3bOSYtiJZzGk2H7pXYa3IlnZ3EC/vXrm9HZLZBQCyGgz2kOm8uREddqTLeST0/XS8xiaS+c3qODc1Acz5rlKMAms3mRHt/P1uWvGnQAOnPWsmL0xIL2v/1BOId9TCDOR5liiajhV8O1J8ZgOMUDaxNnA3Hgcv9sFOg4OmzvlPXg16P8QeNECQv6wyBNMhLf/gr5Lu7wSwJ7TeiaNlLnxzicbCvchJpiUUu1/BTW+dnhjIi6LJyyYwBpQjX78PWD5n4b6ru6Jr10yepRFM3wn0/Tl0K3xSshOHtW7RVfDEcB7R/gyj8p6teHkhVmWHrgr6pW8OW5NWb3S/ciNr4qmRoj/W+C+vUx+OuMZ4rBdqCML3k/FxXSdyNmD4cENrSn/7yCuDItXpFZpOTXkyM5OLd/PyEJSNXJLuUdUsamXpbbq6AHAD81qzIny58g7aQitPYshIhJDfYwice3hom/4r3C1HJz/IvTLY7ZwYVoqSfODUn1T8QL83AEHlzxxoJoHcGf+tmjCXm+rD/BLmH+y+7mJMMyFxJMMYIQQ0xmYGf7j6GGpaw8Rw6BZjZUc6EZ2tzcdf9I0zoeiKW2wzMcjWbvAIbM4qiscOBH5x8eBhHfowtPhH3ani3Fpe5s8zOVLuwdrMZZANfmw7OrHwPKE5TRNVOkOT807RPXBL7zSq2nPMaH9aHdLMRaD2b75AeW6L1qql3EgX9wNir4vItxlserLovbkRZpS9g8xIfzlcMDpH2XUfWYzibRSjMDJkVhWMHVMd5GtHB0RFFbmmRMSCJWaXrv+XeeIPVyQyuupMhCrjutSzjmQRi0hgCchcy+KGhzcAEr+undoGge8U4h1759xFAHaQICDZKFf4lz27CLbrm7rWeZrZRUZXmFZMszOt7Bg8D6Ke7NCMxYl1Nm/n9Cxw2ConhBkS107vpkTvwzA55nfOj2oZXtTQMykDRon8X5xEUh2U3R27BBWyNv8q6tJtcFTp8Kd5j+BkBW9pTQMmJI8KUchZ5OXOywut7mKUOipBTA8Qg3tcutnfuWtaw0kAdE7B41cjvGrgCzE+SY2CTMp1lGpRJLMGsLa9uriYP94S6+OkgwQ+qAX8uJ1o32bc1AgbjJTzQrai1OmUNW2y7D4RznESvQlLL8WK8DADO/rBynKrMA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: P5di3D2gqRr7aE8lBMVuoV+774d4vN4tw8XZUTXniuuPDfOdAP5xd897/+7Z+Yp2MLm1054J8+xHPLmVhn8YpayqfTuOYeOi+o7I2D5S4NpKtuoGuxkQKhleFNWXnN7Zzb86TnB0m/Y8R2i0spFWARGk6Nityo3yaoU/4bObEKWxTvPBuPovTm1qcGxi1Ff26pxjV79svpptAzpr3EIhnHxLd+QO+pYHNcCKPQoE6YZQmlKE/1Cm84YCMwV+HZ5EFH/2/uNv9M+U994fZ9RZd/czbRGrgc/Zjwwl4JO/06+0NbDGo2zOUb1cft4IBiMDNi0TsMcf4A1aGDDGhSIjzb8YoMQuGNMFDEBZ1qKS4N0mBFU2Jjw7UWR6H6q7EcRVBhqf3LxasV7MfOo59jiURnDVDLxB3XayXzVvkki9OYccxl6Sm1vBVYPfw4I8SViFAa32ZmS8srJFDJBJ2EeXYTgWEhphyt0NHHq38yaUmDFElkeXV+8c4p3h9+RcZcysymdf7fp4/fI6jfB9lAx6qcs5GJrxcMnC42kYZI8xxh+owZX2KHd5Y4cSy4PRcSqE9JIKgdr59F8vtpnZqY3hwpeIE5HfYMTgJf01JVODa9Q= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f7f12fac-2d80-43c8-e642-08dd7b06e878 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2025 03:46:15.7971 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ga1Ugbygf2JcPiBZu8+/sQ1s1BlYTsDqt2kOlONXXy7opZj4vr/QBsRSK8qofy1tXEynm+pk1JWCY8bq95ilgRDuLZSgrQGw9cqsOz3KfoA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS4PPFDB583D803 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-14_01,2025-04-10_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504140026 X-Proofpoint-GUID: x2w-Hq3N11mBrFJuEoemsPm3_9WYSaHj X-Proofpoint-ORIG-GUID: x2w-Hq3N11mBrFJuEoemsPm3_9WYSaHj X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E53AD40009 X-Stat-Signature: 41ts6nfe65tqxtf4wkn1wsicjxhwdkye X-Rspam-User: X-HE-Tag: 1744602402-451884 X-HE-Meta: U2FsdGVkX19zKaBdeMK0xesTMSgxJWBl6zrR3IFd9X8a+StaFmQxYYTNBxd1XgP0POw7ptMXCni5dm6w659291MJ2oTk+4aROyJWcrKf4vHSXSL3+LemEZCxidzDxRdA7RvBIAKI22n0nqq0cuKLArGF8r40e7CzC4e3mI7FyWc0p05vJkQ6cek47eBrH9thjpR1lcUxXzeX0shgzgdX3QJf9tpiJ4ONpccSTHVrW/st3vnAveJuYssme42FZjwDFIjl9B/Z+ZDpYWuDwhV+uIC5hSnDEyGBxys4qc0XgM6O6lWgM9TfWPllz3SnuoMf59LeVK0A8WrnsII0LnaWQ9Jivwr5Du00hxljGy4s8hM5UGUyrTkL9dmbo5H8BXWIV482OQqzBpWDkpJFoxYLXsjqjWvPwAmDUf9FXPwt3ATuf9GVZhmMK+pwVPqcvE8A53cA2PsYERWTss5ZTvnesrM2E14zMLku28mUqRV3nL9iLfSQgaQ2w0m0iRMOjLOWszeeswGSXW+hTZo3YLU6yLTAWx9+MrVRWNEUjTG+mntVIM1DuD8wpm0PIPQlfEwUFqON264YnW0irmxnEGlqHIq1WsW5/pL6Q3lGlD6L96+09bnDfj1oJlUsVixB+cxtftabXcBV09F/aTR+lVyztiQsjzgBp+pOo1ddi0Fp9O3b0oJSHjAFT+Szdz8fd5sh9E/WKGn8SrRXE/XUMsVwBfyajqEZVGUyma5OCrdm7lAuq/OW19LhVyXcI3EG2KH22enrIChXp3eQa1lFjCEkodKAqOoIeXg6TVKhrCh8KOFkk0V/uEYfVMl2Z3hhC2NXh4hn5FttibWzcFfQ9Dzw6h5RGiO+oATfWXWYvnCST5tvKxO1azbH7NVpRDSo+/sqk4OhoalpYLtus143yDEpKXUEt1doss+QAqZao/33CLQJpRFnsLUhbnrbfwL+HFDEfuFM9vt3XKIXF8iiE+q m/Nb2CWF Wfg4dfUua/kOQYRnpA5Dt+J2lYSt+H3XdaCnRh4cTHNyGc/SPKZsURAzDmCa6GTJ43HDijy9o9GDV1lmvuhP5VQhJpz/pu4VLvwvtpFTdiMEG3rvMY+DnanViyn0k1ruWMRvyq+lJp6QWyAY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: clear_pages_rep(), clear_pages_erms() use string instructions to zero memory. When operating on more than a single page, we can use these more effectively by explicitly advertising the region-size to the processor, which can use that as a hint to optimize the clearing (ex. by eliding cacheline allocation.) As a secondary benefit, string instructions are typically microcoded, and working with larger regions helps amortize the cost of the decode. When zeroing the 2MB page, maximize spatial locality by clearing in three sections: the faulting page and its immediate neighbourhood, the left and the right regions, with the local neighbourhood cleared last. Performance == Use mmap(MAP_HUGETLB) to demand fault a 64GB region on the local NUMA node. Milan (EPYC 7J13, boost=0, preempt=full|lazy): mm/folio_zero_user x86/folio_zero_user change (GB/s +- stddev) (GB/s +- stddev) pg-sz=2MB 11.89 +- 0.78% 16.12 +- 0.12% + 35.5% pg-sz=1GB 16.51 +- 0.54% 42.80 +- 3.48% + 159.2% Milan uses a threshold of LLC-size (~32MB) for eliding cacheline allocation, so we see a dropoff in cacheline-allocations for pg-sz=1GB. pg-sz=1GB: - 9,250,034,512 cycles # 2.418 GHz ( +- 0.43% ) (46.16%) - 544,878,976 instructions # 0.06 insn per cycle - 2,331,332,516 L1-dcache-loads # 609.471 M/sec ( +- 0.03% ) (46.16%) - 1,075,122,960 L1-dcache-load-misses # 46.12% of all L1-dcache accesses ( +- 0.01% ) (46.15%) + 3,688,681,006 cycles # 2.420 GHz ( +- 3.48% ) (46.01%) + 10,979,121 instructions # 0.00 insn per cycle + 31,829,258 L1-dcache-loads # 20.881 M/sec ( +- 4.92% ) (46.34%) + 13,677,295 L1-dcache-load-misses # 42.97% of all L1-dcache accesses ( +- 6.15% ) (46.32%) That's not the case with pg-sz=2MB, where we also perform better but the number of cacheline allocations remain the same. It's not entirely clear why the performance for pg-sz=2MB improves. We decode fewer instructions and the hardware prefetcher can do a better job, but the perf stats for both of those aren't convincing enough to the extent of ~30%. pg-sz=2MB: - 13,110,306,584 cycles # 2.418 GHz ( +- 0.48% ) (46.13%) - 607,589,360 instructions # 0.05 insn per cycle - 2,416,130,434 L1-dcache-loads # 445.682 M/sec ( +- 0.08% ) (46.19%) - 1,080,187,594 L1-dcache-load-misses # 44.71% of all L1-dcache accesses ( +- 0.01% ) (46.18%) + 9,624,624,178 cycles # 2.418 GHz ( +- 0.01% ) (46.13%) + 277,336,691 instructions # 0.03 insn per cycle + 2,251,220,599 L1-dcache-loads # 565.624 M/sec ( +- 0.01% ) (46.20%) + 1,092,386,130 L1-dcache-load-misses # 48.52% of all L1-dcache accesses ( +- 0.02% ) (46.19%) Icelakex (Platinum 8358, no_turbo=1, preempt=full|lazy): mm/folio_zero_user x86/folio_zero_user change (GB/s +- stddev) (GB/s +- stddev) pg-sz=2MB 7.95 +- 0.30% 10.90 +- 0.26% + 37.10% pg-sz=1GB 8.01 +- 0.24% 11.26 +- 0.48% + 40.57% For both page-sizes, Icelakex, behaves similarly to Milan pg-sz=2MB: we see a drop in cycles but there's no drop in cacheline allocation. Performance for preempt=none|voluntary remains unchanged. Signed-off-by: Ankur Arora --- arch/x86/mm/Makefile | 1 + arch/x86/mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/mm.h | 1 + 3 files changed, 62 insertions(+) create mode 100644 arch/x86/mm/memory.c diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 32035d5be5a0..e61b4d331cdf 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -55,6 +55,7 @@ obj-$(CONFIG_MMIOTRACE_TEST) += testmmiotrace.o obj-$(CONFIG_NUMA) += numa.o numa_$(BITS).o obj-$(CONFIG_AMD_NUMA) += amdtopology.o obj-$(CONFIG_ACPI_NUMA) += srat.o +obj-$(CONFIG_PREEMPTION) += memory.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o diff --git a/arch/x86/mm/memory.c b/arch/x86/mm/memory.c new file mode 100644 index 000000000000..99851c246fcc --- /dev/null +++ b/arch/x86/mm/memory.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include + +#ifndef CONFIG_HIGHMEM +/* + * folio_zero_user_preemptible(): multi-page clearing variant of folio_zero_user(). + * + * Taking inspiration from the common code variant, we split the zeroing in + * three parts: left of the fault, right of the fault, and up to 5 pages + * in the immediate neighbourhood of the target page. + * + * Cleared in that order to keep cache lines of the target region hot. + * + * For gigantic pages, there is no expectation of cache locality so just do a + * straight zero. + */ +void folio_zero_user_preemptible(struct folio *folio, unsigned long addr_hint) +{ + unsigned long base_addr = ALIGN_DOWN(addr_hint, folio_size(folio)); + const long fault_idx = (addr_hint - base_addr) / PAGE_SIZE; + const struct range pg = DEFINE_RANGE(0, folio_nr_pages(folio) - 1); + int width = 2; /* pages cleared last on either side */ + struct range r[3]; + int i; + + if (folio_nr_pages(folio) > MAX_ORDER_NR_PAGES) { + clear_pages(page_address(folio_page(folio, 0)), folio_nr_pages(folio)); + goto out; + } + + /* + * Faulting page and its immediate neighbourhood. Cleared at the end to + * ensure it sticks around in the cache. + */ + r[2] = DEFINE_RANGE(clamp_t(s64, fault_idx - width, pg.start, pg.end), + clamp_t(s64, fault_idx + width, pg.start, pg.end)); + + /* Region to the left of the fault */ + r[1] = DEFINE_RANGE(pg.start, + clamp_t(s64, r[2].start-1, pg.start-1, r[2].start)); + + /* Region to the right of the fault: always valid for the common fault_idx=0 case. */ + r[0] = DEFINE_RANGE(clamp_t(s64, r[2].end+1, r[2].end, pg.end+1), + pg.end); + + for (i = 0; i <= 2; i++) { + int len = range_len(&r[i]); + + if (len > 0) + clear_pages(page_address(folio_page(folio, r[i].start)), len); + } + +out: + /* Explicitly invoke cond_resched() to handle any live patching necessary. */ + cond_resched(); +} + +#endif /* CONFIG_HIGHMEM */ diff --git a/include/linux/mm.h b/include/linux/mm.h index b7f13f087954..b57512da8173 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4114,6 +4114,7 @@ enum mf_action_page_type { }; #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) +void folio_zero_user_preemptible(struct folio *fio, unsigned long addr_hint); void folio_zero_user(struct folio *folio, unsigned long addr_hint); int copy_user_large_folio(struct folio *dst, struct folio *src, unsigned long addr_hint,