From patchwork Tue Jul 12 23:13:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51725C433EF for ; Tue, 12 Jul 2022 23:13:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA2EC9400DB; Tue, 12 Jul 2022 19:13:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D52919400DC; Tue, 12 Jul 2022 19:13:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C423F9400DB; Tue, 12 Jul 2022 19:13:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B5DE7940063 for ; Tue, 12 Jul 2022 19:13:28 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 89F823512A for ; Tue, 12 Jul 2022 23:13:28 +0000 (UTC) X-FDA: 79680001296.07.BD7269F Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf21.hostedemail.com (Postfix) with ESMTP id 657BC1C007B for ; Tue, 12 Jul 2022 23:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667607; x=1689203607; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eoYteC6GODz7y0R2zobgy/O4IaLSlJbZw9u65ssKePw=; b=CmLPlT6aBb7OJCW82/yDxYqiuxirgYQeP/N7S/M2sxJW0A2lxHdxvG+Q LeqW7DERuFhpQGJJ1Pzpchw7s6ECf1fucNJGcHW0OtmUlrVgMY0iTTk/x RSkHTpGdKk1kIxFcIQJ6L2nXahe9Xr6wKcKUS9Y76NoBlRe9KMW0H6Z9t ccheb2Ow6YkDsJEx+HoXQ/0f7CUj+7XvR9PzdLuevFdU/YO6R189CG+EP XN+mBSlwSO0lO8hVQLJr5ovcFkgBZnULTXM6l64Mke6QrlB4gXYBYw1km 85a73mqeB1Wz+4gGoeoAaQ6PnSzi2PZkSTF76RR1BwZPRi2gaxTAh2nUA g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286196154" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286196154" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="599542507" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga007.fm.intel.com with ESMTP; 12 Jul 2022 16:13:21 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id B877BD9; Wed, 13 Jul 2022 02:13:29 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 01/13] x86/mm: Fix CR3_ADDR_MASK Date: Wed, 13 Jul 2022 02:13:16 +0300 Message-Id: <20220712231328.5294-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667608; a=rsa-sha256; cv=none; b=ciNC7fxUh/Ah50BNraYmRIZsjYLzN9foMn8rxlLzi2iKw9pRvXf1/iBv8FAImjtf3/J3Eo bUVgQj3QfuAI/ZwybO9Jh5HeQD1V0+5+koRUSEIf+qeN6jq/m2IinqhKvU6FhveB9YOolj vUIeWJGAv8YUHtkmGkU881rFRzfCrb4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=CmLPlT6a; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KuyAZkoFzY4weTeArp06FMvOHcBT/05M8fqMpTHDt6s=; b=Urh3+Mzck6IzLuxUk2I1FMPdxLZaNEZE/U3hfNvay6AhDZ08BSgyEr3nr9bIxKQdjE4Oaf WbDx0o5nrqdsy/g/IkEyNLm1VvGRvz8xPeKmz+Q+qyerkbqsoqrwr1wY64kVqGajU6jnzp Qx9lWeBkMQJz2GmUEKTl4U/BOw2lMuc= X-Rspamd-Queue-Id: 657BC1C007B Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=CmLPlT6a; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: rckr54gj5jk8p8use3hckaw8sjwskrpb X-HE-Tag: 1657667607-780338 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The mask must not include bits above physical address mask. These bits are reserved and can be used for other things. Bits 61 and 62 are used for Linear Address Masking. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe Reviewed-by: Alexander Potapenko Tested-by: Alexander Potapenko --- arch/x86/include/asm/processor-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index 02c2cbda4a74..a7f3d9100adb 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -35,7 +35,7 @@ */ #ifdef CONFIG_X86_64 /* Mask off the address space ID and SME encryption bits. */ -#define CR3_ADDR_MASK __sme_clr(0x7FFFFFFFFFFFF000ull) +#define CR3_ADDR_MASK __sme_clr(PHYSICAL_PAGE_MASK) #define CR3_PCID_MASK 0xFFFull #define CR3_NOFLUSH BIT_ULL(63) From patchwork Tue Jul 12 23:13:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07518CCA481 for ; Tue, 12 Jul 2022 23:13:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E5D29400DC; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19370940063; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 012EC9400DD; Tue, 12 Jul 2022 19:13:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CA343940063 for ; Tue, 12 Jul 2022 19:13:28 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A531A34309 for ; Tue, 12 Jul 2022 23:13:28 +0000 (UTC) X-FDA: 79680001296.10.381D578 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf18.hostedemail.com (Postfix) with ESMTP id 5C9231C0097 for ; Tue, 12 Jul 2022 23:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667607; x=1689203607; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=95bhHx1Juu5Tt/MFhkDa/aMyBcRcrNVaIQO2CZMJjSk=; b=iXCgBpHOxd7J0q11SmI41evHWVbpkdGQ49NG/VIMtBnLXrp/2Q0msJm+ VkHgtuPQcOkLpl+BJEK0gh7bxW1FhFpusYHI1oTKrNMqJnABflNEVze2f qaU3bdUNRaoSaJVfZ8jxQc5+LR5YLYC2F9PGaWQWkrjE1GL3vePf0vCs+ t/zdR4fVzjAckkaaDOTAlQ6mzCtHVxNsqXbpOxNg+yjEDVwn2cAFjTg4k v7WRXJaeHPYWccT+dB2A0ULinmATeuk2q/iUl0kSy6Qz+7ch4GczJojDj KC0e0pJNgtgn8jCU5rBbEI+tX2CnUsrMR+De8puTZh0oI3OLHd3l8aWg9 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="282616872" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="282616872" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="663128842" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga004.fm.intel.com with ESMTP; 12 Jul 2022 16:13:21 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id BE81516D; Wed, 13 Jul 2022 02:13:29 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 02/13] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Wed, 13 Jul 2022 02:13:17 +0300 Message-Id: <20220712231328.5294-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667608; a=rsa-sha256; cv=none; b=Oox/QZdjMWlBdT7JRXQ7dUKu0dwj1nVJ2H2kp5Wef+7zxdfWu00gAvUJmIoEh25Se5seks cGirKOH6I9y1znMO//cpTip2ZyRa0QHQLJrAQwGz6EpzFoR13w9vHosXWGn0Pi+VJgZCDW q0yuSNdSXfMcvyWSOxilEWkOiONeze0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=iXCgBpHO; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf18.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Gb4J9gA3RCwe+H4n/j6bpgWyiEmyKQ+gRfeiAOgxhhU=; b=JUlO7LcsZmJ/r8GnGlr2JDB/sBv+HYE9VhCsnrYYYfGN5NfTArM61fJzFun28k6kHb/517 eLVc12vPRvSm9WgIqidezdLRnP6SzOAkpt5roodHs1Z1ibfhjs/PM6D32zXIkIqMwmNAzq 1aL1s72id3ycBBcYMGziQQ2ZM9XIznQ= X-Rspam-User: X-Stat-Signature: 1z7ct61z48cjuxs8pe6gnq4z56etirx7 X-Rspamd-Queue-Id: 5C9231C0097 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=iXCgBpHO; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf18.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspamd-Server: rspam03 X-HE-Tag: 1657667607-31436 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. Signed-off-by: Kirill A. Shutemov Reviewed-by: Alexander Potapenko Tested-by: Alexander Potapenko --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 03acc823838a..6ad5841e087f 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -300,6 +300,7 @@ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Tue Jul 12 23:13:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF6E9C43334 for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9338940063; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF6C09400DD; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8265C940063; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 593989400DD for ; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 1DAB91208E6 for ; Tue, 12 Jul 2022 23:13:29 +0000 (UTC) X-FDA: 79680001338.10.8895328 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf31.hostedemail.com (Postfix) with ESMTP id A53D720062 for ; Tue, 12 Jul 2022 23:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667607; x=1689203607; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qG2ghMZQJx407u5RjEBqh0JsX1aGpcVLvNdhKPK6E2U=; b=VeqDLEtNh3kPNIiY5eEfPwa4OFq+VBPwUjWFAHPfvAR72pKzrEm3EetY igrf3NAYjfhvz0pSlmX/khpTpXr2CGcmuRytDzouePq2dHNY5Dkqr6ocl NUlDa8+AlxaWi5BY1YqJpZb0Goqpi0j5vzwyweZEzyRvjiFvgeyHuS8ZJ cipNbqfFgYDluqptuZ5yvqFixYZDSEvyYGUEm2U1ItBHIlbAX3ZKhWeNM 0xJIt71EL4a8AD1DDoc/e+D9WuCXH5SN3a6NN3NpPfOXWcpEd41KDMEdG 5cv3mUOWErGe6dMRPOhjPDS8kXx+Z3E77buV7DcWaQhrulLM6Wc1N2c8I g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="283818837" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="283818837" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="592792963" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga007.jf.intel.com with ESMTP; 12 Jul 2022 16:13:21 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id C8B7618E; Wed, 13 Jul 2022 02:13:29 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 03/13] mm: Pass down mm_struct to untagged_addr() Date: Wed, 13 Jul 2022 02:13:18 +0300 Message-Id: <20220712231328.5294-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667608; a=rsa-sha256; cv=none; b=0JGlMbZ0IhvkEfawAG2hGWmMgaZEnyg4+SnNPkffhSypb5hvLMda0IGQKzGHq46DGWDkD8 JK8BcE/PUibu3xhKJAsy1h0JbDKAJ8jDxDq8sqyPzmDP7UvB7aj8R/slrGz0ZsMnRGw6BW FD/CP4ct/Trm87CMz697WSJ/Gw+8KOc= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VeqDLEtN; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lYqxZEaZCS8bST+cFyWYQS+4SHnVoxQRgp9G8l81jZ0=; b=dyGPgnRuYAPDCgQ+qWzHeuNZZ4X1lcCuIZZglPQKCNdFDx7eU4Xl8H/2SBBMZxrFH7rXKP j9LhnW8wwDpRrC0QjKgEQSo0nfbiO74J9kfF8tW8IuLtDX2pLA8b/CWxRlil4TeZDqfv1b eDucmtoosajJnNldshocx3rwsK034g4= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A53D720062 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VeqDLEtN; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Stat-Signature: d8w5mib9kgqa8j4gubp5me5qmxi943nm X-Rspam-User: X-HE-Tag: 1657667607-306051 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Intel Linear Address Masking (LAM) brings per-mm untagging rules. Pass down mm_struct to the untagging helper. It will help to apply untagging policy correctly. In most cases, current->mm is the one to use, but there are some exceptions, such as get_user_page_remote(). Move dummy implementation of untagged_addr() from to . can override the implementation. Moving the dummy header outside helps to avoid header hell if you need to defer mm_struct within the helper. Signed-off-by: Kirill A. Shutemov Reviewed-by: Rick Edgecombe Reviewed-by: Alexander Potapenko Tested-by: Alexander Potapenko --- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/include/asm/signal.h | 2 +- arch/arm64/include/asm/uaccess.h | 4 ++-- arch/arm64/kernel/hw_breakpoint.c | 2 +- arch/arm64/kernel/traps.c | 4 ++-- arch/arm64/mm/fault.c | 10 +++++----- arch/sparc/include/asm/pgtable_64.h | 2 +- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/radeon/radeon_gem.c | 2 +- drivers/infiniband/hw/mlx4/mr.c | 2 +- drivers/media/common/videobuf2/frame_vector.c | 2 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 2 +- drivers/staging/media/atomisp/pci/hmm/hmm_bo.c | 2 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 15 +++++++++++++++ lib/strncpy_from_user.c | 2 +- lib/strnlen_user.c | 2 +- mm/gup.c | 6 +++--- mm/madvise.c | 2 +- mm/mempolicy.c | 6 +++--- mm/migrate.c | 2 +- mm/mincore.c | 2 +- mm/mlock.c | 4 ++-- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/msync.c | 2 +- virt/kvm/kvm_main.c | 2 +- 33 files changed, 59 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 0af70d9abede..88bee513b74c 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -215,8 +215,8 @@ static inline unsigned long kaslr_offset(void) #define __untagged_addr(addr) \ ((__force __typeof__(addr))sign_extend64((__force u64)(addr), 55)) -#define untagged_addr(addr) ({ \ - u64 __addr = (__force u64)(addr); \ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ __addr &= __untagged_addr(__addr); \ (__force __typeof__(addr))__addr; \ }) diff --git a/arch/arm64/include/asm/signal.h b/arch/arm64/include/asm/signal.h index ef449f5f4ba8..0899c355c398 100644 --- a/arch/arm64/include/asm/signal.h +++ b/arch/arm64/include/asm/signal.h @@ -18,7 +18,7 @@ static inline void __user *arch_untagged_si_addr(void __user *addr, if (sig == SIGTRAP && si_code == TRAP_BRKPT) return addr; - return untagged_addr(addr); + return untagged_addr(current->mm, addr); } #define arch_untagged_si_addr arch_untagged_si_addr diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 63f9c828f1a7..bdcc014bd297 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -44,7 +44,7 @@ static inline int access_ok(const void __user *addr, unsigned long size) */ if (IS_ENABLED(CONFIG_ARM64_TAGGED_ADDR_ABI) && (current->flags & PF_KTHREAD || test_thread_flag(TIF_TAGGED_ADDR))) - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); return likely(__access_ok(addr, size)); } @@ -217,7 +217,7 @@ static inline void __user *__uaccess_mask_ptr(const void __user *ptr) " csel %0, %1, xzr, eq\n" : "=&r" (safe_ptr) : "r" (ptr), "r" (TASK_SIZE_MAX - 1), - "r" (untagged_addr(ptr)) + "r" (untagged_addr(current->mm, ptr)) : "cc"); csdb(); diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c index b29a311bb055..d637cee7b771 100644 --- a/arch/arm64/kernel/hw_breakpoint.c +++ b/arch/arm64/kernel/hw_breakpoint.c @@ -715,7 +715,7 @@ static u64 get_distance_from_watchpoint(unsigned long addr, u64 val, u64 wp_low, wp_high; u32 lens, lene; - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); lens = __ffs(ctrl->len); lene = __fls(ctrl->len); diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 9ac7a81b79be..385612d9890b 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -476,7 +476,7 @@ void arm64_notify_segfault(unsigned long addr) int code; mmap_read_lock(current->mm); - if (find_vma(current->mm, untagged_addr(addr)) == NULL) + if (find_vma(current->mm, untagged_addr(current->mm, addr)) == NULL) code = SEGV_MAPERR; else code = SEGV_ACCERR; @@ -540,7 +540,7 @@ static void user_cache_maint_handler(unsigned long esr, struct pt_regs *regs) int ret = 0; tagged_address = pt_regs_read_reg(regs, rt); - address = untagged_addr(tagged_address); + address = untagged_addr(current->mm, tagged_address); switch (crm) { case ESR_ELx_SYS64_ISS_CRM_DC_CVAU: /* DC CVAU, gets promoted */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index c5e11768e5c1..9577d7e37f36 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -454,7 +454,7 @@ static void set_thread_esr(unsigned long address, unsigned long esr) static void do_bad_area(unsigned long far, unsigned long esr, struct pt_regs *regs) { - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far); /* * If we are in kernel mode at this point, we have no context to @@ -524,7 +524,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, vm_fault_t fault; unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(mm, far); if (kprobe_page_fault(regs, esr)) return 0; @@ -675,7 +675,7 @@ static int __kprobes do_translation_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far); if (is_ttbr0_addr(addr)) return do_page_fault(far, esr, regs); @@ -719,7 +719,7 @@ static int do_sea(unsigned long far, unsigned long esr, struct pt_regs *regs) * UNKNOWN for synchronous external aborts. Mask them out now * so that userspace doesn't see them. */ - siaddr = untagged_addr(far); + siaddr = untagged_addr(current->mm, far); } arm64_notify_die(inf->name, regs, inf->sig, inf->code, siaddr, esr); @@ -809,7 +809,7 @@ static const struct fault_info fault_info[] = { void do_mem_abort(unsigned long far, unsigned long esr, struct pt_regs *regs) { const struct fault_info *inf = esr_to_fault_info(esr); - unsigned long addr = untagged_addr(far); + unsigned long addr = untagged_addr(current->mm, far); if (!inf->fn(far, esr, regs)) return; diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 4679e45c8348..1336d7bfaab9 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -1071,7 +1071,7 @@ static inline unsigned long __untagged_addr(unsigned long start) return start; } -#define untagged_addr(addr) \ +#define untagged_addr(mm, addr) \ ((__typeof__(addr))(__untagged_addr((unsigned long)(addr)))) static inline bool pte_access_permitted(pte_t pte, bool write) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 6b6d46e29e6e..b37199b16643 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -1491,7 +1491,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( if (flags & KFD_IOC_ALLOC_MEM_FLAGS_USERPTR) { if (!offset || !*offset) return -EINVAL; - user_addr = untagged_addr(*offset); + user_addr = untagged_addr(current->mm, *offset); } else if (flags & (KFD_IOC_ALLOC_MEM_FLAGS_DOORBELL | KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP)) { bo_type = ttm_bo_type_sg; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index 8ef31d687ef3..691dfb3f2c0e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -382,7 +382,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, uint32_t handle; int r; - args->addr = untagged_addr(args->addr); + args->addr = untagged_addr(current->mm, args->addr); if (offset_in_page(args->addr | args->size)) return -EINVAL; diff --git a/drivers/gpu/drm/radeon/radeon_gem.c b/drivers/gpu/drm/radeon/radeon_gem.c index 8c01a7f0e027..2c3980677f64 100644 --- a/drivers/gpu/drm/radeon/radeon_gem.c +++ b/drivers/gpu/drm/radeon/radeon_gem.c @@ -371,7 +371,7 @@ int radeon_gem_userptr_ioctl(struct drm_device *dev, void *data, uint32_t handle; int r; - args->addr = untagged_addr(args->addr); + args->addr = untagged_addr(current->mm, args->addr); if (offset_in_page(args->addr | args->size)) return -EINVAL; diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index 04a67b481608..b2860feeae3c 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -379,7 +379,7 @@ static struct ib_umem *mlx4_get_umem_mr(struct ib_device *device, u64 start, * again */ if (!ib_access_writable(access_flags)) { - unsigned long untagged_start = untagged_addr(start); + unsigned long untagged_start = untagged_addr(current->mm, start); struct vm_area_struct *vma; mmap_read_lock(current->mm); diff --git a/drivers/media/common/videobuf2/frame_vector.c b/drivers/media/common/videobuf2/frame_vector.c index 542dde9d2609..7e62f7a2555d 100644 --- a/drivers/media/common/videobuf2/frame_vector.c +++ b/drivers/media/common/videobuf2/frame_vector.c @@ -47,7 +47,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames, if (WARN_ON_ONCE(nr_frames > vec->nr_allocated)) nr_frames = vec->nr_allocated; - start = untagged_addr(start); + start = untagged_addr(mm, start); ret = pin_user_pages_fast(start, nr_frames, FOLL_FORCE | FOLL_WRITE | FOLL_LONGTERM, diff --git a/drivers/media/v4l2-core/videobuf-dma-contig.c b/drivers/media/v4l2-core/videobuf-dma-contig.c index 52312ce2ba05..a1444f8afa05 100644 --- a/drivers/media/v4l2-core/videobuf-dma-contig.c +++ b/drivers/media/v4l2-core/videobuf-dma-contig.c @@ -157,8 +157,8 @@ static void videobuf_dma_contig_user_put(struct videobuf_dma_contig_memory *mem) static int videobuf_dma_contig_user_get(struct videobuf_dma_contig_memory *mem, struct videobuf_buffer *vb) { - unsigned long untagged_baddr = untagged_addr(vb->baddr); struct mm_struct *mm = current->mm; + unsigned long untagged_baddr = untagged_addr(mm, vb->baddr); struct vm_area_struct *vma; unsigned long prev_pfn, this_pfn; unsigned long pages_done, user_address; diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c index 0168f9839c90..863d30a7ad23 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c @@ -913,7 +913,7 @@ static int alloc_user_pages(struct hmm_buffer_object *bo, * and map to user space */ - userptr = untagged_addr(userptr); + userptr = untagged_addr(current->mm, userptr); bo->pages = pages; diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c index f2b1bcefcadd..386be09cb2cd 100644 --- a/drivers/tee/tee_shm.c +++ b/drivers/tee/tee_shm.c @@ -261,7 +261,7 @@ register_shm_helper(struct tee_context *ctx, unsigned long addr, shm->flags = flags; shm->ctx = ctx; shm->id = id; - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); start = rounddown(addr, PAGE_SIZE); shm->offset = addr - start; shm->size = length; diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index c13b9290e357..5ac6c61d7caa 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -561,7 +561,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 2d04e3470d4c..c7d262bd6d6b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1659,7 +1659,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + start_vaddr = untagged_addr(mm, svpfn << PAGE_SHIFT); /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index bc8f326be0ce..f0cb92ff1391 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -94,17 +94,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 5a328cf02b75..46fd816179d7 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,21 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(mm, addr) (addr) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/lib/strncpy_from_user.c b/lib/strncpy_from_user.c index 6432b8c3e431..6e1e2aa0c994 100644 --- a/lib/strncpy_from_user.c +++ b/lib/strncpy_from_user.c @@ -121,7 +121,7 @@ long strncpy_from_user(char *dst, const char __user *src, long count) return 0; max_addr = TASK_SIZE_MAX; - src_addr = (unsigned long)untagged_addr(src); + src_addr = (unsigned long)untagged_addr(current->mm, src); if (likely(src_addr < max_addr)) { unsigned long max = max_addr - src_addr; long retval; diff --git a/lib/strnlen_user.c b/lib/strnlen_user.c index feeb935a2299..abc096a68f05 100644 --- a/lib/strnlen_user.c +++ b/lib/strnlen_user.c @@ -97,7 +97,7 @@ long strnlen_user(const char __user *str, long count) return 0; max_addr = TASK_SIZE_MAX; - src_addr = (unsigned long)untagged_addr(str); + src_addr = (unsigned long)untagged_addr(current->mm, str); if (likely(src_addr < max_addr)) { unsigned long max = max_addr - src_addr; long retval; diff --git a/mm/gup.c b/mm/gup.c index 551264407624..dbe825faf842 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1104,7 +1104,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1285,7 +1285,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; @@ -2865,7 +2865,7 @@ static int internal_get_user_pages_fast(unsigned long start, if (!(gup_flags & FOLL_FAST_ONLY)) might_lock_read(¤t->mm->mmap_lock); - start = untagged_addr(start) & PAGE_MASK; + start = untagged_addr(current->mm, start) & PAGE_MASK; len = nr_pages << PAGE_SHIFT; if (check_add_overflow(start, len, &end)) return 0; diff --git a/mm/madvise.c b/mm/madvise.c index d7b4f2602949..e3c668ddb099 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1373,7 +1373,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); + start = untagged_addr(mm, start); if (!madvise_behavior_valid(behavior)) return -EINVAL; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d39b01fd52fe..a03b4d2bc26a 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1458,7 +1458,7 @@ static long kernel_mbind(unsigned long start, unsigned long len, int lmode = mode; int err; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); err = sanitize_mpol_flags(&lmode, &mode_flags); if (err) return err; @@ -1481,7 +1481,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le unsigned long end; int err = -ENOENT; - start = untagged_addr(start); + start = untagged_addr(mm, start); if (start & ~PAGE_MASK) return -EINVAL; /* @@ -1684,7 +1684,7 @@ static int kernel_get_mempolicy(int __user *policy, if (nmask != NULL && maxnode < nr_node_ids) return -EINVAL; - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); err = do_get_mempolicy(&pval, &nodes, addr, flags); diff --git a/mm/migrate.c b/mm/migrate.c index e51588e95f57..af05049b055b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1714,7 +1714,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); + addr = (unsigned long)untagged_addr(mm, p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) diff --git a/mm/mincore.c b/mm/mincore.c index fa200c14185f..72c55bd9d184 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -236,7 +236,7 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len, unsigned long pages; unsigned char *tmp; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); /* Check the start address: needs to be page-aligned.. */ if (start & ~PAGE_MASK) diff --git a/mm/mlock.c b/mm/mlock.c index 716caf851043..054168d3e648 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -571,7 +571,7 @@ static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t fla unsigned long lock_limit; int error = -ENOMEM; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); if (!can_do_mlock()) return -EPERM; @@ -634,7 +634,7 @@ SYSCALL_DEFINE2(munlock, unsigned long, start, size_t, len) { int ret; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); len = PAGE_ALIGN(len + (offset_in_page(start))); start &= PAGE_MASK; diff --git a/mm/mmap.c b/mm/mmap.c index 61e6135c54ef..1a7baf6b6b8e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2926,7 +2926,7 @@ EXPORT_SYMBOL(vm_munmap); SYSCALL_DEFINE2(munmap, unsigned long, addr, size_t, len) { - addr = untagged_addr(addr); + addr = untagged_addr(current->mm, addr); return __vm_munmap(addr, len, true); } diff --git a/mm/mprotect.c b/mm/mprotect.c index ba5592655ee3..871e954f6155 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -622,7 +622,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, (prot & PROT_READ); struct mmu_gather tlb; - start = untagged_addr(start); + start = untagged_addr(current->mm, start); prot &= ~(PROT_GROWSDOWN|PROT_GROWSUP); if (grows == (PROT_GROWSDOWN|PROT_GROWSUP)) /* can't be both */ diff --git a/mm/mremap.c b/mm/mremap.c index b522cd0259a0..f76648bc4f67 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -906,7 +906,7 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, * * See Documentation/arm64/tagged-address-abi.rst for more information. */ - addr = untagged_addr(addr); + addr = untagged_addr(mm, addr); if (flags & ~(MREMAP_FIXED | MREMAP_MAYMOVE | MREMAP_DONTUNMAP)) return ret; diff --git a/mm/msync.c b/mm/msync.c index 137d1c104f3e..5fe989bd3c4b 100644 --- a/mm/msync.c +++ b/mm/msync.c @@ -37,7 +37,7 @@ SYSCALL_DEFINE3(msync, unsigned long, start, size_t, len, int, flags) int unmapped_error = 0; int error = -EINVAL; - start = untagged_addr(start); + start = untagged_addr(mm, start); if (flags & ~(MS_ASYNC | MS_INVALIDATE | MS_SYNC)) goto out; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index a49df8988cd6..03f7ad0ebc8a 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1876,7 +1876,7 @@ int __kvm_set_memory_region(struct kvm *kvm, return -EINVAL; /* We can read the guest memory with __xxx_user() later on. */ if ((mem->userspace_addr & (PAGE_SIZE - 1)) || - (mem->userspace_addr != untagged_addr(mem->userspace_addr)) || + (mem->userspace_addr != untagged_addr(kvm->mm, mem->userspace_addr)) || !access_ok((void __user *)(unsigned long)mem->userspace_addr, mem->memory_size)) return -EINVAL; From patchwork Tue Jul 12 23:13:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A511CCA483 for ; Tue, 12 Jul 2022 23:13:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25F489400DE; Tue, 12 Jul 2022 19:13:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BC399400DD; Tue, 12 Jul 2022 19:13:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E18779400E0; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D247E9400DE for ; Tue, 12 Jul 2022 19:13:29 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A368B21020 for ; Tue, 12 Jul 2022 23:13:29 +0000 (UTC) X-FDA: 79680001338.31.6B1B585 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf18.hostedemail.com (Postfix) with ESMTP id 075101C009D for ; Tue, 12 Jul 2022 23:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667609; x=1689203609; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xL+V7QRPCo0EgB1wnPrVNvk55kqTmrvYnWwIghDNid4=; b=AZNbAboZM6qyDslSvjzzL96jsjKyQ2B7VwYP1XCCWEKY+lrNYlDt46F2 yOcKLXFVylA8AXxLH91xHhV/TbR7U7JYOn/4Y4gx2k6FeCG3rjqSCzm0+ FbiwvYy2kIAA4nV83hjSA4p3eql8Omw/C8eq9FV+NqYxuXi3sHOnvbL2D 38XvNgbph/6BRR3wfFoemgM854J9NHTPFo1usypQDaMw/ejYzG2RsIb+o yH1rdtW4IRFTDXMERGNMFDE/fZVzgvIvT4Q1GjjS/miosPY1kXbQwuHlC 1dsRn+do4itE7CuFFo5SdxwrcxFbil6JywYTwrob9Gv8NqRUJefB5GPPZ g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="282616875" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="282616875" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="570381832" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga006.jf.intel.com with ESMTP; 12 Jul 2022 16:13:22 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id D2D98366; Wed, 13 Jul 2022 02:13:29 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 04/13] x86/mm: Handle LAM on context switch Date: Wed, 13 Jul 2022 02:13:19 +0300 Message-Id: <20220712231328.5294-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667609; a=rsa-sha256; cv=none; b=flm19UyTP5OYxfNRciU3Kcj6oCeQlVACqO+okj0na6P3wJkXs7N5gTGDpoGDwYfXQhls79 mdDlMA8XcwnmjgkProeJnMwV3kX6GzbIHYSzuvqHKe47MKxG6VwOIWwkn85SAqbOcu0Zwe Q0Urz9PxiNoWPE0CC1DtV4te331ec0E= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=AZNbAboZ; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf18.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667609; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oDidzgRoKtMaAWq7KyCLQGmWbNiggRMWqkxjno6VWfM=; b=ZXQiE4xaYCMKLLn2nJ7kSwWBlBqX2KiUYslRBGigtoobnkoIvhDMCO+MGAABO0E1Kuti3x jUiWCt4kcm4DQeUPkXu1UMh7ZFhl26OLWwSkjQ6MJnTNZ0IbCx5uyf9TR3bw29+NkV/Qp7 qfWppeM5GcyBTWQL0OLYtJflsGfwbPQ= X-Rspam-User: X-Stat-Signature: 5ot5jzh1pd3ezcix9w6jg57itc9gjdho X-Rspamd-Queue-Id: 075101C009D Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=AZNbAboZ; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf18.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspamd-Server: rspam03 X-HE-Tag: 1657667608-982487 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-thread. Add new thread features indicate that the thread has Linear Address Masking enabled. switch_mm_irqs_off() now respects these flags and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 24 +++++++++++++++++ arch/x86/include/asm/tlbflush.h | 36 +++++++++++++++++++++++++ arch/x86/mm/tlb.c | 42 +++++++++++++++++++----------- 4 files changed, 90 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..002889ca8978 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -40,6 +40,9 @@ typedef struct { #ifdef CONFIG_X86_64 unsigned short flags; + + /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ + unsigned long lam_cr3_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b8d40ddeab00..69c943b2ae90 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) } #endif +#ifdef CONFIG_X86_64 +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return mm->context.lam_cr3_mask; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return 0; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ +} +#endif + #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); @@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) { arch_dup_pkeys(oldmm, mm); paravirt_arch_dup_mmap(oldmm, mm); + dup_lam(oldmm, mm); return ldt_dup_context(oldmm, mm); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 4af5579c7ef7..66db94d4daf4 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -100,6 +100,16 @@ struct tlb_state { */ bool invalidate_other; +#ifdef CONFIG_X86_64 + /* + * Active LAM mode. + * + * X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM + * disabled. + */ + u8 lam; +#endif + /* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we @@ -363,4 +373,30 @@ static inline void __native_tlb_flush_global(unsigned long cr4) native_write_cr4(cr4 ^ X86_CR4_PGE); native_write_cr4(cr4); } + +#ifdef CONFIG_X86_64 +static inline unsigned long tlbstate_lam_cr3_mask(void) +{ + unsigned long lam = this_cpu_read(cpu_tlbstate.lam); + + return lam << X86_CR3_LAM_U57_BIT; +} + +static inline void set_tlbstate_cr3_lam_mask(unsigned long mask) +{ + this_cpu_write(cpu_tlbstate.lam, mask >> X86_CR3_LAM_U57_BIT); +} + +#else + +static inline unsigned long tlbstate_lam_cr3_mask(void) +{ + return 0; +} + +static inline void set_tlbstate_cr3_lam_mask(u64 mask) +{ +} +#endif + #endif /* _ASM_X86_TLBFLUSH_H */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index d400b6d9d246..4c93f87a8928 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -154,17 +154,18 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + return __sme_pa(pgd) | kern_pcid(asid) | lam; } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); + return __sme_pa(pgd) | lam; } } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, + unsigned long lam) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* @@ -173,7 +174,7 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return __sme_pa(pgd) | kern_pcid(asid) | lam | CR3_NOFLUSH; } /* @@ -274,15 +275,16 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, unsigned long lam, + bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +493,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + unsigned long prev_lam = tlbstate_lam_cr3_mask(); + unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -520,7 +524,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, prev_lam))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -622,15 +626,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + set_tlbstate_cr3_lam_mask(new_lam); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -691,6 +696,10 @@ void initialize_tlbstate_and_flush(void) /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); + /* LAM expected to be disabled in CR3 and init_mm */ + WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); + WARN_ON(mm_lam_cr3_mask(&init_mm)); + /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization * doesn't work like other CR4 bits because it can only be set from @@ -699,8 +708,8 @@ void initialize_tlbstate_and_flush(void) WARN_ON(boot_cpu_has(X86_FEATURE_PCID) && !(cr4_read_shadow() & X86_CR4_PCIDE)); - /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + /* Disable LAM, force ASID 0 and force a TLB flush. */ + write_cr3(build_cr3(mm->pgd, 0, 0)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -708,6 +717,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); + set_tlbstate_cr3_lam_mask(0); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); @@ -1047,8 +1057,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Tue Jul 12 23:13:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AA9AC43334 for ; Tue, 12 Jul 2022 23:13:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA4069400E0; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2CBA9400DD; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90E1E9400E0; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7FF679400DD for ; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5B29C35126 for ; Tue, 12 Jul 2022 23:13:31 +0000 (UTC) X-FDA: 79680001422.29.E50FE2B Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf03.hostedemail.com (Postfix) with ESMTP id 7D8EC20039 for ; Tue, 12 Jul 2022 23:13:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667610; x=1689203610; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FNDoAfmeIi8uLbuI0UIqM3X0guL9pMr/mCnJr0E5bVU=; b=nWngtr4y64mzfPSW7oXzu0Ybn/Sy0rkzdal/11RGVMeRZzIvt5FsCCQx MXF1Wu1NOpPnwtDU8M+murk/0v0a0CT3VwJji5qoHThK1Pi01H7kOVjyF YnzYn8aFQkGlZqrSl/inDnoyCQ8TZaQyGxOhdGNfNq5fFDbpBIOrYWtpQ OcMfLIntXhIsGErsuFzXDc2ZZewzAhFLz0rlO1M/av3eW1FML8L3sqQy3 k+++1e9g9PEMeNWL3c+3jaGmp0mIJssO99YKrkYBIjqqVuYk9sydIpo0G +hZ2xTTAf7cU3KBXapZNI3of/VwkFEhyGgGkJq/pM6/fjZkaxKdwUbsp1 A==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="268101485" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="268101485" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="628074991" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id E6AAB3ED; Wed, 13 Jul 2022 02:13:29 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 05/13] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Wed, 13 Jul 2022 02:13:20 +0300 Message-Id: <20220712231328.5294-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667610; a=rsa-sha256; cv=none; b=Q+GGIvHfEfuZRYa2C+miG0EfX0upuwehojl6o7fc9A16O/1H2jlupaA9QEDumuxVZ976uX F25LQtRwVvs6CrCwoiBnKB+fyxw1Gn1Lhp01UidPrh6ZEY+RIyck+qBN20JZrfuCyMGgdR ythzSZwfgxsZDq1gGgZEf6Dru31yLfU= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nWngtr4y; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf03.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667610; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=twRlpS3YtzZOcwl3Jo6mKYeJnht9rBQ9NxnDiRzNiRc=; b=AwdsoY5JgYPf2pnej81EGtvRlGF1ktqvY1H9C17bGzIfM7SRlWYryi+IEa7YG0O1UOLQ4c 7Umxddcx+Jem5gvELp4FMvv4dt6ylXCTg4UTOiIZyadQ3TGXnKt7TJZSohWe2qKzkla9rj JJFUkCM8ZcbknTn32fUDrizashU/k2Y= X-Stat-Signature: tsehttx55gxikh1eg87e4yxnqczept5o X-Rspamd-Queue-Id: 7D8EC20039 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nWngtr4y; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf03.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1657667610-601294 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 +++ arch/x86/include/asm/mmu_context.h | 11 ++++++++ arch/x86/include/asm/uaccess.h | 42 +++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 +++ 4 files changed, 56 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 002889ca8978..2fdb390040b5 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -43,6 +43,9 @@ typedef struct { /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 69c943b2ae90..5bd3d46685dc 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 913e593a3b45..803241dfc473 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -6,6 +6,7 @@ */ #include #include +#include #include #include #include @@ -20,6 +21,30 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= (mm)->context.untag_mask | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_ptr(mm, ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(mm, __ptrval); \ + (__force __typeof__(*(ptr)) *)__ptrval; \ +}) +#else +#define untagged_addr(mm, addr) (addr) +#define untagged_ptr(mm, ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -40,7 +65,7 @@ static inline bool pagefault_disabled(void); #define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(current->mm, addr), size)); \ }) #include @@ -125,7 +150,13 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_get_user_call(get_user,x,__ptr_clean); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -222,7 +253,12 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_put_user_call(put_user,x,__ptr_clean); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 9b2772b7e1f3..18b2bfdf7b9b 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM From patchwork Tue Jul 12 23:13:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 620ACCCA47F for ; Tue, 12 Jul 2022 23:13:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C31609400E2; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCAA39400E5; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D28F9400E2; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8D1589400E1 for ; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5A93D350EC for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) X-FDA: 79680001464.02.71038E7 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf23.hostedemail.com (Postfix) with ESMTP id 87083140078 for ; Tue, 12 Jul 2022 23:13:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667611; x=1689203611; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OJDuDjvUGA+HqDGHfFlodTRAfB+eKXuXU7w3WDfuVQk=; b=ZbY5m3QLAzrKT+10cITWjNHlMb7glJn4SxGRi2RfSdBI0O/0UrjF5499 zkEuQo0qYIeHRJNsooRdVwRDqHTjXEnSwa09Ndtvwkfpf9rekvnbhdIMB TFKHOUmjGRAl8NcCAu36nrALm5WVnb0AZu/Lfd5wHD7E7SYi1uIt7WE+R RM4oshCPKUstsNbGxwKbUrfA7CT/3whndlGSeA24JZivom5SySZ9viGj4 AkB8X6pQBJ+hmZXbr171j7m6WWlpryIARFeVD5/g0Z4Bp7wsVnB77DpZ+ C0N3vto6MsZYeFVxCLA+gCZVnZ5tGKqddEXOWd54Nu2nEFDdKac8PQdVh w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="349038543" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="349038543" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="737658278" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga001.fm.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id EC3853E3; Wed, 13 Jul 2022 02:13:29 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 06/13] x86/mm: Provide ARCH_GET_UNTAG_MASK and ARCH_ENABLE_TAGGED_ADDR Date: Wed, 13 Jul 2022 02:13:21 +0300 Message-Id: <20220712231328.5294-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667611; a=rsa-sha256; cv=none; b=gukKVf8GykZ22aE7TdLBmuJAxJCc40o7KanHoEZShch7i2hOgxGmKLsST7Q32o/b0y7cjV W0S8ht2I6sqGXK++2iJFy095O6MWqfyF39zbPSWZ7vFyJNdZpjhqK/24qcuVrVswe6kqJn MHMma0viUZx+Q3R8k4xAG+OSS940CEY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZbY5m3QL; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.100) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wzdj3nCsYMYGs5evgZL4Zh0Cq3r5TjVAZHGW3nheZ7w=; b=SREkGZq9vO2n/f4/P/4U/0Vfvc8JVmidgMSC1ecuQsQhVlUHq6qxNfTU3eNA7kQsxAkEdF e9gmtIolrfLFFdokAzddX4l3v6K44qNE8ysf7jz35t57WiqKaXtDaBLCpLmvio8IvzeB9z WPUn/Wbsd2qRaiCjAD7S0lLNK3Jh1yM= X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 87083140078 X-Rspam-User: Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZbY5m3QL; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.100) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: o1m39c8w4gdbt44jtnjmogbpktqckmpz X-HE-Tag: 1657667611-842239 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a couple of arch_prctl() handles: - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits. - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/uapi/asm/prctl.h | 3 ++ arch/x86/kernel/process_64.c | 60 ++++++++++++++++++++++++++++++- 2 files changed, 62 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..38164a05c23c 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,7 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 1962008fe743..82a19168bfa4 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -742,6 +742,60 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +static void enable_lam_func(void *mm) +{ + struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm); + unsigned long lam_mask; + unsigned long cr3; + + if (loaded_mm != mm) + return; + + lam_mask = READ_ONCE(loaded_mm->context.lam_cr3_mask); + + /* Update CR3 to get LAM active on the CPU */ + cr3 = __read_cr3(); + cr3 &= ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57); + cr3 |= lam_mask; + write_cr3(cr3); + set_tlbstate_cr3_lam_mask(lam_mask); +} + +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) +{ + int ret = 0; + + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENODEV; + + mutex_lock(&mm->context.lock); + + /* Already enabled? */ + if (mm->context.lam_cr3_mask) { + ret = -EBUSY; + goto out; + } + + if (!nr_bits) { + ret = -EINVAL; + goto out; + } else if (nr_bits <= 6) { + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); + } else { + ret = -EINVAL; + goto out; + } + + /* Make lam_cr3_mask and untag_mask visible on other CPUs */ + smp_mb(); + + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); +out: + mutex_unlock(&mm->context.lock); + return ret; +} + long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) { int ret = 0; @@ -829,7 +883,11 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - + case ARCH_GET_UNTAG_MASK: + return put_user(task->mm->context.untag_mask, + (unsigned long __user *)arg2); + case ARCH_ENABLE_TAGGED_ADDR: + return prctl_enable_tagged_addr(task->mm, arg2); default: ret = -EINVAL; break; From patchwork Tue Jul 12 23:13:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88F1ECCA47F for ; Tue, 12 Jul 2022 23:13:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E7AF9400DD; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06D229400E2; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6B359400E1; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 990CE9400DD for ; Tue, 12 Jul 2022 19:13:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6F23160FB0 for ; Tue, 12 Jul 2022 23:13:31 +0000 (UTC) X-FDA: 79680001422.15.2D6ED13 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf21.hostedemail.com (Postfix) with ESMTP id 913C01C007C for ; Tue, 12 Jul 2022 23:13:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667610; x=1689203610; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KcHlgajuJie/y+gq2EtP0KVLDLxzLO/XCoJPUGPXJSQ=; b=C3yEBht7s0joBeCa4xRR7g/TJXVNBc2qPPncpMNfLx92VYuCdnV6zNuY /Tcalduqk+jJX97rHzETmNM80yL9WanNgiQkN05+YXUezTCj3jv0w7WaK 83DMUHfDxwxyP2Oav3CyNn3br4IXMaaUHx2Pupk7eDT7D7XF7Zs7X66/9 rQrJLHfsYcXVVd1kq7C5tUsDNefV9mINFkn3dHIFFmdFEet+6LiWO887q FNa3HkftkajfhqqA7sbo3bfU8msTbD9JwQlOieL58g/sakTI+tTaAbZWx q0RipOfRAYs8VY+qursthGbLk9dNhdjLBUHSnkBlNgCidp950EJKt6D8o A==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286196173" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286196173" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="599542524" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga007.fm.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 0693C4B0; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 07/13] x86: Expose untagging mask in /proc/$PID/arch_status Date: Wed, 13 Jul 2022 02:13:22 +0300 Message-Id: <20220712231328.5294-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667611; a=rsa-sha256; cv=none; b=UsgxqAZMicU2HYCAbgWbygKaeqbP6MiVHwNKondZKy/+hG3pYbr4E1LKJgu+j0An/cOg5i YbimoFtpYPKwiSN0AZB44WVSrCA5t1WFOX3oaDliZCBjYu+qxfvrCzHAZ0j2YNqfVjE2Cu QShgy9cXrbU0wj0R3BnGgBU6kEoce78= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=C3yEBht7; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=clkMDGcG2Np5kuFcqBp/N/syyny0LQSd5EF3+UpqRBM=; b=aQvFuy6njyrkcEi4pvZsLRcFbhe41rDieTgtXrv5Dzvgd8FTa6jkj6mlGHiHZPR0ctMzKI a3RqmFVyutwACcNiW80KDTtAAXbhNgEHW7gcyC68mYSrCt8VcJSHzBoHwxRcLWgAMXXVJj 0VkhxngWBKhawLcXsI8FLwql5BDBZPg= X-Rspamd-Queue-Id: 913C01C007C Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=C3yEBht7; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: d5bdytywusur45xau8b61w9iuc5ya6on X-HE-Tag: 1657667610-383850 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a line in /proc/$PID/arch_status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers. Signed-off-by: Kirill A. Shutemov Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu_context.h | 10 +++++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/fpu/xstate.c | 47 ----------------------- arch/x86/kernel/proc.c | 60 ++++++++++++++++++++++++++++++ 4 files changed, 72 insertions(+), 47 deletions(-) create mode 100644 arch/x86/kernel/proc.c diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 5bd3d46685dc..b0e9ea23758b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -103,6 +103,11 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return mm->context.untag_mask; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { mm->context.untag_mask = -1UL; @@ -119,6 +124,11 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { } diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 4c8b6ae802ac..313f1d8e7783 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev.o +obj-$(CONFIG_PROC_FS) += proc.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index c8340156bfd2..838a6f0627fd 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -10,8 +10,6 @@ #include #include #include -#include -#include #include #include @@ -1745,48 +1743,3 @@ long fpu_xstate_prctl(int option, unsigned long arg2) return -EINVAL; } } - -#ifdef CONFIG_PROC_PID_ARCH_STATUS -/* - * Report the amount of time elapsed in millisecond since last AVX512 - * use in the task. - */ -static void avx512_status(struct seq_file *m, struct task_struct *task) -{ - unsigned long timestamp = READ_ONCE(task->thread.fpu.avx512_timestamp); - long delta; - - if (!timestamp) { - /* - * Report -1 if no AVX512 usage - */ - delta = -1; - } else { - delta = (long)(jiffies - timestamp); - /* - * Cap to LONG_MAX if time difference > LONG_MAX - */ - if (delta < 0) - delta = LONG_MAX; - delta = jiffies_to_msecs(delta); - } - - seq_put_decimal_ll(m, "AVX512_elapsed_ms:\t", delta); - seq_putc(m, '\n'); -} - -/* - * Report architecture specific information - */ -int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, - struct pid *pid, struct task_struct *task) -{ - /* - * Report AVX512 state if the processor and build option supported. - */ - if (cpu_feature_enabled(X86_FEATURE_AVX512F)) - avx512_status(m, task); - - return 0; -} -#endif /* CONFIG_PROC_PID_ARCH_STATUS */ diff --git a/arch/x86/kernel/proc.c b/arch/x86/kernel/proc.c new file mode 100644 index 000000000000..9765b4d05ce4 --- /dev/null +++ b/arch/x86/kernel/proc.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include + +/* + * Report the amount of time elapsed in millisecond since last AVX512 + * use in the task. + */ +static void avx512_status(struct seq_file *m, struct task_struct *task) +{ + unsigned long timestamp = READ_ONCE(task->thread.fpu.avx512_timestamp); + long delta; + + if (!timestamp) { + /* + * Report -1 if no AVX512 usage + */ + delta = -1; + } else { + delta = (long)(jiffies - timestamp); + /* + * Cap to LONG_MAX if time difference > LONG_MAX + */ + if (delta < 0) + delta = LONG_MAX; + delta = jiffies_to_msecs(delta); + } + + seq_put_decimal_ll(m, "AVX512_elapsed_ms:\t", delta); + seq_putc(m, '\n'); +} + +/* + * Report architecture specific information + */ +int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, + struct pid *pid, struct task_struct *task) +{ + struct mm_struct *mm; + unsigned long untag_mask = -1UL; + + /* + * Report AVX512 state if the processor and build option supported. + */ + if (cpu_feature_enabled(X86_FEATURE_AVX512F)) + avx512_status(m, task); + + mm = get_task_mm(task); + if (mm) { + untag_mask = mm_untag_mask(task->mm); + mmput(mm); + } + + seq_printf(m, "untag_mask:\t%#lx\n", untag_mask); + + return 0; +} From patchwork Tue Jul 12 23:13:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8982C43334 for ; Tue, 12 Jul 2022 23:13:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2326E9400E6; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 193879400E5; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDA8C9400E6; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D85359400E5 for ; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AD69FCBE for ; Tue, 12 Jul 2022 23:13:33 +0000 (UTC) X-FDA: 79680001506.24.CE414FB Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf21.hostedemail.com (Postfix) with ESMTP id 061AF1C007D for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667613; x=1689203613; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KhRr1rP86iXpPZMDj7Pl0ZDbvhSc63UeHKOf4/sik00=; b=BwYSqmxaD7+Ipt6v6uV/vfh+RrcAXZdl/VsIzHnksmHpazmSX6JUT2JU jw39s1Odgj18GksnpjUvQWexNZL/+eC2XPWALJ2T2LmF81q0sqCsoL/br 6xsuBQfS7Qxg935/6Z9ZFussFLHQwo1JYKJMvlTmi9JDDsW3JAVl+ESar llLoRxub+NEy/3YcRRk+dsIVyl2ki4CmWOXbd9xQ06BaqHGYSBJcgvYw8 3x08LBLKK1j3EW851tBwtOoqFUB9IYVGme28ZWglqs1zOhRFRoM5gAH00 uRcxEQVnpucfPw96l7EhyKQc7I2tLwAOcJLyIPlddjn37HL5yKqadCEEZ w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="310690827" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="310690827" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="622696276" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga008.jf.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 158674EB; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv5 08/13] selftests/x86/lam: Add malloc test cases for linear-address masking Date: Wed, 13 Jul 2022 02:13:23 +0300 Message-Id: <20220712231328.5294-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667613; a=rsa-sha256; cv=none; b=hxyWhAnnQ1IlW/eEop2k2WRG5L6gEMcUSnMSVBriONuFMhawwLtOd5hM1r9bItixNAIFAS IP8UyVXApjv1zGehgNBXOAKGVUG/LFZYcnaJDryrmAGzGwgLlvW1AUoCbFUt/6g9M3vDRq DD3Rpd7hjf0F51kZwIH5PbxihxYU4n8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=BwYSqmxa; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667613; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eZRcZY0egYrIgTlHOUd7KDOCVZ7X2NN4RYDieBJH9Fc=; b=Zbfz4gyDNfRTR7C7f2lxzApxaqj1F8FmfG9MMG9N4rWqPqRkeJMNGxxSL17tfERORymfTD +IGQ58FgHL5OjYDtnOEggRLuDlLhRy4lcz0h8EP9EX4sTkQd7IJ9Sj+f+K1C2/hxWVXiMU /iu+qJoLGmwh9h+k5PUrq6qZ6fHT6Bs= X-Stat-Signature: snnjq4dt7zaio1m7cp4wwwdecqqn7giz X-Rspamd-Queue-Id: 061AF1C007D Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=BwYSqmxa; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1657667612-50006 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata. LAM includes following mode: - LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata. There are two arch_prctls: ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer. ARCH_GET_UNTAG_MASK: get current untagged mask. The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process. Functions of this test: - LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers. - Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 263 +++++++++++++++++++++++++++ 2 files changed, 264 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 0388c4d60af0..c1a16a9d4f2f 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c new file mode 100644 index 000000000000..4aaf6ad107c3 --- /dev/null +++ b/tools/testing/selftests/x86/lam.c @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 + +#define TEST_MASK 0x1 + +#define MALLOC_LEN 32 + +struct testcases { + unsigned int later; + int expected; /* 2: SIGSEGV Error; 1: other errors */ + unsigned long lam; + uint64_t addr; + int (*test_func)(struct testcases *test); + const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler(int sig) +{ + ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); +} + +static inline int cpu_has_lam(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 1, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[0] & (1 << 26)); +} + +/* + * Set tagged address and read back untag mask. + * check if the untagged mask is expected. + */ +static int set_lam(unsigned long lam) +{ + int ret = 0; + uint64_t ptr = 0; + + if (lam != LAM_U57_BITS && lam != LAM_NONE) + return -1; + + /* Skip check return */ + syscall(SYS_arch_prctl, ARCH_ENABLE_TAGGED_ADDR, lam); + + /* Get untagged mask */ + syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr); + + /* Check mask returned is expected */ + if (lam == LAM_U57_BITS) + ret = (ptr != ~(0x3fULL << 57)); + else if (lam == LAM_NONE) + ret = (ptr != -1ULL); + + return ret; +} + +/* According to LAM mode, set metadata in high bits */ +static uint64_t get_metadata(uint64_t src, unsigned long lam) +{ + uint64_t metadata; + + srand(time(NULL)); + /* Get a random value as metadata */ + metadata = rand(); + + switch (lam) { + case LAM_U57_BITS: /* Set metadata in bits 62:57 */ + metadata = (src & ~(0x3fULL << 57)) | ((metadata & 0x3f) << 57); + break; + default: + metadata = src; + break; + } + + return metadata; +} + +/* + * Set metadata in user pointer, compare new pointer with original pointer. + * both pointers should point to the same address. + */ +static int handle_lam_test(void *src, unsigned int lam) +{ + char *ptr; + + strcpy((char *)src, "USER POINTER"); + + ptr = (char *)get_metadata((uint64_t)src, lam); + if (src == ptr) + return 0; + + /* Copy a string into the pointer with metadata */ + strcpy((char *)ptr, "METADATA POINTER"); + + return (!!strcmp((char *)src, (char *)ptr)); +} + +/* + * Test lam feature through dereference pointer get from malloc. + * @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV + */ +static int handle_malloc(struct testcases *test) +{ + char *ptr = NULL; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) == -1) + return 1; + + ptr = (char *)malloc(MALLOC_LEN); + if (ptr == NULL) { + perror("malloc() failure\n"); + return 1; + } + + /* Set signal handler */ + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) == -1 && ret == 0) + ret = 1; + + free(ptr); + + return ret; +} + +static int fork_test(struct testcases *test) +{ + int ret, child_ret; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + ret = test->test_func(test); + exit(ret); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + } + + return ret; +} + +static void run_test(struct testcases *test, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + struct testcases *t = test + i; + + /* fork a process to run test case */ + ret = fork_test(t); + if (ret != 0) + ret = (t->expected == ret); + else + ret = !(t->expected); + + tests_cnt++; + ksft_test_result(ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negtive] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static void cmd_help(void) +{ + printf("usage: lam [-h] [-t test list]\n"); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); + printf("\t\t0x1:malloc;\n"); + printf("\t-h: help\n"); +} + +int main(int argc, char **argv) +{ + int c = 0; + unsigned int tests = TEST_MASK; + + tests_cnt = 0; + + if (!cpu_has_lam()) { + ksft_print_msg("Unsupported LAM feature!\n"); + return -1; + } + + while ((c = getopt(argc, argv, "ht:")) != -1) { + switch (c) { + case 't': + tests = strtoul(optarg, NULL, 16); + if (!(tests & TEST_MASK)) { + ksft_print_msg("Invalid argument!\n"); + return -1; + } + break; + case 'h': + cmd_help(); + return 0; + default: + ksft_print_msg("Invalid argument\n"); + return -1; + } + } + + if (tests & FUNC_MALLOC) + run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + + ksft_set_plan(tests_cnt); + + return ksft_exit_pass(); +} From patchwork Tue Jul 12 23:13:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED58FC43334 for ; Tue, 12 Jul 2022 23:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF69C9400E8; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA7129400E5; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A46519400E8; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 973849400E5 for ; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 6CB7680E48 for ; Tue, 12 Jul 2022 23:13:34 +0000 (UTC) X-FDA: 79680001548.09.E8D6F94 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf21.hostedemail.com (Postfix) with ESMTP id C5D931C0070 for ; Tue, 12 Jul 2022 23:13:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667613; x=1689203613; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TsGIhct/Z37tuvSy85zzkhbg6Om8lhudHriCZDgaRcc=; b=LQnKu+LzUoDHuA9BYgImtVSMB5NtbeLM5DsbpsKG2T5ytIzp2g4pnGho ECOiHZIdurmkgydwysrifqBbvOH2zV2ZtQ9G72/HRfIBcQaQZFMzl+BOP Nbed2Uy+gb13mezKHI3nrlnJKMtItYMy4tfiegAQZEqR4ULvmFWvcGgeu VwKSkxnwW4ABjcx2B8+OU8qKISEvBJHPbwnQCoZthlTp03j3vYcq3sKDN HPYonfr/keydRGzKWjNFdHKQmdE9eT/DAfUgjRmEL5FJg+TPt1LEZvAZ5 9ZG4Xq4SGB8FgtkvgmeoHLzUn6Zvf1/VRD/6bDy59kS05QNChj/I+JzsU w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="310690829" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="310690829" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="595466249" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga002.jf.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 21147544; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv5 09/13] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking Date: Wed, 13 Jul 2022 02:13:24 +0300 Message-Id: <20220712231328.5294-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667614; a=rsa-sha256; cv=none; b=wonTPzC7tXVCjp4y/hKen5QSYAedHwNTfkn6jlN2KZzMD8tX+7xmlPOYlwIUDZSZgpY4Hr Wd/UU4kkfskpYKhq49C5kRK/uJ1NBLk3R9F1kIaYviYVNOyKLbRXQFMndRSxtCaZairEHi tIpG0U8K/KQnu+uwdIjXWmvjJ1R08S0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=LQnKu+Lz; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oql+tuqZAYfpksO0k6qRom322gGfv/vT5StohHuUhIY=; b=EmCEYbwQpLRHd/nyoYdr4Z4HqTjf+4VZi+9pKm8J7+L7CRwgzQLRcNLD2b85ITbymGTOyP tVesebbfZXI23wNkyxQ5edHZsNSDfaubRrjN4FEhDn24+CoLSKofODoxhVNLRW64911Wi8 9hVs6b2utvGM4mlmOYd6GbNyFeXuGyk= X-Stat-Signature: kds7j6ptra9qq7eeog4g8qcujgnbxmh6 X-Rspamd-Queue-Id: C5D931C0070 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=LQnKu+Lz; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1657667613-205675 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang Add mmap and SYSCALL test cases. SYSCALL test cases: - LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result. - Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT). MMAP test cases: - Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed. - Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 134 +++++++++++++++++++++++++++++- 1 file changed, 132 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 4aaf6ad107c3..70c01cc9386b 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -25,11 +26,18 @@ /* Specified test function bits */ #define FUNC_MALLOC 0x1 +#define FUNC_MMAP 0x2 +#define FUNC_SYSCALL 0x4 -#define TEST_MASK 0x1 +#define TEST_MASK 0x7 + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48) #define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -45,6 +53,7 @@ jmp_buf segv_env; static void segv_handler(int sig) { ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); } @@ -57,6 +66,16 @@ static inline int cpu_has_lam(void) return (cpuinfo[0] & (1 << 26)); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 0, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[2] & (1 << 16)); +} + /* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -161,6 +180,68 @@ static int handle_malloc(struct testcases *test) return ret; } +static int handle_mmap(struct testcases *test) +{ + void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + ptr = mmap((void *)test->addr, PAGE_SIZE, PROT_READ | PROT_WRITE, + flags, -1, 0); + if (ptr == MAP_FAILED) { + if (test->addr == HIGH_ADDR) + if (!cpu_has_la57()) + return 3; /* unsupport LA57 */ + return 1; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + ret = 1; + + if (ret == 0) { + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + } + + munmap(ptr, PAGE_SIZE); + return ret; +} + +static int handle_syscall(struct testcases *test) +{ + struct utsname unme, *pu; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + pu = (struct utsname *)get_metadata((uint64_t)&unme, test->lam); + ret = uname(pu); + if (ret < 0) + ret = 1; + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != -1 && ret == 0) + ret = 1; + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -216,11 +297,54 @@ static struct testcases malloc_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negtive] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = { + { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc;\n"); + printf("\t\t0x1:malloc; 0x2:mmap; 0x4:syscall.\n"); printf("\t-h: help\n"); } @@ -257,6 +381,12 @@ int main(int argc, char **argv) if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + if (tests & FUNC_MMAP) + run_test(mmap_cases, ARRAY_SIZE(mmap_cases)); + + if (tests & FUNC_SYSCALL) + run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Jul 12 23:13:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D7BDCCA47F for ; Tue, 12 Jul 2022 23:13:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 629409400E7; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D94D9400E5; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BC809400E8; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 238D99400E7 for ; Tue, 12 Jul 2022 19:13:34 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EFF587E7 for ; Tue, 12 Jul 2022 23:13:33 +0000 (UTC) X-FDA: 79680001506.27.316ACBF Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf13.hostedemail.com (Postfix) with ESMTP id 9819420078 for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667612; x=1689203612; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HxmYw5jXalfdqzgSfm06GGkTNrOQllOgqCBYn+V6Z4k=; b=ZTJf+PCw5qGqTeu0p9Mx2C6Q6f2Bh3XDe2zooZ5Kg7QYM567mn9W5ZV7 MP8M5SwY+Zxnr3kguyA5EGQyP6enlMcJoRljhQF3G969OfYJD2Ju0iyul 9Cw9Yzuw/9JQjND6k4G5jzq3ZLwubxb9PjcfTGxR8aQeWsNyfsQxShUSV Wu9nHkQslKYNlGrFF65lsITjBMiet1h9xlyuUk9TqLGI5iuv0OA+eUzT3 ZQX+JNA/2c17eX9JN6z4+gKkOUgxKiuwKh2ciwHK2MK0cR1UQWfAkabPR 4Md/nBIvDxDX3fJMXNp5htUjbnkFNiB0x/chrnHstR7XQqzIoUMOSIusD w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="285808449" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="285808449" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="772045849" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga005.jf.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 2B904419; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv5 10/13] selftests/x86/lam: Add io_uring test cases for linear-address masking Date: Wed, 13 Jul 2022 02:13:25 +0300 Message-Id: <20220712231328.5294-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667612; a=rsa-sha256; cv=none; b=xq0Vrzs1ip7neNwgX3qzuOxBleestYv4bFNN1gJTCdzRvEviorykrjl8xwFRuVHpawzigw 5FYHQXAMmyW28BcBccl92VanzCm0+J2oV2lyqTADOY2ulbStrGUitmDZtA/pBaCDtDwBTq 1UsgH5RpshwlXuPRhEPpNEn0L/suB84= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZTJf+PCw; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf13.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667612; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JDXzSxMQwZlX/kSdn2bCuiWSjqi6e3Y5keIrAO2aTzA=; b=1a6lwg5VoG669ndpZBKOAKLB286wRm3dTy/jrvHbLllyTI/9KpO7BIwcLIEK/ak0Ci592W Pbmrm2BWe3wWQMHcSWSaEDmsTPLV02R0Q7UPrxSMeiKMv9ZuD7++1IL/c3gw6BiF2f3okL a2gZI1oA8ihZnc2EmKHDLSDXQwsdeyM= Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZTJf+PCw; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf13.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-Stat-Signature: 6ric413aiqihwqm14n45wgazcjd5zjqf X-Rspamd-Queue-Id: 9819420078 X-Rspamd-Server: rspam08 X-HE-Tag: 1657667612-40550 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer. io_uring can deal with these buffers that pointed to pointers with the metadata in high bits. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 343 +++++++++++++++++++++++++++++- 1 file changed, 340 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 70c01cc9386b..d2ae75b3bdc0 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -9,8 +9,12 @@ #include #include #include +#include +#include #include +#include +#include #include "../kselftest.h" #ifndef __x86_64__ @@ -24,12 +28,13 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 -/* Specified test function bits */ +/* Specified test function bits */ #define FUNC_MALLOC 0x1 #define FUNC_MMAP 0x2 #define FUNC_SYSCALL 0x4 +#define FUNC_URING 0x8 -#define TEST_MASK 0x7 +#define TEST_MASK 0xf #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -38,6 +43,13 @@ #define PAGE_SIZE (4 << 10) +#define barrier() ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -47,6 +59,33 @@ struct testcases { const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { + int file_fd; + off_t file_sz; + struct iovec iovecs[]; +}; + +struct io_uring_queue { + unsigned int *head; + unsigned int *tail; + unsigned int *ring_mask; + unsigned int *ring_entries; + unsigned int *flags; + unsigned int *array; + union { + struct io_uring_cqe *cqes; + struct io_uring_sqe *sqes; + } queue; + size_t ring_sz; +}; + +struct io_ring { + int ring_fd; + struct io_uring_queue sq_ring; + struct io_uring_queue cq_ring; +}; + int tests_cnt; jmp_buf segv_env; @@ -242,6 +281,285 @@ static int handle_syscall(struct testcases *test) return ret; } +int sys_uring_setup(unsigned int entries, struct io_uring_params *p) +{ + return (int)syscall(__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter(int fd, unsigned int to, unsigned int min, unsigned int flags) +{ + return (int)syscall(__NR_io_uring_enter, fd, to, min, flags, NULL, 0); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring(struct io_uring_params p, struct io_ring *s) +{ + struct io_uring_queue *sring = &s->sq_ring; + struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + cring->ring_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe); + + if (p.features & IORING_FEAT_SINGLE_MMAP) { + if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + + void *sq_ptr = mmap(0, sring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_SQ_RING); + + if (sq_ptr == MAP_FAILED) { + perror("sub-queue!"); + return 1; + } + + void *cq_ptr = sq_ptr; + + if (!(p.features & IORING_FEAT_SINGLE_MMAP)) { + cq_ptr = mmap(0, cring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_CQ_RING); + if (cq_ptr == MAP_FAILED) { + perror("cpl-queue!"); + munmap(sq_ptr, sring->ring_sz); + return 1; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + + /* Map a queue as mem map */ + s->sq_ring.queue.sqes = mmap(0, p.sq_entries * sizeof(struct io_uring_sqe), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, + s->ring_fd, IORING_OFF_SQES); + if (s->sq_ring.queue.sqes == MAP_FAILED) { + munmap(sq_ptr, sring->ring_sz); + if (sq_ptr != cq_ptr) { + ksft_print_msg("failed to mmap uring queue!"); + munmap(cq_ptr, cring->ring_sz); + return 1; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + + return 0; +} + +/* Init io_uring queues */ +int setup_io_uring(struct io_ring *s) +{ + struct io_uring_params para; + + memset(¶, 0, sizeof(para)); + s->ring_fd = sys_uring_setup(URING_QUEUE_SZ, ¶); + if (s->ring_fd < 0) + return 1; + + return mmap_io_uring(para, s); +} + +/* + * Get data from completion queue. the data buffer saved the file data + * return 0: success; others: error; + */ +int handle_uring_cq(struct io_ring *s) +{ + struct file_io *fi = NULL; + struct io_uring_queue *cring = &s->cq_ring; + struct io_uring_cqe *cqe; + unsigned int head; + off_t len = 0; + + head = *cring->head; + + do { + barrier(); + if (head == *cring->tail) + break; + /* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; + if (cqe->res < 0) + break; + + int blocks = (int)(fi->file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + for (int i = 0; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + } while (1); + + *cring->head = head; + barrier(); + + return (len != fi->file_sz); +} + +/* + * Submit squeue. specify via IORING_OP_READV. + * the buffer need to be set metadata according to LAM mode + */ +int handle_uring_sq(struct io_ring *ring, struct file_io *fi, unsigned long lam) +{ + int file_fd = fi->file_fd; + struct io_uring_queue *sring = &ring->sq_ring; + unsigned int index = 0, cur_block = 0, tail = 0, next_tail = 0; + struct io_uring_sqe *sqe; + + off_t remain = fi->file_sz; + int blocks = (int)(remain + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + while (remain) { + off_t bytes = remain; + void *buf; + + if (bytes > URING_BLOCK_SZ) + bytes = URING_BLOCK_SZ; + + fi->iovecs[cur_block].iov_len = bytes; + + if (posix_memalign(&buf, URING_BLOCK_SZ, URING_BLOCK_SZ)) + return 1; + + fi->iovecs[cur_block].iov_base = (void *)get_metadata((uint64_t)buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + + barrier(); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags = 0; + sqe->opcode = IORING_OP_READV; + sqe->addr = (unsigned long)fi->iovecs; + sqe->len = blocks; + sqe->off = 0; + sqe->user_data = (uint64_t)fi; + + sring->array[index] = index; + tail = next_tail; + + if (*sring->tail != tail) { + *sring->tail = tail; + barrier(); + } + + if (sys_uring_enter(ring->ring_fd, 1, 1, IORING_ENTER_GETEVENTS) < 0) + return 1; + + return 0; +} + +/* + * Test LAM in async I/O and io_uring, read current binery through io_uring + * Set metadata in pointers to iovecs buffer. + */ +int do_uring(unsigned long lam) +{ + struct io_ring *ring; + struct file_io *fi; + struct stat st; + int ret = 1; + char path[PATH_MAX]; + + /* get current process path */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + return 1; + + int file_fd = open(path, O_RDONLY); + + if (file_fd < 0) + return 1; + + if (fstat(file_fd, &st) < 0) + return 1; + + off_t file_sz = st.st_size; + + int blocks = (int)(file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + fi = malloc(sizeof(*fi) + sizeof(struct iovec) * blocks); + if (!fi) + return 1; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring = malloc(sizeof(*ring)); + if (!ring) + return 1; + + memset(ring, 0, sizeof(struct io_ring)); + + if (setup_io_uring(ring)) + goto out; + + if (handle_uring_sq(ring, fi, lam)) + goto out; + + ret = handle_uring_cq(ring); + +out: + free(ring); + + for (int i = 0; i < blocks; i++) { + if (fi->iovecs[i].iov_base) { + uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); + + switch (lam) { + case LAM_U57_BITS: /* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL << 57)); + break; + } + free((void *)addr); + fi->iovecs[i].iov_base = NULL; + } + } + + free(fi); + + return ret; +} + +int handle_uring(struct testcases *test) +{ + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = do_uring(test->lam); + } else { + ret = 2; + } + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -281,6 +599,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING:[Negtive] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + static struct testcases malloc_cases[] = { { .later = 0, @@ -344,7 +678,7 @@ static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:mmap; 0x4:syscall.\n"); + printf("\t\t0x1:malloc; 0x2:mmap; 0x4:syscall; 0x8:io_uring.\n"); printf("\t-h: help\n"); } @@ -387,6 +721,9 @@ int main(int argc, char **argv) if (tests & FUNC_SYSCALL) run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + if (tests & FUNC_URING) + run_test(uring_cases, ARRAY_SIZE(uring_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Jul 12 23:13:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75FE5CCA47F for ; Tue, 12 Jul 2022 23:13:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 567159400E1; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 516BF9400E5; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B7F39400E1; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 203759400E5 for ; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DADB034BE6 for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) X-FDA: 79680001464.23.C1B6FC7 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf21.hostedemail.com (Postfix) with ESMTP id 2F18D1C0070 for ; Tue, 12 Jul 2022 23:13:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667612; x=1689203612; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MGBbR/k1AssmUS4ywd/fm01GaItlo8jGpggyRDWPYcs=; b=SWzxaMp/XvanRcMmr/w96zZ0xe4wq3bhnr1Lt7KVbEoE8NXcXZUr8Lld mBV0E0akm/oTwnJ+clzAMm15eK+G/RQZXGsVHF5Z7m3yoAbKli4JiL0+3 w+lExHfDEt1WYNTqeOUUy7TSrpL1ij2+p8HHJte4YonQb/JATKAkzVUYJ zUuoAmLWPJ8DhsuP42L+dRqL+XZ2NmZAu71mZM10zKRTmcm/hbKmOVNzr ka1Q/QHIDCqupNFRCaO0W6uGBZwq5CV0XQ10wJkLeAc193k4pBQswggPz VIKzLMTJKhxEUOtykEUX5U9se1MVqbHght9YtmvPsdyhAOBYSYAA9Rs75 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="310690826" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="310690826" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="841547745" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga006.fm.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 39A185F6; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv5 11/13] selftests/x86/lam: Add inherit test cases for linear-address masking Date: Wed, 13 Jul 2022 02:13:26 +0300 Message-Id: <20220712231328.5294-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667612; a=rsa-sha256; cv=none; b=QDjMBMpeMQKTxKE1WltGS5RuetceuxD21WWVpDm30LCVCbW+oEoRhhX2VxUav3ABtdUn8T kWC0mP4ky9CVECyWyZH4xf4t4lRmxFTRAX98c/Rhs4bnKIL0rRtBCwN9XQHoqshVAF3bfr cf8BxYWMXIbEKYBnLChpKYpN4eXp68Y= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SWzxaMp/"; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667612; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4N3iq6346ZyCjOHabos6w+ZqftD1B6E6osWRIZPXXVQ=; b=Ft4kg80495iHAjmb76PxojjqASuqcdq+KTWJ6KOBtiy965tFssRYX4uKwdTpoG6/6MhxcC 3AYEN7aF3GoJFz4FTkJGznuzvUA2Md1teDuiMameNtE+2DqMyUQpaTmG32xd3U3RgFQLij 1JRRTPPK+k2NRFkm6yY476JsqYWU8M8= X-Stat-Signature: hzc1ko1hzyk9gm6m4mebto65ukb1spyz X-Rspamd-Queue-Id: 2F18D1C0070 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="SWzxaMp/"; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.88) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1657667611-928224 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state. There are two test scenarios: - Fork test cases: These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process. - Execve test cases: Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 124 +++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index d2ae75b3bdc0..fcac5feb47d0 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -34,7 +34,9 @@ #define FUNC_SYSCALL 0x4 #define FUNC_URING 0x8 -#define TEST_MASK 0xf +#define FUNC_INHERITE 0x10 + +#define TEST_MASK 0x1f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -142,6 +144,28 @@ static int set_lam(unsigned long lam) return ret; } +/* + * Set tagged address and read back untag mask. + * check if the untag mask is expected. + */ +static int get_lam(void) +{ + uint64_t ptr = 0; + int ret = -1; + /* Get untagged mask */ + if (syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr) == -1) + return -1; + + /* Check mask returned is expected */ + if (ptr == ~(0x3fULL << 57)) + ret = LAM_U57_BITS; + else if (ptr == -1ULL) + ret = LAM_NONE; + + + return ret; +} + /* According to LAM mode, set metadata in high bits */ static uint64_t get_metadata(uint64_t src, unsigned long lam) { @@ -580,6 +604,72 @@ static int fork_test(struct testcases *test) return ret; } +static int handle_execve(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + char path[PATH_MAX]; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + /* Get current binary's path and the binary was run by execve */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + exit(-1); + + /* run binary to get LAM mode and return to parent process */ + if (execlp(path, path, "-t 0x0", NULL) < 0) { + perror("error on exec"); + exit(-1); + } + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + if (ret != LAM_NONE) + return 1; + } + + return 0; +} + +static int handle_inheritance(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + return 1; + } else if (pid == 0) { + /* Set LAM mode in parent process */ + int child_lam = get_lam(); + + exit(child_lam); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + } + + return 0; +} + static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -674,11 +764,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", + }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:mmap; 0x4:syscall; 0x8:io_uring.\n"); + printf("\t\t0x1:malloc; 0x2:mmap; 0x4:syscall; 0x8:io_uring; 0x10:inherit;\n"); printf("\t-h: help\n"); } @@ -698,7 +803,7 @@ int main(int argc, char **argv) switch (c) { case 't': tests = strtoul(optarg, NULL, 16); - if (!(tests & TEST_MASK)) { + if (tests && !(tests & TEST_MASK)) { ksft_print_msg("Invalid argument!\n"); return -1; } @@ -712,6 +817,16 @@ int main(int argc, char **argv) } } + /* + * When tests is 0, it is not a real test case; + * the option used by test case(execve) to check the lam mode in + * process generated by execve, the process read back lam mode and + * check with lam mode in parent process. + */ + if (!tests) + return (get_lam()); + + /* Run test cases */ if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); @@ -724,6 +839,9 @@ int main(int argc, char **argv) if (tests & FUNC_URING) run_test(uring_cases, ARRAY_SIZE(uring_cases)); + if (tests & FUNC_INHERITE) + run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Jul 12 23:13:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E647DC43334 for ; Tue, 12 Jul 2022 23:13:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D3599400E4; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 982E69400E3; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 823569400E2; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 70DFA9400E1 for ; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 2EC8D1208E6 for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) X-FDA: 79680001464.01.1ADF747 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf21.hostedemail.com (Postfix) with ESMTP id 8B53D1C007C for ; Tue, 12 Jul 2022 23:13:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667611; x=1689203611; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6kpq1VPBF3XlDpY7P9nLlIhIxMc1N9UleTXT2cgue9s=; b=jBKMjwZCXu/jKKYXXggynjoweS7LIuWOdWnVO2oRTSrlMD9dg8bCNjya LeC8bXYbbWTY0hbh0X63cVNUFzIE30G01Ftu2i8uVYN7wH3kAXGDNnUfX ogpejjGaIH1OhUxEBTCBqo6go91kmdx7dfE8uv1FC4f2ab+/8CvvMvbAe gCfzWc+jYvKNAsQwY3Z3P0vPizMGMmKYvTM49B9GqXXHJYIa2bHuxQSwL 1J77TGuZ6sZWr266F0d/+d/B2R2NzOL7ej/COpCaJa/VICLhs5jflaL5/ ocO0QIgijPrVzaYQo77WS5gNoTjPU4w0JnFPz7DawC1xLOy4+SY8fICeh w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286196176" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286196176" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="722131606" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga004.jf.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 4227A6CD; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv5 OPTIONAL 12/13] x86/mm: Extend LAM to support to LAM_U48 Date: Wed, 13 Jul 2022 02:13:27 +0300 Message-Id: <20220712231328.5294-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667611; a=rsa-sha256; cv=none; b=LjGocMQ/HaxdhvlDbVphS7QNV0RUCaaAzXpoUZLrYirU2EB78w+6BxBqmU1g1R+c3Cv5qb 6tH+Ia1ucKZpWk9205xhYoL+gGa0befdeD20+dBkhKF98InF9oRDHoM3WaAETmkwoiqipa urjW/bLw3+jvhlEm+TuIcLmq9ocU3Qk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jBKMjwZC; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9ucKyndM4WLt/vUK4/I4AyINLEfZmR1mDMwoFuS059E=; b=C64uA/MqgcX0obtNp4KxN5LX/wl5HQg3F5JSOSP1TWOKNVDi1PxFqa6AQRaumxF6iHUkOB U9+gr9fgQujzr1XHHjYj4VMIDt/w/76Yw2BwEt1k1VQJo49Lrm+2SRWG26lUyd5BWkXPpj vu/q3fb+h18chPxL4amhyKV97BbZs5o= X-Rspamd-Queue-Id: 8B53D1C007C Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jBKMjwZC; spf=none (imf21.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: 65b8g5fut6gsbf6nboqz9uy46356wwuc X-HE-Tag: 1657667611-319346 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: LAM_U48 allows to encode 15 bits of tags into address. LAM_U48 steals bits above 47-bit for tags and makes it impossible for userspace to use full address space on 5-level paging machine. Make these features mutually exclusive: whichever gets enabled first blocks the other one. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/elf.h | 3 ++- arch/x86/include/asm/mmu_context.h | 13 +++++++++++++ arch/x86/kernel/process_64.c | 23 +++++++++++++++++++++++ arch/x86/kernel/sys_x86_64.c | 5 +++-- arch/x86/mm/hugetlbpage.c | 6 ++++-- arch/x86/mm/mmap.c | 10 +++++++++- 6 files changed, 54 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index cb0ff1055ab1..4df13497a770 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -317,7 +317,8 @@ static inline int mmap_is_ia32(void) extern unsigned long task_size_32bit(void); extern unsigned long task_size_64bit(int full_addr_space); extern unsigned long get_mmap_base(int is_legacy); -extern bool mmap_address_hint_valid(unsigned long addr, unsigned long len); +extern bool mmap_address_hint_valid(struct mm_struct *mm, + unsigned long addr, unsigned long len); extern unsigned long get_sigframe_size(void); #ifdef CONFIG_X86_32 diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b0e9ea23758b..3736f41948e9 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -263,6 +263,19 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, unsigned long __get_current_cr3_fast(void); +#ifdef CONFIG_X86_5LEVEL +static inline bool full_va_allowed(struct mm_struct *mm) +{ + /* LAM_U48 steals VA bits above 47-bit for tags */ + return mm->context.lam_cr3_mask != X86_CR3_LAM_U48; +} +#else +static inline bool full_va_allowed(struct mm_struct *mm) +{ + return false; +} +#endif + #include #endif /* _ASM_X86_MMU_CONTEXT_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 82a19168bfa4..cfa2e42a135a 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -761,6 +761,16 @@ static void enable_lam_func(void *mm) set_tlbstate_cr3_lam_mask(lam_mask); } +static bool lam_u48_allowed(void) +{ + struct mm_struct *mm = current->mm; + + if (!full_va_allowed(mm)) + return true; + + return find_vma(mm, DEFAULT_MAP_WINDOW) == NULL; +} + static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) { int ret = 0; @@ -768,6 +778,10 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) if (!cpu_feature_enabled(X86_FEATURE_LAM)) return -ENODEV; + /* lam_u48_allowed() requires mmap_lock */ + if (mmap_write_lock_killable(mm)) + return -EINTR; + mutex_lock(&mm->context.lock); /* Already enabled? */ @@ -782,6 +796,14 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) } else if (nr_bits <= 6) { mm->context.lam_cr3_mask = X86_CR3_LAM_U57; mm->context.untag_mask = ~GENMASK(62, 57); + } else if (nr_bits <= 15) { + if (!lam_u48_allowed()) { + ret = -EBUSY; + goto out; + } + + mm->context.lam_cr3_mask = X86_CR3_LAM_U48; + mm->context.untag_mask = ~GENMASK(62, 48); } else { ret = -EINVAL; goto out; @@ -793,6 +815,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); out: mutex_unlock(&mm->context.lock); + mmap_write_unlock(mm); return ret; } diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c index 8cc653ffdccd..5ea6aaed89ba 100644 --- a/arch/x86/kernel/sys_x86_64.c +++ b/arch/x86/kernel/sys_x86_64.c @@ -21,6 +21,7 @@ #include #include +#include /* * Align a virtual address to avoid aliasing in the I$ on AMD F15h. @@ -182,7 +183,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, /* requesting a specific address */ if (addr) { addr &= PAGE_MASK; - if (!mmap_address_hint_valid(addr, len)) + if (!mmap_address_hint_valid(mm, addr, len)) goto get_unmapped_area; vma = find_vma(mm, addr); @@ -203,7 +204,7 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, * !in_32bit_syscall() check to avoid high addresses for x32 * (and make it no op on native i386). */ - if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall()) + if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall() && full_va_allowed(mm)) info.high_limit += TASK_SIZE_MAX - DEFAULT_MAP_WINDOW; info.align_mask = 0; diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index a0d023cb4292..9fdc8db42365 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -18,6 +18,7 @@ #include #include #include +#include #if 0 /* This is just for testing */ struct page * @@ -103,6 +104,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, unsigned long pgoff, unsigned long flags) { struct hstate *h = hstate_file(file); + struct mm_struct *mm = current->mm; struct vm_unmapped_area_info info; info.flags = VM_UNMAPPED_AREA_TOPDOWN; @@ -114,7 +116,7 @@ static unsigned long hugetlb_get_unmapped_area_topdown(struct file *file, * If hint address is above DEFAULT_MAP_WINDOW, look for unmapped area * in the full address space. */ - if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall()) + if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall() && full_va_allowed(mm)) info.high_limit += TASK_SIZE_MAX - DEFAULT_MAP_WINDOW; info.align_mask = PAGE_MASK & ~huge_page_mask(h); @@ -161,7 +163,7 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, if (addr) { addr &= huge_page_mask(h); - if (!mmap_address_hint_valid(addr, len)) + if (!mmap_address_hint_valid(mm, addr, len)) goto get_unmapped_area; vma = find_vma(mm, addr); diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c index c90c20904a60..aa0086722a38 100644 --- a/arch/x86/mm/mmap.c +++ b/arch/x86/mm/mmap.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "physaddr.h" @@ -35,6 +36,8 @@ unsigned long task_size_32bit(void) unsigned long task_size_64bit(int full_addr_space) { + if (!full_va_allowed(current->mm)) + return DEFAULT_MAP_WINDOW; return full_addr_space ? TASK_SIZE_MAX : DEFAULT_MAP_WINDOW; } @@ -170,6 +173,7 @@ const char *arch_vma_name(struct vm_area_struct *vma) /** * mmap_address_hint_valid - Validate the address hint of mmap + * @mm: Address space * @addr: Address hint * @len: Mapping length * @@ -206,11 +210,15 @@ const char *arch_vma_name(struct vm_area_struct *vma) * the failure of such a fixed mapping request, so the restriction is not * applied. */ -bool mmap_address_hint_valid(unsigned long addr, unsigned long len) +bool mmap_address_hint_valid(struct mm_struct *mm, + unsigned long addr, unsigned long len) { if (TASK_SIZE - len < addr) return false; + if (addr + len > DEFAULT_MAP_WINDOW && !full_va_allowed(mm)) + return false; + return (addr > DEFAULT_MAP_WINDOW) == (addr + len > DEFAULT_MAP_WINDOW); } From patchwork Tue Jul 12 23:13:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 12915804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 025B5C43334 for ; Tue, 12 Jul 2022 23:13:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 05F209400E3; Tue, 12 Jul 2022 19:13:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00CEC9400E1; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF2179400E3; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AEED19400E1 for ; Tue, 12 Jul 2022 19:13:32 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7A31C3430A for ; Tue, 12 Jul 2022 23:13:32 +0000 (UTC) X-FDA: 79680001464.12.81AAAF2 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf31.hostedemail.com (Postfix) with ESMTP id 647F42005E for ; Tue, 12 Jul 2022 23:13:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657667611; x=1689203611; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ccx0RgNBfz7piuZWOrmJMdKV3R7Gsc2VojQmQ6vSEyk=; b=ifL0xduc/fDMqiPNBhFClh9G4EwjNcFoQlebl7l5MhYE6iQUsTBgmgk9 UAqniLm9Sa1UqsLs2BatJb6rtuqDP9rxxEyRFPOB94Gq+6+zCWztIAmax ZapTFsw0bL9i7cPkJ8YBEp6VkQAneDHuanlDOjTSHyph+qy6UWD7vjmF8 zXO1W/8n0KkXtn120Vl/LjDJ7ySeIG2l4R6ZKg+S8NE38dgESgi7vNH5C d6ju6jfVuoem4RqedvC1IiDjf2d6fy9iUltcU53pQzdfTh5GDDVf+bW3e 8erim6MGybFleHelJlulHGawl7Xy5qi+mWWwf6+qkwJ3BhiSi6gQK1G/A g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="283818845" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="283818845" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 16:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="592792970" Received: from black.fi.intel.com ([10.237.72.28]) by orsmga007.jf.intel.com with ESMTP; 12 Jul 2022 16:13:26 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 4ABDA6CE; Wed, 13 Jul 2022 02:13:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv5 OPTIONAL 13/13] selftests/x86/lam: Add tests cases for LAM_U48 Date: Wed, 13 Jul 2022 02:13:28 +0300 Message-Id: <20220712231328.5294-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> References: <20220712231328.5294-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657667612; a=rsa-sha256; cv=none; b=fuWtm6B/Qda19RI1ddpouK20KK1Rx4lo9RC60OAo53MedoglFndzjhiljosUPIrYsFLSTF aaP0q5NOR4HIgKDB8yUS2A55rZtkx/iAL8eyBcUxOEdm8lS70CxL3NrfN/7vlf5H6j9c20 E/oBUAFnXSWAIciQ4Em5LOc5ADN+Dco= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ifL0xduc; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657667612; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=31JxGViB2pQm3/cLN7sZoD3z+SlDfWoeE5FS7dq/Ck4=; b=3AAsUdZxP0jnTflPgYfHbcXJjOSabYaM3vEqV4Ilh3c57uMMmCoqksFGvpjA0n1SI2DuC6 VGIvnePuivki6oqJMW+r9bgKoqZe8S17Hd7ksnN8WrXB6vUSbkom2ld7/7fpWKlY6jUVDY f2MxcZz3oVfruL6mbS0tkKAk2TeCzGE= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 647F42005E Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ifL0xduc; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf31.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=kirill.shutemov@linux.intel.com X-Stat-Signature: 83gbse6kftsyof5x7gdiw1tu4oi8wz6k X-Rspam-User: X-HE-Tag: 1657667611-853290 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Weihong Zhang LAM supports configurations that differ regarding which pointer bits are masked. With LAM_U48, bits 62:48 in pointers can be masked as metadata bits, the width of LAM is 15. Add test cases in existed test scenarios: MALLOC: - Enable LAM_U48, masks bits 48:62 of user pointers as metadata, the process can dereference these pointers. MMAP: - Enable LAM_U48, mmaping with high address (above bits 47) have to be failed, which lead to trigger SIGSEGV. - LAM_U48 can't be enabled if there is a mmaping with high address(above bits 47) before enable LAM_U48. SYSCALL: - LAM supports set metadata in high bits 62:48 (LAM48) of user process, pass these pointers to SYSCALL, SYSCALL can dereference pointers and return correct result. IO_URING: - Add LAM_U48 test on IO_URING, Enable LAM_U48, set metadata in bits 62:48 of buffers, IO_URING can handle these buffers well. FORK/EXEC: - Add LAM_U48 test cases in inherit scenarios. these cases should same as LAM_U57; Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 67 ++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index fcac5feb47d0..b354e57bf072 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -24,6 +24,7 @@ /* LAM modes, these definitions were copied from kernel code */ #define LAM_NONE 0 #define LAM_U57_BITS 6 +#define LAM_U48_BITS 15 /* arch prctl for LAM */ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 @@ -126,7 +127,7 @@ static int set_lam(unsigned long lam) int ret = 0; uint64_t ptr = 0; - if (lam != LAM_U57_BITS && lam != LAM_NONE) + if (lam != LAM_U48_BITS && lam != LAM_U57_BITS && lam != LAM_NONE) return -1; /* Skip check return */ @@ -138,6 +139,8 @@ static int set_lam(unsigned long lam) /* Check mask returned is expected */ if (lam == LAM_U57_BITS) ret = (ptr != ~(0x3fULL << 57)); + else if (lam == LAM_U48_BITS) + ret = (ptr != ~(0x7fffULL << 48)); else if (lam == LAM_NONE) ret = (ptr != -1ULL); @@ -161,6 +164,8 @@ static int get_lam(void) ret = LAM_U57_BITS; else if (ptr == -1ULL) ret = LAM_NONE; + else if (ptr == ~(0x7fffULL << 48)) + ret = LAM_U48_BITS; return ret; @@ -176,6 +181,9 @@ static uint64_t get_metadata(uint64_t src, unsigned long lam) metadata = rand(); switch (lam) { + case LAM_U48_BITS: /* Set metadata in bits 62:48 */ + metadata = (src & ~(0x7fffULL << 48)) | ((metadata & 0x7fff) << 48); + break; case LAM_U57_BITS: /* Set metadata in bits 62:57 */ metadata = (src & ~(0x3fULL << 57)) | ((metadata & 0x3f) << 57); break; @@ -552,6 +560,9 @@ int do_uring(unsigned long lam) uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); switch (lam) { + case LAM_U48_BITS: /* Clear bits 62:48 */ + addr = (addr & ~(0x7fffULL << 48)); + break; case LAM_U57_BITS: /* Clear bits 62:57 */ addr = (addr & ~(0x3fULL << 57)); break; @@ -696,6 +707,12 @@ static struct testcases uring_cases[] = { .test_func = handle_uring, .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", }, + { + .later = 0, + .lam = LAM_U48_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U48. Dereferencing pointer with metadata.\n", + }, { .later = 1, .expected = 1, @@ -712,6 +729,12 @@ static struct testcases malloc_cases[] = { .test_func = handle_malloc, .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", }, + { + .later = 0, + .lam = LAM_U48_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U48. Dereferencing pointer with metadata.\n", + }, { .later = 1, .expected = 2, @@ -728,6 +751,12 @@ static struct testcases syscall_cases[] = { .test_func = handle_syscall, .msg = "SYSCALL: LAM_U57. syscall with metadata\n", }, + { + .later = 0, + .lam = LAM_U48_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U48. syscall with metadata\n", + }, { .later = 1, .expected = 1, @@ -738,6 +767,14 @@ static struct testcases syscall_cases[] = { }; static struct testcases mmap_cases[] = { + { + .later = 0, + .expected = 2, + .lam = LAM_U48_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: [Negtive] First LAM_U48, then High address.\n", + }, { .later = 1, .expected = 0, @@ -746,6 +783,14 @@ static struct testcases mmap_cases[] = { .test_func = handle_mmap, .msg = "MMAP: First mmap high address, then set LAM_U57.\n", }, + { + .later = 1, + .expected = 1, + .lam = LAM_U48_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: [Negtive] First mmap high address, then set LAM_U48.\n", + }, { .later = 0, .expected = 0, @@ -762,6 +807,14 @@ static struct testcases mmap_cases[] = { .test_func = handle_mmap, .msg = "MMAP: First LAM_U57, then Low address.\n", }, + { + .later = 0, + .expected = 0, + .lam = LAM_U48_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U48, then low address.\n", + }, }; static struct testcases inheritance_cases[] = { @@ -771,12 +824,24 @@ static struct testcases inheritance_cases[] = { .test_func = handle_inheritance, .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", }, + { + .expected = 0, + .lam = LAM_U48_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U48, child process should get LAM mode same as parent\n", + }, { .expected = 0, .lam = LAM_U57_BITS, .test_func = handle_execve, .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", }, + { + .expected = 0, + .lam = LAM_U48_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U48, child process should get disabled LAM mode\n", + }, }; static void cmd_help(void)