From patchwork Tue Dec 27 03:08:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 13082048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93620C46467 for ; Tue, 27 Dec 2022 03:09:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B72AA8E000A; Mon, 26 Dec 2022 22:08:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 926478E000D; Mon, 26 Dec 2022 22:08:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5274E8E000A; Mon, 26 Dec 2022 22:08:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 353508E0005 for ; Mon, 26 Dec 2022 22:08:56 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 129731C60A7 for ; Tue, 27 Dec 2022 03:08:56 +0000 (UTC) X-FDA: 80286604272.27.01B2B93 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf23.hostedemail.com (Postfix) with ESMTP id 4242A14000E for ; Tue, 27 Dec 2022 03:08:54 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=j6NpAPRy; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672110534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=+e8B1qGhSypfHkHS/dayIo7FJVoGy1Mn4YDYJhVhukc=; b=7TOc9TIyoONTGlLvXal4X0Z1Jk9YzPRJH/uqKhbuOtVs/wGHRwD0oUyIN/fELIHzYAvtWy IIy7nZnfBFe9OmItdHNe5K47zSeLTaM4SnZTIr9uVqjwakURvXJMj9DVlEY9zCgOTpNNxX 0WyY7bojSW9ArIGMqY4DldHlov/sSYc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=j6NpAPRy; spf=none (imf23.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 134.134.136.24) smtp.mailfrom=kirill.shutemov@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672110534; a=rsa-sha256; cv=none; b=T8aULbIHpKJsKyCrfIXWXW3qrw/TcutIzMg9dy7e/pgemiNFQntkaDzA0VB8gU0BmO8DfW n1zPVnuosmTIZMoejzpXX+CeED4fWtlzxKYm5DdATg2oy8gjd/reHill8KnwLghsdVsfjB q8fpX9hrXyu4Czx2F/UmOSnc3RCwTSs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110534; x=1703646534; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=KrwatNXBHmOV083eFx1LnqNZgGE3IkeAeiegROk1x90=; b=j6NpAPRyF9bnqXNfH25q1MxQmISqReMt5tqYZ5pLtsqwgn8o+6w4fBXj rbJZfi09j0Anodw3rh0EKzPG6sME3WZPWOoDnPhZiu7tO4ZEC6C8LNCwS qrV416NL/0TFYnsLRe79CnRl+Hz7StfGsanrsB8ojoVBiLaWlURXX6YZ0 /grS011yeQoOCX2fel5XPuimzafBwp7P3tC04nQMHdXogqNv8fMTCa3BK VAoBra0sMDdOAK/kKLXOP49fAWkNNKuO5Uxj1jnsuS+5lk6lU2FZqE9G4 BAEbbLx1SM+9GVFg2yuU7A6nwfHp6494nJmz6Z6+NjSWp97QUSLfFtxjx g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869765" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869765" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917524" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917524" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:39 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 7C728109CB8; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 00/16] Linear Address Masking enabling Date: Tue, 27 Dec 2022 06:08:13 +0300 Message-Id: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4242A14000E X-Stat-Signature: 7gs13ssbxx7fudcxi7hmgtr1fnrfmh9c X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1672110534-851228 X-HE-Meta: U2FsdGVkX19UxhenbyE00hxkJ4OiilEmWdt32gJn8lLUY/0ueG/wyq3vKQAVRwXE3LgRakNx6xqLphTHvW0Y/Q8LRL4nKb+oXbD3a2qNSMM05Etk7I/+8L3nM8Ufm8pMUmGb1T6tj7LkpAZekSfN7nDyG00WEzKeMEK40iqioyMpmbOqB++SW/gJPKl9Nt1YHf+ggVETVljlFxfYLJfPQXyqkQME6wIdVfg8/DzWSwx0w8GCYdWilIm1zqyHR/vrwJE1iTmHHm0baUA055UtXLburPWkP3Tauu9fo8dm6d3yVS4UVJchve8Si+AZs3DTn7lQ4uBpOfuyxzxs5mrHcSqqLRDC487lmwhTd8IM4oTXpNea44cKeRVdepoQV3Le4BGLnOFBnGCco6bN5UpVNAtGGF3TTn9v0lA+GDnxwWGx7gMopV0kCa6OUPNeL7HvqqFu0dUZ7XqOLoHQ4ver1ao8s3iMn5jl+5sJwdAbgxjzLf8hSOhV1EXvW0tzxk8giYsQzSpsCAHqD0xAsOTwbJOUDW9Jqtmfmz+Mq5s7XHpdie4PBwI9Wu+HBcF6RVZzz8gk5wqQuq579y7X4DsHtSmKbBNl5msGtoN7HkhznmeYTvgB8FjA770Bp5XhNsrPBdltrP6KtpZw1k0mt8gDkYXafFPeqdRxpH2GziDCFH6lOD4W+qPYY2VpO6eflWdZOECVkh6Cf1HxPR9a2V/EK08AEH6oyV6c2bpPBtBsjqyKvlITWkWC6aeF669gfFfvdKjRVBtV/IYP6H2rRadMwKcaMyDZ7AGko8Vc/+sP7j2M3ippeMQsi+NY/fyLB7NdsKJjAqrR85bMGKg7ZwirH/QDJpyLW09PcWVSP5S+GmP/l50/Wlnn1aq5S1GQoteQio/vbyjsF8FmZDVIoiBmqYfoEiDY8bjMkgfF3fVlMZcPAjsgLhurxdmCHWOdi1iPvXeMYfFbXqQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Linear Address Masking[1] (LAM) modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata. The capability can be used for efficient address sanitizers (ASAN) implementation and for optimizations in JITs and virtual machines. The patchset brings support for LAM for userspace addresses. Only LAM_U57 at this time. Please review and consider applying. git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git lam v13: - Fix race between untagged_addr() and LAM enabling: + Do not allow to enable LAM after the process spawned the second thread; + untagged_addr() untags the address according to rules of the current process; + untagged_addr_remote() can be used for untagging addresses for foreign process. It requires mmap lock for the target process to be taken; v12: - Rebased onto tip/x86/mm; - Drop VM_WARN_ON() that may produce false-positive on race between context switch and LAM enabling; - Adjust comments explain possible race; - User READ_ONCE() in mm_lam_cr3_mask(); - Do not assume &init_mm == mm in initialize_tlbstate_and_flush(); - Ack by Andy; v11: - Move untag_mask to /proc/$PID/status; - s/SVM/SVA/g; - static inline arch_pgtable_dma_compat() instead of macros; - Replace pasid_valid() with mm_valid_pasid(); - Acks from Ashok and Jacob (forgot to apply from v9); v10: - Rebased to v6.1-rc1; - Add selftest for SVM vs LAM; v9: - Fix race between LAM enabling and check that KVM memslot address doesn't have any tags; - Reduce untagged_addr() overhead until the first LAM user; - Clarify SVM vs. LAM semantics; - Use mmap_lock to serialize LAM enabling; v8: - Drop redundant smb_mb() in prctl_enable_tagged_addr(); - Cleanup code around build_cr3(); - Fix commit messages; - Selftests updates; - Acked/Reviewed/Tested-bys from Alexander and Peter; v7: - Drop redundant smb_mb() in prctl_enable_tagged_addr(); - Cleanup code around build_cr3(); - Fix commit message; - Fix indentation; v6: - Rebased onto v6.0-rc1 - LAM_U48 excluded from the patchet. Still available in the git tree; - add ARCH_GET_MAX_TAG_BITS; - Fix build without CONFIG_DEBUG_VM; - Update comments; - Reviewed/Tested-by from Alexander; v5: - Do not use switch_mm() in enable_lam_func() - Use mb()/READ_ONCE() pair on LAM enabling; - Add self-test by Weihong Zhang; - Add comments; v4: - Fix untagged_addr() for LAM_U48; - Remove no-threads restriction on LAM enabling; - Fix mm_struct access from /proc/$PID/arch_status - Fix LAM handling in initialize_tlbstate_and_flush() - Pack tlb_state better; - Comments and commit messages; v3: - Rebased onto v5.19-rc1 - Per-process enabling; - API overhaul (again); - Avoid branches and costly computations in the fast path; - LAM_U48 is in optional patch. v2: - Rebased onto v5.18-rc1 - New arch_prctl(2)-based API - Expose status of LAM (or other thread features) in /proc/$PID/arch_status [1] ISE, Chapter 10. https://cdrdv2.intel.com/v1/dl/getContent/671368 Kirill A. Shutemov (11): x86: Allow atomic MM_CONTEXT flags setting x86: CPUID and CR3/CR4 flags for Linear Address Masking x86/mm: Handle LAM on context switch mm: Introduce untagged_addr_remote() x86/uaccess: Provide untagged_addr() and remove tags before address check x86/mm: Provide arch_prctl() interface for LAM x86/mm: Reduce untagged_addr() overhead until the first LAM user mm: Expose untagging mask in /proc/$PID/status iommu/sva: Replace pasid_valid() helper with mm_valid_pasid() x86/mm/iommu/sva: Make LAM and SVA mutually exclusive selftests/x86/lam: Add test cases for LAM vs thread creation Weihong Zhang (5): selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking selftests/x86/lam: Add io_uring test cases for linear-address masking selftests/x86/lam: Add inherit test cases for linear-address masking selftests/x86/lam: Add ARCH_FORCE_TAGGED_SVA test cases for linear-address masking arch/arm64/include/asm/mmu_context.h | 6 + arch/sparc/include/asm/mmu_context_64.h | 6 + arch/sparc/include/asm/uaccess_64.h | 2 + arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/mmu.h | 16 +- arch/x86/include/asm/mmu_context.h | 49 +- arch/x86/include/asm/processor-flags.h | 2 + arch/x86/include/asm/tlbflush.h | 34 + arch/x86/include/asm/uaccess.h | 52 +- arch/x86/include/uapi/asm/prctl.h | 5 + arch/x86/include/uapi/asm/processor-flags.h | 6 + arch/x86/kernel/process.c | 6 + arch/x86/kernel/process_64.c | 59 +- arch/x86/kernel/traps.c | 6 +- arch/x86/mm/tlb.c | 53 +- drivers/iommu/iommu-sva.c | 8 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/array.c | 6 + fs/proc/task_mmu.c | 2 +- include/linux/ioasid.h | 9 - include/linux/mm.h | 11 - include/linux/mmu_context.h | 14 + include/linux/sched/mm.h | 8 +- include/linux/uaccess.h | 22 + mm/debug.c | 1 + mm/gup.c | 4 +- mm/madvise.c | 5 +- mm/migrate.c | 2 +- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 1241 +++++++++++++++++++ 31 files changed, 1580 insertions(+), 62 deletions(-) create mode 100644 tools/testing/selftests/x86/lam.c