From patchwork Sat Feb 24 15:05:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13570513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17262C54E49 for ; Sat, 24 Feb 2024 15:06:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DDC16B0082; Sat, 24 Feb 2024 10:06:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48D726B0085; Sat, 24 Feb 2024 10:06:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 306DD6B0087; Sat, 24 Feb 2024 10:06:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1D0046B0082 for ; Sat, 24 Feb 2024 10:06:24 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B4887A0360 for ; Sat, 24 Feb 2024 15:06:23 +0000 (UTC) X-FDA: 81827023446.14.3928FDE Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2083.outbound.protection.outlook.com [40.107.96.83]) by imf15.hostedemail.com (Postfix) with ESMTP id 652AFA0024 for ; Sat, 24 Feb 2024 15:06:19 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=tSsTR9Yg; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf15.hostedemail.com: domain of ankita@nvidia.com designates 40.107.96.83 as permitted sender) smtp.mailfrom=ankita@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708787179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=LUZDIzYliEDlkf7KLAlMPoVkGBFu7l2QbwRHkYMVkeU=; b=USsN6nI0f//Xkq4MqSkgUccgB+++zUPd8m8I5KYYKKCcyj+T4dTTvQ0S8yqvMCEWNaqYdD yz2WltBnDAn64s2LhZBva6zq31AUKA1fDAwPD6K6IG6vicW0D22hXyzyer7bTzAMtNH8nj /sR4GhQS37kAq82vSMrNJMv2kQxEk2s= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=tSsTR9Yg; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf15.hostedemail.com: domain of ankita@nvidia.com designates 40.107.96.83 as permitted sender) smtp.mailfrom=ankita@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1708787179; a=rsa-sha256; cv=pass; b=mkBvG41MUeDN3WP0oqew5mb3VZcB66bk4av0xaNiyEYcOWrc3aXtL2dGaPbZT4vhxyjhza WRFEB9QTsZAa4prxrGjd6nh1ZtjHdE3lb505sTUxcqbCaT8s+fYrgKvsP34xD2zRwlD+oo W4bSNZeO+4PFhxx9AQCIaNVy9GaotZk= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FGgjjOf24/2cHpqOuwbkHPeOLrgWSiUzfKnKDbs7BmhlUc7suBQY2LPnF5509SJIxCGw/yYsly1kewFecCnG4xFUyC2iqmi74ikcMNx8rL2JtiiKI12zrkFXxnI1XqHUXguCRadmb8LBOtnKLYaTDDlASNyX86Unb6Avy1PlwmTtrC7EsfcSxTkqnqsGCD0n4ALEV4tRZqS/M34EyuCkkhB3qKVO27mUYx7JIrfGTyvD6IJrHG2nlWxnC0fN0d4SbhzUWK9DgePQGpObSydGo4w0dA/wMiZlYXwi70c+qses2wOcQUwJj7Yekt/SYwzkQb97+OWUFHLm/jB7MXg8Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LUZDIzYliEDlkf7KLAlMPoVkGBFu7l2QbwRHkYMVkeU=; b=lzm1MkhfsNnnLt/9rCDyWRFq9WuBgkEsIyyZeGO4X7paPL1Lxq3nX2AhvLNk5s9+odEeGR+umhoLVldp4dTMfRlFvQmkuj9OcU0oEYau399hUfNSuVpbGzAP4sPdt5aZvm+6iYQS8+2SQa8WsLYLi7L+ZIFLwTmwJ1723K7HDCaNOnzj5UNv5mn/MccxVedytlRD90d6yEPHS/Dx48aMXiY/b01/4hHOrJQ0c592vjSP6GhtGRsBYNiXAKo+ey9sS14k98wx5isRSwiAnUqnnQHyUtmd9pklUCBCWxkmR+8YeLqY4T1688fvwvRQvfMZkh2XTgphemDq7f2u8Lo+QA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LUZDIzYliEDlkf7KLAlMPoVkGBFu7l2QbwRHkYMVkeU=; b=tSsTR9YgA33hI8qhPsemnUL65Vx6oQcWh9EegAN18ZJxG6NIFDXbLojvPO7etL/fjezPf9b+5rzT0K85Wy4A/TL8t0JoWL3BF+3mipkl6zBbpt0fW4OfLL4bYoTUMMqOTDmc2sJjlY0M/sMliQ0XJGtqaO/5Lw7V16Eh8ymxUfBOx8BWOfyGSVvB2Wg6YTiO9M3r0TYTy/f0UHasK1W40fdh6Lb7qsnXROHo1aOWU1Q4xKGmTAfAFtYNflSirqrVseRcOdX06PzjXkRkwGFVuvT2ixRCu03mg0GT+enHeD94IITMiRW6bWhgQQvglpLimRqahHWE8vlNM9W+Kb2adQ== Received: from CH2PR19CA0011.namprd19.prod.outlook.com (2603:10b6:610:4d::21) by DS0PR12MB7678.namprd12.prod.outlook.com (2603:10b6:8:135::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.24; Sat, 24 Feb 2024 15:06:12 +0000 Received: from DS3PEPF000099D7.namprd04.prod.outlook.com (2603:10b6:610:4d:cafe::70) by CH2PR19CA0011.outlook.office365.com (2603:10b6:610:4d::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.48 via Frontend Transport; Sat, 24 Feb 2024 15:06:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF000099D7.mail.protection.outlook.com (10.167.17.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.25 via Frontend Transport; Sat, 24 Feb 2024 15:06:12 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Sat, 24 Feb 2024 07:06:02 -0800 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Sat, 24 Feb 2024 07:06:00 -0800 Received: from sgarnayak-dt.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12 via Frontend Transport; Sat, 24 Feb 2024 07:05:49 -0800 From: To: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , CC: , , , , , , , , , , , , , , Subject: [PATCH v9 0/4] KVM: arm64: Allow the VM to select DEVICE_* and NORMAL_NC for IO memory Date: Sat, 24 Feb 2024 20:35:42 +0530 Message-ID: <20240224150546.368-1-ankita@nvidia.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D7:EE_|DS0PR12MB7678:EE_ X-MS-Office365-Filtering-Correlation-Id: 8b8585ce-199d-4d54-838f-08dc354a23fa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ktApkWdr0NlyvDIrtx6nokZHwdbcm8ERQF4MS21EeZynfmfRI1GDbnOWa3teX3388pJZ8ENP/VjXEDHlD9T0lu6IQnIL0V/+AQ9jzXlIK2oyLWL98vAOCmTf7Bsxivd5iLALVrELUNalpSczNgzSJ/Qpq+oNdZTlU4Ve9Yyma1iwAxET02HF5U8/hC3LbTg5VQkBSQHIe5DhHhmZPN1qsZapRS3e1hi5/qK8ubDTg1D7xQAyQLVpg2iAJcVBslelHT3awCLEcwoAdYueLg5dfybeReKFNxLJo8s2zlXs0l+7/5yjz6/OOe7g3DoonoW2hjReJRb+bgEd0qg0H5KYGjVUTOdUIoHHhIO2/8zh4MzwwuzjRjg6i3bHt/RO+rGdmuXWvDn+U+5Y3BnDdR3nhbsx3Gi7sVjX8LQz7r9bAm2njTNow90Gi7k0A8Qz2uhN588z6db+Malhtq4uh5JAkvZgwdsupZSH5audOVIdKdUR2h42zx/shx5oABuy09Fqj7T5BK+VmaJP+fIx7m6UdQ8l5ymQUU5Z0H+HDmxYnRPf2pT1e6JNREWIstPs52Z9Sfhgg6NM6I/mT+qJIICE7C8r3nkVePtppg4+tzFpIgVijl6JOLaOtlkZEhTu2yxFCgzONnAakhcDedEpVp5Y4DAf23AFbyQy96aYhTSAlCYrwrHfVdf+6maE6kVA8CMdf3BTgYtQYS0DIy0khMS0+Q== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(230273577357003)(36860700004)(40470700004)(46966006)(921011);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2024 15:06:12.4427 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8b8585ce-199d-4d54-838f-08dc354a23fa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D7.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7678 X-Rspamd-Queue-Id: 652AFA0024 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: y9d4hmy6oqi9wper1s5aghh31yzcog3t X-HE-Tag: 1708787179-800102 X-HE-Meta: U2FsdGVkX1+BsJHhYudSRU8yKxUQpu8xdERXYqSV9/rBFaj7cPreoqswKKpb2fD3hscJKQ5RtpHPeamwmO1a1pPVID7hnOJph7eyvZdYs+Pg4qY8SdY103gd8Ybfhk1DK7to/+rBz3h/T+dnMdU1JsIFY0npN9hto84aYTLU6l6FNhewFKTBEImkj+5VtLWduWcHD8lGvxgOw4ulYuAXIncIET3CNfEK/dSj/xO/nMzwqn+vV2P5T8LAlbi4hu4wD7x6LHaWDSA4PqpQlo0VmCTgFMcOeOjyRqVDOdQZSPSyHcP9/HbEBnFpZJORoKjRPZxKfeybmlrfiHQt3LJds0HPO4jJSAXQ1Syk7pXsF2yVZPMBXD78rV/W8Ta142t0VUJSu45e7HIgsgczUXiC80qE2RYmy5JJ6xNhUhHIEzYDTtg/xCM0H1yx2/3XBTT/Yr009lu/x+qWKLdmWGEKHJdcXYzoxtEEMRxLDgAeOEzWUfa/XbBExMH4UoPe8VIv6H94EnVEDFxQvzL9REVB0JWxttk7NKa/NLqVb2JwrNkJmvtkqi+8NaTo6L4AbSM2HipM98DSJMvAnHLSbLnEJNuxW5AuswYo9u41Y0y87DJfQxt/t8+aAngGQbEyvq50pVfW0A22dtmg1JawRdLeFE3x+IY4aM65sCbsLyGLcgIxmMnAZtrG2U2k70B1Wg5qMwZUCG2ccNKjLtQQyKk4UC0E9PBSung5TsVtdjUjjrr/NtejCjh4HPawx6lzJIQff+AAu2sc6eFYaEN+bd7j5j/8eYicsgv7yOtZM2XdOE3tOKnPJfg9a7k8LNvIP+6TtsK8Xriokd7rHUEk6HIRkqpQYibmZ7m8j7C+2Ai8Fh+rjg6iqFum5Ypxe+xHo48MW86NcrjyraoHzSolgAxEU9iei3F9DrSHD14Y0yYn3H4xMWP5su9YRDXixIkHnzlGU3SGBAtUn7Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Ankit Agrawal Currently, KVM for ARM64 maps at stage 2 memory that is considered device with DEVICE_nGnRE memory attributes; this setting overrides (per ARM architecture [1]) any device MMIO mapping present at stage 1, resulting in a set-up whereby a guest operating system cannot determine device MMIO mapping memory attributes on its own but it is always overridden by the KVM stage 2 default. This set-up does not allow guest operating systems to select device memory attributes independently from KVM stage-2 mappings (refer to [1], "Combining stage 1 and stage 2 memory type attributes"), which turns out to be an issue in that guest operating systems (e.g. Linux) may request to map devices MMIO regions with memory attributes that guarantee better performance (e.g. gathering attribute - that for some devices can generate larger PCIe memory writes TLPs) and specific operations (e.g. unaligned transactions) such as the NormalNC memory type. The default device stage 2 mapping was chosen in KVM for ARM64 since it was considered safer (i.e. it would not allow guests to trigger uncontained failures ultimately crashing the machine) but this turned out to be asynchronous (SError) defeating the purpose. For these reasons, relax the KVM stage 2 device memory attributes from DEVICE_nGnRE to Normal-NC. Generalizing to other devices may be problematic, however. E.g. GICv2 VCPU interface, which is effectively a shared peripheral, can allow a guest to affect another guest's interrupt distribution. Hence limit the change to VFIO PCI as caution. This is achieved by making the VFIO PCI core module set a flag that is tested by KVM to activate the code. This could be extended to other devices in the future once that is deemed safe. [1] section D8.5 - DDI0487J_a_a-profile_architecture_reference_manual.pdf Applied over v6.8-rc5. History ======= v8 -> v9 - Collected Reviewed-by and Acked-by. - Updated the commit messages in 2/4 and 4/4 to passive voice and fix spelling error. - Updated subjects to align with convention of using capitalized first letter. - Added links in 1/4 on the previous conversation for tracking purpose. v7 -> v8 - Changed commit message of patches 2/4 and 4/4 to include detailed description of the VM_ALLOW_ANY_UNCACHED flag posted by Jason in the commit message. - Added more detailed comment in the vfio_pci_core about VM_ALLOW_ANY_UNCACHED flag. - Rebased to v6.8-rc5. v6 -> v7 - Changed VM_VFIO_ALLOW_WC to VM_ALLOW_ANY_UNCACHED based on suggestion from Alex Williamson. - Refactored stage2_set_prot_attr() based on Will's suggestion to reorganize the switch cases. Also updated the case to return -EINVAL when both KVM_PGTABLE_PROT_DEVICE and KVM_PGTABLE_PROT_NORMAL_NC set. - Fixed nits pointed by Oliver and Catalin. v5 -> v6 - Rebased to v6.8-rc2 v4 -> v5 - Moved the cover letter description text to patch 1/4. - Cleaned up stage2_set_prot_attr() based on Marc Zyngier suggestions. - Moved the mm header file changes to a separate patch. - Rebased to v6.7-rc3. v3 -> v4 - Moved the vfio-pci change to use the VM_VFIO_ALLOW_WC into separate patch. - Added check to warn on the case NORMAL_NC and DEVICE are set simultaneously. - Fixed miscellaneous nitpicks suggested in v3. v2 -> v3 - Added a new patch (and converted to patch series) suggested by Catalin Marinas to ensure the code changes are restricted to VFIO PCI devices. - Introduced VM_VFIO_ALLOW_WC flag for VFIO PCI to communicate with VMM. - Reverted GIC mapping to DEVICE. v1 -> v2 - Updated commit log to the one posted by Lorenzo Pieralisi (Thanks!) - Added new flag to represent the NORMAL_NC setting. Updated stage2_set_prot_attr() to handle new flag. v8 Link: https://lore.kernel.org/all/20240220072926.6466-1-ankita@nvidia.com/ Suggested-by: Jason Gunthorpe Acked-by: Catalin Marinas Signed-off-by: Ankit Agrawal Ankit Agrawal (4): KVM: arm64: Introduce new flag for non-cacheable IO memory mm: Introduce new flag to indicate wc safe KVM: arm64: Set io memory s2 pte as normalnc for vfio pci device vfio: Convey kvm that the vfio-pci device is wc safe arch/arm64/include/asm/kvm_pgtable.h | 2 ++ arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/kvm/hyp/pgtable.c | 24 +++++++++++++++++++----- arch/arm64/kvm/mmu.c | 14 ++++++++++---- drivers/vfio/pci/vfio_pci_core.c | 19 ++++++++++++++++++- include/linux/mm.h | 14 ++++++++++++++ 6 files changed, 65 insertions(+), 10 deletions(-)