From patchwork Tue Apr 30 17:21:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31B4EC10F16 for ; Tue, 30 Apr 2024 17:22:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hDl8UrBXcp0CLWaj87MR1uszCiRfPf8EHI0uc6W7HVE=; b=g2qzZ7M5ZSuk1p M2AMpFMp2txZCCo7KZPzxAAfC4T55Uunha6GxwBNjH2f9oMSI54S3vw4odfKUSIvmBsRHBD/9+Shq kHiZXsERiOKqtPs0hwEclSyj+KdECAHFKVlsUeJ13XhBVLbAyjRwmP2Vr81q8xVQYwtzr/RV1/ZZY g0I4Tl5dAhUC0kn2W6fQice5rQjsn1UKg9yH+YBfa6JD22/zsYUAsalgENnt8DUUeFW9Vg5ZIEp/S tzE3LXc3GgRr0fhtiW6TamnXMVHdcCqrCmfl9N8mjRr2jlwgai7sKxo78Xn+WD1oIojZXJfJHv8HZ vp3iLFye5W8scsOJstzw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBF-00000007OhO-3F1n; Tue, 30 Apr 2024 17:22:13 +0000 Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rB1-00000007OX2-3anC for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:22:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NbsnZaUM3j5q1aJ6bB0WWzYzDymC2CFHE1J1edegXZUx/A77kwhGTPKOGNU63Fsw9vMwpmZb+fq84iJsw+eNAHpAV0YVEIblOP6PQsD3v33FAchydf1dYFoM/pdnxJ5Q1HGm0AhRS4bdl0Z5c1dA8+UiebSzjQJ/gNjoWUGgEa91mZ5IlKNbYAPEpocDuEg9ivbYUQwnanXNeyJb7f+BnVy1jUbP9mc6La7Zk0tuBeemmMU26epDVqexK0ef+lfu6iSBtPnmxmE6OcSNXbsPmDBeqvHLQXdKbSroTpa9qDyIs4+KyRQuawWWmZyQUIOeQuCs54N0vum+yWutad2HMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NuOQM+tk1hBYURM7kywp08aMkUjx537isuXThrhYqWA=; b=Ap92EnrdsqfOUwPvlLZO4Dm5m9XM9nvyYZvqgtg5IKjTm0wM/kxU3kok6dfso/7nk37eslbS5a4ayiLm5Aa/qV4l4sbczvie98wgZbZkuxVisKAlS84DIb0jta7AgUD3pOSLg+jDDfdOEMn2Chgk3ZSlmLMJHUI9UYcW8wSdAOcFFi/vx0NsFb1aNZZ5yDQwlEtBX+nP/14R8KPhPbe2hQtp/y4ztkC9OU5RsAEK3ZU/4nXWKs0h+2YcEF9jvgMMDhG6qXysdIPlgvZL4SVa4Sd6EcpddfhTMUtgMMkWzK8LHKr9ui/WSrdsSd7UPV4vI2AEmuOW/q1co3yjF0/AYg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NuOQM+tk1hBYURM7kywp08aMkUjx537isuXThrhYqWA=; b=NHNOnKoBdG5TVKL71VcJ/gP1IxM88C42DEzSt/8oQYC8OUVTb3X09AOzRgyUIKmiRLZbaw+Lne5zgoz1NanukODgGlr4NS7BvYpMWhYlbEX+bqGEy7RihM9IJBwTzeqyRlCEZHY7Vi7n1F18vHLClxN7qKZHZ795/LmX/28HkDB2RWCAgs8KV4LWUs2zRB5y4466iou1Aqgc0AjO0UHcebPyr9s0l3ECcHV9gHH082bfU0dshyiI595KgkCxtYEQAjEqKN/B7M2EVY+hupl4WQIic9xl7OEMptkbmEH3SiDBgdvGpJWeTFNg1BLFjz06b8XOKtTAiWhaf5f/S0uoxA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:45 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:45 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 1/9] iommu/arm-smmu-v3: Add an ops indirection to the STE code Date: Tue, 30 Apr 2024 14:21:33 -0300 Message-ID: <1-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BLAPR03CA0139.namprd03.prod.outlook.com (2603:10b6:208:32e::24) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: e567b83b-35f3-43e2-f655-08dc693a014b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: mJRm9jHIgJlHgv700SvHtuO6K45KnnH4C/mHrIttoojFjfU0Sw8mghP20zALtg77SfYbM2v20NgUQ3AAaDYYx2yt1GFxQzZlzrZd1QSrKV1DCsWicIpTrEPyOW8VEgcdc+ftyh31mTPA3A5XsRpG2HDMd5Z9ezHyAPB7yQiVOV34UV0XmiDDEokbhcRVayD+DSKbWNZKAHAOEUCS6vMFGlMOepOyWoATsZ0Lx/TEdbZSlSSmlF/7ayHR+qLT5kTMWm6Mx5kA2ubZFnJf5GCtRc7n3xiF2yBuxeRj4Lcm/d6E7rCUOrJHzr8FOY2NiShoSW7W7vXOleK/HgQlxebFgRniim7JCRr18jhLi73M1tPMey+7sAfSMjm9afss6jmrNLX4RGU62vovdjn8UJ19y2Kj3Lko5TwCs3bGd9HwcsyRjzsETwBMYaO96t823gqMJSLSuxDuoEIPETQ/vwcgpLmrfI1rYQtVTlGavlCN6TNF3SfpzmdsA/TWlSSvJUE/Ugf42H7Q10qL3W/PVPI88LKGXvFUKG7QYSK/FIYne0iSLzXUHjswASeqnDehjABGdzqvPAClmr+dCti+bcAs9VcPYz5DGiExpk/xrY8+ag+RANulRb8g8KJ45Sz31ECUF9GiQ1aj/okAe1vL+wUQqBmpulMk6Hf+nmy+rfpb7E3IaB+1VgWDk25BNJ4pkf3kbL3PdxBEGMRr+/m2Qj/butUtr0yBBPLzfN+/ntqFtIGeJaHMwkIRMBccox3GpG0wljlPsf8B7Vc/8yjqWXlDiCoScQ2qme0Oh/DNwoRf55BUW1UM961ko9nF687K0qkCj1Y87sWWAofGlpnyMJo6n+N/r4NodLqb9GdHJqMbN7bRqQGoY+K8O7T+arXImWdB2is+eZyk/a9T0YtTL4RTMVZR1iqsbpA39EOMsT1HeUWkEBq4JJDiOk9ZDyZwZPiG0f5CgQxcvpw2UramAAjFefDT4fmuy/MKtmDcZdtaca4O6SabZ2aEqF9EgIDxDySV/Aq5E27UAe403mPOtUZOBpyU0hV1V1CQuxuPMGWtoqOc34G4BKuy9LO0zKQZwvhhsNgQ+v2mPX9ReDFu1M5tC0WexrftjKnhf8Q0gfih6n5cD4O8epuvLnFclG57qWQ4r/lIz355Oe/WL84j7oBiuk2PIgtPwg4uWlDEaQ+nMkE8iOOwbwiiGzKQT8oSBFWGm8UjDrGMG11/QZnxtkSnVrkfNSibOSjOlOx57QaC/MclYNYbfp9Vsa6Y7E8u461tKfwNGlWMZY3m/NB9npRntA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: z+OZzhbyUxpk0cI+d3FR6qSTbXIVmLD9vAsT5W6dbHDK7clEbj2ebj83V5UzBmurbCGFhLSkQERnkwZGmiKQiuUcLatd2hzTTLwaIcw5ibuFw+NkaquYyvCb3X0rodo+5SQ2VQhTvcv5/7gDP2vwf9QXZ5KT4zWfuaLaIvhvboZVUYSnpRNmnPE1iTHNKSno2SgUjejcmtUfBNFdDXQ5euq7ADiF05DqJ7JdNfq1j+WxO6QsleAL4ntCwOGFvt97QsVyS+5ZKu3M82Nz/WdSjcc7mP1jFN6uNR1wuv12o++GX+HOiwia52yy0OBnnZIMiGNkjmrxzKL2OuGG3pLqy+w6cAJzuCfkY4NwZNLY3u2vC/+AF/6W4AD3l2PsQBHe4oNto8c35qoK78uS6qlzjO1LakvnsYRsq3zNItsTZ4Qhp4O3Iip4H4OOparkRsWGCd07U2+3bkaFnJecGcNffcIhARfpH2FRze71aHBHSzaDNzHSmyf75O5wrgDRNWLb81wO4m1JHcydofgJbxFBSRB8mC3VDhPAgf0pEVZWEPobWqopwCMk9upAhC4H40UDgiwmUc36pRyDDaL5mj8QbEQs0lpHc+v19+eqzaIt2mhVYXxhyh3Bg6Fm0SI2eP+VUewLraezSpg3sEQpUwlyg+HN2OdfI7hso7HgHl7jkpLw2q4Rf1Vyjr7EjOMkWOFScwny3w+9SD7Zkp0Bz5ggdbSSpLfTOxs5cXK8t9Lqfr9B1q2O4hXiVTNoKMCt2pWblpPTScLjrDk1rsi9p8DjXfLe6ykIIkpSbVGGUbz/kZOXn5n0xL64vbm51CdaeRyRKGLSbrG0KaKKZVwTMl+qtpDpFglmMBbX1XjerPOwhkOum78QwfI3WkciDWSMma/mS+7npgrnJU/Lv3P2myy02mesowSwo9/syFUPg/uAIJ4N/RnmeRsohfr4kG4MCRFBIzNCQHA1kNNNvoNkbEE1nzfEqELw2/GjImGLaCFaVdBE3C42kFc73niM4XoNcs2phBj4d1fK5eH6mR873wMXxmrIKAZ0Xce9649/TFXovt0m66a7fdgmDeGOnbt+DpJr1LKCNlXsqQIEvL2/uHobSNlBF5lQudtIuN/xWFM1vdb/j1lIVBh4AtFPd6gN1Cy9z/V6zbN2L+php7ePkbhv04hgQEhfaBaJP+BfkZSgtRm+otGUOwaaO2+LhcNNCzBIX6SkvLFAmjPsUOMVVFQbJZF4snZfoxj/Ft04yWtUKdoGSG9nH+5qaQtnETMhfTpSkG0p40+2DjJ9qKiNDtcAJkV2tkXKPRgAE8fJbEZrPV0DO+wAe4SwHJxl341DHoEcOJe5xBuYZ0If8250EUU4NowBeETIS6HbMuoVzmVnGKARJB8KNvKQVrBCoeyAINN7dLCr+zqS6rAaiEdFgUPWoNtbxUMJ6kFlV7WWQRvOnAynrAF9Sdk4/uAstNKWJpydSfxfdTEct4DVx/kzSRrO+bldCqEpP3pLhoBhbl2TGjVXS49hM1eqR0Ep6jZ7JrycqO9Ijd0jFDoYVJIYkdnrf4Ascv/O02cGlWVmVvvlNHshRRh5tCoNg9qQCAyvVz2+ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e567b83b-35f3-43e2-f655-08dc693a014b X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:43.2833 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: oorGk7qelFH6JRcmc28o8aMBKMsYVrLfAJeqUVmhfUwk8tuSOe3ncRTzdnVy927t X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102200_037163_7AA68993 X-CRM114-Status: GOOD ( 21.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prepare to put the CD code into the same mechanism. Add an ops indirection around all the STE specific code and make the worker functions independent of the entry content being processed. get_used and sync ops are provided to hook the correct code. Signed-off-by: Michael Shavit Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 176 ++++++++++++-------- 1 file changed, 104 insertions(+), 72 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 79c18e95dd293e..196aeaf280042c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -42,8 +42,19 @@ enum arm_smmu_msi_index { ARM_SMMU_MAX_MSIS, }; -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, - ioasid_t sid); +struct arm_smmu_entry_writer_ops; +struct arm_smmu_entry_writer { + const struct arm_smmu_entry_writer_ops *ops; + struct arm_smmu_master *master; +}; + +struct arm_smmu_entry_writer_ops { + void (*get_used)(const __le64 *entry, __le64 *used); + void (*sync)(struct arm_smmu_entry_writer *writer); +}; + +#define NUM_ENTRY_QWORDS 8 +static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = { [EVTQ_MSI_INDEX] = { @@ -972,43 +983,42 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) * would be nice if this was complete according to the spec, but minimally it * has to capture the bits this driver uses. */ -static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent, - struct arm_smmu_ste *used_bits) +static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) { - unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent->data[0])); + unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0])); - used_bits->data[0] = cpu_to_le64(STRTAB_STE_0_V); - if (!(ent->data[0] & cpu_to_le64(STRTAB_STE_0_V))) + used_bits[0] = cpu_to_le64(STRTAB_STE_0_V); + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V))) return; - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_CFG); + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_CFG); /* S1 translates */ if (cfg & BIT(0)) { - used_bits->data[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT | - STRTAB_STE_0_S1CTXPTR_MASK | - STRTAB_STE_0_S1CDMAX); - used_bits->data[1] |= + used_bits[0] |= cpu_to_le64(STRTAB_STE_0_S1FMT | + STRTAB_STE_0_S1CTXPTR_MASK | + STRTAB_STE_0_S1CDMAX); + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | STRTAB_STE_1_EATS); - used_bits->data[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID); } /* S2 translates */ if (cfg & BIT(1)) { - used_bits->data[1] |= + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_SHCFG); - used_bits->data[2] |= + used_bits[2] |= cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2R); - used_bits->data[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); + used_bits[3] |= cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); } if (cfg == STRTAB_STE_0_CFG_BYPASS) - used_bits->data[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); + used_bits[1] |= cpu_to_le64(STRTAB_STE_1_SHCFG); } /* @@ -1017,57 +1027,55 @@ static void arm_smmu_get_ste_used(const struct arm_smmu_ste *ent, * unused_update is an intermediate value of entry that has unused bits set to * their new values. */ -static u8 arm_smmu_entry_qword_diff(const struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target, - struct arm_smmu_ste *unused_update) +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer, + const __le64 *entry, const __le64 *target, + __le64 *unused_update) { - struct arm_smmu_ste target_used = {}; - struct arm_smmu_ste cur_used = {}; + __le64 target_used[NUM_ENTRY_QWORDS] = {}; + __le64 cur_used[NUM_ENTRY_QWORDS] = {}; u8 used_qword_diff = 0; unsigned int i; - arm_smmu_get_ste_used(entry, &cur_used); - arm_smmu_get_ste_used(target, &target_used); + writer->ops->get_used(entry, cur_used); + writer->ops->get_used(target, target_used); - for (i = 0; i != ARRAY_SIZE(target_used.data); i++) { + for (i = 0; i != NUM_ENTRY_QWORDS; i++) { /* * Check that masks are up to date, the make functions are not * allowed to set a bit to 1 if the used function doesn't say it * is used. */ - WARN_ON_ONCE(target->data[i] & ~target_used.data[i]); + WARN_ON_ONCE(target[i] & ~target_used[i]); /* Bits can change because they are not currently being used */ - unused_update->data[i] = (entry->data[i] & cur_used.data[i]) | - (target->data[i] & ~cur_used.data[i]); + unused_update[i] = (entry[i] & cur_used[i]) | + (target[i] & ~cur_used[i]); /* * Each bit indicates that a used bit in a qword needs to be * changed after unused_update is applied. */ - if ((unused_update->data[i] & target_used.data[i]) != - target->data[i]) + if ((unused_update[i] & target_used[i]) != target[i]) used_qword_diff |= 1 << i; } return used_qword_diff; } -static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid, - struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target, unsigned int start, +static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry, + const __le64 *target, unsigned int start, unsigned int len) { bool changed = false; unsigned int i; for (i = start; len != 0; len--, i++) { - if (entry->data[i] != target->data[i]) { - WRITE_ONCE(entry->data[i], target->data[i]); + if (entry[i] != target[i]) { + WRITE_ONCE(entry[i], target[i]); changed = true; } } if (changed) - arm_smmu_sync_ste_for_sid(smmu, sid); + writer->ops->sync(writer); return changed; } @@ -1097,24 +1105,21 @@ static bool entry_set(struct arm_smmu_device *smmu, ioasid_t sid, * V=0 process. This relies on the IGNORED behavior described in the * specification. */ -static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, - struct arm_smmu_ste *entry, - const struct arm_smmu_ste *target) +static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, + __le64 *entry, const __le64 *target) { - unsigned int num_entry_qwords = ARRAY_SIZE(target->data); - struct arm_smmu_device *smmu = master->smmu; - struct arm_smmu_ste unused_update; + __le64 unused_update[NUM_ENTRY_QWORDS]; u8 used_qword_diff; used_qword_diff = - arm_smmu_entry_qword_diff(entry, target, &unused_update); + arm_smmu_entry_qword_diff(writer, entry, target, unused_update); if (hweight8(used_qword_diff) == 1) { /* * Only one qword needs its used bits to be changed. This is a - * hitless update, update all bits the current STE is ignoring - * to their new values, then update a single "critical qword" to - * change the STE and finally 0 out any bits that are now unused - * in the target configuration. + * hitless update, update all bits the current STE/CD is + * ignoring to their new values, then update a single "critical + * qword" to change the STE/CD and finally 0 out any bits that + * are now unused in the target configuration. */ unsigned int critical_qword_index = ffs(used_qword_diff) - 1; @@ -1123,22 +1128,21 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * writing it in the next step anyways. This can save a sync * when the only change is in that qword. */ - unused_update.data[critical_qword_index] = - entry->data[critical_qword_index]; - entry_set(smmu, sid, entry, &unused_update, 0, num_entry_qwords); - entry_set(smmu, sid, entry, target, critical_qword_index, 1); - entry_set(smmu, sid, entry, target, 0, num_entry_qwords); + unused_update[critical_qword_index] = + entry[critical_qword_index]; + entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS); + entry_set(writer, entry, target, critical_qword_index, 1); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); } else if (used_qword_diff) { /* * At least two qwords need their inuse bits to be changed. This * requires a breaking update, zero the V bit, write all qwords * but 0, then set qword 0 */ - unused_update.data[0] = entry->data[0] & - cpu_to_le64(~STRTAB_STE_0_V); - entry_set(smmu, sid, entry, &unused_update, 0, 1); - entry_set(smmu, sid, entry, target, 1, num_entry_qwords - 1); - entry_set(smmu, sid, entry, target, 0, 1); + unused_update[0] = 0; + entry_set(writer, entry, unused_update, 0, 1); + entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1); + entry_set(writer, entry, target, 0, 1); } else { /* * No inuse bit changed. Sanity check that all unused bits are 0 @@ -1146,18 +1150,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, * compute_qword_diff(). */ WARN_ON_ONCE( - entry_set(smmu, sid, entry, target, 0, num_entry_qwords)); - } - - /* It's likely that we'll want to use the new STE soon */ - if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) { - struct arm_smmu_cmdq_ent - prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, - .prefetch = { - .sid = sid, - } }; - - arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS)); } } @@ -1430,17 +1423,56 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) WRITE_ONCE(*dst, cpu_to_le64(val)); } -static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid) +struct arm_smmu_ste_writer { + struct arm_smmu_entry_writer writer; + u32 sid; +}; + +static void arm_smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *writer) { + struct arm_smmu_ste_writer *ste_writer = + container_of(writer, struct arm_smmu_ste_writer, writer); struct arm_smmu_cmdq_ent cmd = { .opcode = CMDQ_OP_CFGI_STE, .cfgi = { - .sid = sid, + .sid = ste_writer->sid, .leaf = true, }, }; - arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); + arm_smmu_cmdq_issue_cmd_with_sync(writer->master->smmu, &cmd); +} + +static const struct arm_smmu_entry_writer_ops arm_smmu_ste_writer_ops = { + .sync = arm_smmu_ste_writer_sync_entry, + .get_used = arm_smmu_get_ste_used, +}; + +static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, + struct arm_smmu_ste *ste, + const struct arm_smmu_ste *target) +{ + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_ste_writer ste_writer = { + .writer = { + .ops = &arm_smmu_ste_writer_ops, + .master = master, + }, + .sid = sid, + }; + + arm_smmu_write_entry(&ste_writer.writer, ste->data, target->data); + + /* It's likely that we'll want to use the new STE soon */ + if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH)) { + struct arm_smmu_cmdq_ent + prefetch_cmd = { .opcode = CMDQ_OP_PREFETCH_CFG, + .prefetch = { + .sid = sid, + } }; + + arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd); + } } static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target) From patchwork Tue Apr 30 17:21:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D0879C4345F for ; Tue, 30 Apr 2024 17:22:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4i9dPrfNQLQ8FbRKabvJbMQoG7pG48BYmu8laUqLmfE=; b=slgq9DMq3Nhonn 1LHaQqsMyvvGz+p1/LF25zQlft5wrnjuzSGzpPlsnM+CeY2jRSmm0Rs/5qEyzm2/N2i6qQuDO/EM7 eUTlWEnn2G/2WVQ4vS6kMMn8iRco3Nk+JgzC890miGmEbySk/y6JmuknZceynVR5RyGz5KInJEC3H 9dqLwCILjTv3MiZnolWgFMtnA/zmqW5xlndjPuzzuA2cK3IKYgF5VpyRjTE7nnIbilKXkdhzfmFGc neZtp2JjrC0pszLrP0lTpM1Z399sl4GpqTRCjpvcM4fi1Ua4xWCSpGdyzup0BS7BCDGVtP9xG8PvO TNTJZCHPA49lePs8MrHQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rAx-00000007OXT-3a54; Tue, 30 Apr 2024 17:21:55 +0000 Received: from mail-bn8nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2418::600] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rAu-00000007OVf-3Gya for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:21:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YntBr160H3bNPOb/anPl5IJaRmL92srcw+9zdg4MInLuVIVkpTUkhcAlwE9aWBCaQ6PKff/tgI1OEo8ZCDMchF/mHNPkirquBNyv6HCqWTMD7wB3WEZcYz+VEz0EzQpkeNnAMjHiPkHfgVCojQH5PDf7DzMbldM/FBGFYGfqbq2kfJh+nrhQY5k5XcuvgF2FKdl1mnS5pnIdV7g1cG/SNtUOLvFA+/6/bOfb2EJ1FlmCs0eALVfim3U59d1gJ77Be9Lz80J0/eRw+7n9QfdCtxh1uIEA2hmppePqyX4d5IKNrjHtb8Bs7ZtvRS+GVA9QaAavHZo+Xz0gZ65+podPNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u0PSPwTMKz929rCyULVrpInHYArcUbpgRlrKoxMH+xU=; b=dEGSQqceM4eQTAwK1Sj5g4sxY0+Dzbh3+4NIpF+YG0XtDEVej6bEfbIQrGSIPWEclgqT/km0sqFAWv9xCp/6wu3KZT8FClk3ZfjULZKSJFEZZhjDoAcb8gezBoQuJPgwdvynbfKNqRSiqh1ejMM6G+z+gR5p+9NvhrMAqKOHzyZHTnuTEZhYPMFTnYdiuW9kaoXJz5u7ThblgIV7cRKVkUblhYcijhqKDqzRu6IlCtvU35ZVhL0xvCp8sR5UYvIJ1Oq41Xb6IaXDIFyt45Zv2dELQjcRjZ35orDXD2E34X9JK5IyySafOwkysjRrpQa0ao3YWJ7PuglRGEw8n6xVCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u0PSPwTMKz929rCyULVrpInHYArcUbpgRlrKoxMH+xU=; b=XMkGetLzilUdzNIrtkhcSB+0BCylb8myAufjCO1JQldsu7aV4KamOY403dmiGGP3760Z/BUYwoHQLIyaISgRUigzzudrqdKZa0pR04hqMckNsMEWtXVaG8LNnQ4+S0NHlomyyNFtJ5LEtnZ/617Q08mDNWm3llmU3pRzYYrJTuGVo6jfbSKXzvv40rF6bLHnLG5Cmipxn/GlaMDZ5EPRFZVicnJUAT7cYySNFxF7qhJaywz3PDiJX5Q0X0EXZ7IXLZuT0CFyCzgvby3Ex0R94NaHyXVbcR10v9AjVi/DNlDvFOA47Kah5xIugNMbDaLqtepb2/40eMT/yATPZ9WEtg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:42 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:42 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 2/9] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry() Date: Tue, 30 Apr 2024 14:21:34 -0300 Message-ID: <2-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BLAPR03CA0136.namprd03.prod.outlook.com (2603:10b6:208:32e::21) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 90f85db0-51fc-4c90-5d21-08dc693a0105 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: kjVtH45K1ccQRLRTKGbD4qMcl/T9up9bIRtg8ec8uYHsK5QeqUmIrCXsuLt3b2RZVvpTGguWi1WlkzOLXOBCQzik3j3Q62/AAoLQIS+5oirp+iQIAK/455YOAlssYTwBcVzi6VBOVT1YXNVxixMVOsIdblTbC0rLvYg0D27IibcYJZrLGXzXPqpi3abKhA0NjwOrxchI4hU5jMT5SKXToHIMc4dBobFNdcsW3wRcpHxixXYAorcHyOpl4lwlC48RxGT5xWXLS9jfAGNbyaxKJV5VSlM1uBIHlWsNKh6zo5dHrWIbL2KcHWldFt7vzPUADWG9lFUMrR1lkTJGreiWjlu/cuJ4/RO8MDMArOkvIg1G5KNL8uJQMyZn6SuDAMs5ARqCbJ+ubdtlckWgJWxWP3vPCoQ0cnRZ5FuwpXkZ9tPaeOcC5d0mRaxQJXp/4Kl0gcWvLdgwjExRDbrEYkeIjkjAZyHSqXGwEVUqwzliDVj0mZWgZLBj9d5yCHkyiZqVWNNmUIZtvbdIuzJUaDMTjSBQCSHiMvkFoXQe1PJvHj9aTK6JQQGsEy13jILAlMpsYo6wL0XtNJKdgpovMFURdUGZW5Kd97BCsk16VZuGghbIAFnugaz6t6TKcTC6vwUwpl3GJCQ8rhpMF8lgCqq9udgBU+yFfBPQVEucFqjSqPUJowMh+E1pmsWNKIEhcY0mqdJ8LAAyywNS8dpvMDQVJMWfdqEGxoAC6FwCXgsEDzH07V8DPMUmYHvUp1HG9hhlmgxKq9GPlpha//WLJ9wl979/cmFPubas5RRKA0he4dxk3n9SyOc+wycJItcoTAqEVb67BFtFTd6o6uQrnrRAq7sYvUf80IV9iPvlCDXAXuOP/QzL3cCCeZcM/4T3ut3W/4eIzBuNYc/0mL3DWVF/CRE6AitBGbg2UTZEeYaOTO13K/UWH4LfzJcfiCOrJZS0CprV6YbF2mFo0l6heJ4kj4m1OkEP9PNx1XshlJ6j2W+3vxvExDGgeMacz6L51QS5Z/wX0qm63eEB1uWZ6M2CAhljuSIjl3nlj1Bn3N5B5By8wg14vpp7Hg/b50ahadg+f5PYsAc2t5DjAqAxNiK6oco4zezwOCFo/MIhfEnv/Fx0b1DXE9V6DF/YZHEv7XCRt6moQzfq1K+P/XSWNPdNmiRLIeSDoWS1SKzX05tNzdmXfhgrA8wuMYW1TxKnF2pO0U3Kio7BraKc+ZMQFkJqqsS1xa2tE7rYIOPN9E3TmTQFx85jEjqAlpr9p97imz3upFBqn2FJh8M8G2MHmHzvpQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: trP1GRP74cznjEoJP+TqoeJjmNQ+OngzB/a4zX0ghdbfrkA8XZ0L/T1FCJ+PX3U/DGOd5VS6WqbczV+oEivLDQBQwvMOabZOm6S5P4votQmsmWlsARyw3SMOI8GSr9sKdVLQSyeCLjjU3+Xd6ZrFFeZeFhboTKluOrd7TSqjeSRgqSfif/JVa5RBDJcVm2DNbbLZhSRUMOE4u5Lpy5jwe70ZWZfRNMwm+TMNzAH25uaR3w41QJYGEsaaVjXgo3Z4QCeAwNEZS0QIkYLbVdZj0DYu+2dYPG2fF1uEQtFN32LV7U9mofh1cbzBc3rgvj64/sGiTnGcyjL9SW23vnZYFl7aPKwRoJZNsbMYUw5eRPwwIiMSETGmv/qP89TJixx9hMN/kHWuzBU9iP3ySbaRP5w67bRs4XLWF8Q70wWcL++OxJ1w9Yd2KUpsQSRsl7Kr5Ks7ZqQsjjKDL1kF3bk2Wc/kP3/TxqJaCyIbPbZ6fShIFsAiiz7/Ir1zm9PHH+jnKMJozy6Q7vIB183tUK0T2WuvzaKg0IVWPJw98z5rIxoBI4H0zjQDc1hn2BC7KRVHFA18wnk8xZn2CU6ktZFUv4iRr3G2JAClIWA7Kfgg9iSeT/wU9wtoCO9eTezA9b4C9mTafb8HAJfRd0pNK+FqdKMuMmKqSecR1Q4H18SPTfg9IXjkLmvTz/dDTZ6+UcQgF0YUf8TS6gpnojA19tvJke1DN1ppH7oaXg/MdjzHWjYKOWTHrZyVQFul8QA+xxJx3rDOC+jNM3llj4RIENlHqVyZT2wyNDk5yZafqPT9CVUuBw29zjV4XIOdnTGKr/aorwDo7I7Bshb8mSRuEJ51F0Qp5a2XD/pbIeSsjDbm644H+JXkIMbJg/WzmpBPFSmc5jiynP8glH+ObbPFepuys0YGT5NeKJ54fd7Tq4uMScaqIenMQ/FIKSjMuZY2BNbB5/KMUKbqrS9XXvVIeP7Q4pyOxXFyT2IVuXtZVTwq+M7gEkLOAC/qbTrPyDh+kp/st9vn3iLip6V14cph8+yKlowVTLzdaBmhsnIxku8WAYLC3c/lcoPgkSpWxEScFeV8x6pwDGF+O/sizNncK3NggYcEpjlp+uk8Nh3n+hnnLfLrgWc8bHduZdS9+3/fNC62vW5NZyx2hySQzuYpoCQPCzYRSG+dzjyAqXnpYS08nM6F7K9I9+zdh2cNM5IJKuN5Pqt52o1l8ppb3ulwE4Kw5G/6KDmDkGxGmWH8U28RXFU4sl43IPU8qnA4qc+r9PVK/et+ZkCSEPOE5ZnMGVkeNmaTX0lj5nK7AYqsSKTfMSgKtq0fLuRKRshvcBHC3ldM4vDn+xPtrdYG23DYfgujz9SuVxUKskKUWCXVLw7QFYuw5Ya64/ueV0hmxsfAjqsUPmfRhsBwQXhpI0iSd8aZblmB0OB7VRlMPnQs88OF28VXE2iZJY/ylwxEXleSXr3dyVVu010xPuIRCWzbIcZigsqZjiYMgj/Omw7AaBb8rrmKnEYQxk1/JMcCsZXO57z3mIwDa3uH2iKNrM/8DKAqSJ4qYSEJgENj4evBbfkmrOk5sgOlBVU4aeaqcKd7iv5f X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 90f85db0-51fc-4c90-5d21-08dc693a0105 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:42.6978 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SjZ7QG0vBAic+seBuuaAnNyGFZ7Zgpa1OPMBgl+LQIOXCslXAS4g8qeqWuPI/JHa X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102152_974986_74BB3818 X-CRM114-Status: GOOD ( 20.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CD table entries and STE's have the same essential programming sequence, just with different types. Use the new ops indirection to link CD programming to the common writer. In a few more patches all CD writers will call an appropriate make function and then directly call arm_smmu_write_cd_entry(). arm_smmu_write_ctx_desc() will be removed. Until then lightly tweak arm_smmu_write_ctx_desc() to also use the new programmer by using the same logic as right now to build the target CD on the stack, sanitizing it to meet the used rules, and then using the writer. Sanitizing is necessary because the writer expects that the currently programmed CD follows the used rules. Next patches add new make functions and new direct calls to arm_smmu_write_cd_entry() which will require this. Signed-off-by: Michael Shavit Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Moritz Fischer Reviewed-by: Nicolin Chen Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 89 ++++++++++++++++----- 1 file changed, 67 insertions(+), 22 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 196aeaf280042c..3504c25de4306b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -55,6 +55,7 @@ struct arm_smmu_entry_writer_ops { #define NUM_ENTRY_QWORDS 8 static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); +static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64)); static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = { [EVTQ_MSI_INDEX] = { @@ -1230,6 +1231,59 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[idx]; } +struct arm_smmu_cd_writer { + struct arm_smmu_entry_writer writer; + unsigned int ssid; +}; + +static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits) +{ + used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V); + if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V))) + return; + memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd)); + + /* + * If EPD0 is set by the make function it means + * T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED + */ + if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) { + used_bits[0] &= ~cpu_to_le64( + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | + CTXDESC_CD_0_TCR_SH0); + used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); + } +} + +static void arm_smmu_cd_writer_sync_entry(struct arm_smmu_entry_writer *writer) +{ + struct arm_smmu_cd_writer *cd_writer = + container_of(writer, struct arm_smmu_cd_writer, writer); + + arm_smmu_sync_cd(writer->master, cd_writer->ssid, true); +} + +static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = { + .sync = arm_smmu_cd_writer_sync_entry, + .get_used = arm_smmu_get_cd_used, +}; + +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) +{ + struct arm_smmu_cd_writer cd_writer = { + .writer = { + .ops = &arm_smmu_cd_writer_ops, + .master = master, + }, + .ssid = ssid, + }; + + arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data); +} + int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { @@ -1246,26 +1300,34 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, */ u64 val; bool cd_live; - struct arm_smmu_cd *cdptr; + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr = ⌖ + struct arm_smmu_cd *cd_table_entry; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) return -E2BIG; - cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (!cdptr) + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); + if (!cd_table_entry) return -ENOMEM; + target = *cd_table_entry; val = le64_to_cpu(cdptr->data[0]); cd_live = !!(val & CTXDESC_CD_0_V); if (!cd) { /* (5) */ + memset(cdptr, 0, sizeof(*cdptr)); val = 0; } else if (cd == &quiet_cd) { /* (4) */ + val &= ~(CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | + CTXDESC_CD_0_TCR_SH0); if (!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) val &= ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R); val |= CTXDESC_CD_0_TCR_EPD0; + cdptr->data[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); } else if (cd_live) { /* (3) */ val &= ~CTXDESC_CD_0_ASID; val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1278,13 +1340,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, cdptr->data[2] = 0; cdptr->data[3] = cpu_to_le64(cd->mair); - /* - * STE may be live, and the SMMU might read dwords of this CD in any - * order. Ensure that it observes valid values before reading - * V=1. - */ - arm_smmu_sync_cd(master, ssid, true); - val = cd->tcr | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | @@ -1298,18 +1353,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (cd_table->stall_enabled) val |= CTXDESC_CD_0_S; } - - /* - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3 - * "Configuration structures and configuration invalidation completion" - * - * The size of single-copy atomic reads made by the SMMU is - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single - * field within an aligned 64-bit span of a structure can be altered - * without first making the structure invalid. - */ - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val)); - arm_smmu_sync_cd(master, ssid, true); + cdptr->data[0] = cpu_to_le64(val); + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); return 0; } From patchwork Tue Apr 30 17:21:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D9D4DC4345F for ; Tue, 30 Apr 2024 17:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YV72S7bwaN4MoXhgHyHBy9bxslzb9X6Uwul83mnfh/0=; b=HWuHeijTEkE5EH 21kCJNpESDs3ZxstPJHOw9tor2vzEGhnOgvGTmak/y9m4BiZw4SUnPT+Y+nfvgf6qQ14uPh6AypSV EyEsnBGWaFis857j8rPV1BeOFynWPH6gxoyh4WJQAJ47BmUX6IqLoJWVueAQyivvMUKpdCSddzfuG SFyN60fV7dqjBuwgTRF6cuVeSYzQ/Mxrwj6dHK09RIC0TKw7NJY1xe0cYCBh9VqKdmD9TMTMIB++B RrFn+TSjljPnTAZLwT7Xkeor8tzGiJ/FTw0VSz7wgypRGfdCPsinng4gwtaLqFpz5t2CBYIpKwkxW BYoPaDa8jqW77JCptS+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBN-00000007OmX-3y60; Tue, 30 Apr 2024 17:22:21 +0000 Received: from mail-bn8nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2418::600] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rB4-00000007OVf-3o0l for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:22:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l8qniOxV27Aq56tCCAJAMw5rAm4cD2djMybN2WYoAF4c15q3FsjTN+OVkweN13M2baMZxZtRqwTnW7ReOAJor/s4oY9buTI01nhKAMYf4rKkitUkuQI24LDb5dQ9W09D4WKtNtlBh3UlsSzCciHTZeloZsbw7sQ7yxBmVESPY4ZO1uD+HRIgJEPbmPK/Ytcq6dEnj6xsvfPwIb+MeCIjq4blh3QXFyvONIaGyaPoEMXoavgBQApSNhvnUZ7YWsGEGnKoD5K4gtVzIM74aRwT0DHvEoRGpJnBeECBbXNudddgGKzxdZJuEN9wqY1T41AeGGrZLguCi2RzAZ4vRtvrWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rfiowHDG2quWv/lUvO0KpNtFbwdW1jxLsjjmq+AoO34=; b=gPqEK8zxfO8MMZWEpk02+7EEuB3mf6zLQQbgRgIxRePjjc0rGATXdbrvm8mlvDoHo5mtaHzz3N555B7E7TWfAkYbLlL/LciXzHC4h2i4l5un+9HJLE60m7v/YVCy/A/yzAhO0pzKoSIdPplGBXxsQepyH7hhohBnZVo7PEGW11UVEShRsGovzoZKp4JrGJpSR5+1NOjXkzij5wa3109TTq0BNLa1Gac7q8ogLfJrhEVzPOIWtYO0FGEI4hiqnFvV0iZ3jSdz2qhRHuHMsp+wp6cgToAl03c1/kMBt5kMfypaFvk+EF1EnvA3Po4Qdtr/S79taRwKyvGCDU3HC32Mlw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rfiowHDG2quWv/lUvO0KpNtFbwdW1jxLsjjmq+AoO34=; b=fp1zuSWyzzu230r8mCJ3Uannj/xo6iz/e7mr/RiIHW+ygW2P3HvbCO8VnOua1rMEdWRYJiov0zGQxuanpSNgQaIzIPpui3u724sWE5ldI2tDfFPLNxj3lCtQQDXKuacuomzuy4gnImqmbx4UK5XvR7fk0R/iVS1oivOp5LBnIgnSyX47GjPFhSaqQFBoRCIU7oLEHsLHSCXhuwDSqcbFGTaJQBseyQuhl6eVPlCX9muWOGrCVxwX9i0kOWhmYnOwG5yGC4WkxF0yH1kosmFnJnX9Cy5OhxEmvgddPtS/9lgE6p/a098VG/9zqu6xAH01oUwVstb8YcBp1Q/FzaNouA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:46 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:46 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 3/9] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function Date: Tue, 30 Apr 2024 14:21:35 -0300 Message-ID: <3-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0027.namprd17.prod.outlook.com (2603:10b6:208:15e::40) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 5e38b2ed-f088-4dd3-3242-08dc693a01b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: DSPquYM2oG3Q9bJxjivAyKg8A9gqmxwHNa3f0xHaG7P3GYd1qoppEbxz+WkRfYaJ2myQyhpx70UyQncfgwzHOHv4Vkn8GSbivokV70k+gPmtqwDqZYLUrCr5zem16x7G+f2jhA+duKHW9kSrQlcaN8/5VRNEl46GC0LOLEh2b01JmG6l9ALIkTZXJo9x4pf26SkbtSE//azfRSz99sb4mHehgq8Ry6rRnHkdJT2XYGMpblI07tgcRv/RNnOhw+Xqg/IfB+kQ7IkMIB3jBxn64KwDBYT+mRZv2dWHAlKew6bnz1OLQBfB//N4uz1xVrePd/G8N3HtZavEL9sGWMat80ewE4HgbK6D+LvKW722qZiJh4tSZ7drF81sjaMUZ7Tae6SL9sERAKaHSrBLTKQJX9KxGRpJoNEYgrsfzo9EKiBgRyxT5hjxtkoKxV+JaJrM8i1snYf43CUgb8Eaz2ORtAU/VPW9krvcQC/Ef3w16tAeqtRnR+wgTPj9tju98AgI6bgia270sPCGtz46x/0VZh+sjwUjsPaYddkRfJOuHEI1Lu+Hifz4ZrB5eiN0L+bsO+J9odPnZJJlHhjZ4aVPIU891W89WLDbMxhWJpYrJzPONJXsclSab+CpKNIFUbbKzaHIpUP+xutE655aaJhLp+qls4jV2NNus3Azag7789l7s7Obnk3798C5syN4BP8OvF9eNBqXdF8C8839xDgDtOVxQFdWeWA3NHPBukQ9T+LIfRLjQb7Pg6HekX5nSRi3gQytFNcgCQJFum0HDjqBMLFuG4YTXlaLswlHWaSZi87pLfPjOfUpZMcdx8ie1y8Gn7N5c1Ouz5od35j4ZZxOIeJru6LSo0EsKmuwQnsfotvLfmOenVSD3/M+pBoH6qfwdNHqxgIMy2QGTxM4Zg0BJtA0Ykvgz2UZs0tOkZZ5QBOqBfvS11xjkl1929ZPtLn9Yn0dgSfSV7cYEugs/wGL7dzE0RVvk5MTTxi5rOgv8jIhnS0O2k6WFlBuRiTTvHiGCLur51iPvcO3gEqMR+9Vx2dOZ3V+29RnCF8OHYnVnwDThimsPKv8yYiytjh1oja+eiwNr1TlRzOUMJlCQHg2bio9ScqtJane4DBKJqfXvDRIZcOx5dH0IRzyKoTDiv5abe4RwcPUsXF8Nw3BMLu4w4KpmypfS6EcvSKdhQ5YYZy627VysyOigmMKRTmZ1wZn5m+QFHoFdoNB8LcVNcB4utB+V4yoF/+msouWMUM2mT2K7yYdKcVYeJ73Fj8wDXoVB758W7QrldfU4YdalET2LQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ug1MMhX6RdaTlK68cp9+VInfuYDv0xn4lr3Fgfs+bcJFmbZx7lWxCeOPpCJjQySbhloNiqE1izZ5Fzs0dkjOHevXfrO7m/h8YEflkkTMRiUmGmpIm8A/3kYDfg+yM3IJu5/hylk89St2nrmXbIA+8n9vVxHJutv2QILzN7jzlfirMDpiKSl3OEp4sy+hebXa1AR79YLK0uP0mjcQleuRFC6dbL1GmCh8jL9FhiP4iBwXiN8k4TIqKPAFWN68YQPz9ZTRGd1bELYKNL3BvEkdUhRj4NhNIN3ytRR2/6vW2fu9uEBTATUHhFQkTqFJbhQ5QB6UjE6ivZrFtR5puP50UYSigkvnvTwH5H7Yvor6he3BvGlGpyu9glqz+FdLgEPD9l7K8SwbeicO4+uC8kSm29ahQ/oxi3lK0P5bWJmS1HepEvnLRD0fQZ6o10O9OWailRRPPiQT8Om3OjNbK8/bAFnvgIxq/u1rA2hwrWS3m9kUG6R0pcv9IJaTR/R6WSQfwVWFq3pOLyrRGLvglYPgKgXgmPGkJltN8neA3oj44yUnP0Xy8c2l1VJuDEpYyvpu9wcE0B81NVy2rt2g8B8BYK6gHF/cb3GkhiBeUd+Zz0WpqL1nHvvPXAROI08yl+hrCObn0RpYdXa2PRbhfxFVM5vdxlNudfNIN8ylyW7ho1GvdSKC+FXFtSz56vvnfX6MvfEP0+6QlTISgcXwP3+OtnUJE6hk4BlT9VGhazb9pzWwKiRvIfvdzZYLQqZdHJyMrdt34cnNjDYrfnBKd30PNgkH0R9Q52cfYwB1W7V/47IaVBt7BU8B8LlpOFrnag84gP/zHG2O3jEMKpWSej7EaOq3V6HtieICCEqS3JX6QplKdAaFtL9AcnogEg/fKeZ9p7xF3uYCOxUynZx+3yCgvn4NObMqvDJ0mU+LZMJLidY1ZzfPE0sISRILYkl1JIx4r87WUpIrOz6HXzBZZvopgl7cZWcxgU2RAzwoDC54X2tKlnB5dMvgCKEXEu4NvFoLQr3i8+bcZCc/g9tSoDBeeYzHFquWd8Jizcic8ar/E9TgsLF28cGoI3GZXnJKgNnwL+tcLiWalNlHP5G5LXV8Dt9BlYE3WOGqfBACJxo7iafvwawqQetLBMjEdNtC1ns54Eu0LDN+YZclfdYIXl99iWe8YBosEYQ3es4qq/0QqinUbVfwVj9J1V2VGUlrED9fryOT7or0H4Nm8EEhRZEvrzGZaMOE1s6ehn8tbaVVih5mHebnHR7pUFKL7UknBKuoC3+96D5p+EDpc+1NbaHosCZTimfbYUNn9EIBicoYuJFQ/pCU2+mV7YsyfKjAQ8mQ8UWsPAFKpIC/FOwRf5QgqEAujRxxp2P/WJjiCvOzUfYqbjY7031dZSO49y6wmZaYuxNjh6XnJjUita36A9a2HzJoKbqdgd3u3bZbWZF8vW5Ju0NFkqf2RDZ06JsrwlCzUrXhQ9aEWU4qiKT/0v9Q4AShOxY1N7Q2qbkAAB8E/gly1G0+jb37ABLOj77y5hcLC+Hz63fZ5iZme12BRi14ttfymEqQTRrFY/d/oanW2R+GRmIrVFfSaFxq3T/9TqV3 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5e38b2ed-f088-4dd3-3242-08dc693a01b9 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:43.9917 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7PJomkDWRupVB8uiu6bHgI8Zh576DxqQdWAbmXVLl8nxSqiU673VW04tz0YJuNQO X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102203_148537_E9BF75EA X-CRM114-Status: GOOD ( 22.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain, and reorganize all the places programming S1 domain CD table entries to call it. Split arm_smmu_update_s1_domain_cd_entry() from arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call chain separate from the unrelated SVA path. arm_smmu_update_s1_domain_cd_entry() only works on S1 domains attached to RIDs and refreshes all their CDs. Remove case (3) from arm_smmu_write_ctx_desc() as it is now handled by directly calling arm_smmu_write_cd_entry(). Remove the forced clear of the CD during S1 domain attach, arm_smmu_write_cd_entry() will do this automatically if necessary. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 81 ++++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 +++ 3 files changed, 86 insertions(+), 29 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 41b44baef15e80..d159f60480935e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -53,6 +53,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); } +static void +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_master *master; + struct arm_smmu_cd target_cd; + unsigned long flags; + + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + struct arm_smmu_cd *cdptr; + + /* S1 domains only support RID attachment right now */ + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (WARN_ON(!cdptr)) + continue; + + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + &target_cd); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +} + /* * Check if the CPU ASID is available on the SMMU side. If a private context * descriptor is using it, try to replace it. @@ -96,7 +119,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) * be some overlap between use of both ASIDs, until we invalidate the * TLB. */ - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd); + arm_smmu_update_s1_domain_cd_entry(smmu_domain); /* Invalidate TLB entries previously associated with that context */ arm_smmu_tlb_inv_asid(smmu, asid); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3504c25de4306b..a80758d72d6a7b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1203,8 +1203,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, WRITE_ONCE(*dst, cpu_to_le64(val)); } -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, - u32 ssid) +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { __le64 *l1ptr; unsigned int idx; @@ -1269,9 +1269,9 @@ static const struct arm_smmu_entry_writer_ops arm_smmu_cd_writer_ops = { .get_used = arm_smmu_get_cd_used, }; -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, - struct arm_smmu_cd *cdptr, - const struct arm_smmu_cd *target) +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) { struct arm_smmu_cd_writer cd_writer = { .writer = { @@ -1284,6 +1284,42 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, arm_smmu_write_entry(&cd_writer.writer, cdptr->data, target->data); } +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; + + memset(target, 0, sizeof(*target)); + + target->data[0] = cpu_to_le64( + cd->tcr | +#ifdef __BIG_ENDIAN + CTXDESC_CD_0_ENDI | +#endif + CTXDESC_CD_0_V | + CTXDESC_CD_0_AA64 | + (master->stall_enabled ? CTXDESC_CD_0_S : 0) | + CTXDESC_CD_0_R | + CTXDESC_CD_0_A | + CTXDESC_CD_0_ASET | + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) + ); + + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); + target->data[3] = cpu_to_le64(cd->mair); +} + +static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) +{ + struct arm_smmu_cd used = {}; + int i; + + arm_smmu_get_cd_used(target->data, used.data); + for (i = 0; i != ARRAY_SIZE(target->data); i++) + target->data[i] &= used.data[i]; +} + int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { @@ -1292,14 +1328,11 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, * * (1) Install primary CD, for normal DMA traffic (SSID = IOMMU_NO_PASID = 0). * (2) Install a secondary CD, for SID+SSID traffic. - * (3) Update ASID of a CD. Atomically write the first 64 bits of the - * CD, then invalidate the old entry and mappings. * (4) Quiesce the context without clearing the valid bit. Disable * translation, and ignore any translation fault. * (5) Remove a secondary CD. */ u64 val; - bool cd_live; struct arm_smmu_cd target; struct arm_smmu_cd *cdptr = ⌖ struct arm_smmu_cd *cd_table_entry; @@ -1315,7 +1348,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, target = *cd_table_entry; val = le64_to_cpu(cdptr->data[0]); - cd_live = !!(val & CTXDESC_CD_0_V); if (!cd) { /* (5) */ memset(cdptr, 0, sizeof(*cdptr)); @@ -1328,13 +1360,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, val &= ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R); val |= CTXDESC_CD_0_TCR_EPD0; cdptr->data[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); - } else if (cd_live) { /* (3) */ - val &= ~CTXDESC_CD_0_ASID; - val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); - /* - * Until CD+TLB invalidation, both ASIDs may be used for tagging - * this substream's traffic - */ } else { /* (1) and (2) */ cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); cdptr->data[2] = 0; @@ -2633,29 +2658,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); switch (smmu_domain->stage) { - case ARM_SMMU_DOMAIN_S1: + case ARM_SMMU_DOMAIN_S1: { + struct arm_smmu_cd target_cd; + struct arm_smmu_cd *cdptr; + if (!master->cd_table.cdtab) { ret = arm_smmu_alloc_cd_tables(master); if (ret) goto out_list_del; - } else { - /* - * arm_smmu_write_ctx_desc() relies on the entry being - * invalid to work, clear any existing entry. - */ - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, - NULL); - if (ret) - goto out_list_del; } - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd); - if (ret) + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (!cdptr) { + ret = -ENOMEM; goto out_list_del; + } + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + &target_cd); arm_smmu_make_cdtable_ste(&target, master); arm_smmu_install_ste_for_dev(master, &target); break; + } case ARM_SMMU_DOMAIN_S2: arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); arm_smmu_install_ste_for_dev(master, &target); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 4b767e0eeeb682..bb08f087ba39e4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -751,6 +751,15 @@ extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid); +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain); +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target); + int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); From patchwork Tue Apr 30 17:21:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52A8CC10F16 for ; Tue, 30 Apr 2024 17:22:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Y12gIvHJyQ71BEgxe966AhVsaD/2KLhuHmsgO2WlE+c=; b=Eb2XWCmOUyvu0t 3W5UAldLagYt7duOR2hiQMBM1z+/RJvskp8x4y6pBIa9JHszhK9W9nARmt6kjV/X0VOwfGAfodXC0 hkvpIPa/7MCfnXe6Q8lcCOWJ4gDqZguHmpmPz03lJ4zhS03lFPVCn+M7RRVfol66plcg3nYfJePUd OjtMib3o1QfMLADiBgw/0Eqk+slybn6ednTX2elrsohZhLtymtTax80hGl6utxiUWs4yDUBy3nJgn C2d5lZb/eWr+Ou+SUzHAt7rUnW+Gnrfqdg47E/ZtjsQcCruGkaAPTecZLcJHtJdOCTVS9XfTECD8f tlIRQM9gqre2SNTLIjHQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBE-00000007Ogf-2Rr7; Tue, 30 Apr 2024 17:22:12 +0000 Received: from mail-bn8nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2418::600] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rAz-00000007OVf-2j8w for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:21:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DQJ8koSpISrKyCRtXiUgPlYHVhKKk/Pgoi1ZjF19tvas/+mS185AFCl3foLDpEfrufatywSh2yKCPwqz07y8x/6tbhnimamcWKYwXbqun2Th0g5GjiVI2+eDcYLvOKmgu0XIaEF6fquAxezLjs/GxdL3PZ7if7SJLG2HjiQ7lIF0EFbnoPyMwct362h89UwfE3quL2Ouw4t4Pcb8YSy/nHbc2EUDdJgBbpXB4QKWnAnpUTwksVbpj1hhfaiW2M68HR33wm/6yDzhW1KVgaC845ZUYhCY6+ahsKjfwmnz3p1CPLHXtKWpIlA0zsBDZHoc2aAmaC3h1vBKDsaZsGqZ9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P/d8Ky9DmRDFyDlbnkO/237pszRuKBySk7B1YH+zJ4k=; b=b8aT6zU43Ssx75CmcS5HEBpuNCBRCJSLpZiLB7ivJLyvM782cFBxSZ3uv+Au7r88Djk5fRYgktOF/kYJsstNXwaxyF8LsBOffizNCZ8MD/GCcNF4L3Wh4uVAoqJisFxELcu1rh0TLfeeU5G/mQRjudZcIc0QjwS/mRE8MoDMWm7Rl4Y5LbTv5baSg/e1X5NMCrnZR5Q4BRZ2ImL0onGS8WkvSeIsJokj6wl91BLIpssGbp6l/To8hbXl+uma1yC7R4QAsz0TWw3B14lw/pYa4xjtlSIKZXLseL1+u8ePVQtPhW6B6tAmYJxINnVQqf54OLuBfe5k8yT+UO9DGEvVWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P/d8Ky9DmRDFyDlbnkO/237pszRuKBySk7B1YH+zJ4k=; b=RM+p3eKt6gUvLmV+iuMePH5gfEyIOMarhLjlzvQVkBzwLQBk18u/JAyyJqlYQ97doTBMGyUoQJRo17i32oUZuFOUuAtmAmrcB4gQuBKMb7ZGz8+SprCdgpmlufXaZulwwoL2s4QOGoRDeot09d4R9L5ykn97VkcrvF/ZB7ooH+iqXifKO+pxda1WGz5hBMgT2stc8N0fDJf+DyeE1XZ1hzoL39qH5YXVDYohgAM9GjC/pO1RYt6bm4EtHRIr/OdHPTpZhdRZ7TrwJf1Fz7f0uKWmOnTb2jA42wYoL7fmv8M6CFxdIM+OUy0GbxqTwpnbClC+be0wfcsYMoMUamchRg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:44 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:44 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 4/9] iommu/arm-smmu-v3: Consolidate clearing a CD table entry Date: Tue, 30 Apr 2024 14:21:36 -0300 Message-ID: <4-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0406.namprd13.prod.outlook.com (2603:10b6:208:2c2::21) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 779c15cd-762e-41d8-63d3-08dc693a0139 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: ZNSHxggfTw2zKB2AbEMoI2CVimDjDcf0h08XCzloDNsqav1iy5WwT4nE041Ekd4ifnEm7q/d+LmLjFD+uoYatsD4P5P7DvPpzt+aEZt3w94IYfCEc7kwUYbUSaLh3MjH77IOV1bPLN2xRE3gXdhMFRy+tUCsgPuTicfVuajZHB503frj8lL4yrhK+HJqvrVUSmibu5E1CYf9ayXOjnovRGq0kc0Nnp0vULNKraEI6keUlYEmZ10H3jP5Hu14kjoSkPf3HewrJqha7Pb2ocz7bIqmcwHbm5QQ11EvDa8kTGlZvECGGjsD+ypKpTe3eR9uohc9jWVsxnU/Y0v3yqlj+b8wKkvFp82BtTYxqhn1Pa4Wo07Oe81an1JyqlTtIEIO89DyUboQ0tira5QkBu9smzE3l5IWKLgCe2Sj3D0jqCR5gpwrC5l7n3Yahnl858KXLpNUcs0lQ6A7e8xAQhIn54Xaqv+/6tuc4QoFmERiWmpvUE7nEtmCB3ipgTbEDTzJH647PrGVVTkTYK6atFRQHnhHrzCsTSlU7ZbGB78v7Txu5qTvxuU1E6Y/E+A7fSbpER4dUWz7BJLd309jCQiGDQxF1vCJnkDP/sZVHctf/+iEAYLJA/hCd1nfYYYLZys1U1aiM/xNDjQ4+kswGszD64hJAgk4E5yyF4S/3pb/8agmyo1FBRd1t6Rj6jzJYqI8jMorrwFTlFwsfP13tMmpXd/YNSzomqfOip2LJzkZyUq1bcAwlXXu1o7ZDSTqKa602iAMv+M16sSNV20ytP0EDBEmnqnlFU5+w2aCX0mCo6H7732gsHuQCH2SeP4JefjqBK4rIeBZ4v7u4/q3RhwRj9EoiYx//mxb7WS1km8bD0g06CdickNH8cFbXV1+9oZVFYQx1gSOJr5XOerO7wO+cTbQEz6pZFsHBMmv2uROzECvRSHj0BItx+BEtKJzkalxymTqIbUFnBhDkqiBjfjDAaOeCblhxOQtqtDJmhmRpla+DTZ0duwqAW21E1qU+QcXnbwRSFhalrU8vvuG7MDgK+YRJdaxFg9+2d9hhs6t/nuTzf6V23Ha1sau6frur4wZb9mdsmpZYZt5T07uEteUbUu3MelxOS32nOb/2+xpdBULco88XQrh76uYbE5BTqadBdshEWyLcJLD9x9g+6lE7pJaFM5j6MxeCZqjYTq/tCXPBDCMr2sOZSMTyl+OT/c2KejipRyJ9sOIXm7TZmS0QFUpiINlKWjyMa5cFQZ1FlDzoQlZo9pPDMGgpeW379eCXcuOxnbYBl/3/1T7xIYJUw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vL+SNgp+vmyEyZ4zrFMzf9HUEOyF8YTrIzFL274eU+IiqYVceBqHYnFgb6ZYuJwioubXYBHftzCR+RSI2nOlQsKizznROrHEDyAjhpYwfRBQtIKJ8i9uqxSqbunCNotSFIWsXUwz22MvIU7iznLKFAfy8hstzWZfYbyvG4MGQdmmr5WZelU694od6RGhJ9viLQ7pfi8j+gWYuAZEZmCw75DHQa5dQ+hH5f1LKxrCJ1KST8lmpnpqwSITRS5gDDj9yqVKQ+aAGFUwMSVZbo7Hj+i8IkpqCNEg4pUSj4REvH/wgLkdt4KQzfy+yJFWSJCaS7V+RaGMuJBvHVfhl1XCBqMzBiCeTD0yxXP4YURQMxuarPRVEJLzr4uKHky5x+lXLKWjY3R6BwW0OW3JKvLZaZQVrO7Gf+JuIo4LWz8oCne7TeY7CRiBMyFBFQgqMP+G1coyYTFMsow5i5d7kFa5Jb6RcQ/2vZxiWVWVtraBYWPZk+paELnHas1K9bvwDBrkjKLKtuBldW310lR3aOXk+GVeARgfvSVHEafJ6Skl/BQUXHm2FmRcae7szoxpADTyoFz5Oe/5dYxMCpIMhSxTWx/B61qg7pMjHLsNKCLbnQy8hq327SB1E2aGJq4Xv7sPIdHUg847ESkQmc/7sowgbsMhzMcZ5+2PL8X2f7jVbPCWvEssou6kyuOnsQMhmocXZVOUwpxgh3R8F9b1y1zBthr1Xy1YDZLqb1a1zVrenQMWseuz5YJu+9NAiKdenGeW+H6jZ6wiJY7aniIHuLAwv65JOAJbFVvVF8TbmWL2l7hezqlD8ERoxhPFAq+Nq2MNx7XbkbGpbgGt4QY41801YXVxzr3FYoqw6I0SPD1J+uFoBeTZ3fFaHzNJxsU2JUPRWCHhhGz1E1Fp3uiIfWPuWUVyTrOLjj5WKtbaHz3Qpgk6mKJ0Ux52V/+UD5ATKSKku4vKO4LVu916CGYpIT2MwMU6+VyfHpP9rNXogiJQfQHBGFwNFeYYB44H5qMQwd5/ZXYOChrFI/jqK+qBV9MaXtUg9VWXORX0eHTAXa1Dh+HsuCPbtkTYkZuMKKZ+1as6gDZTlHkFNmKOj64Rv/8o+zVGrjE2QfJR2HPs9KWozxNMSNzlp9GoTEZumbVNcVTET2ox+N2EAf+ZTyUMlMWPN8VTt4vtMtFYyy2Y5VoDv32fp4tuSt4uLasRWVEhZU2VrMqf9S78ekFyWsgymBtOBqr5wKNCzFs0L++o1agilHyYM10EDB9VmMj/s+MBsm/3zCkm+YMIqGnb6pyN+Vp6SfAtAXzhaRH4W6D3nYpoIx4ecUNBKBRir7bfOEQVjq/Havik01Gaq6r4aeN6P6BwJs7acQyN4exi6/WWMnxEpHB07pWMwWW/OQn+AkWOBQH5tgrFrQXN724TSSlm18ovpoNZ632buiqAj7CYefvhJI7Hrge3YE2YIc236dhb+i03RVYkewjfFzr5P3TztkL5CSPavEyNLVZjY6XONL1DH7+Ns/vUgYQEUq1R0m98QRA50fgMTBeUikjijKDolVGIIcA4RlfSBPEBVwWAQ14qdv5LYMvEWLUI1zZP41EsY200 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 779c15cd-762e-41d8-63d3-08dc693a0139 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:43.1211 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: i6Z3ItqZLtGz2v3OGehi+hfyvj9l8FCUPI/163JP8oD06JGoSBB3AzJRr0Te1Sbk X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102157_763541_1BB76B08 X-CRM114-Status: GOOD ( 14.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence. If we are clearing an entry and for some reason it is not already allocated in the CD table then something has gone wrong. Remove case (5) from arm_smmu_write_ctx_desc(). Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Moritz Fischer Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 3 files changed, 18 insertions(+), 11 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index d159f60480935e..7cf286f7a009fb 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -569,7 +569,7 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, mutex_lock(&sva_lock); - arm_smmu_write_ctx_desc(master, id, NULL); + arm_smmu_clear_cd(master, id); list_for_each_entry(t, &master->bonds, list) { if (t->mm == mm) { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a80758d72d6a7b..8a6c8aa247eb38 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1310,6 +1310,19 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, target->data[3] = cpu_to_le64(cd->mair); } +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) +{ + struct arm_smmu_cd target = {}; + struct arm_smmu_cd *cdptr; + + if (!master->cd_table.cdtab) + return; + cdptr = arm_smmu_get_cd_ptr(master, ssid); + if (WARN_ON(!cdptr)) + return; + arm_smmu_write_cd_entry(master, ssid, cdptr, &target); +} + static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) { struct arm_smmu_cd used = {}; @@ -1330,7 +1343,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, * (2) Install a secondary CD, for SID+SSID traffic. * (4) Quiesce the context without clearing the valid bit. Disable * translation, and ignore any translation fault. - * (5) Remove a secondary CD. */ u64 val; struct arm_smmu_cd target; @@ -1349,10 +1361,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, target = *cd_table_entry; val = le64_to_cpu(cdptr->data[0]); - if (!cd) { /* (5) */ - memset(cdptr, 0, sizeof(*cdptr)); - val = 0; - } else if (cd == &quiet_cd) { /* (4) */ + if (cd == &quiet_cd) { /* (4) */ val &= ~(CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | CTXDESC_CD_0_TCR_SH0); @@ -2684,9 +2693,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) case ARM_SMMU_DOMAIN_S2: arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); arm_smmu_install_ste_for_dev(master, &target); - if (master->cd_table.cdtab) - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, - NULL); + arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; } @@ -2734,8 +2741,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * arm_smmu_domain->devices to avoid races updating the same context * descriptor from arm_smmu_share_asid(). */ - if (master->cd_table.cdtab) - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL); + arm_smmu_clear_cd(master, IOMMU_NO_PASID); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index bb08f087ba39e4..99fd6f24caa818 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -751,6 +751,7 @@ extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; +void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, From patchwork Tue Apr 30 17:21:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4ECDBC4345F for ; Tue, 30 Apr 2024 17:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Z8QKIBzp/LndW+ePuFGXgtBb8QsIKlY9kk38BVGEleY=; b=rpze14iC/x+7Wr NHpvng0QhGszvnH+YiVfdcSqSxKmPpZmQ/HbXXvYEd6pgWa0OPcTtQtqHiW4zoOlX1GS2pwojmDRh 64951Rae5rtOhSEeRO6F8j8hd4u30eiRrcDV17Cr47yzF98JOPQbU06zltoSdgPMnZoZTEyWUFESD jsVDru+39JaLV2RMPV2rtx1Vfcaygqz0Yfqe66HwLliqmh/N3rQTJUuruqCwSqitGBpaiYAMD4ETc sqCbbI9ykuM0ts5wQ8jmUAfv8xsTdq1dJfaXGuCOXJikr0nkD4cpymEAxULIngPXsMHtBeTCJUEKA fKVbSnZioJHR7qhCjXDw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBB-00000007Oey-3I77; Tue, 30 Apr 2024 17:22:09 +0000 Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rAx-00000007OX2-2mXG for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:21:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KvX8dat27HkB8X2m3pq18HqyF6gqBFXJhwEDvGvZJfu5KYHp+JGhfM0elluuMWLC1JZs5zua8pYFOyHXqyGHtmx8qGWjVIY0yktslOqk0EJ53tnLfZfcg4mRw4uVfP7UitR6RZzXYAAGFBhNihhNbkXvH07odKViYZZa+m8EUbP9imrw3ZQQ87vssSFD6dZnApwNfdu8wOJ6ImXz+xKFF8u4zOr5pBG20iSSgBtAEFKsLkLxEDSa74CwK5ApEOgsqe/oA4dHXpKEdUIz1yx8ycbwDFOul8Ln/WjOntBSgqro2FRJmMD96VaD13z/x+qJNZn7CvxSu/PazCi+I1vEXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wGze6D26bkOaVaTHFLcgfCu+3JgXqXg1DoOzkI1N41Y=; b=OvFQxZ+gRFsJZ0o1Komml80S9PvUfvONvrurg2TcNhrELZo+2sNaM381CCowPl1vJFGgK1DD0JDbLhXvN4ZuTlq18tFS5+IGvo5LegwSQ+kRf2CG6JoE+2mMSSHoRkFJzUDa2/ln8t/JI0I177o6mbuvvFPH0oKVsGiO6eSuosvQrFAEXWCqw+dHpmAqJC2XTj06EjqQEdqI+F5V/PCI7Rpv2Yqh+wxROr+POHO0TwkTsn+AP8aPPLOPJbG8RmNHKAJ7aab5Ug55jJr4j5sWEVjG2NsGB8I1p57Q/B5Cav+gSjPWeKpYhmEMdimae9JOsJkQEdoSWsIrVW0KT4l8bQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wGze6D26bkOaVaTHFLcgfCu+3JgXqXg1DoOzkI1N41Y=; b=TdYFce/v4IQX4GKw8lYWrwJgSotqTOZnLbS1bSAN3DCrv9eQQ7NwCD21IM14KCNp5ajITUfOu6u5PeTOBUSaJJlRQDbnygeuvMSCxkgbVUxfSnZjDFG9tS9gluk5exjKGbsjR0QbNUfezBsuV+ZG5cdVY5m+qY9xvJrlqv7ky+NgNWIYfRLMJ82DEiEFtoHS09YM6eDjMLmO0YpCvK3CiaA3wdMVkegfCslYNVxVNYve5vIaXzdvUft0uN9PBSqD1/+JCD74dX+vCFCW9PHnuoN/d/sb5zZUnYeQA4e6PKlM7BCiwyDKWIYUO6tIGsDSR/5143kEDymLyfblZtjHWQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:43 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:43 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 5/9] iommu/arm-smmu-v3: Make arm_smmu_alloc_cd_ptr() Date: Tue, 30 Apr 2024 14:21:37 -0300 Message-ID: <5-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR01CA0046.prod.exchangelabs.com (2603:10b6:208:23f::15) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: f2177a4e-fb1a-4a58-4024-08dc693a010a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: b1fpW/ybhoyNvE15fULq+m+qUA5Go0NbgkRUAGOgdZHTL1J50nkRipBJ9L2rpJFLBLtAtf7azU8K07jDbN/jc2HJaDj5vFs66kv80ojzPb/5/FG49C2ul4u8JSmKybOnOFGbmHA0NFbPmbCB17LAiq9l1QjQPEB+z4ugpBdQFheD8wCap/bZ+RLZp1o4amkFxWeHxxdWW5KKBaIwCLl6KdocFJQC5+FAQhGnPN8IXqrAD9DrkH535BuUzQnPfmjcgOQ6U3Iu3eq6fJlSQD06bCf5xBfgwthzpZmYl2mhzpWpdAx/kCNHUeMvocPGQ2f3X/Wh8GcixvLrrKUj50fveLHt7G71EhdhFB5hYDdErekvC9TD4NseZQs02vO3O4m1ol9553SxqwkROkPmzn+qrTkRLY5FCzpDfFKBFj5Fy7fpbZbuPRE8AYpGv4Xaf1SjwOUHHfXK//zHKJLTWkEHfc9Ymil9kVulCcQoR95F80SXupsgTSwnNk9xh+dmSfBAI6JpVTLuiZvDvka6B2+mArX5W7hi82w5FGc/+KvaQxHEeCsbRfdhMbvZY7wTh9by+SCTjtAiezMlQy9+CTsnDIaz6zqzyLyLpbE/H8nbGDo4MLrEltFDYHegEDgHXvzNsvCWMESIAAqLVBy5yEl1Z+cSYOl09ApsWoHf6I6Ri0fHEPPMvPFL4F2x1VBCP55/HWAkvqIj+nqo8V1kkTixd1e+BIOr0KZ8e5+EIXBWdlpgLfmlh07mNaUeWkAAe10U2sMFr70OzTdS0RU6GjArK4nZroylZl69BedLrUsQNQ57eM4QNtdcWkoMYhGwIASZTaxSmdMOFPOjAnZjcm2eZOIxfyIZgNAxDhYdz+5xqDEWKZv+Uk6OiPX6pVew5SKdg2faVsq3S3Y5MYHu4LkpM9a/4PQWWtYGJJ0qktG48gVRZdSI7eBfflfZNMbvjyaBLIEpW8zL+cbTVoA+hjqYJ900kbXnlnipKRIceZllYEko6OvTGbIIPf2TJbQr49p0Ab/VOYxuigA2SbuPCKf1RCfXKtJnoMkAfivpl7XrrwIkBB5PmqyLzckyH73StYhn+HXhltMs4cC5vARvVC15IPzhlkTLVPPcUuQ4UuxxqfuKCZbRpQafPy8wSksfwBWTkADuCcn4JlVW7aaFkljaWcV0seaWSHKGSvHHos8Rr4+ZUEbnuH2ZBF+refMAxG29qsoqhPqsciqlIiGy13CMqQyJcqABIeMjzQ7+XwmPs57O9hr9IbuP1QE5hLQ3iDhlxbvm2bwlXpxiHr14t4WdFQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dqRlVkrcG+mSBso8HTIed/UMFQ9rjaLZ1PVDkHbwL25wMH/4eaKr13LY/5p5OUcb1yUGgrfe61mGC5UBgD2ZblOgnu3wrhVZkWO6JFm0yLPyJtVJnDf9fDtTIMrodlF+hfP5HWtW4qFVF4PE8usiDie2UBx/XLlXFO67MEOSV5Hj4j5Vz37Hb85eiaRaQeDChAQ8EwRXfmLkPMOuYOXr+fpWD/+h0/RoY5Ff8qiSmLCvvEtUIZoVgWflQDJEvo8AC3tHNC/iyMBHgUZiGfdoDXh+4cQBBjZ+1/HDcLWTdL9VNPRtEi+rVRNbRJJBcJ39CcWvdT1f2gdJptIsRVaCJx6uCgIYCWhNH3+YvzqsqpTG2hsMEGtgpGI/eRUJZDTdfmjKja1RWO+wye+38PA5tlMCWajwga+U8ongxRNAddPwQvrLQaEmVE58k3/8ori05WzVjoBg7aZMrJHObwsJCr5IAvxU70ghayps0Hxfzd0B1x9CUQsgp03gzXss8tJVHl8B8HWiPhWuE8/nqQVZJOQOXF+IHMXLPU2u05ze3Lc7P0F5iIV2Kd2SkFKCZv8nRlmnTR+MMZgLgR3xOsCsDs9uD1bc6eEnkLyeXmYDT9L+A9sjNb7JiQogARSAxFzdveLBrOAB0SSWbNHzqE1TUycW+03gdcJxUYdwYJB+Nh1at0g/ujgq4/3/pr93Qbxg0VESj8PuT3ywolli1uNPidfD06/P4MCmH15zqh/xgydcUyepki3OpoB9cljvEkkH8mMrNVTM8L3hZWJmZX+bIwx26rH0I+N85XRqo/Yr4k1quM49HjZTLynrTIaJeKuTfpwUc5k6jeODIUPQLo7yFANNZYVzzX8dYPz3pmJ4J/N08+R1SlYKzVQETyBGPD+4wuBv58skEP3PQm/TSMmyzuJnQYcryMlRa0UxmG5wIsrv9OLrvCEhpQKM8FE5augs8bDyCgyKyMcWUG8H8UJoGVlyrcDR4WrC/RHqzOHH0HTRkKtQZqBObYv7suOSfSF8QJJjrBnXF+H87HxdB/cRltFCd5F+PamxEeX86ToYWwWWRF6BtLcOIYXZaom1DYG5p/ZQBPavOAaCCGx1H7gRJv8kj+mv2Qsb3fV1mGrU78RgMQVzCD5hx4YJGfHESvxNTLt0ZE7S6gP+1t+ULxixSojJRlHIPQgmI5+QDAs8R1lTazuZwTF6Bzkqt7b/DRfIahsHcd9gZFrAtmn8HMag6JSY5H4Ti0gLAdBRTJIe1WDvzcm9e4J1Wga3PngJErGK0F7DMae4v94EdeQgGm3nbcxPduH/EtZZSkXhGblOIrMUjEVtJaor46bEvjQobmNwqGe8AshI6Q50uF7F0Ys82inyahc+zMZGFsBwgJeoJvOu5pzDQqTtx/tuOvmhQzMcrFZAHFDchiYqKdH1LtOlTUYivUSVZgM+1cXrkSZzJrloT8cyqaP0A6ktccc+OFsZqEyE1KJ8PQM/Su8lLXAo0N3yNBQDI/Rg3faGBXxwnZcWIQiA52Siikc3UH6YtUGIhaBa2IH58gMlfNGomTazoi72HJ6O17YwNcQIJPRsGuF7u5D2pXgcrXJtYAOEGEUj X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f2177a4e-fb1a-4a58-4024-08dc693a010a X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:42.7648 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0baSJfp/LCwWXZj6QunsQKy4yWHEUUm28mbTFSOzXpjOI9XKrpsbnChC7SynyzKp X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102155_790961_2D05EF48 X-CRM114-Status: GOOD ( 16.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Only the attach callers can perform an allocation for the CD table entry, the other callers must not do so, they do not have the correct locking and they cannot sleep. Split up the functions so this is clear. arm_smmu_get_cd_ptr() will return pointer to a CD table entry without doing any kind of allocation. arm_smmu_alloc_cd_ptr() will allocate the table and any required leaf. A following patch will add lockdep assertions to arm_smmu_alloc_cd_ptr() once the restructuring is completed and arm_smmu_alloc_cd_ptr() is never called in the wrong context. Tested-by: Nicolin Chen Reviewed-by: Nicolin Chen Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 61 +++++++++++++-------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +- 2 files changed, 40 insertions(+), 24 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 8a6c8aa247eb38..7dce9cf52ee37c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -97,6 +97,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = { static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, struct arm_smmu_device *smmu); +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master); static void parse_driver_options(struct arm_smmu_device *smmu) { @@ -1206,29 +1207,51 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) { - __le64 *l1ptr; - unsigned int idx; struct arm_smmu_l1_ctx_desc *l1_desc; - struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; + if (!cd_table->cdtab) + return NULL; + if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR) return (struct arm_smmu_cd *)(cd_table->cdtab + ssid * CTXDESC_CD_DWORDS); - idx = ssid >> CTXDESC_SPLIT; - l1_desc = &cd_table->l1_desc[idx]; - if (!l1_desc->l2ptr) { - if (arm_smmu_alloc_cd_leaf_table(smmu, l1_desc)) - return NULL; + l1_desc = &cd_table->l1_desc[ssid / CTXDESC_L2_ENTRIES]; + if (!l1_desc->l2ptr) + return NULL; + return &l1_desc->l2ptr[ssid % CTXDESC_L2_ENTRIES]; +} - l1ptr = cd_table->cdtab + idx * CTXDESC_L1_DESC_DWORDS; - arm_smmu_write_cd_l1_desc(l1ptr, l1_desc); - /* An invalid L1CD can be cached */ - arm_smmu_sync_cd(master, ssid, false); +static struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid) +{ + struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; + struct arm_smmu_device *smmu = master->smmu; + + if (!cd_table->cdtab) { + if (arm_smmu_alloc_cd_tables(master)) + return NULL; } - idx = ssid & (CTXDESC_L2_ENTRIES - 1); - return &l1_desc->l2ptr[idx]; + + if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_64K_L2) { + unsigned int idx = ssid / CTXDESC_L2_ENTRIES; + struct arm_smmu_l1_ctx_desc *l1_desc; + + l1_desc = &cd_table->l1_desc[idx]; + if (!l1_desc->l2ptr) { + __le64 *l1ptr; + + if (arm_smmu_alloc_cd_leaf_table(smmu, l1_desc)) + return NULL; + + l1ptr = cd_table->cdtab + idx * CTXDESC_L1_DESC_DWORDS; + arm_smmu_write_cd_l1_desc(l1ptr, l1_desc); + /* An invalid L1CD can be cached */ + arm_smmu_sync_cd(master, ssid, false); + } + } + return arm_smmu_get_cd_ptr(master, ssid); } struct arm_smmu_cd_writer { @@ -1354,7 +1377,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) return -E2BIG; - cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); + cd_table_entry = arm_smmu_alloc_cd_ptr(master, ssid); if (!cd_table_entry) return -ENOMEM; @@ -2671,13 +2694,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_cd target_cd; struct arm_smmu_cd *cdptr; - if (!master->cd_table.cdtab) { - ret = arm_smmu_alloc_cd_tables(master); - if (ret) - goto out_list_del; - } - - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); if (!cdptr) { ret = -ENOMEM; goto out_list_del; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 99fd6f24caa818..c5c55d3e281865 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -275,8 +275,7 @@ struct arm_smmu_ste { * 2lvl: at most 1024 L1 entries, * 1024 lazy entries per table. */ -#define CTXDESC_SPLIT 10 -#define CTXDESC_L2_ENTRIES (1 << CTXDESC_SPLIT) +#define CTXDESC_L2_ENTRIES 1024 #define CTXDESC_L1_DESC_DWORDS 1 #define CTXDESC_L1_DESC_V (1UL << 0) From patchwork Tue Apr 30 17:21:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 81F73C4345F for ; Tue, 30 Apr 2024 17:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KCQ3Y9gKeLAXyopGp+HXG99hvLQnhf+zIwtaT6M3PJ0=; b=HaNzufndQ7n2dO yuw6ZEHchlxlIXbUr9Jd3Clp6KWaZkapb1WfsAtRSx6GuWsZLQ9LlLiStRVFgWVXCG6KqgnF2nLsc 4JPmFdcVYvDDOWPXfJc2Bgjr/cNX0zCjDIN453TMhMr5hs6OE0AHA7HzeNR9DaIRFmux4yeZWdxyP W8I9yfrJK6DQgmDV25+c/9mg8RMa7XHt/gWAEpTq2JARj3KLhRAM8OQQf/EY8deU+QD7XSy2LHkDw hAGSVGVRPULChr36qL6by3cAxRT9PZg6QGxBlAZEWp9N+sX1jGvyCa0JmydKdja5ASBfPmjhXq401 Fr1a8XVvkr1ac0QJvMaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBD-00000007Og1-1iYr; Tue, 30 Apr 2024 17:22:11 +0000 Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rAz-00000007OX2-27VO for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:21:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LWNOSwUxKAO8UU3K8Rtsk+8v4yIoXBfHIeJKTnIfna8w6/4SAOMRhpzlVzF8aStibbQaws9qH1EaPZ1hb5zvaqrpZJuXrQLQWRGE9ARE8axv6+SwljBEqf2CISH+ckZ8A8C/qs6CP49HaDoum6Hf782S/Pj/MIM+Er0Ck72a6KVxWVUyOiglRE/iuAd2ailK6m0CDjxCMfW78t9J8xGx87z/ScmFezcu3gvjQWIRkuqW+ymXpPGPqcYnTE14Hgdgv/DMTQFtbDwDMECj1wRwZEL6euNJmyHXecTUuHFANf4dQHw9ii9RgLL48DOTu3z+caQHetsrfzlfbh2+n1pJpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Erk8uyaJedaiNiUmPnCRWPE/PXYm9GKIR7KkBe5beww=; b=CKF35W8y7CpvNjUeCfgCdBKb1KN5xCGitHYWv7vPYvgjVU0/Di/CwKwElhibo+vH3IlAwa4GpLA1JPAMPUVXjebimiyaoWPRiHKG2vsF7117AVpCYjXQE2C9wET8S/VYE/qo9DWi+joV17yPMFouKGRf2Npt3r1TJrQjm9J2OH95lL4r3g6EXRC27n0l2iMCw8G6/KCP7Y7/4u6EbDtrF3INBN1YDCdjF/aBXh/4Yo2LT7YLOS4mJZVAt1tzoCSs+KSB05BQmUznLY6EQUxUOzKipLdPBVtX0Z+N1mXy5lPEDhPeg/c1KFAnEiWfxh0bgoKk8axZScDUcpLxe1T1dg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Erk8uyaJedaiNiUmPnCRWPE/PXYm9GKIR7KkBe5beww=; b=BTNQ2e1SpVUPOZlDJGZFFr9vIx/MnrtFTx9MTdZvxXuoQn2mAKzXm3nPmUwgIKKSB1w661x5wX2MQOw3qKdQzZw0y87G4iCUojr116v5RTEjz+AV9LV6koL2jtMR4hU0hewXup0heboYv0reW0ooH/ygJHH68rpGtumrBJrnod4S4BlrOQO/SS7fq36mhkqA4EUFeeuII1O2n6MQ+KoQ25HTlT7mGXqokaesqweZbqlTT7MWPFJqXEFFXkaCZ5cBPm5csMN/9cA6HLfo70Ymz2BdP4hvmpH99qWm1X5Gv+kdW8DvjvYaik3uHtwotgeqH9tjmygxNi/XZW873axiuw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:44 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:43 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 6/9] iommu/arm-smmu-v3: Allocate the CD table entry in advance Date: Tue, 30 Apr 2024 14:21:38 -0300 Message-ID: <6-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0009.namprd17.prod.outlook.com (2603:10b6:208:15e::22) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 795f9bf4-a842-47ab-1268-08dc693a012f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: ea6goo/l0ry7WfFidEr8VxYCdnuGMEMa9GxVXIylLIu3YikaQ1UAZOrH6jAnjZV6rEkQJ4WiyRBl+aJ4ig43sLOqcjAMaYjw1CF2N57RPoKYVRfuEZSmG/lXOQhGjBJqdvHmVTFKmJvfWBxIym8aQuayDf/HkBXnzEbfcUtrwQX9EyKSusywx6wMYiKQGHczKIN550MkHQfMFDUmUHSfXBz6+BqkpMZBjj2pZa0i8re8dh2E9JQbz41r9bC9ljkZf4tF5VNDKXM73B9B9+cNDXLBG47JOydti7VAIcxaGIfcUk7dgsw7SRcB5jDBNq/kYyRQtxZN32u33yxZ2VISVii2jtPJWZ2rWbq4u2+L84Ioh/M8X40PpSZbAkCHg+atRj/DL2MSFmUVzeBRQPmwlk+rFgnBsQz4js91DY2COPesPDu8zUyJ9lPhBhjInojpqk8fgPKcWwOGfUxjKP8XVNjNV/XFLBNp5H9NJXjFewB35zIHHaxgTZQqQalIFHdkynN6sURiKhHGEsLaDfxnkOzCgvl9h8Li2Y1OZHLw4fmxiP0O6oz/3zJ/5abd7ckB0+ZIXxye1TqsLkkVmNnh5+kn5Ox3YgAU5zO62dBAeCrLUMp586iUF/D4WNupsMVE0j0YeFvbYCOJtAX6z36L1V00V37lRS5FQc2wd4cBFp5d8WqHj99iPm0FMa7xI/Bt00/SYHywoSI7pFZyJS/YQ9FW1OpMmLg1N+HgXPQxrO0SyT2wCM2ESoLjPBybDc223ZXGgHBUQw0hFZvmt+BCkAoFLGzVzEX2EjjQ7iatTF7lEOw1Q9JWlPa8YBaOuY7Z5pmUL379I5YaczfUKVEtZp7hfo0vNG8lJiP3RYcXJurEC5Tf45z4OKOqdS2mWW7G5/zY1iaxRbgAUUavWRo7ZNXKP4tfoLsGAWO9H03aDAa7/JqE0RkTx0219OGtv38EA7QAoBkf0eBF9sGJleK4fzpj/TG76DxaQaOzdHYXhtCTlJwxUGL53/z5xU4yGUjryVFvgqR/F3sWRTGgHNVyCOKoCDAZl2P9r8/zFjpp56Gh8vcTbIl5ql2Ldvbqibs7qFYwzvrZFR1rgiucZ2fZlj4zeZHBa1qac/AdGmCAkF7yUr4C9Q+3KfATvnY7EqpIvCBHTZzJc/HM2qW6vzPU3ZkOAXEH5ulKGj35d6EWRJ0Mg1ajU8rJ5QgnaCdUrxNaawCSzt+E4TbjyeWIZu9ScElLsLzW0vKU5eTZNh86bdYUVtqHaPdFiCXyfXBmycDRSFmZjZfHzisxuSJeVB07sA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bmeA4iOJ2lP5ERL+WJCeOPno6zrevVDeAikmpUQcKqUqSLil6gR2QMy/rA+18RQxliq3cUrl+D2nzmQpnTjKTl0DUiT3CQafO5PgsXey6LqWKBX41iotZcLwTInc3Ya2cRdhf7xUkjBftqBHHd1WYJFhyg8IhknbkMr/XiFi6D27tXubce+fNcMgTQgZd8v+a+fLV0NjG+eoKVgE5tY6dan6n5lRWyz7/jG6bY1MTY8d1I91UeBe5YXsstNI/XLl3aS2KIaYZIsudLv3xZ3CkAWCaFrQQ1gDxmJITFEkO4YucP6kYDVpnp6hFkV3KN0HolzYr3/EAfe0b0FpPyPYPVcuGNuMmo1c+RJjsp4rK3wFI4j+JyPQaVghw7c9oBYCFL2VTTvt3L7NzAb132HFB9FlBX0iabKFpvpj6ieS+9P6+nUzId4H5ttyNfaVUa6t4l5nK6n+QFleYQt1sAMGKk/4i8Z0MCYHkU2A8vAOdvK0YLWG1mecnH5Dt6/dPWylKEC9ER/4Znk0h4tblCEaPv9lj9XepHI3bwNJLZr9gcBaYxjih5FyYuw9mD74gCakzN5y2DM60qTrTeRi/fSkhLWX7Mhk+MbcVWdGK9klw3+XzNfigt0rY23iQ3NUeNMZrs4HnBZAh8HbMFnw9AyMThxJ6uJKqR6NEBM/wznBou8RoJbRw3gVc4HWxzRSwN02aUDxzgnrFpzD82YEuXlpTMmy9geKq5AhrS/2t/9Q6QAuali2uAK21Prw2q81NiHjt92P4pIlPRdu+MK+nOs9G3dBZ0/sEaF62ryAGSOdi2CUDMrdS0L8SGSrHtUbzmOOb/l2ogSMnSUZUfRqCO3Zy4sJ9R7WwRaz+HvV3weKxlRp3KEDEYnMXiy1KurGRYK/k/A7UnoQkofgBST8HbPEtHIIA+ku+ZKP/GIl0rF6A6HUTPZYCyyGzYCM1QwTV5BWND0RdMxOQblsG7kMubtL1aDlRkTp5FxYUmt9Xvsq4+5gh+JLI5fn/6sIW49Py5Z4ntaj6X91VhfOk42ZQRnvb3R+skPecbAVRTCRHTUNJX1aKfI/mIxR77cC9ClO6VdVtw89zLyKpENF4cNVD3amfHMdISnISZIW6by4qMxLyrfJYSFNveefSreh+3vuuAgdE14ufTBLPKWkLIFH2YdALhYL9y/RQGFLVVxVMwMo4pmlbhD+zZfYpZDLoCMlgquL9QEhsV7S49cY9RnIdK1EaHE2WH2qXJSoYlSMC1jLZe1H0bx99wLTlmbfUPMk1kTYgXTljFd7+Czwd8WgdGsIHm8MTuE2oDkm7Gg15a+EoSJkhAynoaLrmUjUgP7hN2/XSnEuUvZ+KZ6X0/E8qdnOHkkLtReFuyZyD6z0Nxa+M42vrKJKuCUtZBCUtdi1UaHKjQWsyvvNeLBSXWwPBu0cCqSYuIjvpNm31TrmsNDzddzk1lXYHZ8o+YVxhqUJ6mNwfP3YnGaKf0KK+BsckDCr6VIPVTdl2z/W5AnrX4sMCbZ8OuyBqZrxECjLP0372dn1MbLvnY0X3de3qGTv3F29mv4mdMUImB8jLAkymm8K9PB/5jOYmLuoO3ljG+LFhipO X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 795f9bf4-a842-47ab-1268-08dc693a012f X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:42.9652 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cc9GXGC4E1BzlzaKh9jP2KKGngucn0nWHuvN6s3vm6buBRSOS6H9nAxnuMoo7FXd X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102157_666764_17ED4B1B X-CRM114-Status: GOOD ( 15.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Avoid arm_smmu_attach_dev() having to undo the changes to the smmu_domain->devices list, acquire the cdptr earlier so we don't need to handle that error. Now there is a clear break in arm_smmu_attach_dev() where all the prep-work has been done non-disruptively and we commit to making the HW change, which cannot fail. This completes transforming arm_smmu_attach_dev() so that it does not disturb the HW if it fails. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Mostafa Saleh Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++-------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7dce9cf52ee37c..d0ab49300acde9 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2645,6 +2645,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master; + struct arm_smmu_cd *cdptr; if (!fwspec) return -ENOENT; @@ -2673,6 +2674,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (ret) return ret; + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); + if (!cdptr) + return -ENOMEM; + } + /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2692,13 +2699,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) switch (smmu_domain->stage) { case ARM_SMMU_DOMAIN_S1: { struct arm_smmu_cd target_cd; - struct arm_smmu_cd *cdptr; - - cdptr = arm_smmu_alloc_cd_ptr(master, IOMMU_NO_PASID); - if (!cdptr) { - ret = -ENOMEM; - goto out_list_del; - } arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, @@ -2715,16 +2715,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } arm_smmu_enable_ats(master, smmu_domain); - goto out_unlock; - -out_list_del: - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_del_init(&master->domain_head); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - -out_unlock: mutex_unlock(&arm_smmu_asid_lock); - return ret; + return 0; } static int arm_smmu_attach_dev_ste(struct device *dev, From patchwork Tue Apr 30 17:21:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF81DC4345F for ; Tue, 30 Apr 2024 17:22:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RKIKVeb41eFv2Rqcf6XWkF4uVIkLn0g4RJ0c/YNqvKA=; b=OQIx5sRieAs6jV 8N3keF8YeVwPhor8J/0unqDNDLoqm8z+POQ1Vq0pTNMMrdTQg0YZSeIzG4FWGIhj8H8cfwHIAJPgj UGnssq5n5KVUMbg2QG4DWV8bcY0b25CKr4d1pZWpZQkO7iJazMzCrUPy2auOch9Waew9OAcHQq6yX UUHWthNzLkq5iik6t6tYX8ibB8g759XkUy80V4qDagypHka7cgH4Wfm3h9EJ4KAIHxIBvri/qxFT+ S+NWanLlVywuXSeUpEcKj6AZOKXVXeeFAppUWvVPjusEmXL6g1Ri4kALgTjvP4gsQG69PeCLOFFs5 rVfvOL6yIM6MkrdhQTPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBH-00000007Oik-3LXp; Tue, 30 Apr 2024 17:22:15 +0000 Received: from mail-bn8nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2418::600] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rB1-00000007OVf-3vN3 for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:22:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NqXmDn8Gk8ufhFSkbXJMsG0ftkOadP0BxUYd/FhdlJgJ3a2puAN8bYMo37gEhLkAtAXoShNmkxG2cp3gqLOdiEi4GWULT854FNrb3nTioYmGzUBHGE1w/bFgM0A4EQ3Kg4MV8CkCFRR/nKAhsR7WuetKiiwa3WlcudYxf4Am+SQ183VAUn3XjCTwntFQr+xp32I0d3HOa5bR3B05Ykx7egWGw10wDGREVbNuaeqOAjYOpQbYKYHt5AUy9avlcx0TtNdxLbw1bOAQ8GDCDYN/h27Pi97xHigWEkFltYa8fBxmusCBi1kDLoTvBsgtAvhXKd536EnrK+Evjf8iL1S3Ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6NXyUR/KFc6V0DsyTcvLnSk//4Fu7kVY7vCq8LiQfQk=; b=fQozcMziF55X7hauoWL6pZLm3fdyX3fZXvqeYp9Gst0sCzsbZ+LlNroH7cRhAijaSAe6Wd2vlZJhAut3nwkOMRn+e97slonMgaIUjgL995f3+5EYsjvhZeEEHrBd38ss6XHH+Wc+Dns3Fm9KaGG2jdLJCBFifbm8YRQx1KhIPIMl3gt62ChxE53S9DC0eHTyajBpCqZ2tZ/Pt+VgA1jOH8+YbiVrQHCibDWkp6GRFGU4rrw1abW3vAJzOXZlZ8NT9HUzlwCAnQrAOQ5v6U1V6xCwXGD5noCLfVvFV2fFGqe596XmfF79Js9uow0kc+GbyTW2U5+LtCoKbmi5cUVjxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6NXyUR/KFc6V0DsyTcvLnSk//4Fu7kVY7vCq8LiQfQk=; b=G+fWmkOuYAzTBnXubFp5ASXtswAFQFXXyR7p1Zv9cOhK0ENgLq3JefgzTjIdtvUhb3s6x7wHrcI8yW2r/FYMXq4j0SuY7zfUhXkiD54VrZr9GdqBfoMb+sk+wiY+Rn47NdfaNMtm+o1x8B1lC3mVrVLQoEM2G9Plb62kQKPmU9HNvKRa0DvnzgDzEi8axgzhKKt7zZtfSOCfI2PU6bSpupZeah62mEkB8OK2UVuhrCeOXmn3MepBXHsM9RGFzwSQ9GjVr+rDSl2giV5mUKHNQkAGhFCsi1puR5m8V/Zwrjn4Dn0GXncmxKw7DChvjUyUDzYzT0MiNPj8nobHeo2+AQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:45 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:45 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 7/9] iommu/arm-smmu-v3: Move the CD generation for SVA into a function Date: Tue, 30 Apr 2024 14:21:39 -0300 Message-ID: <7-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BLAPR03CA0140.namprd03.prod.outlook.com (2603:10b6:208:32e::25) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 9444d65a-de30-4862-7d46-08dc693a0126 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: xvXGF0En2MmDdaEYWYO5EAkWSJA/OTiPHGE5oxmRrahhyuuFRW5FJ9E1Xj8tRKRwBOwXW846jY0hAy48Hg+CiFhs0C6wgSwUIA5d/jdaW/OXr9KUYvbNVgUF5P3BJEiTw9qzyyocQnsLHnLQIN9vWegbi3Rjqx5Xzc5b95Z/MXmVcaQJNVL5F7WH1bdDMiAhCWZw7RPmEYNvHU8GSKGozAALo3bnNh4ehSC/nUrafYYo2wkNVFgdAv6R4kR7crjHPPzsnDWdg7L5hLKvBbtr0lWJhRmFAesRxna8qhWUKV6csIKEbpKg9KWxfIqrGFRFzoOKz6jf8oAjYXOhjLKqxuziUTVTqxrvFB+rOWgtOjkuEn/txp7EoVmWm8myy5j9+0TE1gGDkipkXqaz7wbQ7Whnn02Lr/SKPzCo7Zt4Ef4tzAP7KrPgFCVtfXTgy7aW74wsSmqV4gs69J5gs4Js1mJDIjMova1Q6WNPH25rTEcHWyMXGGQWCqgub/u8iPAYVa+qqE4d3wqfZZVwgDYBchdshILEjqp/dK2Rncmz7cNUK/Jtl1jFmkdv8y8GtdkvXkiRf4rnItfZFqmCKpRIE2lbEO/VFL4bcsaGb0RPtrFPglym+2Y1p04DF5+O5LbXrcCkhCcJSXe/1ZFanhd1zUwFYCBjkKChVUF9yu9xn7Eo0zCfacmgBBxISHHkmYD5VCFn+82kJuKCwgj59fga+DBvXYNCf9MpgZISN3Fgwe/mZgWRn/mjz2NenS+aZu8JYX3eodLKX+XB6PhfGmO+omtyItMzApIB545I+6+BVPHzLPiKK61LKqIzOBmt8usWyZDA3sUUBi9xJk3uPrNTXhJRXM3Nof3ZRvazQqWn6RJAFhf3c5QeXjdq2hmFA67oCxSUGRSM7TS148XsB3tcMaeMkHcEL+BqxTqIaPkcIJV7/bQ1pT4w/5EDA0a5tauzQNjudKaB0jjCKY1Nxw0dIN+PFUigs7TreFcL+O45PzcrdPEv2105xuM79Onz0nV+MoGkRgNqlTVx3wyTqXs2GmbP3LrNGWxFc5YEDwyRkhUPNqKOKtYlgc81MLeriUZYpvljgoYssNPgs5sGw+riv0E8ZrQDGdEW7Bc4x0hMf34ESNw76PnRFjlbQC0N9gQtl4r7inhIvsaZU/yjQV7z1XQuQSrybWfjjTWTCwqb5soN1RI+23d50Xb5qeE0cE9bCjTKBdKRh6AL3pqYmRKXKrAnx3UNqA8EbVaOJJ8wNmC9NcIKZG5xG0c1lS3/J4n9tKXLYyKSps4hTCBSbi3bUg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 24W14yEwcaQ+3KctYOjoyjumgWHPqYve2lY5c4vpN3o3btfItPR3gPa8PsSqWth18XC1lTvMXkXwP/7DxiGTk3luEr3xqb5OMOYl8aAvcPxiehmeFf+SzlLf2MjxvcdJCf6P7+k29Iz3uUK2rw16iUhYEU/8AxR/bMftn6l6IBYHFKD/vjjWiYaPcS8WlQ2pH63buD4zVxfJbTkAPJTuJD4tpt+CA+u4R7A1O9jhXEYRBy4T06cvIyjCV2h06XwJPyz/judl1o4xeNhB0W17TpIiACARwNVk2FW2McXB34MF82o5QrYjNDvaM4IrSEnIHSAu7RR1+PVRO2zjQvhUDCUGcpc+gD5cswNjuOkRCVPgIiA7hjynT1UAuzl8j+fASWmAxBgYVippduX/XZcw6NWiUVHljYqjLYyRl15YjWT725fcRSJfh8VxiXPSLZ8FQfSzqIH3tNTmERuaVE0IhNwD7rJSIU7UsNXpJkcw18wdA9ZyS4g4kyYYfv3i/93JDYuES3O1tyaY9S/nRNjKl2c4QtoVDLlOzE1ozN7RL2djuwn74XFx8OviZvvPQQUG6Cgv7evNeM90oorGfFrprBWCrRi3DBFYeJkFbR3oII9MPjLtDJHf5aZX19xUhMYJYbAiY9dvUG8mmuQ0uI+5PPfdKmbG4hohjn1xGh7VPkwulYr8RTYqgR7CnRAEHsUl0e8UOGZWUUEIuDsYQFJ1m6qWLzn7cT5mqKP3tNKZ0cevT5AHXI2Wd1+jNgw+XQqiIQprqdWB4iZ8Yy0ncQ9pytEO6DKV6/z8LAuCNeotYy3hHtlS2XwTO1N44dY8S8gkSdBU0ooj3aE69j+eCGD5xtGsv/TR90I/CLVrYCXWQApzND9hu3YU8lH1bGwNr9HiuVCwsr34kUnoXuotR58c+FJr8nUEjC0TVr09VfPLGdpFlP+VWO0dxxN7tc2XUFosp4DROkf73jcuhVF773ATME+9memlBQx+nfkmXTlyGBedyAsos7B+2HstU8s191RUAQKV8txzEYh8ntACsrG+//qW434iTqN41jILtC2+ox3YqTbCCsxdmkTd3CG5oMdssd3REqiCFbV+XA6XvZeWFw2AQNFmE78ZMr47AGeATa9dytaMTYTkloBD4iSDV8DFvh3YuRA3XyJrRxn/OQOsSWVWnexJ+aO0O+weqCKjVqVH34udKHyAX99AFZOy8ko7fSEoHFPg2ci01OcLCTB9sV6WuFkgc8ymqnIrXgEJj6DmX5xfclOmVvtO/OvCpEUYYik6cQ/pjacVvPL7pES50wplHL5tHzu5I55JqI7nX40XFPHpYgq1D4IYZfGCtuDd20zQGeTi9oTfypH2U5kJGuNa/7oabT4Dz6A9lW0LAMSMVlM/dUvXnpXqI7KML+d4hDuvzZmu36OQKSeW3aJy5LoOr6btBcljxPYOtMBRtXX3gxC5Xl3FiqeZuBzeYlWbGU8gOH7iTzqkj3bdyoOCSk8VkZE0Fm6G5Wkh0Y/7O2fwFIIUiX6l3bd4/6PJ3iXPD0vxYIMp3ghrZY+lNjX8TVAollrHXe76iCv7HSgbVA5ji6D2hHb4JsXQdU1aKJMF X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9444d65a-de30-4862-7d46-08dc693a0126 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:43.1787 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8MueHRrEvduCq9UjSoaGGT8gfkuEKt3yULEr1fIluGDIQRjP9jHJVw+aETXdnmWF X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102200_095171_84E93BD5 X-CRM114-Status: GOOD ( 24.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pull all the calculations for building the CD table entry for a mmu_struct into arm_smmu_make_sva_cd(). Call it in the two places installing the SVA CD table entry. Open code the last caller of arm_smmu_update_ctx_desc_devices() and remove the function. Remove arm_smmu_write_ctx_desc() since all callers are gone. Add the locking assertions to arm_smmu_alloc_cd_ptr() since arm_smmu_update_ctx_desc_devices() was the last problematic caller. Remove quiet_cd since all users are gone, arm_smmu_make_sva_cd() creates the same value. The behavior of quiet_cd changes slightly, the old implementation edited the CD in place to set CTXDESC_CD_0_TCR_EPD0 assuming it was a SVA CD entry. This version generates a full CD entry with a 0 TTB0 and relies on arm_smmu_write_cd_entry() to install it hitlessly. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Nicolin Chen Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 155 +++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 87 +--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +- 3 files changed, 107 insertions(+), 142 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 7cf286f7a009fb..8730a7043909e3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -34,25 +34,6 @@ struct arm_smmu_bond { static DEFINE_MUTEX(sva_lock); -/* - * Write the CD to the CD tables for all masters that this domain is attached - * to. Note that this is only used to update existing CD entries in the target - * CD table, for which it's assumed that arm_smmu_write_ctx_desc can't fail. - */ -static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain, - int ssid, - struct arm_smmu_ctx_desc *cd) -{ - struct arm_smmu_master *master; - unsigned long flags; - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - arm_smmu_write_ctx_desc(master, ssid, cd); - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); -} - static void arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { @@ -128,11 +109,85 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return NULL; } +static u64 page_size_to_cd(void) +{ + static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K || + PAGE_SIZE == SZ_64K); + if (PAGE_SIZE == SZ_64K) + return ARM_LPAE_TCR_TG0_64K; + if (PAGE_SIZE == SZ_16K) + return ARM_LPAE_TCR_TG0_16K; + return ARM_LPAE_TCR_TG0_4K; +} + +static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct mm_struct *mm, u16 asid) +{ + u64 par; + + memset(target, 0, sizeof(*target)); + + par = cpuid_feature_extract_unsigned_field( + read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1), + ID_AA64MMFR0_EL1_PARANGE_SHIFT); + + target->data[0] = cpu_to_le64( + CTXDESC_CD_0_TCR_EPD1 | +#ifdef __BIG_ENDIAN + CTXDESC_CD_0_ENDI | +#endif + CTXDESC_CD_0_V | + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par) | + CTXDESC_CD_0_AA64 | + (master->stall_enabled ? CTXDESC_CD_0_S : 0) | + CTXDESC_CD_0_R | + CTXDESC_CD_0_A | + CTXDESC_CD_0_ASET | + FIELD_PREP(CTXDESC_CD_0_ASID, asid)); + + /* + * If no MM is passed then this creates a SVA entry that faults + * everything. arm_smmu_write_cd_entry() can hitlessly go between these + * two entries types since TTB0 is ignored by HW when EPD0 is set. + */ + if (mm) { + target->data[0] |= cpu_to_le64( + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, + 64ULL - vabits_actual) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd()) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, + ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, + ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS)); + + target->data[1] = cpu_to_le64(virt_to_phys(mm->pgd) & + CTXDESC_CD_1_TTB0_MASK); + } else { + target->data[0] |= cpu_to_le64(CTXDESC_CD_0_TCR_EPD0); + + /* + * Disable stall and immediately generate an abort if stall + * disable is permitted. This speeds up cleanup for an unclean + * exit if the device is still doing a lot of DMA. + */ + if (!(master->smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) + target->data[0] &= + cpu_to_le64(~(CTXDESC_CD_0_S | CTXDESC_CD_0_R)); + } + + /* + * MAIR value is pretty much constant and global, so we can just get it + * from the current CPU register + */ + target->data[3] = cpu_to_le64(read_sysreg(mair_el1)); +} + static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; int err = 0; - u64 tcr, par, reg; struct arm_smmu_ctx_desc *cd; struct arm_smmu_ctx_desc *ret = NULL; @@ -166,39 +221,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) if (err) goto out_free_asid; - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - - switch (PAGE_SIZE) { - case SZ_4K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); - break; - case SZ_16K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); - break; - case SZ_64K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); - break; - default: - WARN_ON(1); - err = -EINVAL; - goto out_free_asid; - } - - reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_EL1_PARANGE_SHIFT); - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); - - cd->ttbr = virt_to_phys(mm->pgd); - cd->tcr = tcr; - /* - * MAIR value is pretty much constant and global, so we can just get it - * from the current CPU register - */ - cd->mair = read_sysreg(mair_el1); cd->asid = asid; cd->mm = mm; @@ -276,6 +298,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_master *master; + unsigned long flags; mutex_lock(&sva_lock); if (smmu_mn->cleared) { @@ -287,8 +311,19 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. */ - arm_smmu_update_ctx_desc_devices(smmu_domain, mm_get_enqcmd_pasid(mm), - &quiet_cd); + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr; + + cdptr = arm_smmu_get_cd_ptr(master, mm_get_enqcmd_pasid(mm)); + if (WARN_ON(!cdptr)) + continue; + arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid); + arm_smmu_write_cd_entry(master, mm_get_enqcmd_pasid(mm), cdptr, + &target); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); arm_smmu_atc_inv_domain(smmu_domain, mm_get_enqcmd_pasid(mm), 0, 0); @@ -383,6 +418,8 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid, struct mm_struct *mm) { int ret; + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr; struct arm_smmu_bond *bond; struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct iommu_domain *domain = iommu_get_domain_for_dev(dev); @@ -409,9 +446,13 @@ static int __arm_smmu_sva_bind(struct device *dev, ioasid_t pasid, goto err_free_bond; } - ret = arm_smmu_write_ctx_desc(master, pasid, bond->smmu_mn->cd); - if (ret) + cdptr = arm_smmu_alloc_cd_ptr(master, mm_get_enqcmd_pasid(mm)); + if (!cdptr) { + ret = -ENOMEM; goto err_put_notifier; + } + arm_smmu_make_sva_cd(&target, master, mm, bond->smmu_mn->cd->asid); + arm_smmu_write_cd_entry(master, pasid, cdptr, &target); list_add(&bond->list, &master->bonds); return 0; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index d0ab49300acde9..f021268dab4763 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -83,12 +83,6 @@ struct arm_smmu_option_prop { DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); DEFINE_MUTEX(arm_smmu_asid_lock); -/* - * Special value used by SVA when a process dies, to quiesce a CD without - * disabling it. - */ -struct arm_smmu_ctx_desc quiet_cd = { 0 }; - static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1200,7 +1194,7 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, u64 val = (l1_desc->l2ptr_dma & CTXDESC_L1_DESC_L2PTR_MASK) | CTXDESC_L1_DESC_V; - /* See comment in arm_smmu_write_ctx_desc() */ + /* The HW has 64 bit atomicity with stores to the L2 CD table */ WRITE_ONCE(*dst, cpu_to_le64(val)); } @@ -1223,12 +1217,15 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[ssid % CTXDESC_L2_ENTRIES]; } -static struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, - u32 ssid) +struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; + might_sleep(); + iommu_group_mutex_assert(master->dev); + if (!cd_table->cdtab) { if (arm_smmu_alloc_cd_tables(master)) return NULL; @@ -1346,75 +1343,6 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) arm_smmu_write_cd_entry(master, ssid, cdptr, &target); } -static void arm_smmu_clean_cd_entry(struct arm_smmu_cd *target) -{ - struct arm_smmu_cd used = {}; - int i; - - arm_smmu_get_cd_used(target->data, used.data); - for (i = 0; i != ARRAY_SIZE(target->data); i++) - target->data[i] &= used.data[i]; -} - -int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, - struct arm_smmu_ctx_desc *cd) -{ - /* - * This function handles the following cases: - * - * (1) Install primary CD, for normal DMA traffic (SSID = IOMMU_NO_PASID = 0). - * (2) Install a secondary CD, for SID+SSID traffic. - * (4) Quiesce the context without clearing the valid bit. Disable - * translation, and ignore any translation fault. - */ - u64 val; - struct arm_smmu_cd target; - struct arm_smmu_cd *cdptr = ⌖ - struct arm_smmu_cd *cd_table_entry; - struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; - struct arm_smmu_device *smmu = master->smmu; - - if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) - return -E2BIG; - - cd_table_entry = arm_smmu_alloc_cd_ptr(master, ssid); - if (!cd_table_entry) - return -ENOMEM; - - target = *cd_table_entry; - val = le64_to_cpu(cdptr->data[0]); - - if (cd == &quiet_cd) { /* (4) */ - val &= ~(CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | - CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | - CTXDESC_CD_0_TCR_SH0); - if (!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) - val &= ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R); - val |= CTXDESC_CD_0_TCR_EPD0; - cdptr->data[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); - } else { /* (1) and (2) */ - cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - cdptr->data[2] = 0; - cdptr->data[3] = cpu_to_le64(cd->mair); - - val = cd->tcr | -#ifdef __BIG_ENDIAN - CTXDESC_CD_0_ENDI | -#endif - CTXDESC_CD_0_R | CTXDESC_CD_0_A | - (cd->mm ? 0 : CTXDESC_CD_0_ASET) | - CTXDESC_CD_0_AA64 | - FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | - CTXDESC_CD_0_V; - - if (cd_table->stall_enabled) - val |= CTXDESC_CD_0_S; - } - cdptr->data[0] = cpu_to_le64(val); - arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); - return 0; -} - static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) { int ret; @@ -1423,7 +1351,6 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; - cd_table->stall_enabled = master->stall_enabled; cd_table->s1cdmax = master->ssid_bits; max_contexts = 1 << cd_table->s1cdmax; @@ -1521,7 +1448,7 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, desc->span); val |= desc->l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - /* See comment in arm_smmu_write_ctx_desc() */ + /* The HW has 64 bit atomicity with stores to the L2 STE table */ WRITE_ONCE(*dst, cpu_to_le64(val)); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index c5c55d3e281865..5540609069fcd0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -608,8 +608,6 @@ struct arm_smmu_ctx_desc_cfg { u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; - /* Whether CD entries in this table have the stall bit set. */ - u8 stall_enabled:1; }; struct arm_smmu_s2_cfg { @@ -748,11 +746,12 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; -extern struct arm_smmu_ctx_desc quiet_cd; void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); +struct arm_smmu_cd *arm_smmu_alloc_cd_ptr(struct arm_smmu_master *master, + u32 ssid); void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain); @@ -760,8 +759,6 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, struct arm_smmu_cd *cdptr, const struct arm_smmu_cd *target); -int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, - struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Tue Apr 30 17:21:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D124AC4345F for ; Tue, 30 Apr 2024 17:22:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=L/aedMc0V7/9vG5UeOpx55sOYHwXQQRWo6JbrhSPPGQ=; b=oUdvz4km5j1udd 3M56jyU96uI845Q/xwAy67by0VQ8WMkvPGqNJsIALahK0UyAdNVcKf829fEWSu1NZOXRYVcMuJckU K4Xi00RgZmmrV7gVJqG46kAYXzYvcsbj4hDUCnxDSjCTxb74nC0pl6b0UtBlAuG9aAknbmQ3Ijb/U 3xlVNATcAFNX3+zX403jdm7ll32LwMgI4Mrb+rbfFY9+d7eKNQE1iNMfO5rOKIu24BL9shPBB7W/L ttV0LrDx53JokfNovpOAjSy4KGe851xwbCvaZzF6iheP73Q35GeEHSK90Zq5/SOnI6tQB09bNSlVx tqeMzAQsI2b7EAHt6HoA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBC-00000007OfJ-2BYE; Tue, 30 Apr 2024 17:22:10 +0000 Received: from mail-bn8nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2418::600] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rAw-00000007OVf-1yj7 for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:21:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NpDtcfD/t1TeUVpPfYQEwOBiakauJWFlEfRXUn1w8VNi6uhs8xZHTz4b3CVp640i2xedQoL5hezyGZmQDU5fzNFQ1WaU4SZWLPVwqRpzLEfQwpw6KWRR7yUTN2AvPxXmlOMIZEdgjt68aLvlMilVcBLB4mUW9wD1SvP+aQUtNrtkIIhTSIA01sUZmMAi/AZqO+GGBoPaCM2OVQxUM4uBibrb1wAsOzbn+mUeB3NMUwpcCgIS6r5gaj/Zgaeqd7h8YP6wdl9tvY2rqBNViQFpTg1eR0cv8G7tN6kEp0HYPiYwXCVcblvkypV68i552/2xHMFXTX3d8jLz0+4Oa6Tlog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P9Gc0ljTfkZiDfNYJqusXC0QJBdqdT4d4ljYV+FyCtw=; b=DJbg9T3DRgqoj9rF9VhdeecSe95M3rGZ1u33eqfo4GRGLkoTWbJPUrbt6/zFmvUJs+8GmMoPnXV840ZIaFg03PF9aUT4NaUB0OxRuGvA9VR49eqoH5lXDURwzw2j/Etz5UsQQcG4xTzcy1dsHXj9pN7TjJTlsgMz750cYEHXDu4BVfuaaXC3m5423hhp1omjsS+HNl2naNwmwp4pdjFW/PkGRVFXXkb8w0ngQoY+vIfPJksHWjDyHNkRBq5KwCcXdvSxJnxhVPAmR8LnU15LqQWRLOeIjfqhxk0To/2c0UUTa+0dw1hC9uK7AEmVrL+jh2W9h01CZ4y5Y+h65T4fUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P9Gc0ljTfkZiDfNYJqusXC0QJBdqdT4d4ljYV+FyCtw=; b=bppu1QebjTl792WWmbgatin3EdNzn0fZ5+0RJLMkN63mIX42Cd93nGQX3UffH7/55XLW/dwDDwmRApr/cRVJwb1VBpNb0ebIcSBF77nuQktmj4rEzqhoCDswQBnQmdlXofDCIpEk0YKI2KAAFE5hOH5UneSCsr6AEa0+efbF06hCrxL7WY2Tj7gnpPcMmbYcoQnUfFTwA8zgArwpSIR1PF3JjEfaBdWyr4G+eDWVBNa4cZ3oRbP87VnMW9mTLyhrlWSPEGc7/MOj0b0JLMInVN1hjGwdbRE2MYS9hm2XUxUZD1AtfeDzs+W1J9FT2Y7nSCjmFIYS4HeA47zcNNDnhA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:43 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:43 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 8/9] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd() Date: Tue, 30 Apr 2024 14:21:40 -0300 Message-ID: <8-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0001.namprd17.prod.outlook.com (2603:10b6:208:15e::14) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 9f0de155-62b7-4edc-c4e1-08dc693a0114 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: zA1rzPIVp/tALkc5JFnesolGay2D61kb3LBD0dfLgGBrWy+enVZmL9hPpe58Q+H04IBcQgcOgICcNed8btzHfyuQ9qNLL2PLptItjkOeXVy52OQjGZbfH+bMGlGjeZLxoe5L2fGePo0KJRQkdZ1A/DS0bl955X8XiwCQwBIRFfzPd6M4gZBlg1DuWDs0bSHEgi5I9YCXDX8EztZABJC7IgBX/wnsn8z8WKvY76oPkMwtVXQSZdUUEbRPcJQbyEtJmhWVnTM05sVdCwDiQnHRaOvhby4374uYnn1NonjHV/vV0wTHDSIcO2zyhZp/lJ6Pc0g3I6nTE8Oa3rzotLS9hxOeqFmBxHrB/+jX/R1yk/MVolhDBSckM+d280tVBIULUPC0LnrX4FIWB3NZFv8uFgv168rBXgon7F6axeNvDocEbYX/s/83xoQE1hjWomqu7UbhYnxiVC3lKPjo+tjsFJMKMunh0fZzlScOFD1HOqmt5k+yH4YGGcLX6TYpEege0m0QRRpPRw/15D52PZ+yjMWEkKvNmA4hTIk3Vf8wyNd7RuK4OFOv95ted8wAwaz6cyYtnJxvPVaPlEeR5znukrDYZZfN4j+AMdtPFAvcRUIxGt7hoGO18cfwxI+lonMMd57zUa0GfEj5a0jtqzXnWGIwaEfi/3xx8ozE7DEEqjaZw3DZUCepIlbYa6Kpft1IAqRKxgRSxEAw4/ZFb8rNqaCOT41gmrUgw7j4lkHJQDsCYJCln68abMbichGPRBJwtjXZ40u35zLT4SwfbX03iUttZYrHHb64LbBTdiOXSG+ayj/+kW6iQFX2bLAqR8CLwK2TTBNWR1UnLAtppaUwrZXpUzP89n92CE2mSLUPkFfpn1vCF7NKKRQ/sn9Fd6obeUOoBP1Kth2N9EnmB00gL9jf5A6QchAeRdgOw+66uGLa1feMHQ0QtjiJPlw4w1KXDRMczidH+w7Sgpov/TkzLM86dygw8jdanafQR85RZPTLmgzQc4mZJ7mD0EJFkKm4nrxEegwJumbq5pw1jpqP/AjAuEBNN12FoklTXQJt04LRTe3NUU6CNJXEW2dHUQDLuXcM3igkgOxU2+4Z8k2YQUgZmfwSobLP7IJ8+vhYBt95+p0dCEb1lYi9vXpD77apshNrHDpHYj9PY4WbM9C/PUVcXMThek6et5klQ82I8GBLilE3vC5R1K+cygvS36k3fVWbnS7/XQKyUQ4MQmJg+IzlCkQxr9isJfks7wEh7yXcd9EwGWEIVL2XIc/RHSui/U74gh+XtgJmVn3+baHHFg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: oy+MTZ0NUvNfrOLqlr75y2msnutpBwmV5Ih6srzdX44dduQkMUgL+UGBF4XUET9s2eUV34oyhlNQZTpvFqZkH3pQlDRwdDOjrsMgcd6/zRw3h2taD5ASOgdEXGA7wGE4YhxY9LWCSHooyLLBjaglosNgUZuG3dOb8gdUwCE1Iy+qXUPcyNB5Ws8NTyOM3Hyfj9/5m8g7VBV3BXp/U7YzWUUlSPvcgv55je10/kg7NRg2EA2xkXMPxYkisIP6O7zjmZr2H7LNqh1fMECBDTRDCvEAlwSRwwlcFau2+mzQ1XGm2PZ4+xB871oJdCMIk4TMlI4qtHhP+nvv8TzNNnQlJlt8AnqfcB4GtmEbTqgmLC/3sKJ9wHVYtJ37N7SJdjXCK5DQ8XihUkx+YsmgehEVGSL1Wi3weDxPrbz2q3fd/KXmY1XA3oBRIOjuiOENcZY673iXl2en67oZhrmV6aBNyqZ1Gs6CXwoYvfR37AQJOoJOAYtb2vDbCW42Iuu9TMT+RFq/8dWHoe5YFMO/eAc+vniATp7CiXJ7VdiC7DNTebUcVZxU77lGtljvfv0cAvsWtFo2WKPnq6utoaV64u+rU5SRwG2aPpXxYreeNPGFxHyPhWu6dYbdhJeetxfU09ZS055BDnaecOMmBVSLVxT93bOBo8Q5Q5ziS4zweUuQxYOYYfMXk1FsvkqHFK+LUamstoAaGiur4qIWaSr1KVf5gWQqKbbXIQm4Gip+AemneVu9623YK4egBAn0i1JOVJ6CUHApg/zHIOcJp/BQNHs8R6tEaWN0jVuBbxdL6Mg6p5fQS5ksjjf1xOsHNDoXQKSgrN26xs2dv6ggbX/jsV5gv3to5gHLW9UkvlCHqLJCiv0C+z1JmOK1hI8VGDz1fgCjur3t3G0smEBoPsY9qpmUs+YQNzScjWSEWXtDPeckwZZ/tot6vEKl8YeovnGcTrcRerU26IO0EuWonXLv1az/pQ7NgYXU7QHYd+1kevZi4sY17edYV3W4vGA2VdmS3vnm110Slwi48JzOnKPXiiSynYCKXMBsLbdp6rL9gic9Fy/C2nefSp+gnG6tmCvetdwFrRecAuP06EuG8ccFPoJ8TYnnh6ZY1NFJfeSOKpwd/PA4Sli8ys9mhZvTXv58oS2SL4Ecyb7YfXAXyt2Fs+e+QS3yK0GxSAEPKAisAqX7GbD/QRt/WHLX9VlWjQRsI03JIBs1qwwyW0rKcRZA7gTcFp5L3SH+p8/6wyQ0UqMOewapWpaR+3tPmOSjiI7IJQBMmKhMDulMK4M14HQM+WUtMCe6AElRGY7yucoBaDoUxIBhDq6W1qoLlgUXBcr/BNAU03RkkbOTCEo6l7meO0OBKwayaxZ14q3nI96ioIDITGWfOaX7I9/OEbRbeTmlcN3BDY2Oi1Lzf/Eh4n0zd+c9aM59uCodevdIL4hT3JyS9SB+TNCCy9bCebLW7Sl5f+J0O6r1MhSmX9YHF1xqQ8VZgja5XF2FliACeAwDwyv9yUxqXY5BA3ZY8eXVjVKTcDj6HJBejJHjC9mCjqArz8lv/9gtScdJEkniLAQkrFp76BrCoHF0EI5EvrkY2PhBMIfy X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9f0de155-62b7-4edc-c4e1-08dc693a0114 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:42.8413 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ET7ppnV3UMdLqQbyN2e9XojGALFLn+fhOUL7Ib0VqeJhaGgANoutt6nxH2dYx1p7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102154_544497_B6D9B4E2 X-CRM114-Status: GOOD ( 13.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Half the code was living in arm_smmu_domain_finalise_s1(), just move it here and take the values directly from the pgtbl_ops instead of storing copies. Tested-by: Nicolin Chen Tested-by: Shameer Kolothum Reviewed-by: Michael Shavit Reviewed-by: Mostafa Saleh Reviewed-by: Nicolin Chen Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 -- 2 files changed, 18 insertions(+), 32 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f021268dab4763..3817bc99d7e319 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1309,15 +1309,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, struct arm_smmu_domain *smmu_domain) { struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; + const struct io_pgtable_cfg *pgtbl_cfg = + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg; + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = + &pgtbl_cfg->arm_lpae_s1_cfg.tcr; memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( - cd->tcr | + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | #endif + CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_V | + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | CTXDESC_CD_0_AA64 | (master->stall_enabled ? CTXDESC_CD_0_S : 0) | CTXDESC_CD_0_R | @@ -1325,9 +1335,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, CTXDESC_CD_0_ASET | FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) ); - - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - target->data[3] = cpu_to_le64(cd->mair); + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr & + CTXDESC_CD_1_TTB0_MASK); + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair); } void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) @@ -2284,13 +2294,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) } static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg) + struct arm_smmu_domain *smmu_domain) { int ret; u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; refcount_set(&cd->refs, 1); @@ -2298,31 +2306,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, mutex_lock(&arm_smmu_asid_lock); ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - if (ret) - goto out_unlock; - cd->asid = (u16)asid; - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr; - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair; - - mutex_unlock(&arm_smmu_asid_lock); - return 0; - -out_unlock: mutex_unlock(&arm_smmu_asid_lock); return ret; } static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg) + struct arm_smmu_domain *smmu_domain) { int vmid; struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; @@ -2346,8 +2336,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, struct io_pgtable_cfg pgtbl_cfg; struct io_pgtable_ops *pgtbl_ops; int (*finalise_stage_fn)(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg); + struct arm_smmu_domain *smmu_domain); /* Restrict the stage to what we can actually support */ if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1)) @@ -2390,7 +2379,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1; smmu_domain->domain.geometry.force_aperture = true; - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg); + ret = finalise_stage_fn(smmu, smmu_domain); if (ret < 0) { free_io_pgtable_ops(pgtbl_ops); return ret; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 5540609069fcd0..392130b840d55b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc { struct arm_smmu_ctx_desc { u16 asid; - u64 ttbr; - u64 tcr; - u64 mair; refcount_t refs; struct mm_struct *mm; From patchwork Tue Apr 30 17:21:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13649712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3DB45C19F4F for ; Tue, 30 Apr 2024 17:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yPxp7sZA1B/t46jZZY9CbiaCsDX8hNKNUaQSDI2xyQM=; b=W+XfHtNblk7Tyf oLTK/XTNvxe0DI5whNbISPoT33tEx2Yvw2OBTgjeXhpwKzFxRlvied8ZRU7eVRNulFWId+nCHvkRv YmRbDvGgK72EO3kzhIUra86L3QhliVzMfSjTpAFjWFisOuME1DCF8RYfeheZ6yMIA/0nQu0P2uY+g Q3mi0GlOBiq9q5F4faWDoJ4f2IXRU227V4Ea8JLo4UGTcSVHNmORDiq1EF6LfXmO1fzpOLVxKf6Fi q/fuddZuFreFcAMzddTQGG+hhjB4HUrcIFkD7yLGoS9wMl+a5tErLzG3leP4Ob1q3KiIJxtTmCqeD YnsK6/yZ848b6GchT6nA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rBP-00000007Onw-27yd; Tue, 30 Apr 2024 17:22:23 +0000 Received: from mail-bn8nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2418::601] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s1rB6-00000007OX2-2gH5 for linux-arm-kernel@lists.infradead.org; Tue, 30 Apr 2024 17:22:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NcWL0LgvM89Esn++mS7e5oG3/1cyr0nk6o1bKKuHVjNiUaIP8nwgSN0F/zqcBL4xgkUUDRVwhdSh9RpCQaMcbPuQmjUlMqf6JqD5KHLar77WslqC+Qi1MdfKu1KiVIieNIzCqG/wRDq8QgtTSeZxhJDDy0h3ObDwuyEauyFaW+v3FzfqjCp6Bt9iDyLM1TSPCGGWJ/0yFJvQgv0Iaj05HL9QEhlFNNr0K7IzBsOf2eJeFjl6c91Fgji7Tmk7/aa6bp/oxD821oxSAOLNAVGQJ4R72J5E/FGKUD2c2O9y6LGHJIaMVbD9zHBlW3K+Irk4Pfckw8rg0GnoqErD14VV1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7pTKawxbgzlCZqkYCwcBYfS0qJC+JBhkt0ODlLyj3Ys=; b=BayGO0Gum5UGdt9aJO2XUaPUgOyee970UKj6Xhay45lZ0tC55QoDj2p2DjJwKk6uFh6JB3n3VC+TvG4Aq8isrkkmiS3BJgORVIOjjiBAjN4dbblIGm6Dn0oALcan9hwcRsao/Fj7sjVKYRmGmUkT/xjZKfsP5EZNRvImgmfHLQftaWD2U0JWsUUI5xtjyQzvsv3hdjXw+cq5dD1AXrh46hN8+k2bsWml4EBqKvtvH2rWn87orDxp968+w6X6GftpKGGpHHhS73iKLmmVhlwP+m0OdmAtzkiHecRhqjnZgSwD0Gs8OVx7g4asFTZA5dUNkZm0YdgvI12pTQwNlN/xMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7pTKawxbgzlCZqkYCwcBYfS0qJC+JBhkt0ODlLyj3Ys=; b=Ah1F5JxpYuIUGxqqrTqVnNDOsXhe09467XkeqtxscX7mfUZNOcRcm6QXd+kqQ//9A9MU9sBB1VHrDicwNM1sS26UDkqBkEz+loKSfTQI3px0OtIVqukwMhvBBnBNOO9k2Pz1CcymrlDym++TtGvNV1ZYk8Ik1hAt7a6HZ1Rt5onczjzq8BUYgeUus6ODZBPU0b/5Fv3e/ccCbZr3wJAtafc39cP7A+B8xlBhwPPRUFKv5Ml+ggK7mzxU7ZDf7UbrJtKGfJs3VWkM2dkhVsGB07wN48S+967mv4fR754fXY/+/bkljlhqETApUK3fNekfMzJVJypn6HcZMRZU5pWTeg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.35; Tue, 30 Apr 2024 17:21:46 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::c296:774b:a5fc:965e%3]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 17:21:46 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Moritz Fischer , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameer Kolothum , Mostafa Saleh Subject: [PATCH v9 9/9] iommu/arm-smmu-v3: Add unit tests for arm_smmu_write_entry Date: Tue, 30 Apr 2024 14:21:41 -0300 Message-ID: <9-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v9-5040dc602008+177d7-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0417.namprd13.prod.outlook.com (2603:10b6:208:2c2::32) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f870336-7de9-406c-7c70-08dc693a0163 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|376005|7416005|366007; X-Microsoft-Antispam-Message-Info: XGVdXwNUoPtVoHpZ9Wg2UsOVGgLtD+IfkaJ37f1cF/MBsOlq1GakHhNTTf8OM2hIm7Yh7WKKCUaQADftka8iBUDzfkZSHPv8TzGXhKp0TJS/73d77O8rcPBPsilL8wuxnU7elQQvQq2hic0lq3D117i5Jqpo92MLDCr01P8Ym3Y2oV/dWBavIOfoD+3OUekG2AOvNU/5T2h8jG63Pfwfo2QImna/amo57+32xVJqyzot1ESiFU0s/0vV+z6GhC2gciFATf0jSEqU0VS4HhCGxJcqd0AnPmUOc4Y7bMW6huRQvQ61e+DxfJum/32e8+1OwaTWcIvgjhCMnc8LRGHintlOj1Hvzyzui8Lf9UF+bhNXlcII2r7Ho0dRR3ptaiO4KAN66kVasrBtfPQ8CluwMAAJ9cbPHfV7pclx/M43ehg3+DDRkvGuP3vxP5B1w5p+DqxlV3mO5CWGDzrJwxNYD549QrvFErvim3jNXLTmSKPeYNWCevF3DTLOSSGKo3SScOXYLmOYD0R2sd1hccLa82/bffN5cOPmu37P4rwuvEN/KEloP8B3fzbmqXfvyiq0tKNam0UTelPYUvjYXjnm4Xpi7iG5sfKxfnY8FNMWnMaXmub/n102ETAThMtQuQzOSgy61iLXWjqeWQIRGbMZA2DM2H+bkCkJZ6khHDoIaELnLXiXDSsvrqYA4OCNHeRFweSzbkKzLIigWw+dOuqtHleQcWhdTHQn8ne0MEL7SyhQdIEYqXJPbE6kP8fpGTz4tIQ/SxbmHDHbar0HGDSfkAegvRGANjSwqidiEbLkZ72sABuBwev8vtFu3Gj6Cmbg8DguWLyyA2K9JaNzckTP3M6XBLHDf/TWlXam6T0E3hLpTaIEzfapJ+YWpVwIbAruKO4otmWFviG8yaGEEQw9cvdwKkKGECQmtF8eB4O/hQeRPiy7jVkHGXTfts3jg7fwAUbMSxFWVMKitiX1CpOXyQTQd8RrO9Ohoq2gzAHXELt6OmKzFBHXvgIXTcqmOM3bANeWVJCqzCW2tx5BUqC07gyzjsWcrEJlCsX2IcDxyhl+Q8hAkp3HsUrkKkP8jqd36N1ajzFlMjHlKnjiVmV8cI5JLwyFtDKJVy+hrMaS4PRz1Or9o9LnHO/GNEaDMyx3SrjGSl4hskcrwZFfU5e64zt34CvQtswJ45riIR/SYOXxL7X3RgxFoNhX6KbyMNJtTRpikXcQqEIVTlg6ADXvEBfDCAv6RNG8BYHPN10k0O5f0YlDEDfw43RvTuSfDjPEN8jpNoj/u2AjtewqAJVvDw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(376005)(7416005)(366007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: p8ymBiNdRczuqyRJCRbCfPyY/8am+yaCm5XUIoMdxHYmQUjP5MokPovFW/qOXUauJjH/pv+L5+WNGhlwF6HKbTjiTy75MwX0OY9d6nVfNXjn3Ti7Q1oTqadAtJBcryaHVPB9sqNU96GCYbv+wlgTYWGdodH1Hc2NQfpWKDE8/9wgcSyFJUU+37RrYVIKQoC/JZzX5KxhA7d850tGjM4VIlVhDFai6AYnROABK4alYPCM39D7iurCF3FKJE4XC3Bv6cl6CzS7IqjacFtZwLeNBNLVv/+W4RR8izRA9jLUXhC4jtkYvee5FcyWpHwtt7029KtCR5P0okkAQP7w9iiwMvYn6MBUFE48zvwanMU0HIEln/x/FYC/Y5FT2cuLjkuxTNONW/JcvzwM4dPVpzTXac9B8qelhm18mJ9sXlHyyvqQjqJIAvxdqIR4B7PebTezggvgMfmH6E3R+FV5VhHtIZD4BkA+PPsXyxF5FSFdVcaJgMg/EHulEOcU9GWyZx9zDqcGoQG682IbqZO8VqI4rpHIjTynYf3YPIqrYKkP2cKLgt/+aMq8ZJ5PbIlFaGoxs+3bD4CJN8SuF07cSm91E9P0XsTKDAqgc1hs0ZiAAhH3sJObuuD0jBuU77xj5h4CT6jFtJh9gL4O/+NLAx6p9UwDX/kl+HOOfRisUwwcHdbV6PFk4L5SpUN6WrSEZzWX8OKBHQhJYp5CUhwalO1tXlI8SSNKugga/O6GRMHKjM73yc2KnvGuTQAt9NUk4rE12XPGF0GbOUG4hSOE1OeUNtoLw8Z2nabcHp2Wb7VKL/O97K2PFna9KysxjKn1BbqTgCXiB9Pqm6sdm/0feUc0lQ/vMwmVdvqWDA072WVa6Ug2EINB+dvt87YPKTK3z4XVwneP67AXzxq2Y8AoJvSl84JAhHvVm9UNw4TTPUgcWWARS3ucpnfhKrD680gRCIEGXCa+3gRRsd4qPwuJFB0jOWlm2zkRai1Bw0pSua9js1emtS7rAd6J9Q/jdeFCrR5h4YL0iZsiA5+pzKog49O42NntTgkde7EabARQEPiB9o0tvF5CcdLPJv1C8tdl6XbyP18Dnhlr0sJEJSLtUZfbcQaMNkPjHS0MthHQK/UB8uogXaLqCMpgBgDqpFc8RZfzDRedDW2E0Inr/5Bk6vvmY6mlmdTlYPVDehejn2HYt+Tx4stKeqva5AfNOUlUhSM8H8ki2q71MFn3B2HZed8sQvX0G/7ka7JAzJYsMPVX2GVvDZ2zP4zajXVEB1TU9Xvm5sSRIs9+wVCuF5m8EJFoapK1BTUvAPT35ZW0mZ9g6mWMeMFSb88Yij4H+/wKvCLPhYc/iWIRcxNtS8qJi2EXyEz3ppU7kYdVjFvjH10Njd6Tx9MZ1BMDxlcsCidw5rGw2GLE6VmvKoiT980D4i/7Qjcd1Wg/pM7WV0jCLSoKlTyyENtBn+lRzc8TNW9H632H3dicxZaEpgK1JqQgn5kXxTlEOMU3GHanDhpGBdE01TEOB2Vx0wWhLo9cEs56mi0u3E5FeAMbCcTOVtYhKDhJhPD+650OMDdJWrgL1k31eY0nBgZFMTjCbNs+LmaBJWoc X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1f870336-7de9-406c-7c70-08dc693a0163 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 17:21:43.4043 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: W3xEYTf2dNIyZYT7jeXlwUV5JPmeU+UrVN6iZlMC4kDtZNfuaUGnJ+Q+SOouVzE7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240430_102204_863419_BF2852EA X-CRM114-Status: GOOD ( 23.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add tests for some of the more common STE update operations that we expect to see, as well as some artificial STE updates to test the edges of arm_smmu_write_entry. These also serve as a record of which common operation is expected to be hitless, and how many syncs they require. arm_smmu_write_entry implements a generic algorithm that updates an STE/CD to any other abritrary STE/CD configuration. The update requires a sequence of write+sync operations with some invariants that must be held true after each sync. arm_smmu_write_entry lends itself well to unit-testing since the function's interaction with the STE/CD is already abstracted by input callbacks that we can hook to introspect into the sequence of operations. We can use these hooks to guarantee that invariants are held throughout the entire update operation. Link: https://lore.kernel.org/r/20240106083617.1173871-3-mshavit@google.com Tested-by: Nicolin Chen Signed-off-by: Michael Shavit Signed-off-by: Jason Gunthorpe --- drivers/iommu/Kconfig | 13 +- drivers/iommu/arm/arm-smmu-v3/Makefile | 1 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 8 +- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 465 ++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 43 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 30 ++ 6 files changed, 533 insertions(+), 27 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 0af39bbbe3a30e..f872aeccd82041 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -397,9 +397,9 @@ config ARM_SMMU_V3 Say Y here if your system includes an IOMMU device implementing the ARM SMMUv3 architecture. +if ARM_SMMU_V3 config ARM_SMMU_V3_SVA bool "Shared Virtual Addressing support for the ARM SMMUv3" - depends on ARM_SMMU_V3 select IOMMU_SVA select IOMMU_IOPF select MMU_NOTIFIER @@ -410,6 +410,17 @@ config ARM_SMMU_V3_SVA Say Y here if your system supports SVA extensions such as PCIe PASID and PRI. +config ARM_SMMU_V3_KUNIT_TEST + bool "KUnit tests for arm-smmu-v3 driver" if !KUNIT_ALL_TESTS + depends on KUNIT + depends on ARM_SMMU_V3_SVA + default KUNIT_ALL_TESTS + help + Enable this option to unit-test arm-smmu-v3 driver functions. + + If unsure, say N. +endif + config S390_IOMMU def_bool y if S390 && PCI depends on S390 && PCI diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm-smmu-v3/Makefile index 54feb1ecccad89..0b97054b3929b7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -2,4 +2,5 @@ obj-$(CONFIG_ARM_SMMU_V3) += arm_smmu_v3.o arm_smmu_v3-objs-y += arm-smmu-v3.o arm_smmu_v3-objs-$(CONFIG_ARM_SMMU_V3_SVA) += arm-smmu-v3-sva.o +arm_smmu_v3-objs-$(CONFIG_ARM_SMMU_V3_KUNIT_TEST) += arm-smmu-v3-test.o arm_smmu_v3-objs := $(arm_smmu_v3-objs-y) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 8730a7043909e3..34a977a0767d46 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "arm-smmu-v3.h" #include "../../io-pgtable-arm.h" @@ -120,9 +121,10 @@ static u64 page_size_to_cd(void) return ARM_LPAE_TCR_TG0_4K; } -static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, - struct arm_smmu_master *master, - struct mm_struct *mm, u16 asid) +VISIBLE_IF_KUNIT +void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, struct mm_struct *mm, + u16 asid) { u64 par; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c new file mode 100644 index 00000000000000..417804392ff089 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c @@ -0,0 +1,465 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2024 Google LLC. + */ +#include +#include + +#include "arm-smmu-v3.h" + +struct arm_smmu_test_writer { + struct arm_smmu_entry_writer writer; + struct kunit *test; + const __le64 *init_entry; + const __le64 *target_entry; + __le64 *entry; + + bool invalid_entry_written; + unsigned int num_syncs; +}; + +#define NUM_ENTRY_QWORDS 8 +#define NUM_EXPECTED_SYNCS(x) x + +static struct arm_smmu_ste bypass_ste; +static struct arm_smmu_ste abort_ste; +static struct arm_smmu_device smmu = { + .features = ARM_SMMU_FEAT_STALLS | ARM_SMMU_FEAT_ATTR_TYPES_OVR +}; +static struct mm_struct sva_mm = { + .pgd = (void *)0xdaedbeefdeadbeefULL, +}; + +static bool arm_smmu_entry_differs_in_used_bits(const __le64 *entry, + const __le64 *used_bits, + const __le64 *target, + unsigned int length) +{ + bool differs = false; + unsigned int i; + + for (i = 0; i < length; i++) { + if ((entry[i] & used_bits[i]) != target[i]) + differs = true; + } + return differs; +} + +static void +arm_smmu_test_writer_record_syncs(struct arm_smmu_entry_writer *writer) +{ + struct arm_smmu_test_writer *test_writer = + container_of(writer, struct arm_smmu_test_writer, writer); + __le64 *entry_used_bits; + + entry_used_bits = kunit_kzalloc( + test_writer->test, sizeof(*entry_used_bits) * NUM_ENTRY_QWORDS, + GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test_writer->test, entry_used_bits); + + pr_debug("STE value is now set to: "); + print_hex_dump_debug(" ", DUMP_PREFIX_NONE, 16, 8, + test_writer->entry, + NUM_ENTRY_QWORDS * sizeof(*test_writer->entry), + false); + + test_writer->num_syncs += 1; + if (!test_writer->entry[0]) { + test_writer->invalid_entry_written = true; + } else { + /* + * At any stage in a hitless transition, the entry must be + * equivalent to either the initial entry or the target entry + * when only considering the bits used by the current + * configuration. + */ + writer->ops->get_used(test_writer->entry, entry_used_bits); + KUNIT_EXPECT_FALSE( + test_writer->test, + arm_smmu_entry_differs_in_used_bits( + test_writer->entry, entry_used_bits, + test_writer->init_entry, NUM_ENTRY_QWORDS) && + arm_smmu_entry_differs_in_used_bits( + test_writer->entry, entry_used_bits, + test_writer->target_entry, + NUM_ENTRY_QWORDS)); + } +} + +static void +arm_smmu_v3_test_debug_print_used_bits(struct arm_smmu_entry_writer *writer, + const __le64 *ste) +{ + __le64 used_bits[NUM_ENTRY_QWORDS] = {}; + + arm_smmu_get_ste_used(ste, used_bits); + pr_debug("STE used bits: "); + print_hex_dump_debug(" ", DUMP_PREFIX_NONE, 16, 8, used_bits, + sizeof(used_bits), false); +} + +static const struct arm_smmu_entry_writer_ops test_ste_ops = { + .sync = arm_smmu_test_writer_record_syncs, + .get_used = arm_smmu_get_ste_used, +}; + +static const struct arm_smmu_entry_writer_ops test_cd_ops = { + .sync = arm_smmu_test_writer_record_syncs, + .get_used = arm_smmu_get_cd_used, +}; + +static void arm_smmu_v3_test_ste_expect_transition( + struct kunit *test, const struct arm_smmu_ste *cur, + const struct arm_smmu_ste *target, unsigned int num_syncs_expected, + bool hitless) +{ + struct arm_smmu_ste cur_copy = *cur; + struct arm_smmu_test_writer test_writer = { + .writer = { + .ops = &test_ste_ops, + }, + .test = test, + .init_entry = cur->data, + .target_entry = target->data, + .entry = cur_copy.data, + .num_syncs = 0, + .invalid_entry_written = false, + + }; + + pr_debug("STE initial value: "); + print_hex_dump_debug(" ", DUMP_PREFIX_NONE, 16, 8, cur_copy.data, + sizeof(cur_copy), false); + arm_smmu_v3_test_debug_print_used_bits(&test_writer.writer, cur->data); + pr_debug("STE target value: "); + print_hex_dump_debug(" ", DUMP_PREFIX_NONE, 16, 8, target->data, + sizeof(cur_copy), false); + arm_smmu_v3_test_debug_print_used_bits(&test_writer.writer, + target->data); + + arm_smmu_write_entry(&test_writer.writer, cur_copy.data, target->data); + + KUNIT_EXPECT_EQ(test, test_writer.invalid_entry_written, !hitless); + KUNIT_EXPECT_EQ(test, test_writer.num_syncs, num_syncs_expected); + KUNIT_EXPECT_MEMEQ(test, target->data, cur_copy.data, sizeof(cur_copy)); +} + +static void arm_smmu_v3_test_ste_expect_hitless_transition( + struct kunit *test, const struct arm_smmu_ste *cur, + const struct arm_smmu_ste *target, unsigned int num_syncs_expected) +{ + arm_smmu_v3_test_ste_expect_transition(test, cur, target, + num_syncs_expected, true); +} + +static const dma_addr_t fake_cdtab_dma_addr = 0xF0F0F0F0F0F0; + +static void arm_smmu_test_make_cdtable_ste(struct arm_smmu_ste *ste, + const dma_addr_t dma_addr) +{ + struct arm_smmu_master master = { + .cd_table.cdtab_dma = dma_addr, + .cd_table.s1cdmax = 0xFF, + .cd_table.s1fmt = STRTAB_STE_0_S1FMT_64K_L2, + .smmu = &smmu, + }; + + arm_smmu_make_cdtable_ste(ste, &master); +} + +static void arm_smmu_v3_write_ste_test_bypass_to_abort(struct kunit *test) +{ + /* + * Bypass STEs has used bits in the first two Qwords, while abort STEs + * only have used bits in the first QWord. Transitioning from bypass to + * abort requires two syncs: the first to set the first qword and make + * the STE into an abort, the second to clean up the second qword. + */ + arm_smmu_v3_test_ste_expect_hitless_transition( + test, &bypass_ste, &abort_ste, NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_abort_to_bypass(struct kunit *test) +{ + /* + * Transitioning from abort to bypass also requires two syncs: the first + * to set the second qword data required by the bypass STE, and the + * second to set the first qword and switch to bypass. + */ + arm_smmu_v3_test_ste_expect_hitless_transition( + test, &abort_ste, &bypass_ste, NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_cdtable_to_abort(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_cdtable_ste(&ste, fake_cdtab_dma_addr); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &ste, &abort_ste, + NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_abort_to_cdtable(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_cdtable_ste(&ste, fake_cdtab_dma_addr); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &abort_ste, &ste, + NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_cdtable_to_bypass(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_cdtable_ste(&ste, fake_cdtab_dma_addr); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &ste, &bypass_ste, + NUM_EXPECTED_SYNCS(3)); +} + +static void arm_smmu_v3_write_ste_test_bypass_to_cdtable(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_cdtable_ste(&ste, fake_cdtab_dma_addr); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &bypass_ste, &ste, + NUM_EXPECTED_SYNCS(3)); +} + +static void arm_smmu_test_make_s2_ste(struct arm_smmu_ste *ste, + bool ats_enabled) +{ + struct arm_smmu_master master = { + .smmu = &smmu, + .ats_enabled = ats_enabled, + }; + struct io_pgtable io_pgtable = {}; + struct arm_smmu_domain smmu_domain = { + .pgtbl_ops = &io_pgtable.ops, + }; + + io_pgtable.cfg.arm_lpae_s2_cfg.vttbr = 0xdaedbeefdeadbeefULL; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.ps = 1; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.tg = 2; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.sh = 3; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.orgn = 1; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.irgn = 2; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.sl = 3; + io_pgtable.cfg.arm_lpae_s2_cfg.vtcr.tsz = 4; + + arm_smmu_make_s2_domain_ste(ste, &master, &smmu_domain); +} + +static void arm_smmu_v3_write_ste_test_s2_to_abort(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_s2_ste(&ste, true); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &ste, &abort_ste, + NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_abort_to_s2(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_s2_ste(&ste, true); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &abort_ste, &ste, + NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_s2_to_bypass(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_s2_ste(&ste, true); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &ste, &bypass_ste, + NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_ste_test_bypass_to_s2(struct kunit *test) +{ + struct arm_smmu_ste ste; + + arm_smmu_test_make_s2_ste(&ste, true); + arm_smmu_v3_test_ste_expect_hitless_transition(test, &bypass_ste, &ste, + NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_test_cd_expect_transition( + struct kunit *test, const struct arm_smmu_cd *cur, + const struct arm_smmu_cd *target, unsigned int num_syncs_expected, + bool hitless) +{ + struct arm_smmu_cd cur_copy = *cur; + struct arm_smmu_test_writer test_writer = { + .writer = { + .ops = &test_cd_ops, + }, + .test = test, + .init_entry = cur->data, + .target_entry = target->data, + .entry = cur_copy.data, + .num_syncs = 0, + .invalid_entry_written = false, + + }; + + pr_debug("CD initial value: "); + print_hex_dump_debug(" ", DUMP_PREFIX_NONE, 16, 8, cur_copy.data, + sizeof(cur_copy), false); + arm_smmu_v3_test_debug_print_used_bits(&test_writer.writer, cur->data); + pr_debug("CD target value: "); + print_hex_dump_debug(" ", DUMP_PREFIX_NONE, 16, 8, target->data, + sizeof(cur_copy), false); + arm_smmu_v3_test_debug_print_used_bits(&test_writer.writer, + target->data); + + arm_smmu_write_entry(&test_writer.writer, cur_copy.data, target->data); + + KUNIT_EXPECT_EQ(test, test_writer.invalid_entry_written, !hitless); + KUNIT_EXPECT_EQ(test, test_writer.num_syncs, num_syncs_expected); + KUNIT_EXPECT_MEMEQ(test, target->data, cur_copy.data, sizeof(cur_copy)); +} + +static void arm_smmu_v3_test_cd_expect_non_hitless_transition( + struct kunit *test, const struct arm_smmu_cd *cur, + const struct arm_smmu_cd *target, unsigned int num_syncs_expected) +{ + arm_smmu_v3_test_cd_expect_transition(test, cur, target, + num_syncs_expected, false); +} + +static void arm_smmu_v3_test_cd_expect_hitless_transition( + struct kunit *test, const struct arm_smmu_cd *cur, + const struct arm_smmu_cd *target, unsigned int num_syncs_expected) +{ + arm_smmu_v3_test_cd_expect_transition(test, cur, target, + num_syncs_expected, true); +} + +static void arm_smmu_test_make_s1_cd(struct arm_smmu_cd *cd, unsigned int asid) +{ + struct arm_smmu_master master = { + .smmu = &smmu, + }; + struct io_pgtable io_pgtable = {}; + struct arm_smmu_domain smmu_domain = { + .pgtbl_ops = &io_pgtable.ops, + .cd = { + .asid = asid, + }, + }; + + io_pgtable.cfg.arm_lpae_s1_cfg.ttbr = 0xdaedbeefdeadbeefULL; + io_pgtable.cfg.arm_lpae_s1_cfg.tcr.ips = 1; + io_pgtable.cfg.arm_lpae_s1_cfg.tcr.tg = 2; + io_pgtable.cfg.arm_lpae_s1_cfg.tcr.sh = 3; + io_pgtable.cfg.arm_lpae_s1_cfg.tcr.orgn = 1; + io_pgtable.cfg.arm_lpae_s1_cfg.tcr.irgn = 2; + io_pgtable.cfg.arm_lpae_s1_cfg.tcr.tsz = 4; + io_pgtable.cfg.arm_lpae_s1_cfg.mair = 0xabcdef012345678ULL; + + arm_smmu_make_s1_cd(cd, &master, &smmu_domain); +} + +static void arm_smmu_v3_write_cd_test_s1_clear(struct kunit *test) +{ + struct arm_smmu_cd cd = {}; + struct arm_smmu_cd cd_2; + + arm_smmu_test_make_s1_cd(&cd_2, 1997); + arm_smmu_v3_test_cd_expect_non_hitless_transition( + test, &cd, &cd_2, NUM_EXPECTED_SYNCS(2)); + arm_smmu_v3_test_cd_expect_non_hitless_transition( + test, &cd_2, &cd, NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_cd_test_s1_change_asid(struct kunit *test) +{ + struct arm_smmu_cd cd = {}; + struct arm_smmu_cd cd_2; + + arm_smmu_test_make_s1_cd(&cd, 778); + arm_smmu_test_make_s1_cd(&cd_2, 1997); + arm_smmu_v3_test_cd_expect_hitless_transition(test, &cd, &cd_2, + NUM_EXPECTED_SYNCS(1)); + arm_smmu_v3_test_cd_expect_hitless_transition(test, &cd_2, &cd, + NUM_EXPECTED_SYNCS(1)); +} + +static void arm_smmu_test_make_sva_cd(struct arm_smmu_cd *cd, unsigned int asid) +{ + struct arm_smmu_master master = { + .smmu = &smmu, + }; + + arm_smmu_make_sva_cd(cd, &master, &sva_mm, asid); +} + +static void arm_smmu_test_make_sva_release_cd(struct arm_smmu_cd *cd, + unsigned int asid) +{ + struct arm_smmu_master master = { + .smmu = &smmu, + }; + + arm_smmu_make_sva_cd(cd, &master, NULL, asid); +} + +static void arm_smmu_v3_write_cd_test_sva_clear(struct kunit *test) +{ + struct arm_smmu_cd cd = {}; + struct arm_smmu_cd cd_2; + + arm_smmu_test_make_sva_cd(&cd_2, 1997); + arm_smmu_v3_test_cd_expect_non_hitless_transition( + test, &cd, &cd_2, NUM_EXPECTED_SYNCS(2)); + arm_smmu_v3_test_cd_expect_non_hitless_transition( + test, &cd_2, &cd, NUM_EXPECTED_SYNCS(2)); +} + +static void arm_smmu_v3_write_cd_test_sva_release(struct kunit *test) +{ + struct arm_smmu_cd cd; + struct arm_smmu_cd cd_2; + + arm_smmu_test_make_sva_cd(&cd, 1997); + arm_smmu_test_make_sva_release_cd(&cd_2, 1997); + arm_smmu_v3_test_cd_expect_hitless_transition(test, &cd, &cd_2, + NUM_EXPECTED_SYNCS(2)); + arm_smmu_v3_test_cd_expect_hitless_transition(test, &cd_2, &cd, + NUM_EXPECTED_SYNCS(2)); +} + +static struct kunit_case arm_smmu_v3_test_cases[] = { + KUNIT_CASE(arm_smmu_v3_write_ste_test_bypass_to_abort), + KUNIT_CASE(arm_smmu_v3_write_ste_test_abort_to_bypass), + KUNIT_CASE(arm_smmu_v3_write_ste_test_cdtable_to_abort), + KUNIT_CASE(arm_smmu_v3_write_ste_test_abort_to_cdtable), + KUNIT_CASE(arm_smmu_v3_write_ste_test_cdtable_to_bypass), + KUNIT_CASE(arm_smmu_v3_write_ste_test_bypass_to_cdtable), + KUNIT_CASE(arm_smmu_v3_write_ste_test_s2_to_abort), + KUNIT_CASE(arm_smmu_v3_write_ste_test_abort_to_s2), + KUNIT_CASE(arm_smmu_v3_write_ste_test_s2_to_bypass), + KUNIT_CASE(arm_smmu_v3_write_ste_test_bypass_to_s2), + KUNIT_CASE(arm_smmu_v3_write_cd_test_s1_clear), + KUNIT_CASE(arm_smmu_v3_write_cd_test_s1_change_asid), + KUNIT_CASE(arm_smmu_v3_write_cd_test_sva_clear), + KUNIT_CASE(arm_smmu_v3_write_cd_test_sva_release), + {}, +}; + +static int arm_smmu_v3_test_suite_init(struct kunit_suite *test) +{ + arm_smmu_make_bypass_ste(&smmu, &bypass_ste); + arm_smmu_make_abort_ste(&abort_ste); + return 0; +} + +static struct kunit_suite arm_smmu_v3_test_module = { + .name = "arm-smmu-v3-kunit-test", + .suite_init = arm_smmu_v3_test_suite_init, + .test_cases = arm_smmu_v3_test_cases, +}; +kunit_test_suites(&arm_smmu_v3_test_module); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3817bc99d7e319..15bad76cf84a61 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -26,6 +26,7 @@ #include #include #include +#include #include "arm-smmu-v3.h" #include "../../dma-iommu.h" @@ -42,17 +43,6 @@ enum arm_smmu_msi_index { ARM_SMMU_MAX_MSIS, }; -struct arm_smmu_entry_writer_ops; -struct arm_smmu_entry_writer { - const struct arm_smmu_entry_writer_ops *ops; - struct arm_smmu_master *master; -}; - -struct arm_smmu_entry_writer_ops { - void (*get_used)(const __le64 *entry, __le64 *used); - void (*sync)(struct arm_smmu_entry_writer *writer); -}; - #define NUM_ENTRY_QWORDS 8 static_assert(sizeof(struct arm_smmu_ste) == NUM_ENTRY_QWORDS * sizeof(u64)); static_assert(sizeof(struct arm_smmu_cd) == NUM_ENTRY_QWORDS * sizeof(u64)); @@ -979,7 +969,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) * would be nice if this was complete according to the spec, but minimally it * has to capture the bits this driver uses. */ -static void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) +VISIBLE_IF_KUNIT +void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) { unsigned int cfg = FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0])); @@ -1101,8 +1092,9 @@ static bool entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry, * V=0 process. This relies on the IGNORED behavior described in the * specification. */ -static void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, - __le64 *entry, const __le64 *target) +VISIBLE_IF_KUNIT +void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *entry, + const __le64 *target) { __le64 unused_update[NUM_ENTRY_QWORDS]; u8 used_qword_diff; @@ -1256,7 +1248,8 @@ struct arm_smmu_cd_writer { unsigned int ssid; }; -static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits) +VISIBLE_IF_KUNIT +void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits) { used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V); if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V))) @@ -1514,7 +1507,8 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid, } } -static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target) +VISIBLE_IF_KUNIT +void arm_smmu_make_abort_ste(struct arm_smmu_ste *target) { memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( @@ -1522,8 +1516,9 @@ static void arm_smmu_make_abort_ste(struct arm_smmu_ste *target) FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT)); } -static void arm_smmu_make_bypass_ste(struct arm_smmu_device *smmu, - struct arm_smmu_ste *target) +VISIBLE_IF_KUNIT +void arm_smmu_make_bypass_ste(struct arm_smmu_device *smmu, + struct arm_smmu_ste *target) { memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( @@ -1535,8 +1530,9 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_device *smmu, STRTAB_STE_1_SHCFG_INCOMING)); } -static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, - struct arm_smmu_master *master) +VISIBLE_IF_KUNIT +void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, + struct arm_smmu_master *master) { struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; struct arm_smmu_device *smmu = master->smmu; @@ -1585,9 +1581,10 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, } } -static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, - struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) +VISIBLE_IF_KUNIT +void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) { struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg; const struct io_pgtable_cfg *pgtbl_cfg = diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 392130b840d55b..1242a086c9f948 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -736,6 +736,36 @@ struct arm_smmu_domain { struct list_head mmu_notifiers; }; +/* The following are exposed for testing purposes. */ +struct arm_smmu_entry_writer_ops; +struct arm_smmu_entry_writer { + const struct arm_smmu_entry_writer_ops *ops; + struct arm_smmu_master *master; +}; + +struct arm_smmu_entry_writer_ops { + void (*get_used)(const __le64 *entry, __le64 *used); + void (*sync)(struct arm_smmu_entry_writer *writer); +}; + +#if IS_ENABLED(CONFIG_KUNIT) +void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits); +void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *cur, + const __le64 *target); +void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits); +void arm_smmu_make_abort_ste(struct arm_smmu_ste *target); +void arm_smmu_make_bypass_ste(struct arm_smmu_device *smmu, + struct arm_smmu_ste *target); +void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, + struct arm_smmu_master *master); +void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain); +void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, struct mm_struct *mm, + u16 asid); +#endif + static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) { return container_of(dom, struct arm_smmu_domain, domain);